Repository: googleapis/google-cloud-python

Issue:
Vision: Google Cloud Storage Client project=None bootstrapping problem
Hello,

In using the Google Cloud API's, I noticed a small discrepancy in the documentation's expected behavior, and the actual behavior. 
The documentation and Google cloud examples code typically assumes that credentials should be created using a JSON key store saved on persistent storage. However, this isn't always an option, as with deployments using ephemeral storage. Luckily, a Credentials object can be created directly from a dictionary, which houses the same mapping that the JSON keystore would hold. This is the procedure that I use in my project (the specifics of which, is shown below).


The [Storage.Client documentation](https://googlecloudplatform.github.io/google-cloud-python/latest/storage/client.html#google.cloud.storage.client.Client) suggests that the `project` parameter is optional, and thus may be None. We found, however, that when it is not specified, an error is thrown, as a result of a sort of bootstrapping problem in which the Google Cloud APIs check to see if the project is None, and if so, attempt to create it (which is impossible, without first having the keys).
To elaborate:

1. When we make a call to Storage.Client(), we step into [storage/google/cloud/storage/client.py](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/). 
2. This class calls `super(Client, self).__init__(project=project, credentials=credentials, _http=_http)`, which takes us to [core/google/cloud/client.py](https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/core/google/cloud/client.py) .
3. This calls `_ClientProjectMixin.__init__(self, project=project)`, where we see the following call:
```
        project = self._determine_default(project)
```
4. Following this call, we see:
```
    if project is None:
        _, project = google.auth.default()
```


So, there is an attempt to infer the project parameter based on the environment credentials. However, as discussed above, we are using a dictionary-created Credentials object as our credentials,  so no such credentials exist in the environment. **This is a problem because the documentation claims that project is optional, when, in fact, it is not if the project is not stored in the environment.**

Note, however, that this seems to be a problem specific with Storage.Client(), and not other API clients. Trying this procedure (see the code below) for `google.cloud.vision.ImageAnnotatorClient` seems to work perfectly fine with just a credentials object, without the project argument. 


----



1. **OS:** Ubuntu 16.04
2. **Python Version:** Python 2.7.10
3. **Google APIs:**
google-auth==1.1.1
google-cloud-core==0.27.1
google-cloud-storage==1.4.0
google-cloud-vision==0.27.0
google-gax==0.15.15
google-resumable-media==0.2.3
googleapis-common-protos==1.5.3
oauth2client==4.1.2


4. **Stacktrace:**


```
Traceback (most recent call last):
  File "storage_client.py", line 23, in <module>
    client = _get_client()
  File "storage_client.py", line 18, in _get_client
    client = storage.Client(credentials=creds)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/storage/client.py", line 59, in __init__
    _http=_http)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 211, in __init__
    _ClientProjectMixin.__init__(self, project=project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 165, in __init__
    project = self._determine_default(project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/client.py", line 178, in _determine_default
    return _determine_default_project(project)
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/cloud/_helpers.py", line 179, in _determine_default_project
    _, project = google.auth.default()
  File "/Users/admin/testproject/venv/lib/python2.7/site-packages/google/auth/_default.py", line 286, in default
    raise exceptions.DefaultCredentialsError(_HELP_MESSAGE)
google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or
explicitly create credential and re-run the application. For more
information, please see
https://developers.google.com/accounts/docs/application-default-credentials.

```


5. **Steps to reproduce:**



- So, create an OAuth2 credentials object from a python dictionary (let's call it `ketfile_dict` using `google.oauth2.service_account.Credentials.from_service_account_info(keyfile_dict)`.

- Create an instance of a storage client using `google.cloud.storage.Client(credentials=creds)`

- Although the documentation states that the "project" argument is optional, the API will in fact throw an error when no project argument is presented.

6. Code example

Assume that in the same directory, you create a file, `constants.py`, with the following fields defined from the appropriate fields of the service account key JSON.

constants.GCP_SCOPE
constants.GCP_TYPE
constants.GCP_CLIENT_EMAIL
constants.GCP_PRIVATE_KEY
constants.GCP_PRIVATE_KEY_ID
constants.GCP_CLIENT_ID
constants.GCP_TOKEN_URI
constants.GCP_PROJECT_ID



```
from google.cloud import storage
from google.oauth2 import service_account
import constants


# Return an Oauth2 credentials instance
def get_credentials(with_keyfile=False):
    keyfile_dict = _get_keyfile_dict()
    scope = [constants.GCP_SCOPE]
    creds = service_account.Credentials.from_service_account_info(keyfile_dict).with_scopes(scope)

    if not with_keyfile:
        return creds
    else:
        # Return creds and the keyfile.
        return creds, keyfile_dict


# Construct a dictionary of the required key:value pairs.
def _get_keyfile_dict():
    keyfile_dict = {}
    keyfile_dict['type'] = constants.GCP_TYPE
    keyfile_dict['client_email'] = constants.GCP_CLIENT_EMAIL
    keyfile_dict['private_key'] = constants.GCP_PRIVATE_KEY
    keyfile_dict['private_key_id'] = constants.GCP_PRIVATE_KEY_ID
    keyfile_dict['client_id'] = constants.GCP_CLIENT_ID
    keyfile_dict['token_uri'] = constants.GCP_TOKEN_URI
    keyfile_dict['project_id'] = constants.GCP_PROJECT_ID
    return keyfile_dict


def _get_client():
    creds, keyfile_dict = get_credentials(with_keyfile=True)
    project = keyfile_dict['project_id']
    #client = storage.Client(project=project, credentials=creds) # This WILL work because a project is passed in.
    #client = vision.ImageAnnotatorClient(credentials=creds) #Note that the Vision client will work without a project argument
    client = storage.Client(credentials=creds) # This will throw an error because project=None.
    return client


# Instantiates a client
client = _get_client()
```


Let me know if you have any other questions. :)







Discussion:
@jonparrott Is this just a documentation issue, or should the client be willing to infer the project from the passed `credentials`?
This is interesting, because storage effectively doesn't need a project to operate. Buckets are global resources.
Places which touch `client.project`:
- `list_buckets` is the only client method which uses the project:  I guess we could add an optional `project` argument, falling back to the client's project.  Likely we should raise an exception if neither is set: `project` is a [required parameter for 'buckets.list'](https://cloud.google.com/storage/docs/json_api/v1/buckets/list#project).
- `Bucket.create` is the only bucket method which uses it: it is [likewise required](https://cloud.google.com/storage/docs/json_api/v1/buckets/insert#project), so we could allow an optional `project` argument and fall back / fail in the same way.  We would also need to plumb it through to `Client.create_bucket`.
- `BucketNotification.__init__` takes ~~and~~ **an** optional `topic_project` parameter, and falls back to the client's project.  We would need to have it raise if neither was set.

To implement this, we could have the default for the `project` argument be a marker, and then treat `None` specially (pass through a dummy to the `super()` version and then assign `None`.
Sounds reasonable, @tseaver.

Resolving patch:
diff --git a/storage/google/cloud/storage/bucket.py b/storage/google/cloud/storage/bucket.py
--- a/storage/google/cloud/storage/bucket.py
+++ b/storage/google/cloud/storage/bucket.py
@@ -251,7 +251,7 @@ def exists(self, client=None):
         except NotFound:
             return False
 
-    def create(self, client=None):
+    def create(self, client=None, project=None):
         """Creates current bucket.
 
         If the bucket already exists, will raise
@@ -265,12 +265,28 @@ def create(self, client=None):
                       ``NoneType``
         :param client: Optional. The client to use.  If not passed, falls back
                        to the ``client`` stored on the current bucket.
+
+        :type project: str
+        :param project: (Optional) the project under which the  bucket is to
+                        be created.  If not passed, uses the project set on
+                        the client.
+        :raises ValueError: if :attr:`user_project` is set.
+        :raises ValueError: if ``project`` is None and client's
+                            :attr:`project` is also None.
         """
         if self.user_project is not None:
             raise ValueError("Cannot create bucket with 'user_project' set.")
 
         client = self._require_client(client)
-        query_params = {'project': client.project}
+
+        if project is None:
+            project = client.project
+
+        if project is None:
+            raise ValueError(
+                "Client project not set:  pass an explicit project.")
+
+        query_params = {'project': project}
         properties = {key: self._properties[key] for key in self._changes}
         properties['name'] = self.name
         api_response = client._connection.api_request(
diff --git a/storage/google/cloud/storage/client.py b/storage/google/cloud/storage/client.py
--- a/storage/google/cloud/storage/client.py
+++ b/storage/google/cloud/storage/client.py
@@ -26,10 +26,13 @@
 from google.cloud.storage.bucket import Bucket
 
 
+_marker = object()
+
+
 class Client(ClientWithProject):
     """Client to bundle configuration needed for API requests.
 
-    :type project: str
+    :type project: str or None
     :param project: the project which the client acts on behalf of. Will be
                     passed when creating a topic.  If not passed,
                     falls back to the default inferred from the environment.
@@ -55,10 +58,19 @@ class Client(ClientWithProject):
              'https://www.googleapis.com/auth/devstorage.read_write')
     """The scopes required for authenticating as a Cloud Storage consumer."""
 
-    def __init__(self, project=None, credentials=None, _http=None):
+    def __init__(self, project=_marker, credentials=None, _http=None):
         self._base_connection = None
+        if project is None:
+            no_project = True
+            project = '<none>'
+        else:
+            no_project = False
+        if project is _marker:
+            project = None
         super(Client, self).__init__(project=project, credentials=credentials,
                                      _http=_http)
+        if no_project:
+            self.project = None
         self._connection = Connection(self)
         self._batch_stack = _LocalStack()
 
@@ -216,7 +228,7 @@ def lookup_bucket(self, bucket_name):
         except NotFound:
             return None
 
-    def create_bucket(self, bucket_name, requester_pays=None):
+    def create_bucket(self, bucket_name, requester_pays=None, project=None):
         """Create a new bucket.
 
         For example:
@@ -238,17 +250,22 @@ def create_bucket(self, bucket_name, requester_pays=None):
             (Optional) Whether requester pays for API requests for this
             bucket and its blobs.
 
+        :type project: str
+        :param project: (Optional) the project under which the  bucket is to
+                        be created.  If not passed, uses the project set on
+                        the client.
+
         :rtype: :class:`google.cloud.storage.bucket.Bucket`
         :returns: The newly created bucket.
         """
         bucket = Bucket(self, name=bucket_name)
         if requester_pays is not None:
             bucket.requester_pays = requester_pays
-        bucket.create(client=self)
+        bucket.create(client=self, project=project)
         return bucket
 
     def list_buckets(self, max_results=None, page_token=None, prefix=None,
-                     projection='noAcl', fields=None):
+                     projection='noAcl', fields=None, project=None):
         """Get all buckets in the project associated to the client.
 
         This will not populate the list of blobs available in each
@@ -284,11 +301,24 @@ def list_buckets(self, max_results=None, page_token=None, prefix=None,
             response with just the next page token and the language of each
             bucket returned: 'items/id,nextPageToken'
 
+        :type project: str
+        :param project: (Optional) the project whose buckets are to be listed.
+                        If not passed, uses the project set on the client.
+
         :rtype: :class:`~google.api_core.page_iterator.Iterator`
+        :raises ValueError: if both ``project`` is ``None`` and the client's
+                            project is also ``None``.
         :returns: Iterator of all :class:`~google.cloud.storage.bucket.Bucket`
                   belonging to this project.
         """
-        extra_params = {'project': self.project}
+        if project is None:
+            project = self.project
+
+        if project is None:
+            raise ValueError(
+                "Client project not set:  pass an explicit project.")
+
+        extra_params = {'project': project}
 
         if prefix is not None:
             extra_params['prefix'] = prefix
diff --git a/storage/google/cloud/storage/notification.py b/storage/google/cloud/storage/notification.py
--- a/storage/google/cloud/storage/notification.py
+++ b/storage/google/cloud/storage/notification.py
@@ -76,6 +76,11 @@ def __init__(self, bucket, topic_name,
 
         if topic_project is None:
             topic_project = bucket.client.project
+
+        if topic_project is None:
+            raise ValueError(
+                "Client project not set:  pass an explicit topic_project.")
+
         self._topic_project = topic_project
 
         self._properties = {}


Test patch:
diff --git a/storage/tests/unit/test_bucket.py b/storage/tests/unit/test_bucket.py
--- a/storage/tests/unit/test_bucket.py
+++ b/storage/tests/unit/test_bucket.py
@@ -239,6 +239,33 @@ def test_create_w_user_project(self):
         with self.assertRaises(ValueError):
             bucket.create()
 
+    def test_create_w_missing_client_project(self):
+        BUCKET_NAME = 'bucket-name'
+        USER_PROJECT = 'user-project-123'
+        connection = _Connection()
+        client = _Client(connection, project=None)
+        bucket = self._make_one(client, BUCKET_NAME)
+
+        with self.assertRaises(ValueError):
+            bucket.create()
+
+    def test_create_w_explicit_project(self):
+        PROJECT = 'PROJECT'
+        BUCKET_NAME = 'bucket-name'
+        OTHER_PROJECT = 'other-project-123'
+        DATA = {'name': BUCKET_NAME}
+        connection = _Connection(DATA)
+        client = _Client(connection, project=PROJECT)
+        bucket = self._make_one(client, BUCKET_NAME)
+
+        bucket.create(project=OTHER_PROJECT)
+
+        kw, = connection._requested
+        self.assertEqual(kw['method'], 'POST')
+        self.assertEqual(kw['path'], '/b')
+        self.assertEqual(kw['query_params'], {'project': OTHER_PROJECT})
+        self.assertEqual(kw['data'], DATA)
+
     def test_create_hit(self):
         PROJECT = 'PROJECT'
         BUCKET_NAME = 'bucket-name'
diff --git a/storage/tests/unit/test_client.py b/storage/tests/unit/test_client.py
--- a/storage/tests/unit/test_client.py
+++ b/storage/tests/unit/test_client.py
@@ -75,6 +75,38 @@ def test_ctor_connection_type(self):
         self.assertIsNone(client.current_batch)
         self.assertEqual(list(client._batch_stack), [])
 
+    def test_ctor_wo_project(self):
+        from google.cloud.storage._http import Connection
+
+        PROJECT = 'PROJECT'
+        CREDENTIALS = _make_credentials()
+
+        ddp_patch = mock.patch(
+            'google.cloud.client._determine_default_project',
+            return_value=PROJECT)
+
+        with ddp_patch:
+            client = self._make_one(credentials=CREDENTIALS)
+
+        self.assertEqual(client.project, PROJECT)
+        self.assertIsInstance(client._connection, Connection)
+        self.assertIs(client._connection.credentials, CREDENTIALS)
+        self.assertIsNone(client.current_batch)
+        self.assertEqual(list(client._batch_stack), [])
+
+    def test_ctor_w_project_explicit_none(self):
+        from google.cloud.storage._http import Connection
+
+        CREDENTIALS = _make_credentials()
+
+        client = self._make_one(project=None, credentials=CREDENTIALS)
+
+        self.assertIsNone(client.project)
+        self.assertIsInstance(client._connection, Connection)
+        self.assertIs(client._connection.credentials, CREDENTIALS)
+        self.assertIsNone(client.current_batch)
+        self.assertEqual(list(client._batch_stack), [])
+
     def test_create_anonymous_client(self):
         from google.auth.credentials import AnonymousCredentials
         from google.cloud.storage._http import Connection
@@ -286,6 +318,7 @@ def test_create_bucket_conflict(self):
         from google.cloud.exceptions import Conflict
 
         PROJECT = 'PROJECT'
+        OTHER_PROJECT = 'OTHER_PROJECT'
         CREDENTIALS = _make_credentials()
         client = self._make_one(project=PROJECT, credentials=CREDENTIALS)
 
@@ -294,7 +327,7 @@ def test_create_bucket_conflict(self):
             client._connection.API_BASE_URL,
             'storage',
             client._connection.API_VERSION,
-            'b?project=%s' % (PROJECT,),
+            'b?project=%s' % (OTHER_PROJECT,),
         ])
         data = {'error': {'message': 'Conflict'}}
         sent = {'name': BUCKET_NAME}
@@ -302,7 +335,9 @@ def test_create_bucket_conflict(self):
             _make_json_response(data, status=http_client.CONFLICT)])
         client._http_internal = http
 
-        self.assertRaises(Conflict, client.create_bucket, BUCKET_NAME)
+        with self.assertRaises(Conflict):
+            client.create_bucket(BUCKET_NAME, project=OTHER_PROJECT)
+
         http.request.assert_called_once_with(
             method='POST', url=URI, data=mock.ANY, headers=mock.ANY)
         json_sent = http.request.call_args_list[0][1]['data']
@@ -337,6 +372,13 @@ def test_create_bucket_success(self):
         json_sent = http.request.call_args_list[0][1]['data']
         self.assertEqual(sent, json.loads(json_sent))
 
+    def test_list_buckets_wo_project(self):
+        CREDENTIALS = _make_credentials()
+        client = self._make_one(project=None, credentials=CREDENTIALS)
+
+        with self.assertRaises(ValueError):
+            client.list_buckets()
+
     def test_list_buckets_empty(self):
         from six.moves.urllib.parse import parse_qs
         from six.moves.urllib.parse import urlparse
@@ -371,6 +413,41 @@ def test_list_buckets_empty(self):
         uri_parts = urlparse(requested_url)
         self.assertEqual(parse_qs(uri_parts.query), expected_query)
 
+    def test_list_buckets_explicit_project(self):
+        from six.moves.urllib.parse import parse_qs
+        from six.moves.urllib.parse import urlparse
+
+        PROJECT = 'PROJECT'
+        OTHER_PROJECT = 'OTHER_PROJECT'
+        CREDENTIALS = _make_credentials()
+        client = self._make_one(project=PROJECT, credentials=CREDENTIALS)
+
+        http = _make_requests_session([_make_json_response({})])
+        client._http_internal = http
+
+        buckets = list(client.list_buckets(project=OTHER_PROJECT))
+
+        self.assertEqual(len(buckets), 0)
+
+        http.request.assert_called_once_with(
+            method='GET', url=mock.ANY, data=mock.ANY, headers=mock.ANY)
+
+        requested_url = http.request.mock_calls[0][2]['url']
+        expected_base_url = '/'.join([
+            client._connection.API_BASE_URL,
+            'storage',
+            client._connection.API_VERSION,
+            'b',
+        ])
+        self.assertTrue(requested_url.startswith(expected_base_url))
+
+        expected_query = {
+            'project': [OTHER_PROJECT],
+            'projection': ['noAcl'],
+        }
+        uri_parts = urlparse(requested_url)
+        self.assertEqual(parse_qs(uri_parts.query), expected_query)
+
     def test_list_buckets_non_empty(self):
         PROJECT = 'PROJECT'
         CREDENTIALS = _make_credentials()
diff --git a/storage/tests/unit/test_notification.py b/storage/tests/unit/test_notification.py
--- a/storage/tests/unit/test_notification.py
+++ b/storage/tests/unit/test_notification.py
@@ -74,6 +74,13 @@ def _make_bucket(self, client, name=BUCKET_NAME, user_project=None):
          bucket.user_project = user_project
          return bucket
 
+    def test_ctor_w_missing_project(self):
+        client = self._make_client(project=None)
+        bucket = self._make_bucket(client)
+
+        with self.assertRaises(ValueError):
+            self._make_one(bucket, self.TOPIC_NAME)
+
     def test_ctor_defaults(self):
         from google.cloud.storage.notification import NONE_PAYLOAD_FORMAT