Repository: googleapis/google-cloud-python

Issue:
Is our authentication broken?
Blog post has a pretty sad story, which isn't at all what is supposed to happen...

Link: https://synack.me/blog/app-engine-to-kubernetes

> The blog is a fairly simplistic use case for datastore, so I decided to move forward with porting the handful of queries over from the ORM to the GQL interface. As I began testing these changes, I ran into authentication errors when trying to query datastore. There are three authentication methods supported by the gcloud library: Explicit credentials, credentials from a file, or a service account. Service accounts are a really nice feature of Google Cloud, allowing you to automatically provide access to other Google services from Compute Engine instances running in your project. I found that my development instance didn't have the proper scopes enabled for datastore access when it was created, so I had to delete the instance (keeping the disks) and re-create it with the service auth checkbox checked. After spending a few hours debugging an unrelated issue where I had broken the boot scripts on my dev instance, I tried to access datastore with the service account again and still got an unauthorized error. I did a fair bit more debugging, watching HTTP requests fly back and forth, ensuring that the gcloud library was getting an OAuth token using the service account and passing that in a header to the datastore service, but never managed to get this to work. I suspect there's something broken in either the gcloud library or the public datastore service preventing service accounts from authenticating properly. I generated a new service account in the Cloud Console, exported it as a json file, and passed that to the gcloud.datastore.Client.from_service_account_json classmethod to authenticate my app instead. This worked on the first try. Unfortunately, now I have a file full of secrets to worry about...

Are we not properly detecting Datastore permissions ? Maybe he was missing the project ID (dataset ID) because we don't auto-detect that?

/cc @JeremyGrosser



Discussion:
Just as an FYI:

``` python
>>> from gcloud import datastore
>>> c = datastore.Client()
>>> c.project
'jjg-cloud-research'
>>> c.get(c.key('Person', 'jimmy'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/dist-packages/gcloud/datastore/client.py", line 262, in get
    deferred=deferred)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/datastore/client.py", line 303, in get_multi
    transaction_id=transaction and transaction.id,
  File "/usr/local/lib/python2.7/dist-packages/gcloud/datastore/client.py", line 136, in _extended_lookup
    transaction_id=transaction_id,
  File "/usr/local/lib/python2.7/dist-packages/gcloud/datastore/connection.py", line 200, in lookup
    _datastore_pb2.LookupResponse)
  File "/usr/local/lib/python2.7/dist-packages/gcloud/datastore/connection.py", line 118, in _rpc
    data=request_pb.SerializeToString())
  File "/usr/local/lib/python2.7/dist-packages/gcloud/datastore/connection.py", line 91, in _request
    method='POST', headers=headers, body=data)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/client.py", line 598, in new_request
    self._refresh(request_orig)
  File "/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.py", line 102, in _refresh
    raise HttpAccessTokenRefreshError(content, status=response.status)
oauth2client.client.HttpAccessTokenRefreshError
```

And here's a screenshot from the GCE instance's access panel....

![access](https://cloud.githubusercontent.com/assets/112928/13117654/e3d3a7a2-d56e-11e5-9116-5a24fd64fe9f.png)

``` python
>>> c.connection.credentials
<oauth2client.contrib.gce.AppAssertionCredentials object at 0x7fbc1c977650>
>>> creds = c.connection.credentials
>>> creds.scope
'https://www.googleapis.com/auth/datastore https://www.googleapis.com/auth/userinfo.email'
```

Alllll right, looks like some Compute Engine metadata ... did some crazy stuff...

We're trying to access `/computeMetadata/v1/instance/service-accounts/default/acquire?scope=...`. Which isn't found...

``` sh
jjg@gcloud-test-1:~$ curl metadata/computeMetadata/v1/instance/service-accounts/default/ -H "Metadata-Flavor: Google"
aliases
email
scopes
token
```

This URL _does_ however exist somehow at a different path...

``` sh
jjg@gcloud-test-1:~$ curl metadata/0.1/meta-data/service-accounts/default/ -H "Metadata-Flavor: Google"
acquire
```

This is all up in oauth2client... so ... Who do we bother in that library to fix this...

I can fix.

Yay :)

Danny's spelunking found this https://github.com/google/oauth2client/pull/387 which looks to be breaking GCE auth. /cc @nathanielmanistaatgoogle

/cc @Galabar001

@jgeewax can you also list the scopes that your instance has via `gcloud compute instances describe`?

``` sh
$ gcloud --project jjg-cloud-research compute instances describe us-east1-b/gcloud-test-1
canIpForward: false
cpuPlatform: Intel Haswell
creationTimestamp: '2016-02-17T07:50:49.888-08:00'
description: ''
disks:
- autoDelete: true
  boot: true
  deviceName: gcloud-test-1
  index: 0
  interface: SCSI
  kind: compute#attachedDisk
  licenses:
  - https://www.googleapis.com/compute/v1/projects/ubuntu-os-cloud/global/licenses/ubuntu-1510-wily
  mode: READ_WRITE
  source: https://www.googleapis.com/compute/v1/projects/jjg-cloud-research/zones/us-east1-b/disks/gcloud-test-1
  type: PERSISTENT
id: '5495952314270583994'
kind: compute#instance
machineType: https://www.googleapis.com/compute/v1/projects/jjg-cloud-research/zones/us-east1-b/machineTypes/n1-standard-1
metadata:
  fingerprint: A2bOqA2ht4o=
  kind: compute#metadata
name: gcloud-test-1
networkInterfaces:
- accessConfigs:
  - kind: compute#accessConfig
    name: External NAT
    natIP: 104.196.117.27
    type: ONE_TO_ONE_NAT
  name: nic0
  network: https://www.googleapis.com/compute/v1/projects/jjg-cloud-research/global/networks/default
  networkIP: 10.240.0.5
scheduling:
  automaticRestart: true
  onHostMaintenance: MIGRATE
  preemptible: false
selfLink: https://www.googleapis.com/compute/v1/projects/jjg-cloud-research/zones/us-east1-b/instances/gcloud-test-1
serviceAccounts:
- email: <redacted>@developer.gserviceaccount.com
  scopes:
  - https://www.googleapis.com/auth/cloud-platform
status: RUNNING
tags:
  fingerprint: 42WmSpB8rSM=
zone: https://www.googleapis.com/compute/v1/projects/jjg-cloud-research/zones/us-east1-b
```

To add another layer of fun here `cloud-platform` isn't sufficient to access Cloud Datastore. You additionally need `userinfo.email`. This won't be true once `v1beta3` comes out, but it's true for now.

@jonparrott Jinx! Also ISTM that sending `?scope=FOO` when acquiring a token is pointless

Pinging `https://www.googleapis.com/oauth2/v2/tokeninfo?access_token=ya29...` with a token returned from the metadata service just gives

```
{
 "issued_to": "anonymous",
 "audience": "anonymous",
 "scope": "https://www.googleapis.com/auth/cloud-platform",
 "expires_in": 1623,
 "access_type": "offline"
}
```

no matter what I asked for. (This is because I didn't add the `email` scope for the VM when I created it. Silly me.)

It seems like once you click that checkbox in the UI though ("give me access to everything") it doesn't let you also add the User Info permission... I'm going to walk over and ask some folks who work on the cloud console to take a look.....

Thanks.

> Also ISTM that sending ?scope=FOO

Yep, scopes are fixed at instance creation time and you can't request a sub or superset of them.

> It seems like once you click that checkbox in the UI though ("give me access to everything") it doesn't let you also add the User Info permission...

Yep, it's really annoying. Our getting started guides advise users to use `gcloud` to create instances specifically for this reason.

FYI, Managed VMs is unaffected as we added `userinfo.email` to the default set of scopes for MVMs months ago.

OK, so the fix for right now (aka, you can actually do this) is....
1. Take a snapshot of your disk from instance-1
2. Clone instance-1, boot from the snapshot, and then in the access permissions thing, don't check the box, but instead go through manually and "enable" everything in all of the dropdowns (aka, click until you hate your mouse)
3. Great success. Datastore should now work automagically.

The fix on our side is...
1. When I check that box saying "give all cloud permissions", it should actually mean that, and include the `userinfo.email` scope as well as the `cloud-platform` scope.

I'm really sorry that this is all we can do now... The team in Warsaw is probably gone for the day, so this will have to wait until tomorrow. :(

![1324dfd733535e58dba70264e6d05c9b70346204d2cacef65abef9c702746d1c](https://cloud.githubusercontent.com/assets/520669/13120437/ad669c76-d562-11e5-9909-8cb026914651.jpg)


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -14,7 +14,7 @@
 REQUIREMENTS = [
     'httplib2 >= 0.9.1',
     'googleapis-common-protos',
-    'oauth2client >= 2.0.0',
+    'oauth2client >= 2.0.0.post1',
     'protobuf >= 3.0.0b2',
     'pyOpenSSL',
     'six',
@@ -22,7 +22,7 @@
 
 setup(
     name='gcloud',
-    version='0.10.0',
+    version='0.10.1',
     description='API Client library for Google Cloud',
     author='Google Cloud Platform',
     author_email='jjg+gcloud-python@google.com',


Test patch: