Repository: googleapis/google-auth-library-python

Issue:
Breaking changes for impersonated_credentials between 1.6.3 and 1.7.0+
#### Environment details

  - OS: MacOS Mojave
  - Python version: 3.7.3
  - pip version: 19.0.3
  - `google-auth` version: 1.6.3, 1.7.0, 1.10.0

The default credentials are my user credentials (with the ServiceAccountTokenCreator role on the service account). 

#### Steps to reproduce
The following code works and produces a valid access token for the service account in version 1.6.3.

```python
from google.auth import impersonated_credentials, default
from google.auth.transport.requests import Request

sa = "<sa>@<project>.iam.gserviceaccount.com"
source_credentials, _ = default()
scopes = ["https://www.googleapis.com/auth/cloud-platform"]
creds = impersonated_credentials.Credentials(
    source_credentials=source_credentials, target_principal=sa, target_scopes=scopes
)

creds.refresh(Request())
print(creds.token)
```
After upgrading to 1.10.0, it fails to authenticate the service account due to invalid scopes.

```
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    creds.refresh(Request())
  File "/google/lib/python3.7/site-packages/google/auth/impersonated_credentials.py", line 218, in refresh
    self._update_token(request)
  File "/google/lib/python3.7/site-packages/google/auth/impersonated_credentials.py", line 234, in _update_token
    self._source_credentials.refresh(request)
  File "/google/lib/python3.7/site-packages/google/oauth2/credentials.py", line 152, in refresh
    self._scopes,
  File "/google/lib/python3.7/site-packages/google/oauth2/_client.py", line 241, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/google/lib/python3.7/site-packages/google/oauth2/_client.py", line 115, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/google/lib/python3.7/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_scope: Bad Request', '{\n  "error": "invalid_scope",\n  "error_description": "Bad Request"\n}')
```

I traced this back to an google-auth upgrade from 1.6.3 to 1.7.0 (same error occurs).


Discussion:
Facing the same issue. Seems to be related with the fact that `impersonated_credentials` set the source credentials scopes to `['https://www.googleapis.com/auth/iam']` which is [undocumented](https://developers.google.com/identity/protocols/googlescopes) [ref](https://github.com/googleapis/google-auth-library-python/blob/master/google/auth/impersonated_credentials.py#L208).

A work around is to reset the scope of the source credential

```python
user_cred, _ = default()
credentials = Credentials(user_cred, sa, scopes)
credentials._source_credentials._scopes = user_cred.scopes
creds.refresh(Request())
```
Thanks @gpoulin - I confirmed the workaround as well.

Seeing the exact same issue - macOS impersonated creds scopes break on google-auth>1.6.3
Adding some more context to this, the scope was always there but scopes were not actually requested until this change was made:
https://github.com/googleapis/google-auth-library-python/commit/49a18c43a5b39615bad90d23ab33a105fb656a6a?diff=split#diff-1b020acb6247d690d2ca64fcb04c8f53R241-R242
Thank you all for the investigation and details you've provided.

The general documentation for impersonated credentials lives [here](https://cloud.google.com/iam/docs/creating-short-lived-service-account-credentials). 

These docs state that `auth/iam/` scope and the `auth/cloud-platform/` scopes are valid. It doesn't go into detail as to if there's a difference between the two.

> scope: The OAuth 2.0 scope for the request. The following scopes are valid when calling the generateAccessToken API:
        * https://www.googleapis.com/auth/iam
        * https://www.googleapis.com/auth/cloud-platform

As @gpoulin points out the scope is not mentioned on the list of OAuth2 scopes at https://developers.google.com/identity/protocols/googlescopes, but it may just be out of date.

I will do some more investigation internally and update this issue.
Alright, so I think I've figured out the root cause. Impersonated credentials modify the scope of the source credential. This follows the guide  [here](https://cloud.google.com/iam/docs/creating-short-lived-service-account-credentials#sa-credentials-oauth). 

https://github.com/googleapis/google-auth-library-python/blob/31299e231912fe3c7a0e43dd3677c9f9391471c5/google/auth/impersonated_credentials.py#L208

Howecer `google.oauth2.credentials` *cannot* have their scopes modified after creation, so the source credential refresh fails with `invalid_scope`.
https://github.com/googleapis/google-auth-library-python/blob/31299e231912fe3c7a0e43dd3677c9f9391471c5/google/auth/impersonated_credentials.py#L224-L252
https://github.com/googleapis/google-auth-library-python/blob/31299e231912fe3c7a0e43dd3677c9f9391471c5/google/oauth2/service_account.py#L232-L239




Resolving patch:
diff --git a/google/auth/impersonated_credentials.py b/google/auth/impersonated_credentials.py
--- a/google/auth/impersonated_credentials.py
+++ b/google/auth/impersonated_credentials.py
@@ -205,7 +205,11 @@ def __init__(
         super(Credentials, self).__init__()
 
         self._source_credentials = copy.copy(source_credentials)
-        self._source_credentials._scopes = _IAM_SCOPE
+        # Service account source credentials must have the _IAM_SCOPE
+        # added to refresh correctly. User credentials cannot have
+        # their original scopes modified.
+        if isinstance(self._source_credentials, credentials.Scoped):
+            self._source_credentials = self._source_credentials.with_scopes(_IAM_SCOPE)
         self._target_principal = target_principal
         self._target_scopes = target_scopes
         self._delegates = delegates


Test patch:
diff --git a/system_tests/conftest.py b/system_tests/conftest.py
--- a/system_tests/conftest.py
+++ b/system_tests/conftest.py
@@ -25,6 +25,9 @@
 
 HERE = os.path.dirname(__file__)
 DATA_DIR = os.path.join(HERE, "data")
+IMPERSONATED_SERVICE_ACCOUNT_FILE = os.path.join(
+    DATA_DIR, "impersonated_service_account.json"
+)
 SERVICE_ACCOUNT_FILE = os.path.join(DATA_DIR, "service_account.json")
 AUTHORIZED_USER_FILE = os.path.join(DATA_DIR, "authorized_user.json")
 URLLIB3_HTTP = urllib3.PoolManager(retries=False)
@@ -39,6 +42,12 @@ def service_account_file():
     yield SERVICE_ACCOUNT_FILE
 
 
+@pytest.fixture
+def impersonated_service_account_file():
+    """The full path to a valid service account key file."""
+    yield IMPERSONATED_SERVICE_ACCOUNT_FILE
+
+
 @pytest.fixture
 def authorized_user_file():
     """The full path to a valid authorized user file."""
diff --git a/system_tests/noxfile.py b/system_tests/noxfile.py
--- a/system_tests/noxfile.py
+++ b/system_tests/noxfile.py
@@ -170,7 +170,8 @@ def configure_cloud_sdk(session, application_default_credentials, project=False)
 # Test sesssions
 
 TEST_DEPENDENCIES = ["pytest", "requests"]
-PYTHON_VERSIONS=['2.7', '3.7']
+PYTHON_VERSIONS = ["2.7", "3.7"]
+
 
 @nox.session(python=PYTHON_VERSIONS)
 def service_account(session):
@@ -186,6 +187,13 @@ def oauth2_credentials(session):
     session.run("pytest", "test_oauth2_credentials.py")
 
 
+@nox.session(python=PYTHON_VERSIONS)
+def impersonated_credentials(session):
+    session.install(*TEST_DEPENDENCIES)
+    session.install(LIBRARY_DIR)
+    session.run("pytest", "test_impersonated_credentials.py")
+
+
 @nox.session(python=PYTHON_VERSIONS)
 def default_explicit_service_account(session):
     session.env[EXPLICIT_CREDENTIALS_ENV] = SERVICE_ACCOUNT_FILE
diff --git a/system_tests/secrets.tar.enc b/system_tests/secrets.tar.enc
Binary files a/system_tests/secrets.tar.enc and b/system_tests/secrets.tar.enc differ
diff --git a/system_tests/test_impersonated_credentials.py b/system_tests/test_impersonated_credentials.py
new file mode 100644
--- /dev/null
+++ b/system_tests/test_impersonated_credentials.py
@@ -0,0 +1,99 @@
+# Copyright 2020 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+import pytest
+
+import google.oauth2.credentials
+from google.oauth2 import service_account
+import google.auth.impersonated_credentials
+from google.auth import _helpers
+
+
+GOOGLE_OAUTH2_TOKEN_ENDPOINT = "https://oauth2.googleapis.com/token"
+
+
+@pytest.fixture
+def service_account_credentials(service_account_file):
+    yield service_account.Credentials.from_service_account_file(service_account_file)
+
+
+@pytest.fixture
+def impersonated_service_account_credentials(impersonated_service_account_file):
+    yield service_account.Credentials.from_service_account_file(
+        impersonated_service_account_file
+    )
+
+
+def test_refresh_with_user_credentials_as_source(
+    authorized_user_file,
+    impersonated_service_account_credentials,
+    http_request,
+    token_info,
+):
+    with open(authorized_user_file, "r") as fh:
+        info = json.load(fh)
+
+    source_credentials = google.oauth2.credentials.Credentials(
+        None,
+        refresh_token=info["refresh_token"],
+        token_uri=GOOGLE_OAUTH2_TOKEN_ENDPOINT,
+        client_id=info["client_id"],
+        client_secret=info["client_secret"],
+        # The source credential needs this scope for the generateAccessToken request
+        # The user must also have `Service Account Token Creator` on the project
+        # that owns the impersonated service account.
+        # See https://cloud.google.com/iam/docs/creating-short-lived-service-account-credentials
+        scopes=["https://www.googleapis.com/auth/cloud-platform"],
+    )
+
+    source_credentials.refresh(http_request)
+
+    target_scopes = [
+        "https://www.googleapis.com/auth/devstorage.read_only",
+        "https://www.googleapis.com/auth/analytics",
+    ]
+    target_credentials = google.auth.impersonated_credentials.Credentials(
+        source_credentials=source_credentials,
+        target_principal=impersonated_service_account_credentials.service_account_email,
+        target_scopes=target_scopes,
+        lifetime=100,
+    )
+
+    target_credentials.refresh(http_request)
+    assert target_credentials.token
+
+
+def test_refresh_with_service_account_credentials_as_source(
+    http_request,
+    service_account_credentials,
+    impersonated_service_account_credentials,
+    token_info,
+):
+    source_credentials = service_account_credentials.with_scopes(["email"])
+    source_credentials.refresh(http_request)
+    assert source_credentials.token
+
+    target_scopes = [
+        "https://www.googleapis.com/auth/devstorage.read_only",
+        "https://www.googleapis.com/auth/analytics",
+    ]
+    target_credentials = google.auth.impersonated_credentials.Credentials(
+        source_credentials=source_credentials,
+        target_principal=impersonated_service_account_credentials.service_account_email,
+        target_scopes=target_scopes,
+    )
+
+    target_credentials.refresh(http_request)
+    assert target_credentials.token
diff --git a/tests/test_impersonated_credentials.py b/tests/test_impersonated_credentials.py
--- a/tests/test_impersonated_credentials.py
+++ b/tests/test_impersonated_credentials.py
@@ -26,6 +26,7 @@
 from google.auth import impersonated_credentials
 from google.auth import transport
 from google.auth.impersonated_credentials import Credentials
+from google.oauth2 import credentials
 from google.oauth2 import service_account
 
 DATA_DIR = os.path.join(os.path.dirname(__file__), "", "data")
@@ -102,17 +103,30 @@ class TestImpersonatedCredentials(object):
     SOURCE_CREDENTIALS = service_account.Credentials(
         SIGNER, SERVICE_ACCOUNT_EMAIL, TOKEN_URI
     )
+    USER_SOURCE_CREDENTIALS = credentials.Credentials(token="ABCDE")
 
-    def make_credentials(self, lifetime=LIFETIME, target_principal=TARGET_PRINCIPAL):
+    def make_credentials(
+        self,
+        source_credentials=SOURCE_CREDENTIALS,
+        lifetime=LIFETIME,
+        target_principal=TARGET_PRINCIPAL,
+    ):
 
         return Credentials(
-            source_credentials=self.SOURCE_CREDENTIALS,
+            source_credentials=source_credentials,
             target_principal=target_principal,
             target_scopes=self.TARGET_SCOPES,
             delegates=self.DELEGATES,
             lifetime=lifetime,
         )
 
+    def test_make_from_user_credentials(self):
+        credentials = self.make_credentials(
+            source_credentials=self.USER_SOURCE_CREDENTIALS
+        )
+        assert not credentials.valid
+        assert credentials.expired
+
     def test_default_state(self):
         credentials = self.make_credentials()
         assert not credentials.valid