Repository: googleapis/google-auth-library-python

Issue:
Use extra for asyncio dependencies
Hello!  The latest release for this library pulls in aiohttp and its dependencies unconditionally, which adds non-trivial burden to projects that don’t need it.  Would you consider using a packaging extra so that people can opt-in?
TODO: undo pin of 'aiohttp' once 'aioresponses' releases a fix
Environment details

  - OS: $ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.6
BuildVersion:   18G6020

  - Python version: 3.6, 3.7, 3.8
  - pip version: pip 20.2.4
  - `google-auth` version: 5906c8583ca351b5385a079a30521a9a8a0c7c59

#### Steps to reproduce

  1. nox -s unit


There are 9 tests that fail, all with the same error:

`TypeError: __init__() missing 1 required positional argument: 'limit'`


```
====================================================== short test summary info =======================================================
FAILED tests_async/transport/test_aiohttp_requests.py::TestCombinedResponse::test_content_compressed - TypeError: __init__() missin...
FAILED tests_async/transport/test_aiohttp_requests.py::TestResponse::test_headers_prop - TypeError: __init__() missing 1 required p...
FAILED tests_async/transport/test_aiohttp_requests.py::TestResponse::test_status_prop - TypeError: __init__() missing 1 required po...
FAILED tests_async/transport/test_aiohttp_requests.py::TestAuthorizedSession::test_request - TypeError: __init__() missing 1 requir...
FAILED tests_async/transport/test_aiohttp_requests.py::TestAuthorizedSession::test_ctx - TypeError: __init__() missing 1 required p...
FAILED tests_async/transport/test_aiohttp_requests.py::TestAuthorizedSession::test_http_headers - TypeError: __init__() missing 1 r...
FAILED tests_async/transport/test_aiohttp_requests.py::TestAuthorizedSession::test_regexp_example - TypeError: __init__() missing 1...
FAILED tests_async/transport/test_aiohttp_requests.py::TestAuthorizedSession::test_request_no_refresh - TypeError: __init__() missi...
FAILED tests_async/transport/test_aiohttp_requests.py::TestAuthorizedSession::test_request_refresh - TypeError: __init__() missing ...
============================================ 9 failed, 609 passed, 12 warnings in 33.41s =============================================
```

Here is the traceback for one of the failing tests:


```
____________________________________________ TestCombinedResponse.test_content_compressed ____________________________________________

self = <tests_async.transport.test_aiohttp_requests.TestCombinedResponse object at 0x108803160>
urllib3_mock = <function decompress at 0x10880a820>

    @mock.patch(
        "google.auth.transport._aiohttp_requests.urllib3.response.MultiDecoder.decompress",
        return_value="decompressed",
        autospec=True,
    )
    @pytest.mark.asyncio
    async def test_content_compressed(self, urllib3_mock):
        rm = core.RequestMatch(
            "url", headers={"Content-Encoding": "gzip"}, payload="compressed"
        )
>       response = await rm.build_response(core.URL("url"))

tests_async/transport/test_aiohttp_requests.py:72: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../.virtualenv/google-auth-library-python/lib/python3.8/site-packages/aioresponses/core.py:192: in build_response
    resp = self._build_response(
../../../.virtualenv/google-auth-library-python/lib/python3.8/site-packages/aioresponses/core.py:173: in _build_response
    resp.content = stream_reader_factory(loop)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

loop = <Mock id='4437587472'>

    def stream_reader_factory(  # noqa
        loop: 'Optional[asyncio.AbstractEventLoop]' = None
    ):
        protocol = ResponseHandler(loop=loop)
>       return StreamReader(protocol, loop=loop)
E       TypeError: __init__() missing 1 required positional argument: 'limit'

../../../.virtualenv/google-auth-library-python/lib/python3.8/site-packages/aioresponses/compat.py:48: TypeError
========================================================== warnings summary ==========================================================
```

The root cause is a change in aiohttp version 3.7.0 which was released a few hours ago. The signature for StreamReader has changed, making the optional argument `limit` a required argument.

https://github.com/aio-libs/aiohttp/blob/56e78836aa7c67292ace9e256711699d51d57285/aiohttp/streams.py#L106

This change breaks aioresponses:

https://github.com/pnuckowski/aioresponses/blob/e61977f42a0164e0c572031dfb18ae95ba198df0/aioresponses/compat.py#L44


Add support for Python 3.9



Discussion:

Resolving patch:
diff --git a/google/auth/_default.py b/google/auth/_default.py
--- a/google/auth/_default.py
+++ b/google/auth/_default.py
@@ -195,28 +195,9 @@ def _get_gcloud_sdk_credentials():
     return credentials, project_id
 
 
-def _get_explicit_environ_credentials(request=None, scopes=None, default_scopes=None):
+def _get_explicit_environ_credentials():
     """Gets credentials from the GOOGLE_APPLICATION_CREDENTIALS environment
-    variable.
-
-    Args:
-        request (Optional[google.auth.transport.Request]): An object used to make
-            HTTP requests. This is used to determine the associated project ID
-            for a workload identity pool resource (external account credentials).
-            If not specified, then it will use a
-            google.auth.transport.requests.Request client to make requests.
-        scopes (Optional[Sequence[str]]): The list of scopes for the credentials. If
-            specified, the credentials will automatically be scoped if
-            necessary.
-        default_scopes (Optional[Sequence[str]]): Default scopes passed by a
-            Google client library. Use 'scopes' for user-defined scopes.
-
-    Returns:
-        Tuple[Optional[google.auth.credentials.Credentials], Optional[str]]: Loaded
-            credentials and the project ID. Authorized user credentials do not
-            have the project ID information. External account credentials project
-            IDs may not always be determined.
-    """
+    variable."""
     explicit_file = os.environ.get(environment_vars.CREDENTIALS)
 
     _LOGGER.debug(
@@ -225,11 +206,7 @@ def _get_explicit_environ_credentials(request=None, scopes=None, default_scopes=
 
     if explicit_file is not None:
         credentials, project_id = load_credentials_from_file(
-            os.environ[environment_vars.CREDENTIALS],
-            scopes=scopes,
-            default_scopes=default_scopes,
-            quota_project_id=None,
-            request=request,
+            os.environ[environment_vars.CREDENTIALS]
         )
 
         return credentials, project_id
@@ -450,9 +427,11 @@ def default(scopes=None, request=None, quota_project_id=None, default_scopes=Non
     )
 
     checkers = (
-        lambda: _get_explicit_environ_credentials(
-            request=request, scopes=scopes, default_scopes=default_scopes
-        ),
+        # Avoid passing scopes here to prevent passing scopes to user credentials.
+        # with_scopes_if_required() below will ensure scopes/default scopes are
+        # safely set on the returned credentials since requires_scopes will
+        # guard against setting scopes on user credentials.
+        _get_explicit_environ_credentials,
         _get_gcloud_sdk_credentials,
         _get_gae_credentials,
         lambda: _get_gce_credentials(request),
@@ -464,6 +443,17 @@ def default(scopes=None, request=None, quota_project_id=None, default_scopes=Non
             credentials = with_scopes_if_required(
                 credentials, scopes, default_scopes=default_scopes
             )
+
+            # For external account credentials, scopes are required to determine
+            # the project ID. Try to get the project ID again if not yet
+            # determined.
+            if not project_id and callable(
+                getattr(credentials, "get_project_id", None)
+            ):
+                if request is None:
+                    request = google.auth.transport.requests.Request()
+                project_id = credentials.get_project_id(request=request)
+
             if quota_project_id:
                 credentials = credentials.with_quota_project(quota_project_id)
 


Test patch:
diff --git a/tests/test__default.py b/tests/test__default.py
--- a/tests/test__default.py
+++ b/tests/test__default.py
@@ -83,6 +83,14 @@
 MOCK_CREDENTIALS = mock.Mock(spec=credentials.CredentialsWithQuotaProject)
 MOCK_CREDENTIALS.with_quota_project.return_value = MOCK_CREDENTIALS
 
+
+def get_project_id_side_effect(self, request=None):
+    # If no scopes are set, this will always return None.
+    if not self.scopes:
+        return None
+    return mock.sentinel.project_id
+
+
 LOAD_FILE_PATCH = mock.patch(
     "google.auth._default.load_credentials_from_file",
     return_value=(MOCK_CREDENTIALS, mock.sentinel.project_id),
@@ -91,7 +99,7 @@
 EXTERNAL_ACCOUNT_GET_PROJECT_ID_PATCH = mock.patch.object(
     external_account.Credentials,
     "get_project_id",
-    return_value=mock.sentinel.project_id,
+    side_effect=get_project_id_side_effect,
     autospec=True,
 )
 
@@ -231,7 +239,8 @@ def test_load_credentials_from_file_external_account_identity_pool(
     credentials, project_id = _default.load_credentials_from_file(str(config_file))
 
     assert isinstance(credentials, identity_pool.Credentials)
-    assert project_id is mock.sentinel.project_id
+    # Since no scopes are specified, the project ID cannot be determined.
+    assert project_id is None
     assert get_project_id.called
 
 
@@ -242,7 +251,8 @@ def test_load_credentials_from_file_external_account_aws(get_project_id, tmpdir)
     credentials, project_id = _default.load_credentials_from_file(str(config_file))
 
     assert isinstance(credentials, aws.Credentials)
-    assert project_id is mock.sentinel.project_id
+    # Since no scopes are specified, the project ID cannot be determined.
+    assert project_id is None
     assert get_project_id.called
 
 
@@ -259,6 +269,7 @@ def test_load_credentials_from_file_external_account_with_user_and_default_scope
     )
 
     assert isinstance(credentials, identity_pool.Credentials)
+    # Since scopes are specified, the project ID can be determined.
     assert project_id is mock.sentinel.project_id
     assert credentials.scopes == ["https://www.google.com/calendar/feeds"]
     assert credentials.default_scopes == [
@@ -277,7 +288,8 @@ def test_load_credentials_from_file_external_account_with_quota_project(
     )
 
     assert isinstance(credentials, identity_pool.Credentials)
-    assert project_id is mock.sentinel.project_id
+    # Since no scopes are specified, the project ID cannot be determined.
+    assert project_id is None
     assert credentials.quota_project_id == "project-foo"
 
 
@@ -300,10 +312,13 @@ def test_load_credentials_from_file_external_account_explicit_request(
     config_file = tmpdir.join("config.json")
     config_file.write(json.dumps(IDENTITY_POOL_DATA))
     credentials, project_id = _default.load_credentials_from_file(
-        str(config_file), request=mock.sentinel.request
+        str(config_file),
+        request=mock.sentinel.request,
+        scopes=["https://www.googleapis.com/auth/cloud-platform"],
     )
 
     assert isinstance(credentials, identity_pool.Credentials)
+    # Since scopes are specified, the project ID can be determined.
     assert project_id is mock.sentinel.project_id
     get_project_id.assert_called_with(credentials, request=mock.sentinel.request)
 
@@ -321,34 +336,7 @@ def test__get_explicit_environ_credentials(load, monkeypatch):
 
     assert credentials is MOCK_CREDENTIALS
     assert project_id is mock.sentinel.project_id
-    load.assert_called_with(
-        "filename",
-        scopes=None,
-        default_scopes=None,
-        quota_project_id=None,
-        request=None,
-    )
-
-
-@LOAD_FILE_PATCH
-def test__get_explicit_environ_credentials_with_scopes_and_request(load, monkeypatch):
-    scopes = ["one", "two"]
-    monkeypatch.setenv(environment_vars.CREDENTIALS, "filename")
-
-    credentials, project_id = _default._get_explicit_environ_credentials(
-        request=mock.sentinel.request, scopes=scopes
-    )
-
-    assert credentials is MOCK_CREDENTIALS
-    assert project_id is mock.sentinel.project_id
-    # Request and scopes should be propagated.
-    load.assert_called_with(
-        "filename",
-        scopes=scopes,
-        default_scopes=None,
-        quota_project_id=None,
-        request=mock.sentinel.request,
-    )
+    load.assert_called_with("filename")
 
 
 @LOAD_FILE_PATCH
@@ -664,7 +652,8 @@ def test_default_environ_external_credentials(get_project_id, monkeypatch, tmpdi
     credentials, project_id = _default.default()
 
     assert isinstance(credentials, identity_pool.Credentials)
-    assert project_id is mock.sentinel.project_id
+    # Without scopes, project ID cannot be determined.
+    assert project_id is None
 
 
 @EXTERNAL_ACCOUNT_GET_PROJECT_ID_PATCH
@@ -691,14 +680,17 @@ def test_default_environ_external_credentials_with_user_and_default_scopes_and_q
 
 
 @EXTERNAL_ACCOUNT_GET_PROJECT_ID_PATCH
-def test_default_environ_external_credentials_explicit_request(
+def test_default_environ_external_credentials_explicit_request_with_scopes(
     get_project_id, monkeypatch, tmpdir
 ):
     config_file = tmpdir.join("config.json")
     config_file.write(json.dumps(IDENTITY_POOL_DATA))
     monkeypatch.setenv(environment_vars.CREDENTIALS, str(config_file))
 
-    credentials, project_id = _default.default(request=mock.sentinel.request)
+    credentials, project_id = _default.default(
+        request=mock.sentinel.request,
+        scopes=["https://www.googleapis.com/auth/cloud-platform"],
+    )
 
     assert isinstance(credentials, identity_pool.Credentials)
     assert project_id is mock.sentinel.project_id