Repository: googleapis/google-api-python-client

Issue:
BatchHttpRequest and google-auth
Hi,

I'm using this client library with google-auth to make calls to several Google APIs (Drive mainly), it works fine however it seems that when using a Batch request it doesn't use the credentials of each request in the batch, only the credentials from the first one.

Which means that when I impersonate two different users in two different requests in the same batch, both requests are made impersonating the first user.

For example here's how I proceed to list the files of two users:

```
creds_one = google.oauth2.service_account.Credentials.from_service_account_file(path_to_json, scopes=scopes, subject='user1@xxx.com')
creds_two = google.oauth2.service_account.Credentials.from_service_account_file(path_to_json, scopes=scopes, subject='user2@xxx.com')

drive_service_one = build('drive', 'v3', credentials=creds_one)
drive_service_two = build('drive', 'v3', credentials=creds_two)

_batch = BatchHttpRequest()
_batch.add(request=drive_service_one.files().list(), request_id='u1')
_batch.add(request=drive_service_two.files().list(), request_id='u2')

_batch.execute()
```

What I observe in this case is that both requests return the list of files from the first user.
Am I missing something here ?

Thanks



Discussion:
@sbrieuc is this perhaps a duplicate of #350?
@jonparrott Well it doesn't look like it.

To me issue #350 relates to credentials having no valid access token and using oauth2client. From what I can see in my case the issue is specific with using credentials from google-auth library. With oauth2client credentials it works perfectly.

Looking a bit more in the code, when building a service from a credentials object there are two cases:
```
if HAS_GOOGLE_AUTH and isinstance(
            credentials, google.auth.credentials.Credentials):
        return google_auth_httplib2.AuthorizedHttp(credentials,
                                                   http=build_http())
else:
        return credentials.authorize(build_http())
```

1/ If the Credentials object is from oauth2client library, then the authorized http object returned is an Http() instance from httplib2 where the _request_ method has been overridden and on which a _credentials_ attribute is set with the actual credentials provided

```
def authorize(self, http):
        ...
        # The closure that will replace 'httplib2.Http.request'.
        def new_request(uri, method='GET', body=None, headers=None,
                        redirections=httplib2.DEFAULT_MAX_REDIRECTS,
                        connection_type=None):
            ...
            
        # Replace the request method with our own closure.
        http.request = new_request

        # Set credentials as a property of the request method.
        setattr(http.request, 'credentials', self)

        return http
```

2/ If the Credentials object however is from google-auth library, then the authorized http object returned is an instance of AuthorizedHttp. This object has its own _request_ method which does not have a _credentials_ attribute

```
class AuthorizedHttp(object):
    def __init__(self, credentials, http=None,
                 refresh_status_codes=transport.DEFAULT_REFRESH_STATUS_CODES,
                 max_refresh_attempts=transport.DEFAULT_MAX_REFRESH_ATTEMPTS):
        
        if http is None:
            http = _make_default_http()

        self.http = http
        self.credentials = credentials
        self._refresh_status_codes = refresh_status_codes
        self._max_refresh_attempts = max_refresh_attempts
        # Request instance used by internal methods (for example,
        # credentials.refresh).
        self._request = Request(self.http)

    def request(self, uri, method='GET', body=None, headers=None,
                **kwargs):
        ... 
```

Now, when making a regular request, this works fine, as the _request_ method from AuthorizedHttp takes care of refreshing and applying the credentials to the headers.

However In BatchHttpRequest, when serializing requests it works a bit differently from what I can see. The _request_ method from the http object is expected to have a _credentials_ attribute, and only in that case the credentials are applied to the headers. 

```
if request.http is not None and hasattr(request.http.request,
        'credentials'):
      request.http.request.credentials.apply(headers)
```
This is where the issue comes up, as with google-auth the AuthorizedHttp object _request_ method does not have this _credentials_ attribute. Therefore the credentials cannot be applied to the headers and it falls back on using the credentials from the first request in the batch as we can see in the execute() method of BatchHttpRequest:

```
# If http is not supplied use the first valid one given in the requests.
    if http is None:
      for request_id in self._order:
        request = self._requests[request_id]
        if request is not None:
          http = request.http
          break
```

This is as far as I understand what's going on. Maybe I missed something but it seems to me that google-auth credentials are not fully supported in the case of batch request.

Thanks, @sbrieuc. This gives me a good idea of what's going on. It seems to me that google_auth_httplib2 should be modified to support a per-request credentials. I'll look into that.

Resolving patch:
diff --git a/googleapiclient/_auth.py b/googleapiclient/_auth.py
--- a/googleapiclient/_auth.py
+++ b/googleapiclient/_auth.py
@@ -14,6 +14,8 @@
 
 """Helpers for authentication using oauth2client or google-auth."""
 
+import httplib2
+
 try:
     import google.auth
     import google.auth.credentials
@@ -29,8 +31,6 @@
 except ImportError:  # pragma: NO COVER
     HAS_OAUTH2CLIENT = False
 
-from googleapiclient.http import build_http
-
 
 def default_credentials():
     """Returns Application Default Credentials."""
@@ -84,9 +84,49 @@ def authorized_http(credentials):
         Union[httplib2.Http, google_auth_httplib2.AuthorizedHttp]: An
             authorized http client.
     """
+    from googleapiclient.http import build_http
+
     if HAS_GOOGLE_AUTH and isinstance(
             credentials, google.auth.credentials.Credentials):
         return google_auth_httplib2.AuthorizedHttp(credentials,
                                                    http=build_http())
     else:
         return credentials.authorize(build_http())
+
+
+def refresh_credentials(credentials):
+    # Refresh must use a new http instance, as the one associated with the
+    # credentials could be a AuthorizedHttp or an oauth2client-decorated
+    # Http instance which would cause a weird recursive loop of refreshing
+    # and likely tear a hole in spacetime.
+    refresh_http = httplib2.Http()
+    if HAS_GOOGLE_AUTH and isinstance(
+            credentials, google.auth.credentials.Credentials):
+        request = google_auth_httplib2.Request(refresh_http)
+        return credentials.refresh(request)
+    else:
+        return credentials.refresh(refresh_http)
+
+
+def apply_credentials(credentials, headers):
+    # oauth2client and google-auth have the same interface for this.
+    return credentials.apply(headers)
+
+
+def is_valid(credentials):
+    if HAS_GOOGLE_AUTH and isinstance(
+            credentials, google.auth.credentials.Credentials):
+        return credentials.valid
+    else:
+        return not credentials.access_token_expired
+
+
+def get_credentials_from_http(http):
+    if http is None:
+        return None
+    elif hasattr(http.request, 'credentials'):
+        return http.request.credentials
+    elif hasattr(http, 'credentials'):
+        return http.credentials
+    else:
+        return None
diff --git a/googleapiclient/http.py b/googleapiclient/http.py
--- a/googleapiclient/http.py
+++ b/googleapiclient/http.py
@@ -62,6 +62,7 @@
 except ImportError:
   from oauth2client import _helpers as util
 
+from googleapiclient import _auth
 from googleapiclient import mimeparse
 from googleapiclient.errors import BatchError
 from googleapiclient.errors import HttpError
@@ -1126,21 +1127,25 @@ def _refresh_and_apply_credentials(self, request, http):
     # If there is no http per the request then refresh the http passed in
     # via execute()
     creds = None
-    if request.http is not None and hasattr(request.http.request,
-        'credentials'):
-      creds = request.http.request.credentials
-    elif http is not None and hasattr(http.request, 'credentials'):
-      creds = http.request.credentials
+    request_credentials = False
+
+    if request.http is not None:
+      creds = _auth.get_credentials_from_http(request.http)
+      request_credentials = True
+
+    if creds is None and http is not None:
+      creds = _auth.get_credentials_from_http(http)
+
     if creds is not None:
       if id(creds) not in self._refreshed_credentials:
-        creds.refresh(http)
+        _auth.refresh_credentials(creds)
         self._refreshed_credentials[id(creds)] = 1
 
     # Only apply the credentials if we are using the http object passed in,
     # otherwise apply() will get called during _serialize_request().
-    if request.http is None or not hasattr(request.http.request,
-        'credentials'):
-      creds.apply(request.headers)
+    if request.http is None or not request_credentials:
+      _auth.apply_credentials(creds, request.headers)
+
 
   def _id_to_header(self, id_):
     """Convert an id to a Content-ID header value.
@@ -1200,9 +1205,10 @@ def _serialize_request(self, request):
     msg = MIMENonMultipart(major, minor)
     headers = request.headers.copy()
 
-    if request.http is not None and hasattr(request.http.request,
-        'credentials'):
-      request.http.request.credentials.apply(headers)
+    if request.http is not None:
+      credentials = _auth.get_credentials_from_http(request.http)
+      if credentials is not None:
+        _auth.apply_credentials(credentials, headers)
 
     # MIMENonMultipart adds its own Content-Type header.
     if 'content-type' in headers:
@@ -1409,11 +1415,11 @@ def execute(self, http=None):
 
     # Special case for OAuth2Credentials-style objects which have not yet been
     # refreshed with an initial access_token.
-    if getattr(http.request, 'credentials', None) is not None:
-      creds = http.request.credentials
-      if not getattr(creds, 'access_token', None):
+    creds = _auth.get_credentials_from_http(http)
+    if creds is not None:
+      if not _auth.is_valid(creds):
         LOGGER.info('Attempting refresh to obtain initial access_token')
-        creds.refresh(http)
+        _auth.refresh_credentials(creds)
 
     self._execute(http, self._order, self._requests)
 


Test patch:
diff --git a/tests/test_http.py b/tests/test_http.py
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -62,12 +62,17 @@
 
 class MockCredentials(Credentials):
   """Mock class for all Credentials objects."""
-  def __init__(self, bearer_token):
+  def __init__(self, bearer_token, expired=False):
     super(MockCredentials, self).__init__()
     self._authorized = 0
     self._refreshed = 0
     self._applied = 0
     self._bearer_token = bearer_token
+    self._access_token_expired = expired
+
+  @property
+  def access_token_expired(self):
+    return self._access_token_expired
 
   def authorize(self, http):
     self._authorized += 1
@@ -1106,10 +1111,7 @@ def test_execute_request_body(self):
   def test_execute_initial_refresh_oauth2(self):
     batch = BatchHttpRequest()
     callbacks = Callbacks()
-    cred = MockCredentials('Foo')
-
-    # Pretend this is a OAuth2Credentials object
-    cred.access_token = None
+    cred = MockCredentials('Foo', expired=True)
 
     http = HttpMockSequence([
       ({'status': '200',