Repository: googleapis/google-api-python-client

Issue:
`TypeError` when trying to issue `BatchHttpRequest` when service was built using application default credentials
This issue is very similar to #211.  My team has an app-engine application that uses batch requests in the Gmail API.  After an upgrade of this client library, we started seeing failures:

```
...
  File "/base/data/home/apps/s~app-id/modname:version/path/to/my/code.py", line 241, in _user_method
    return batch_request.execute(http=self._http)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/oauth2client/_helpers.py", line 133, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/googleapiclient/http.py", line 1417, in execute
    self._execute(http, self._order, self._requests)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/googleapiclient/http.py", line 1333, in _execute
    body = self._serialize_request(request)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/googleapiclient/http.py", line 1204, in _serialize_request
    request.http.request.credentials.apply(headers)
  File "/base/data/home/apps/s~app-id/modname:version/third_party/oauth2client/client.py", line 558, in apply
    headers['Authorization'] = 'Bearer ' + self.access_token
TypeError: cannot concatenate 'str' and 'NoneType' objects
```

Our usage of this client library looks like the following:

```python
from googleapiclient.discovery import build
gmail_service = build('gmail', 'v1')  # Uses application credentials

...

authorized_http = user_credentials.authorize(httplib2.Http())  # Uses end user credentials

batch_request = gmail_service.new_batch_http_request()
batch_request.add(
    gmail_service.users().threads().get(id=thread_id, userId='me'),
    callback=callback,
    request_id=thread_id)

batch_request.execute(http=authorized_http)
```

The `gmail_service` is cached at the application level to avoid needing to do an API call for each request (which is why it ends up with the application credentials).  What I believe is happening is that

```py
gmail_service.users().threads().get(id=thread_id, userId='me')
```

ends up creating an `HttpRequest` that uses the same `http` that was used to construct `gmail_service` (i.e. the application credentials) and I believe that the application credentials have no `access_token`.  This results in the `TypeError` seen above.  The fix in [#232](https://github.com/google/google-api-python-client/pull/232/files) won't help (the `http` that is getting passed to `batch_request.execute` is valid).

It seems to me that the application of credentials in [_serialize_request](https://github.com/google/google-api-python-client/blob/master/googleapiclient/http.py#L1202) should be conditional on the credentials actually having an access token:

```python
if request.http is not None and hasattr(request.http.request,
    'credentials'):
    if request.http.request.credentials.access_token:
        request.http.request.credentials.apply(headers)
```

Otherwise, if the sub-request isn't authenticated then the authentication from the outer request should be used (at least, if the [gmail docs](https://developers.google.com/gmail/api/guides/batch) are any indicator):

> The HTTP headers for the outer batch request, except for the Content- headers such as Content-Type, apply to every request in the batch. If you specify a given HTTP header in both the outer request and an individual call, then the individual call header's value overrides the outer batch request header's value. The headers for an individual call apply only to that call.

> For example, if you provide an Authorization header for a specific call, then that header applies only to that call. If you provide an Authorization header for the outer request, then that header applies to all of the individual calls unless they override it with Authorization headers of their own.

--------

In the event that this behavior is actually working as intended or if my proposed fix isn't satisfactory and a real fix would be too difficult, a work-around to this issue is possible by updating the requests that you add to the batch with an appropriately authorized http instance.  i.e changing the above to:

```python

batch_request = gmail_service.new_batch_http_request()

request =  gmail_service.users().threads().get(id=thread_id, userId='me')
request.http = authorized_http  # explicitly set the authorization on the request.

batch_request.add(
    request,
    callback=callback,
    request_id=thread_id)

batch_request.execute(http=authorized_http)
```

Seems to fix the issue.


Discussion:
@mgilson thanks for this really detailed bug report. Are you willing to send a PR to fix this issue? If so, I'll be happy to review.
Sure, I can submit a PR.  I've got a patch ready [here](https://github.com/google/google-api-python-client/compare/master...mgilson:app-default-creds?expand=1), but I don't know anything about your test setup.  I tried downloading `tox` in my virtual environment (`pip install tox`) and running it, but I seem to be running into all sorts of errors when I run it.  The log file looks like this:

```
$ cat /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/log/py27-oauth2client1-0.log
actionid: py27-oauth2client1
msg: getenv
cmdargs: ['/Users/mgilson/anaconda/bin/python', '-m', 'virtualenv', '--python', '/Users/mgilson/anaconda/bin/python2.7', 'py27-oauth2client1']
env: {'TERM_PROGRAM_VERSION': '361.1', 'LOGNAME': 'mgilson', 'USER': 'mgilson', 'HOME': '/Users/mgilson', 'PATH': '/Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/bin:/Users/mgilson/bin/google-cloud-sdk/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Users/mgilson/anaconda/bin:/Library/Frameworks/Python.framework/Versions/3.6/bin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/Users/mgilson/bin/scala-2.11.8/bin:/Users/mgilson/bin/spark-1.6.1-bin-hadoop2.6/bin:/Users/mgilson/bin', 'VIRTUALENVWRAPPER_SCRIPT': '/usr/local/bin/virtualenvwrapper.sh', 'DISPLAY': '/private/tmp/com.apple.launchd.Bxzp5coMLF/org.macosforge.xquartz:0', ... 'Apple_PubSub_Socket_Render': '/private/tmp/com.apple.launchd.rSbwT8LUMA/Render', 'PS1': '\\u:.../\\W \\[\x1b[36m\\]$git_branch\\[\x1b[31m\\]$git_dirty\\[\x1b(B\x1b[0m\\]$ ', '_': '/Users/mgilson/anaconda/bin/tox', 'VIRTUALENVWRAPPER_PROJECT_FILENAME': '.project', 'LSCOLORS': 'ExFxCxDxBxegedabagacad', 'VIRTUALENVWRAPPER_HOOK_DIR': '/Users/mgilson/Envs', 'OLDPWD': '/Users/mgilson/git', 'BASH_RC_SOURCED': '1', 'CLICOLOR': '1', '__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x0', 'PWD': '/Users/mgilson/git/google-api-python-client', 'VIRTUALENVWRAPPER_WORKON_CD': '1'}

New python executable in /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/bin/python2.7
Also creating executable in /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/bin/python
Installing setuptools, pip, wheel...
  Complete output from command /Users/mgilson/git/g...lient1/bin/python2.7 - setuptools pip wheel:
  Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "/Users/mgilson/anaconda/lib/python2.7/tempfile.py", line 32, in <module>
    import io as _io
  File "/Users/mgilson/anaconda/lib/python2.7/io.py", line 51, in <module>
    import _io
ImportError: dlopen(/Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/lib/python2.7/lib-dynload/_io.so, 2): Symbol not found: __PyCodecInfo_GetIncrementalDecoder
  Referenced from: /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/lib/python2.7/lib-dynload/_io.so
  Expected in: flat namespace
 in /Users/mgilson/git/google-api-python-client/.tox/py27-oauth2client1/lib/python2.7/lib-dynload/_io.so
----------------------------------------
...Installing setuptools, pip, wheel...done.
Traceback (most recent call last):
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 2328, in <module>
    main()
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 713, in main
    symlink=options.symlink)
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 945, in create_environment
    download=download,
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 901, in install_wheel
    call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
  File "/Users/mgilson/anaconda/lib/python2.7/site-packages/virtualenv.py", line 797, in call_subprocess
    % (cmd_desc, proc.returncode))
OSError: Command /Users/mgilson/git/g...lient1/bin/python2.7 - setuptools pip wheel failed with error code 1
Running virtualenv with interpreter /Users/mgilson/anaconda/bin/python2.7
```

Have you ever seen anything like this before?
this bit me recently when i upgraded to oauth2client 4.1.1 and google-api-python-client 1.6.2. patching #351 into my local lib fixes it. i'd love to see that PR merged so it gets into the next release!
Resolved by #376.

Resolving patch:
diff --git a/googleapiclient/_auth.py b/googleapiclient/_auth.py
--- a/googleapiclient/_auth.py
+++ b/googleapiclient/_auth.py
@@ -120,6 +120,8 @@ def refresh_credentials(credentials):
 
 def apply_credentials(credentials, headers):
     # oauth2client and google-auth have the same interface for this.
+    if not is_valid(credentials):
+        refresh_credentials(credentials)
     return credentials.apply(headers)
 
 
@@ -128,7 +130,9 @@ def is_valid(credentials):
             credentials, google.auth.credentials.Credentials):
         return credentials.valid
     else:
-        return not credentials.access_token_expired
+        return (
+            credentials.access_token is not None and
+            not credentials.access_token_expired)
 
 
 def get_credentials_from_http(http):


Test patch:
diff --git a/tests/test_http.py b/tests/test_http.py
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -70,6 +70,10 @@ def __init__(self, bearer_token, expired=False):
     self._bearer_token = bearer_token
     self._access_token_expired = expired
 
+  @property
+  def access_token(self):
+    return self._bearer_token
+
   @property
   def access_token_expired(self):
     return self._access_token_expired