Repository: googleapis/google-api-python-client

Issue:
Failure when uploading binary files on Python 3 when resumable=False
Seems like `MediaFileUpload` (or something else deep inside) is failing to handling binary files properly on Python 3.  My guess is that the contents of the file is mistakenly passed as a Unicode string (rather than as a byte-string) into `http.client`, which dies as soon as it encounters a non-Latin-1 character.  This only happens when `resumable=False`.

Code to reproduce the problem:

``` py
import apiclient
service = apiclient.discovery.build("drive", "v2", …)
filename = "some-file.tar.xz"
media = apiclient.http.MediaFileUpload(filename)
service.files().insert(body={"title": os.path.basename(filename)}, media_body=media).execute()
```

Typical error:

```
Traceback (most recent call last):
  …
  File "/usr/lib/python3.5/site-packages/oauth2client/util.py", line 142, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/lib/python3.5/site-packages/googleapiclient/http.py", line 722, in execute
    body=self.body, headers=self.headers)
  File "/usr/lib/python3.5/site-packages/oauth2client/client.py", line 589, in new_request
    redirections, connection_type)
  File "/usr/lib/python3.5/site-packages/httplib2/__init__.py", line 1313, in request
    (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
  File "/usr/lib/python3.5/site-packages/httplib2/__init__.py", line 1063, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/lib/python3.5/site-packages/httplib2/__init__.py", line 987, in _conn_request
    conn.request(method, request_uri, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1127, in _send_request
    body = body.encode('iso-8859-1')
UnicodeEncodeError: 'latin-1' codec can't encode character '\ufffd' in position 350: ordinal not in range(256)
```

(Using Python 3.5.0 on Arch Linux 4.2.2.)



Discussion:
I see that the stack trace goes through oauth2client and httplib2 code as well as this library's code - with what versions of the three libraries are you seeing this problem?

```
apiclient 1.4.2
oauth2client 1.5.1
httplib2 0.9.2
```

There are ways you could have answered that question that would have made my life easier and that is not one of them. :-P And all three libraries formally support Python 3, so that doesn't lead to an easy answer either. I agree with you that "Seems like `MediaFileUpload` (or something else deep inside) is failing to handling binary files properly on Python 3.". I won't soon have time to investigate this, but I suspect that a proper investigation is warranted. Are you able to look into it?


Resolving patch:
diff --git a/googleapiclient/discovery.py b/googleapiclient/discovery.py
--- a/googleapiclient/discovery.py
+++ b/googleapiclient/discovery.py
@@ -28,14 +28,17 @@
     'key2param',
     ]
 
-from six import StringIO
+from six import BytesIO
 from six.moves import http_client
 from six.moves.urllib.parse import urlencode, urlparse, urljoin, \
   urlunparse, parse_qsl
 
 # Standard library imports
 import copy
-from email.generator import Generator
+try:
+  from email.generator import BytesGenerator
+except ImportError:
+  from email.generator import Generator as BytesGenerator
 from email.mime.multipart import MIMEMultipart
 from email.mime.nonmultipart import MIMENonMultipart
 import json
@@ -102,6 +105,10 @@
 # Library-specific reserved words beyond Python keywords.
 RESERVED_WORDS = frozenset(['body'])
 
+# patch _write_lines to avoid munging '\r' into '\n'
+# ( https://bugs.python.org/issue18886 https://bugs.python.org/issue19003 )
+class _BytesGenerator(BytesGenerator):
+  _write_lines = BytesGenerator.write
 
 def fix_method_name(name):
   """Fix method names to avoid reserved word conflicts.
@@ -797,8 +804,8 @@ def method(self, **kwargs):
           msgRoot.attach(msg)
           # encode the body: note that we can't use `as_string`, because
           # it plays games with `From ` lines.
-          fp = StringIO()
-          g = Generator(fp, mangle_from_=False)
+          fp = BytesIO()
+          g = _BytesGenerator(fp, mangle_from_=False)
           g.flatten(msgRoot, unixfrom=False)
           body = fp.getvalue()
 


Test patch:
diff --git a/tests/test_discovery.py b/tests/test_discovery.py
--- a/tests/test_discovery.py
+++ b/tests/test_discovery.py
@@ -35,6 +35,7 @@
 import json
 import os
 import pickle
+import re
 import sys
 import unittest2 as unittest
 
@@ -787,7 +788,21 @@ def test_multipart_media_good_upload(self):
     request = zoo.animals().insert(media_body=datafile('small.png'), body={})
     self.assertTrue(request.headers['content-type'].startswith(
         'multipart/related'))
-    self.assertEquals('--==', request.body[0:4])
+    with open(datafile('small.png'), 'rb') as f:
+      contents = f.read()
+    boundary = re.match(b'--=+([^=]+)', request.body).group(1)
+    self.assertEqual(
+      request.body.rstrip(b"\n"), # Python 2.6 does not add a trailing \n
+      b'--===============' + boundary + b'==\n' +
+      b'Content-Type: application/json\n' +
+      b'MIME-Version: 1.0\n\n' +
+      b'{"data": {}}\n' +
+      b'--===============' + boundary + b'==\n' +
+      b'Content-Type: image/png\n' +
+      b'MIME-Version: 1.0\n' +
+      b'Content-Transfer-Encoding: binary\n\n' +
+      contents +
+      b'\n--===============' + boundary + b'==--')
     assertUrisEqual(self,
         'https://www.googleapis.com/upload/zoo/v1/animals?uploadType=multipart&alt=json',
         request.uri)