Repository: googleapis/google-cloud-python

Issue:
Cloud Storage: Overflow error: Python int too large to convert to C long from blob.py
# Thank you for reporting an issue to google-cloud-python!

If you are reporting an issue or requesting a feature, please search the
existing open and closed issues to see if there is already work being done.

- https://github.com/GoogleCloudPlatform/google-cloud-python/issues
- http://stackoverflow.com/questions/tagged/google-cloud-python

If you can provide us with as much of the following information as possible it
will help us identify the cause of your issue more quickly.

1. Specify the API at the beginning of the title (for example, "BigQuery: ...")
   General, Core, and Other are also allowed as types

2. OS type and version 
Microsoft Windows [Version 6.1.7601]

3. Python version and virtual environment information `python --version`
2.7.13

4. google-cloud-python version `pip show google-cloud`, `pip show google-<service>` or `pip freeze`

google-cloud-storage==1.8.0

5. Stacktrace if available
```
Traceback (most recent call last):
  File "C:/Dev/Python/widdler/widdler.py", line 605, in <module>
    sys.exit(main())
  File "C:/Dev/Python/widdler/widdler.py", line 593, in main
    result = args.func(args)
  File "C:/Dev/Python/widdler/widdler.py", line 408, in call_upload
    uploaded_files = b.upload_workflow_input_files(args.wdl, args.json)
  File "C:\Dev\Python\widdler\src\SingleBucket.py", line 189, in upload_workflow_input_files
    self.upload_files(files_to_upload)
  File "C:\Dev\Python\widdler\src\SingleBucket.py", line 107, in upload_files
    self.upload_file(f, f.replace('\\', '/'))
  File "C:\Dev\Python\widdler\src\SingleBucket.py", line 91, in upload_file
    blob.upload_from_filename(source_file_name)
  File "C:\Miniconda3\envs\Widdler\lib\site-packages\google\cloud\storage\blob.py", line 992, in upload_from_filename
    size=total_bytes)
  File "C:\Miniconda3\envs\Widdler\lib\site-packages\google\cloud\storage\blob.py", line 946, in upload_from_file
    client, file_obj, content_type, size, num_retries)
  File "C:\Miniconda3\envs\Widdler\lib\site-packages\google\cloud\storage\blob.py", line 867, in _do_upload
    client, stream, content_type, size, num_retries)
  File "C:\Miniconda3\envs\Widdler\lib\site-packages\google\cloud\storage\blob.py", line 676, in _do_multipart_upload
    data = stream.read(size)
OverflowError: Python int too large to convert to C long
```
Also, additional output from my debug console window:

```
Connected to pydev debugger (build 173.4674.37)
Error while evaluating expression: if (size == 3799968885)
Traceback (most recent call last):
  File "_pydevd_bundle\pydevd_cython_win32_27_64.pyx", line 172, in _pydevd_bundle.pydevd_cython_win32_27_64.handle_breakpoint_condition
  File "<string>", line 1
    if (size == 3799968885)
     ^
SyntaxError: invalid syntax
Error while evaluating expression: if (size == 3799968885)
Traceback (most recent call last):
  File "_pydevd_bundle\pydevd_cython_win32_27_64.pyx", line 172, in _pydevd_bundle.pydevd_cython_win32_27_64.handle_breakpoint_condition
  File "<string>", line 1
    if (size == 3799968885)
     ^
SyntaxError: invalid syntax
2018-04-09 14:49:38,200 - widdler.SingleBucket.SingleBucket - CRITICAL - Python int too large to convert to C long
```


6. Steps to reproduce

Use blob.upload_from_filename(source_file_name) to upload a file that's 3.8GB in size. Guessing anything
larger than 2.1 GB will do as this appears to be an issue of the file size being handled as an int when it should be handled as a long. 

7. Code example

This is my function:
```
    def upload_file(self, source_file_name, destination_blob_name):
        """
        Upload a file to a bucket. Once they are in the bucket, they are called blobs.
        :param source_file_name: Full path and name of source file.
        :param destination_blob_name: name to give file in bucket.
        :return:
        """
        blob = self.bucket.blob(destination_blob_name)

        try:
            blob.upload_from_filename(source_file_name)
        except Exception as e:
            print_log_exit(str(e))
```
It is part of a class I made for implementing various cloud storage functionality for a single bucket. When attempting to upload a large file (3.8GB in size), I get the overflow error in the stack trace above. While tracking it down in debugger, the error appears to be caused by line 676 of blob.py in the google cloud storage library:

`            data = stream.read(size)
`




Discussion:
@Osiris1975, hrm, I wonder if this happens in Python 3 as well?
I was able to repro this on mac also

from shell:
```
dd if=/dev/zero of=4GFile bs=1M count=4096
```

from python3 repl: (python 3.6.5 - 64 bit)
```
>>> from google.cloud import storage
>>> c = storage.Client('crwilcox-test')
>>> b.blob('4gfile')
<Blob: crwilcox-test, 4gfile>
>>> blob = b.blob('4gfile')
>>> blob.upload_from_filename('4GFile')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "USER/miniconda3/envs/playground/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 992, in upload_from_filename
    size=total_bytes)
  File "USER/miniconda3/envs/playground/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 946, in upload_from_file
    client, file_obj, content_type, size, num_retries)
  File "USER/miniconda3/envs/playground/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 867, in _do_upload
    client, stream, content_type, size, num_retries)
  File "USER/miniconda3/envs/playground/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 676, in _do_multipart_upload
    data = stream.read(size)
OSError: [Errno 22] Invalid argument
```


Expanding. Python2 does this:
```
python2 upload_huge_file.py
Traceback (most recent call last):
  File "upload_huge_file.py", line 5, in <module>
    blob.upload_from_filename('4GFile')
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 992, in upload_from_filename
    size=total_bytes)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 946, in upload_from_file
    client, file_obj, content_type, size, num_retries)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 867, in _do_upload
    client, stream, content_type, size, num_retries)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 700, in _do_multipart_upload
    transport, data, object_metadata, content_type)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/google/resumable_media/requests/upload.py", line 97, in transmit
    retry_strategy=self._retry_strategy)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/google/resumable_media/requests/_helpers.py", line 101, in http_request
    func, RequestsMixin._get_status_code, retry_strategy)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/google/resumable_media/_helpers.py", line 146, in wait_and_retry
    response = func()
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/google/auth/transport/requests.py", line 201, in request
    method, url, data=data, headers=request_headers, **kwargs)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "USER/git/PLAYGROUND/env27/lib/python2.7/site-packages/urllib3/connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "USER/.pyenv/versions/2.7.14/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "USER/.pyenv/versions/2.7.14/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "USER/.pyenv/versions/2.7.14/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "USER/.pyenv/versions/2.7.14/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "USER/.pyenv/versions/2.7.14/lib/python2.7/httplib.py", line 858, in send
    self.sock.sendall(data)
  File "USER/.pyenv/versions/2.7.14/lib/python2.7/ssl.py", line 753, in sendall
    v = self.send(data[count:])
  File "USER/.pyenv/versions/2.7.14/lib/python2.7/ssl.py", line 719, in send
    v = self._sslobj.write(data)
OverflowError: string longer than 2147483647 bytes
```
@crwilcox want to take point on fixing this?
@jonparrott and @crwilcox thanks for jumping on this so quickly. Hopefully a quick and easy fix. Once fixed will you be releasing a new version anytime soon?

Resolving patch:
diff --git a/storage/google/cloud/storage/blob.py b/storage/google/cloud/storage/blob.py
--- a/storage/google/cloud/storage/blob.py
+++ b/storage/google/cloud/storage/blob.py
@@ -94,6 +94,9 @@
     'Size {:d} was specified but the file-like object only had '
     '{:d} bytes remaining.')
 
+_DEFAULT_CHUNKSIZE = 104857600  # 1024 * 1024 B * 100 = 100 MB
+_MAX_MULTIPART_SIZE = 8388608  # 8 MB
+
 
 class Blob(_PropertyMixin):
     """A wrapper around Cloud Storage's concept of an ``Object``.
@@ -120,7 +123,6 @@ class Blob(_PropertyMixin):
     """
 
     _chunk_size = None  # Default value for each instance.
-
     _CHUNK_SIZE_MULTIPLE = 256 * 1024
     """Number (256 KB, in bytes) that must divide the chunk size."""
 
@@ -175,7 +177,9 @@ def chunk_size(self, value):
         :raises: :class:`ValueError` if ``value`` is not ``None`` and is not a
                  multiple of 256 KB.
         """
-        if value is not None and value % self._CHUNK_SIZE_MULTIPLE != 0:
+        if value is not None and \
+                value > 0 and \
+                value % self._CHUNK_SIZE_MULTIPLE != 0:
             raise ValueError('Chunk size must be a multiple of %d.' % (
                 self._CHUNK_SIZE_MULTIPLE,))
         self._chunk_size = value
@@ -669,8 +673,6 @@ def _do_multipart_upload(self, client, stream, content_type,
                              size, num_retries, predefined_acl):
         """Perform a multipart upload.
 
-        Assumes ``chunk_size`` is :data:`None` on the current blob.
-
         The content type of the upload will be determined in order
         of precedence:
 
@@ -795,6 +797,8 @@ def _initiate_resumable_upload(self, client, stream, content_type,
         """
         if chunk_size is None:
             chunk_size = self.chunk_size
+            if chunk_size is None:
+                chunk_size = _DEFAULT_CHUNKSIZE
 
         transport = self._get_transport(client)
         info = self._get_upload_arguments(content_type)
@@ -876,9 +880,9 @@ def _do_upload(self, client, stream, content_type,
                    size, num_retries, predefined_acl):
         """Determine an upload strategy and then perform the upload.
 
-        If the current blob has a ``chunk_size`` set, then a resumable upload
-        will be used, otherwise the content and the metadata will be uploaded
-        in a single multipart upload request.
+        If the size of the data to be uploaded exceeds 5 MB a resumable media
+        request will be used, otherwise the content and the metadata will be
+        uploaded in a single multipart upload request.
 
         The content type of the upload will be determined in order
         of precedence:
@@ -914,7 +918,7 @@ def _do_upload(self, client, stream, content_type,
                   **only** response in the multipart case and it will be the
                   **final** response in the resumable case.
         """
-        if self.chunk_size is None:
+        if size is not None and size <= _MAX_MULTIPART_SIZE:
             response = self._do_multipart_upload(
                 client, stream, content_type,
                 size, num_retries, predefined_acl)


Test patch:
diff --git a/storage/tests/unit/test_blob.py b/storage/tests/unit/test_blob.py
--- a/storage/tests/unit/test_blob.py
+++ b/storage/tests/unit/test_blob.py
@@ -21,6 +21,7 @@
 import tempfile
 import unittest
 
+import google.cloud.storage.blob
 import mock
 import six
 from six.moves import http_client
@@ -1087,14 +1088,18 @@ def test__do_multipart_upload_bad_size(self):
 
     def _initiate_resumable_helper(
             self, size=None, extra_headers=None, chunk_size=None,
-            num_retries=None, user_project=None, predefined_acl=None):
+            num_retries=None, user_project=None, predefined_acl=None,
+            blob_chunk_size=786432):
         from google.resumable_media.requests import ResumableUpload
 
         bucket = _Bucket(name='whammy', user_project=user_project)
         blob = self._make_one(u'blob-name', bucket=bucket)
         blob.metadata = {'rook': 'takes knight'}
-        blob.chunk_size = 3 * blob._CHUNK_SIZE_MULTIPLE
-        self.assertIsNotNone(blob.chunk_size)
+        blob.chunk_size = blob_chunk_size
+        if blob_chunk_size is not None:
+            self.assertIsNotNone(blob.chunk_size)
+        else:
+            self.assertIsNone(blob.chunk_size)
 
         # Need to make sure **same** dict is used because ``json.dumps()``
         # will depend on the hash order.
@@ -1135,7 +1140,11 @@ def _initiate_resumable_helper(
             self.assertIsNot(upload._headers, extra_headers)
         self.assertFalse(upload.finished)
         if chunk_size is None:
-            self.assertEqual(upload._chunk_size, blob.chunk_size)
+            if blob_chunk_size is None:
+                self.assertEqual(upload._chunk_size,
+                                 google.cloud.storage.blob._DEFAULT_CHUNKSIZE)
+            else:
+                self.assertEqual(upload._chunk_size, blob.chunk_size)
         else:
             self.assertNotEqual(blob.chunk_size, chunk_size)
             self.assertEqual(upload._chunk_size, chunk_size)
@@ -1182,6 +1191,9 @@ def test__initiate_resumable_upload_with_user_project(self):
         user_project = 'user-project-123'
         self._initiate_resumable_helper(user_project=user_project)
 
+    def test__initiate_resumable_upload_without_chunk_size(self):
+        self._initiate_resumable_helper(blob_chunk_size=None)
+
     def test__initiate_resumable_upload_with_chunk_size(self):
         one_mb = 1048576
         self._initiate_resumable_helper(chunk_size=one_mb)
@@ -1327,7 +1339,8 @@ def test__do_resumable_upload_with_predefined_acl(self):
         self._do_resumable_helper(predefined_acl='private')
 
     def _do_upload_helper(
-            self, chunk_size=None, num_retries=None, predefined_acl=None):
+            self, chunk_size=None, num_retries=None, predefined_acl=None,
+            size=None):
         blob = self._make_one(u'blob-name', bucket=None)
 
         # Create a fake response.
@@ -1346,14 +1359,15 @@ def _do_upload_helper(
         client = mock.sentinel.client
         stream = mock.sentinel.stream
         content_type = u'video/mp4'
-        size = 12345654321
+        if size is None:
+            size = 12345654321
         # Make the request and check the mocks.
         created_json = blob._do_upload(
             client, stream, content_type, size, num_retries, predefined_acl)
         self.assertIs(created_json, mock.sentinel.json)
         response.json.assert_called_once_with()
-        if chunk_size is None:
-
+        if size is not None and \
+                size <= google.cloud.storage.blob._MAX_MULTIPART_SIZE:
             blob._do_multipart_upload.assert_called_once_with(
                 client, stream, content_type, size, num_retries,
                 predefined_acl)
@@ -1364,12 +1378,14 @@ def _do_upload_helper(
                 client, stream, content_type, size, num_retries,
                 predefined_acl)
 
-    def test__do_upload_without_chunk_size(self):
-        self._do_upload_helper()
+    def test__do_upload_uses_multipart(self):
+        self._do_upload_helper(
+            size=google.cloud.storage.blob._MAX_MULTIPART_SIZE)
 
-    def test__do_upload_with_chunk_size(self):
-        chunk_size = 1024 * 1024 * 1024  # 1GB
-        self._do_upload_helper(chunk_size=chunk_size)
+    def test__do_upload_uses_resumable(self):
+        self._do_upload_helper(
+            chunk_size=256 * 1024,  # 256KB
+            size=google.cloud.storage.blob._MAX_MULTIPART_SIZE + 1)
 
     def test__do_upload_with_retry(self):
         self._do_upload_helper(num_retries=20)