Repository: googleapis/google-cloud-python

Issue:
Blob.download_to_file() downloads to RAM instead
Hello,

Blob.download_to_file() with no chunk size set causes the file to be pulled entirely into RAM before starting to write it to the supplied file object. This is due to google/resumable_media/download.py's "sans I/O" approach.

I can work around it by setting the chunk size parameter, however this changes both what I am billed for (multiple GETs rather than a single GET) and application's latency profile to paper over an obvious deficiency in the client library, so if you don't consider this a design flaw then at least consider it a performance bug.

Either way, I've lost 24 hours of data due to an OOM triggered by this, the docstring should at a minimum be updated to warn the user that the library cares more about some trend from last week than doing what you'd expect the simplest possible HTTP client talking to Gcloud could have done in the early 90s.

Apologies for the angst, but I'm leaving it in here, because you really need to know how disappointing it is to have a crash of this sort caused by a library function so conceptually simple.


pip freeze:

````
gapic-google-cloud-pubsub-v1==0.15.4
gapic-google-cloud-speech-v1==0.15.3
google-api-python-client==1.6.2
google-auth==1.0.1
google-auth-httplib2==0.0.2
google-cloud-core==0.25.0
google-cloud-language==0.25.0
google-cloud-pubsub==0.26.0
google-cloud-speech==0.25.1
google-cloud-storage==1.1.1
google-cloud-translate==0.24.0
google-gax==0.15.13
google-resumable-media==0.0.2
googleapis-common-protos==1.5.2
grpc-google-iam-v1==0.11.1
proto-google-cloud-pubsub-v1==0.15.4
proto-google-cloud-speech-v1==0.15.3
````




Discussion:
> the library cares more about some trend from last week than doing what you'd expect the simplest possible HTTP client

I don't think that's called for, though I do appreciate you talking the time to file the issue. I designed `google-resumable-media-python`, so you can put the blame 100% on my shoulders.

The sans-I/O approach has nothing to do with the "missing feature" of piping the HTTP payload directly into a file. What's more, the behavior of the **old** code (i.e. `google.cloud.streaming`) is mostly the same, and that was written 3+ years ago.

The real issue is that the `google-cloud-storage` library doesn't do some kind of "smart chunking". It's definitely worth discussing. The reason there is no smart chunking is because we may not "a priori" know the size of the file, and it would be an **extra** round trip to determine the size before the first request. (Right now the `google-cloud-storage` package does chunked or full download, depending [only on the chunk size][2].)

We have [discussed][1] strategies to reduce the amount of a file that ends up in RAM, but haven't had a chance to implement it yet.

[1]: https://github.com/GoogleCloudPlatform/google-resumable-media-python/issues/5
[2]: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/ce9cd83e6a65212c7e26a41344d38a53c770c2b7/storage/google/cloud/storage/blob.py#L420-L426
It seems the sans-IO approach requires a logical abstraction for the entire response, which necessitates it living temporarily on disk or in RAM

I'm confused about this concept of smart chunking -- both producer and consumer of the stream already apply chunking (libc / Python io lib's buffering on the consumer side), and kernel's TCP windowing on the producer side, it doesn't seem like you should be implementing any kind of policy in your code, considering both kernel and IO library are trivial to adjust as needs require.

To repeat, your library should not be buffering whatsoever, and the current implementation should never have been checked in.

Apologies for the grumpy bug report earlier, but this has cost me dearly and most of that code should not have existed in the first instance.

Resolving patch:
diff --git a/storage/google/cloud/storage/blob.py b/storage/google/cloud/storage/blob.py
--- a/storage/google/cloud/storage/blob.py
+++ b/storage/google/cloud/storage/blob.py
@@ -414,9 +414,8 @@ def _do_download(self, transport, file_obj, download_url, headers):
         :param headers: Optional headers to be sent with the request(s).
         """
         if self.chunk_size is None:
-            download = Download(download_url, headers=headers)
-            response = download.consume(transport)
-            file_obj.write(response.content)
+            download = Download(download_url, stream=file_obj, headers=headers)
+            download.consume(transport)
         else:
             download = ChunkedDownload(
                 download_url, self.chunk_size, file_obj, headers=headers)
diff --git a/storage/setup.py b/storage/setup.py
--- a/storage/setup.py
+++ b/storage/setup.py
@@ -53,7 +53,7 @@
 REQUIREMENTS = [
     'google-cloud-core >= 0.25.0, < 0.26dev',
     'google-auth >= 1.0.0',
-    'google-resumable-media >= 0.2.1',
+    'google-resumable-media >= 0.2.2',
     'requests >= 2.0.0',
 ]
 


Test patch:
diff --git a/storage/tests/unit/test_blob.py b/storage/tests/unit/test_blob.py
--- a/storage/tests/unit/test_blob.py
+++ b/storage/tests/unit/test_blob.py
@@ -375,13 +375,20 @@ def test__get_download_url_on_the_fly_with_generation(self):
         self.assertEqual(download_url, expected_url)
 
     @staticmethod
-    def _mock_requests_response(status_code, headers, content=b''):
+    def _mock_requests_response(
+            status_code, headers, content=b'', stream=False):
         import requests
 
         response = requests.Response()
         response.status_code = status_code
         response.headers.update(headers)
-        response._content = content
+        if stream:
+            response.raw = io.BytesIO(content)
+            response._content = False
+        else:
+            response.raw = None
+            response._content = content
+
         response.request = requests.Request(
             'POST', 'http://example.com').prepare()
         return response
@@ -429,7 +436,9 @@ def test__do_download_simple(self):
         transport.request.return_value = self._mock_requests_response(
             http_client.OK,
             {'content-length': '6', 'content-range': 'bytes 0-5/6'},
-            content=b'abcdef')
+            content=b'abcdef',
+            stream=True,
+        )
         file_obj = io.BytesIO()
         download_url = 'http://test.invalid'
         headers = {}
@@ -438,7 +447,7 @@ def test__do_download_simple(self):
         self.assertEqual(file_obj.getvalue(), b'abcdef')
 
         transport.request.assert_called_once_with(
-            'GET', download_url, data=None, headers=headers)
+            'GET', download_url, data=None, headers=headers, stream=True)
 
     def test__do_download_chunked(self):
         blob_name = 'blob-name'
@@ -493,7 +502,7 @@ def test_download_to_file_with_failure(self):
         self.assertEqual(file_obj.tell(), 0)
         # Check that the transport was called once.
         transport.request.assert_called_once_with(
-            'GET', blob.media_link, data=None, headers={})
+            'GET', blob.media_link, data=None, headers={}, stream=True)
 
     def test_download_to_file_wo_media_link(self):
         blob_name = 'blob-name'
@@ -535,7 +544,9 @@ def _download_to_file_helper(self, use_chunks=False):
             single_chunk_response = self._mock_requests_response(
                 http_client.OK,
                 {'content-length': '6', 'content-range': 'bytes 0-5/6'},
-                content=b'abcdef')
+                content=b'abcdef',
+                stream=True,
+            )
             transport.request.side_effect = [single_chunk_response]
 
         file_obj = io.BytesIO()
@@ -546,7 +557,7 @@ def _download_to_file_helper(self, use_chunks=False):
             self._check_session_mocks(client, transport, media_link)
         else:
             transport.request.assert_called_once_with(
-                'GET', media_link, data=None, headers={})
+                'GET', media_link, data=None, headers={}, stream=True)
 
     def test_download_to_file_default(self):
         self._download_to_file_helper()