Repository: googleapis/google-cloud-python

Issue:
Can't download encrypted files larger than 1 MB from cloud storage
I'm using `upload_from_file` with `encryption_key` to store files inside a Google Cloud Storage bucket. This works very well.

Now I'm trying to download these files again. Using `download_to_file` with the same `encryption_key`. This crashes with the following stack trace:

```
Traceback (most recent call last):
  File "restore.py", line 40, in <module>
    main()
  File "restore.py", line 37, in main
    download_from_storage(args.file, args.bucket, args.credentials, key)
  File "restore.py", line 17, in download_from_storage
    blob.download_to_file(my_file, encryption_key=encryption_key)
  File "/Users/daniel/Projects/python/gcs-backup/env/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 354, in download_to_file
    download.initialize_download(request, client._connection.http)
  File "/Users/daniel/Projects/python/gcs-backup/env/lib/python2.7/site-packages/google/cloud/streaming/transfer.py", line 390, in initialize_download
    self.stream_file(use_chunks=True)
  File "/Users/daniel/Projects/python/gcs-backup/env/lib/python2.7/site-packages/google/cloud/streaming/transfer.py", line 614, in stream_file
    response = self._process_response(response)
  File "/Users/daniel/Projects/python/gcs-backup/env/lib/python2.7/site-packages/google/cloud/streaming/transfer.py", line 528, in _process_response
    raise TransferRetryError(response.content)
google.cloud.streaming.exceptions.TransferRetryError: The target object is encrypted by a customer-supplied encryption key.
```

I'm using `google-cloud-storage==0.20.0`.

This only occurs when downloading files larger than 1mb. Any file less than 1mb is downloaded and decrypted as expected. A file larger than 1mb might be downloaded in chunks and only the first 1mb chunk is saved to disk.

**Steps to reproduce**
1) Upload a file larger than 1mb using a custom encryption key
2) Try to download this file using the same encryption key
3) Find a 1mb chunk of your file at the destination

**demo code**
```
#!/usr/bin/env python

from tempfile import NamedTemporaryFile
import os

from google.cloud import storage
from google.cloud.storage import Blob

CREDENTIALS = './credentials.json'
BUCKET_NAME = '< SET BUCKET >'
ENCRYPTION_KEY = 'v3CtoFyEJmj52RGsSqze7C8fD6mzgpnd'
FILE_NAME = 'enc-test-file'

client = storage.Client.from_service_account_json(CREDENTIALS)
bucket = client.get_bucket(BUCKET_NAME)


def upload_to_storage(filename):
    blob = Blob(FILE_NAME, bucket)

    with open(filename, 'rb') as my_file:
        blob.upload_from_file(my_file, encryption_key=ENCRYPTION_KEY)


def download_from_storage(filename):
    blob = Blob(FILE_NAME, bucket)

    with open(filename, 'wb') as my_file:
        blob.download_to_file(my_file, encryption_key=ENCRYPTION_KEY)


def main():
    size = 2097152  # 2mb file

    f = NamedTemporaryFile(delete=False)
    f.write("\0" * size)
    f.close()

    print('Uploading {} to Google Cloud Storage...'.format(f.name))
    upload_to_storage(f.name)

    print('Downloading {} from Google Cloud Storage...'.format(f.name))
    download_from_storage(f.name)

    os.remove(f.name)

if __name__ == '__main__':
    main()
```

This might be because the encryption headers aren't added to the request for the next chunk.


Discussion:
@dbanck thanks for reporting!

@dbanck, I _think_ that the encryption key shouldn't get passed to `download_to_file()` but rather is passed to the `Blob` constructor.

See: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/491519dc54cf2204703374ff374cabbd4cd14476/storage/google/cloud/storage/blob.py#L310-L311

This might be true for the current _master_, but the docs have a different example: https://googlecloudplatform.github.io/google-cloud-python/stable/storage-blobs.html#google.cloud.storage.blob.Blob.download_to_file

The files inside my local `site-packages` are looking like this: https://github.com/GoogleCloudPlatform/google-cloud-python/blob/umbrella-0.20.0/storage/google/cloud/storage/blob.py#L287

Should I update to a newer version? But there is no newer version than `0.20.0` on pypi: https://pypi.python.org/pypi/google-cloud-storage/0.20.0

@dbanck, thanks! I forgot that it changed since the last release.

> This might be because the encryption headers aren't added to the request for the next chunk.

I'll take a look at this and see if I can track down the issue.

@dbanck sorry for the slow update on this. I've been playing with it and I thought it was completely fixed on master but it appears that it may not be. I'm confirming that and then I'll try and get a fix in.

Don't worry, take your time. Thanks for the feedback and looking into it :)


Resolving patch:
diff --git a/core/google/cloud/streaming/transfer.py b/core/google/cloud/streaming/transfer.py
--- a/core/google/cloud/streaming/transfer.py
+++ b/core/google/cloud/streaming/transfer.py
@@ -387,7 +387,7 @@ def initialize_download(self, http_request, http):
         # Unless the user has requested otherwise, we want to just
         # go ahead and pump the bytes now.
         if self.auto_transfer:
-            self.stream_file(use_chunks=True)
+            self.stream_file(use_chunks=True, headers=http_request.headers)
 
     def _normalize_start_end(self, start, end=None):
         """Validate / fix up byte range.
@@ -487,7 +487,7 @@ def _compute_end_byte(self, start, end=None, use_chunks=True):
 
         return end_byte
 
-    def _get_chunk(self, start, end):
+    def _get_chunk(self, start, end, headers=None):
         """Retrieve a chunk of the file.
 
         :type start: int
@@ -496,11 +496,14 @@ def _get_chunk(self, start, end):
         :type end: int
         :param end: (Optional) end byte of the range.
 
+        :type headers: dict
+        :param headers: (Optional) Headers to be used for the ``Request``.
+
         :rtype: :class:`google.cloud.streaming.http_wrapper.Response`
         :returns: response from the chunk request.
         """
         self._ensure_initialized()
-        request = Request(url=self.url)
+        request = Request(url=self.url, headers=headers)
         self._set_range_header(request, start, end=end)
         return make_api_request(
             self.bytes_http, request, retries=self.num_retries)
@@ -589,7 +592,7 @@ def get_range(self, start, end=None, use_chunks=True):
                 raise TransferRetryError(
                     'Zero bytes unexpectedly returned in download response')
 
-    def stream_file(self, use_chunks=True):
+    def stream_file(self, use_chunks=True, headers=None):
         """Stream the entire download.
 
         Writes retrieved bytes into :attr:`stream`.
@@ -598,6 +601,9 @@ def stream_file(self, use_chunks=True):
         :param use_chunks: If False, ignore :attr:`chunksize`
                            and stream this download in a single request.
                            If True, streams via chunks.
+
+        :type headers: dict
+        :param headers: (Optional) Headers to be used for the ``Request``.
         """
         self._ensure_initialized()
         while True:
@@ -607,7 +613,8 @@ def stream_file(self, use_chunks=True):
             else:
                 end_byte = self._compute_end_byte(self.progress,
                                                   use_chunks=use_chunks)
-                response = self._get_chunk(self.progress, end_byte)
+                response = self._get_chunk(self.progress, end_byte,
+                                           headers=headers)
             if self.total_size is None:
                 self._set_total(response.info)
             response = self._process_response(response)


Test patch:
diff --git a/system_tests/storage.py b/system_tests/storage.py
--- a/system_tests/storage.py
+++ b/system_tests/storage.py
@@ -149,6 +149,7 @@ def tearDown(self):
 
 
 class TestStorageWriteFiles(TestStorageFiles):
+    ENCRYPTION_KEY = 'b23ff11bba187db8c37077e6af3b25b8'
 
     def test_large_file_write_from_stream(self):
         blob = self.bucket.blob('LargeFile')
@@ -163,6 +164,29 @@ def test_large_file_write_from_stream(self):
             md5_hash = md5_hash.encode('utf-8')
         self.assertEqual(md5_hash, file_data['hash'])
 
+    def test_large_encrypted_file_write_from_stream(self):
+        blob = self.bucket.blob('LargeFile',
+                                encryption_key=self.ENCRYPTION_KEY)
+
+        file_data = self.FILES['big']
+        with open(file_data['path'], 'rb') as file_obj:
+            blob.upload_from_file(file_obj)
+            self.case_blobs_to_delete.append(blob)
+
+        md5_hash = blob.md5_hash
+        if not isinstance(md5_hash, six.binary_type):
+            md5_hash = md5_hash.encode('utf-8')
+        self.assertEqual(md5_hash, file_data['hash'])
+
+        temp_filename = tempfile.mktemp()
+        with open(temp_filename, 'wb') as file_obj:
+            blob.download_to_file(file_obj)
+
+        with open(temp_filename, 'rb') as file_obj:
+            md5_temp_hash = _base64_md5hash(file_obj)
+
+        self.assertEqual(md5_temp_hash, file_data['hash'])
+
     def test_small_file_write_from_filename(self):
         blob = self.bucket.blob('SmallFile')