Repository: spotify/luigi

Issue:
GCSClient.download()/.put() is unusably slow for large files
**TL;DR**: It takes 20 _MINUTES_ to download a `1.7 GB` files with `GCSClient()` but `gsutil` does it in about 20 seconds.  Working on a PR to parameterize `chunk_size` and suggest the default be changed to `10 MB`.  Not sure why the Google docs recommend such a small chunk size.

`GCSClient.download()` downloads files in [`1 MB` chunks](https://github.com/spotify/luigi/blob/672856d3b01428681c390773f5e15346c49f3ec5/luigi/contrib/gcs.py#L336), which is what the [Google Cloud Storage API docs use](https://cloud.google.com/storage/docs/json_api/v1/objects/get).  Select `python` from the menu under `Examples`.  `GCSClient.put()` uses a [`2 MB`](https://github.com/spotify/luigi/blob/672856d3b01428681c390773f5e15346c49f3ec5/luigi/contrib/gcs.py#L245) [chunk size](https://github.com/spotify/luigi/blob/672856d3b01428681c390773f5e15346c49f3ec5/luigi/contrib/gcs.py#L55).

Obviously this works but is incredibly slow for large files as each API call costs about `0.5` seconds.  I ran some download tests against `gsutil` on a [`n1-standard-2`](https://cloud.google.com/compute/docs/machine-types) machine with a `50GB` boot disk in zone `us-central-a` with a file that is `1.7 GB`.  Did not test upload but it appears to be about as slow.

| Chunk Size | Time (m:s) |
| :-: | :-: |
| gsutil | 00:21 |
| 1 mb | 20:04 |
| 10 mb | 01:01 |
| 100 mb | 00:44 |

Test code:

``` python
import json
import httplib2
import oauth2client.client

from googleapiclient import errors
from googleapiclient import discovery
from googleapiclient import http
import io
from apiclient.http import MediaIoBaseDownload

oauth_credentials = oauth2client.client.GoogleCredentials.get_application_default()

client = discovery.build('storage', 'v1', credentials=oauth_credentials, http=httplib2.Http())

bucket_name='{bucket}'
object_name='2015-01-01.nmea'

# Get Metadata
req = client.objects().get(
        bucket=bucket_name,
        object=object_name)
resp = req.execute()
#print json.dumps(resp, indent=2)

# Get Payload Data
req = client.objects().get_media(
        bucket=bucket_name,
        object=object_name)
# The BytesIO object may be replaced with any io.Base instance.
fh = io.FileIO(object_name, 'w')
downloader = MediaIoBaseDownload(fh, req, chunksize=10*1024*1024)
done = False
while not done:
    status, done = downloader.next_chunk()
```



Discussion:
wow that's crazy. let's bump chunk size then.

what does gsutil do that is so much faster?

Not sure.  I looked around in the source code a bit late in the day yesterday and haven't had time to figure it out today - will revisit before I submit a PR.  Here's the [`cp` command implementation](https://github.com/GoogleCloudPlatform/gsutil/blob/3f3e582dac01f3988ce12d605dc431b7302a43d1/gslib/commands/cp.py#L688) if anyone else wants to look around.

wanna send a PR for higher chunk size?

Yep - and to parameterize the `chunksize` so you can up it on larger machines.  Already made the changes and am running the test suite locally right now.  A `10 mb` default is still slower than I would like for what I'm working with.


Resolving patch:
diff --git a/luigi/contrib/gcs.py b/luigi/contrib/gcs.py
--- a/luigi/contrib/gcs.py
+++ b/luigi/contrib/gcs.py
@@ -52,7 +52,7 @@
 NUM_RETRIES = 5
 
 # Number of bytes to send/receive in each request.
-CHUNKSIZE = 2 * 1024 * 1024
+CHUNKSIZE = 10 * 1024 * 1024
 
 # Mimetype to use if one can't be guessed from the file extension.
 DEFAULT_MIMETYPE = 'application/octet-stream'
@@ -97,6 +97,9 @@ class GCSClient(luigi.target.FileSystem):
              scope='https://www.googleapis.com/auth/devstorage.read_write')
          client = GCSClient(oauth_credentials=credentails)
 
+        The chunksize parameter specifies how much data to transfer when downloading
+        or uploading files.
+
     .. warning::
       By default this class will use "automated service discovery" which will require
       a connection to the web. The google api client downloads a JSON file to "create" the
@@ -104,7 +107,9 @@ class GCSClient(luigi.target.FileSystem):
       contents of this file (currently found at https://www.googleapis.com/discovery/v1/apis/storage/v1/rest )
       as the ``descriptor`` argument.
     """
-    def __init__(self, oauth_credentials=None, descriptor='', http_=None):
+    def __init__(self, oauth_credentials=None, descriptor='', http_=None,
+                 chunksize=CHUNKSIZE):
+        self.chunksize = chunksize
         http_ = http_ or httplib2.Http()
 
         if not oauth_credentials:
@@ -238,11 +243,12 @@ def remove(self, path, recursive=True):
 
         return False
 
-    def put(self, filename, dest_path, mimetype=None):
+    def put(self, filename, dest_path, mimetype=None, chunksize=None):
+        chunksize = chunksize or self.chunksize
         resumable = os.path.getsize(filename) > 0
 
         mimetype = mimetype or mimetypes.guess_type(dest_path)[0] or DEFAULT_MIMETYPE
-        media = http.MediaFileUpload(filename, mimetype, chunksize=CHUNKSIZE, resumable=resumable)
+        media = http.MediaFileUpload(filename, mimetype, chunksize=chunksize, resumable=resumable)
 
         self._do_put(media, dest_path)
 
@@ -320,7 +326,8 @@ def listdir(self, path):
         for it in self._list_iter(bucket, obj_prefix):
             yield self._add_path_delimiter(path) + it['name'][obj_prefix_len:]
 
-    def download(self, path):
+    def download(self, path, chunksize=None):
+        chunksize = chunksize or self.chunksize
         bucket, obj = self._path_to_bucket_and_key(path)
 
         with tempfile.NamedTemporaryFile(delete=False) as fp:
@@ -333,7 +340,7 @@ def download(self, path):
                 return return_fp
 
             request = self.client.objects().get_media(bucket=bucket, object=obj)
-            downloader = http.MediaIoBaseDownload(fp, request, chunksize=1024 * 1024)
+            downloader = http.MediaIoBaseDownload(fp, request, chunksize=chunksize)
 
             attempts = 0
             done = False


Test patch: