Repository: iterative/dvc

Issue:
Failed to upload to GCS due to connection time out
**Please provide information about your setup**
DVC version(i.e. `dvc --version`), Platform and method of installation (pip, homebrew, pkg Mac, exe (Windows), DEB(Linux), RPM(Linux))

DVC version: `0.61.2`
Method of installation: pip
Platform: Mac

As discussed in Discord, I'm having an issue uploading a dataset to GCS with size of ~50MB. I'm using `google-cloud-storage=1.19.0`. Below is the error message using `-v`.

```
DEBUG: PRAGMA user_version;
DEBUG: fetched: [(3,)]
DEBUG: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
DEBUG: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
DEBUG: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
DEBUG: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
DEBUG: PRAGMA user_version = 3;
DEBUG: Preparing to upload data to 'gs://dse/dvc/dataset_20191004'
DEBUG: Preparing to collect status from gs://dse/dvc/dataset_20191004
DEBUG: Collecting information from local cache...
DEBUG: Path .dvc/cache/a7/a404b4826cca3b01dd5d8e6326de3e inode 7438277                                                                                                                                  
DEBUG: SELECT mtime, size, md5, timestamp from state WHERE inode=?                                                                                                                                      
DEBUG: fetched: [('1570168433714557952', '54785677', 'a7a404b4826cca3b01dd5d8e6326de3e', '1570174167312645120')]                                                                                        
DEBUG: UPDATE state SET timestamp = ? WHERE inode = ?                                                                                                                                                   
DEBUG: cache '.dvc/cache/a7/a404b4826cca3b01dd5d8e6326de3e' expected 'a7a404b4826cca3b01dd5d8e6326de3e' actual 'a7a404b4826cca3b01dd5d8e6326de3e'                                                       
DEBUG: Collecting information from remote cache...                                                                                                                                                      
DEBUG: Uploading '.dvc/cache/a7/a404b4826cca3b01dd5d8e6326de3e' to 'gs://dse/dvc/dataset_20191004/a7/a404b4826cca3b01dd5d8e6326de3e'                 
ERROR: failed to upload '.dvc/cache/a7/a404b4826cca3b01dd5d8e6326de3e' to 'gs://dse/dvc/dataset_20191004/a7/a404b4826cca3b01dd5d8e6326de3e' - ('Connection aborted.', timeout('The write operation timed out',))

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
DEBUG: SELECT count from state_info WHERE rowid=?
DEBUG: fetched: [(2,)]
DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?
ERROR: failed to push data to the cloud - 1 files failed to upload
------------------------------------------------------------
Traceback (most recent call last):
  File "dvc/command/data_sync.py", line 50, in run
  File "dvc/repo/__init__.py", line 33, in wrapper
  File "dvc/repo/push.py", line 28, in push
  File "dvc/data_cloud.py", line 63, in push
  File "dvc/remote/local/__init__.py", line 403, in push
  File "dvc/remote/local/__init__.py", line 393, in _process
dvc.exceptions.UploadError: 1 files failed to upload
```
I have tried to push using option `-j 1`. But still encounter the same issue.


Discussion:
For the record: Another user is experiencing a similar issue: https://discordapp.com/channels/485586884165107732/485596304961962003/629269604614799370

Not able to reproduce this on linux, will try on mac later.
For the record: both users are from the same team.
For the record: tried running minimal example of:
```
from google.cloud.storage import Client

client = Client("myprojectname")

bucket = client.bucket("mybucket")
blob = bucket.blob("mypath")
blob.upload_from_filename("path/to/my/local/file/to/upload")
```

and user was still experiencing the same error. Though, turned out, that lowering chunk_size from default 100M to minimal 256KB made this work. ~8M worked too in this example, but `dvc push` was still failing, as it seems like our multithreaded upload was overwhelming the network.

It is also important to note that `gsutil` works differently(it is not even using google-cloud-storage package) and has dynamic chunk size https://github.com/googleapis/google-cloud-dotnet/issues/1480#issuecomment-330962961 . It might be worth it implementing a similar thing for us too(maybe even consider contributing it back to google-cloud-storage).
@syahrulhamdani How is it going, need any help? :slightly_smiling_face: 
Another user is experiencing this issue https://discordapp.com/channels/485586884165107732/485596304961962003/636843168356368387 . Bumping up the priority.

Resolving patch:
diff --git a/dvc/remote/gs.py b/dvc/remote/gs.py
--- a/dvc/remote/gs.py
+++ b/dvc/remote/gs.py
@@ -2,9 +2,11 @@
 
 import logging
 from datetime import timedelta
+from functools import wraps
+
 from funcy import cached_property
-from dvc.utils.compat import FileNotFoundError
 
+from dvc.utils.compat import FileNotFoundError  # skipcq: PYL-W0622
 from dvc.remote.base import RemoteBASE
 from dvc.config import Config
 from dvc.exceptions import DvcException
@@ -14,6 +16,41 @@
 logger = logging.getLogger(__name__)
 
 
+def dynamic_chunk_size(func):
+    @wraps(func)
+    def wrapper(*args, **kwargs):
+        import requests
+        from google.cloud.storage.blob import Blob, _DEFAULT_CHUNKSIZE
+
+        # Default chunk size for gs is 100M, which might be too much for
+        # particular network (see [1]). So if we are getting ConnectionError,
+        # we should try lowering the chunk size until we reach the minimum
+        # allowed chunk size of 256K. Also note that `chunk_size` must be a
+        # multiple of 256K per the API specification.
+        #
+        # [1] https://github.com/iterative/dvc/issues/2572
+
+        # skipcq: PYL-W0212
+        multiplier = int(_DEFAULT_CHUNKSIZE / Blob._CHUNK_SIZE_MULTIPLE)
+        while True:
+            try:
+                # skipcq: PYL-W0212
+                chunk_size = Blob._CHUNK_SIZE_MULTIPLE * multiplier
+                return func(*args, chunk_size=chunk_size, **kwargs)
+            except requests.exceptions.ConnectionError:
+                multiplier = int(multiplier / 2)
+                if not multiplier:
+                    raise
+
+    return wrapper
+
+
+@dynamic_chunk_size
+def _upload_to_bucket(bucket, from_file, to_info, **kwargs):
+    blob = bucket.blob(to_info.path, **kwargs)
+    blob.upload_from_filename(from_file)
+
+
 class RemoteGS(RemoteBASE):
     scheme = Schemes.GS
     path_cls = CloudURLInfo
@@ -88,8 +125,7 @@ def exists(self, path_info):
 
     def _upload(self, from_file, to_info, **_kwargs):
         bucket = self.gs.bucket(to_info.bucket)
-        blob = bucket.blob(to_info.path)
-        blob.upload_from_filename(from_file)
+        _upload_to_bucket(bucket, from_file)
 
     def _download(self, from_info, to_file, **_kwargs):
         bucket = self.gs.bucket(from_info.bucket)


Test patch:
diff --git a/tests/unit/remote/test_gs.py b/tests/unit/remote/test_gs.py
--- a/tests/unit/remote/test_gs.py
+++ b/tests/unit/remote/test_gs.py
@@ -1,7 +1,9 @@
 import mock
+import pytest
+import requests
 from unittest import TestCase
 
-from dvc.remote.gs import RemoteGS
+from dvc.remote.gs import RemoteGS, dynamic_chunk_size
 
 
 class TestRemoteGS(TestCase):
@@ -36,3 +38,27 @@ def test_gs_no_credspath(self, mock_client):
         remote = RemoteGS(None, config)
         remote.gs()
         mock_client.assert_called_with(self.PROJECT)
+
+
+def test_dynamic_chunk_size():
+    chunk_sizes = []
+
+    @dynamic_chunk_size
+    def upload(chunk_size=None):
+        chunk_sizes.append(chunk_size)
+        raise requests.exceptions.ConnectionError()
+
+    with pytest.raises(requests.exceptions.ConnectionError):
+        upload()
+
+    assert chunk_sizes == [
+        104857600,
+        52428800,
+        26214400,
+        13107200,
+        6553600,
+        3145728,
+        1572864,
+        786432,
+        262144,
+    ]