Repository: iterative/dvc

Issue:
s3: introduce timeout config option
# Bug Report

## Description

I am trying to push files with approximately 1GB per file to an internal s3 storage. For other (smaller) files this works perfectly fine, however with the 1GB files I am experiencing a timeout, while the logs from the storage do not show any error and keep uploading even after the timeout. 

### Reproduce

adding 5 1GB files to the repository and then using dvc push -v -j 1

### Expected

The files should be pushed to the remote.

### Environment information

Microsoft Windows 10 Enterprise Ver. 10.0.18363 Build 18363 


**Output of `dvc doctor`:**

```console
$ dvc doctor

DVC version: 2.4.1 (pip)
---------------------------------
Platform: Python 3.7.4 on Windows-10-10.0.18362-SP0
Supports: http, https, s3, ssh
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: s3
Workspace directory: NTFS on C:\
Repo: dvc, git


```


**Additional Information (if any):**

```
2021-06-28 17:29:15,903 DEBUG: Uploading '.dvc\cache\a2\2fe6fed4cf9e93cd9938b6cb3dc911' to 's3://mystorage/dvc/a2/2fe6fed4cf9e93cd9938b6cb3dc911'
2021-06-28 17:34:59,468 ERROR: failed to upload '.dvc\cache\a2\2fe6fed4cf9e93cd9938b6cb3dc911' to 's3://mystorage/dvc/a2/2fe6fed4cf9e93cd9938b6cb3dc911' - 'NoneType' object is not subscriptable
------------------------------------------------------------
Traceback (most recent call last):
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "c:\util\anaconda3\lib\http\client.py", line 1336, in getresponse
    response.begin()
  File "c:\util\anaconda3\lib\http\client.py", line 306, in begin
    version, status, reason = self._read_status()
  File "c:\util\anaconda3\lib\http\client.py", line 267, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "c:\util\anaconda3\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
  File "c:\util\anaconda3\lib\ssl.py", line 1071, in recv_into
    return self.read(nbytes, buffer)
  File "c:\util\anaconda3\lib\ssl.py", line 929, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\util\anaconda3\lib\site-packages\botocore\httpsession.py", line 323, in send
    chunked=self._chunked(request.headers),
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 756, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "c:\util\anaconda3\lib\site-packages\urllib3\util\retry.py", line 507, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "c:\util\anaconda3\lib\site-packages\urllib3\packages\six.py", line 735, in reraise
    raise value
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 706, in urlopen
    chunked=chunked,
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "c:\util\anaconda3\lib\site-packages\urllib3\connectionpool.py", line 337, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='myhost', port=443): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\util\anaconda3\lib\site-packages\dvc\fs\s3.py", line 156, in wrapper
    func(*args, **kwargs)
  File "c:\util\anaconda3\lib\site-packages\dvc\fs\s3.py", line 214, in _upload
    Config=self._transfer_config,
  File "c:\util\anaconda3\lib\site-packages\boto3\s3\inject.py", line 280, in object_upload_file
    ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)
  File "c:\util\anaconda3\lib\site-packages\boto3\s3\inject.py", line 131, in upload_file
    extra_args=ExtraArgs, callback=Callback)
  File "c:\util\anaconda3\lib\site-packages\boto3\s3\transfer.py", line 279, in upload_file
    future.result()
  File "c:\util\anaconda3\lib\site-packages\s3transfer\futures.py", line 106, in result
    return self._coordinator.result()
  File "c:\util\anaconda3\lib\site-packages\s3transfer\futures.py", line 265, in result
    raise self._exception
  File "c:\util\anaconda3\lib\site-packages\s3transfer\tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "c:\util\anaconda3\lib\site-packages\s3transfer\tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "c:\util\anaconda3\lib\site-packages\s3transfer\upload.py", line 722, in _main
    Body=body, **extra_args)
  File "c:\util\anaconda3\lib\site-packages\botocore\client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "c:\util\anaconda3\lib\site-packages\botocore\client.py", line 663, in _make_api_call
    operation_model, request_dict, request_context)
  File "c:\util\anaconda3\lib\site-packages\botocore\client.py", line 682, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 137, in _send_request
    success_response, exception):
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 256, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "c:\util\anaconda3\lib\site-packages\botocore\hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "c:\util\anaconda3\lib\site-packages\botocore\hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "c:\util\anaconda3\lib\site-packages\botocore\hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 251, in __call__
    caught_exception)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 317, in __call__
    caught_exception)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 223, in __call__
    attempt_number, caught_exception)
  File "c:\util\anaconda3\lib\site-packages\botocore\retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "c:\util\anaconda3\lib\site-packages\botocore\endpoint.py", line 269, in _send
    return self.http_session.send(request)
  File "c:\util\anaconda3\lib\site-packages\botocore\httpsession.py", line 349, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://myhost/mystorage/dvc/a2/2fe6fed4cf9e93cd9938b6cb3dc911?uploadId=2~pIdriVwtHnYe2HUsrEJrSn0wQy7HSW8&partNumber=1"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\util\anaconda3\lib\site-packages\dvc\remote\base.py", line 36, in wrapper
    func(from_info, to_info, *args, **kwargs)
  File "c:\util\anaconda3\lib\site-packages\dvc\fs\base.py", line 235, in upload
    no_progress_bar=no_progress_bar,
  File "c:\util\anaconda3\lib\site-packages\dvc\fs\s3.py", line 160, in wrapper
    raise translate_boto_error(exc)
  File "c:\util\anaconda3\lib\site-packages\s3fs\errors.py", line 142, in translate_boto_error
    code = error.response["Error"].get("Code")
TypeError: 'NoneType' object is not subscriptable
```

Any hints would be greatly appreciated.


Discussion:
Hi @JenniferHem

What s3 storage are you using? minio?
We are using a Ceph Storage Cluster
@JenniferHem Are you able to upload that file by hand using aws CLI? Just trying to figure out where the problem is.
Yes the GUI works and I can upload the files. 
@JenniferHem Not sure what you mean by GUI. What about aws CLI though(e.g. `aws s3 cp ...`)? 
@efiop Hi sorry I have to internally figure things out, that is why I am taking a bit of time. So we have a GUI which uses boto3 for the upload. However, currently I am seeing no Error in the GUI but using the CLI via boto3 in a script I again see the timeout. So I am investigating on our side again. I will report back here with more information
@efiop So we tested internally, I can use boto3 for the upload if I increase the timeout of the Connection:
```
from botocore.client import Config
import boto3

config = Config(connect_timeout=1200, read_timeout=1200,
                retries={'max_attempts': 1})
s3 = boto3.resource(
    's3',
    endpoint_url='https://host:port',
    aws_access_key_id='key',
    aws_secret_access_key='secret',
    config=config
)
```
is there such a possibility in DVC? 
@JenniferHem Unfortunately, there is no such option right now. In order to add that, one needs to add those two options to our config (`dvc/config_schema.py`) and then pass it to `S3FileSystem` (`dvc/fs/s3.py`) which will then pass it to `s3fs` https://github.com/dask/s3fs/blob/fc4489cfe749b2d3de62bde926ed93c05e27ce7a/s3fs/core.py#L329

Maybe we could also consider increasing defaults for it to provide a better experience for users in more cases.

Btw, I suppose `1200` is just an arbitrary value that you've used? It would be pretty interesting to see values that are slightly above minimal working ones, just to get a sense of how bad the timeouts are.

I've pushed a very dirty hack with just increasing the timeouts to my repo, you could try it out with:

```
pip install "git+https://github.com/efiop/dvc@fix-6241#egg=dvc[s3]"
```

just to see if that's the only problem. Please let us know how it goes.
Thanks for the fast reply! 

@efiop I tried to get a useful timeout value and 400 worked, whereas 300 did not. I did not narrow it down further yet. My guess is my pretty lazy internet connection + a (necessary) VPN which does not really increase speed. 

I have installed your Version, however the timeout error persists (I am after installation on Version dvc 2.4.3+bc67f8). If you need more information or anything I should try, let me know.
> @JenniferHem Unfortunately, there is no such option right now. In order to add that, one needs to add those two options to our config (`dvc/config_schema.py`) and then pass it to `S3FileSystem` (`dvc/fs/s3.py`) which will then pass it to `s3fs` https://github.com/dask/s3fs/blob/fc4489cfe749b2d3de62bde926ed93c05e27ce7a/s3fs/core.py#L329
> 
> Maybe we could also consider increasing defaults for it to provide a better experience for users in more cases.
> 
> Btw, I suppose `1200` is just an arbitrary value that you've used? It would be pretty interesting to see values that are slightly above minimal working ones, just to get a sense of how bad the timeouts are.
> 
> I've pushed a very dirty hack with just increasing the timeouts to my repo, you could try it out with:
> 
> ```
> pip install "git+https://github.com/efiop/dvc@fix-6241#egg=dvc[s3]"
> ```
> 
> just to see if that's the only problem. Please let us know how it goes.

It is a universal argument in all of remotes (maybe except the local remote )
@karajan1001 Good point! There is a formal difference between read and connect timeouts in s3, but in awscli sources it defaults to botocore's DEFAULT_TIMEOUT for both (60sec), so indeed it is tempting to just introduce one `timeout` for both in our s3 config :slightly_smiling_face: So I agree that we should just go this way.

@JenniferHem Thanks for the info! 400 sec is a lot, probably way too much to set it as default. I also wonder if you won't have other problems with this, like if your connection is even capable enough to handle any non-negligible amount of data. To judge that, you could try to use `awscli` to upload/download your typical data sizes. Also I wonder what is the minimal read timeout needed for your case compared to minimal connection timeout.
@efiop so far 1GB is the largest filesize I am trying to upload, I believe it will not get larger than this. 

I agree that 400 is much more than should be needed as a timeout, so I guess it would not be the useful default to set, however it might be helpful for users to be able to define a timeout for cases where bad Internet + a VPN is making life hard. 
I will try to set different timeouts, however I will not manage to do this within this week. 

Resolving patch:
diff --git a/dvc/config_schema.py b/dvc/config_schema.py
--- a/dvc/config_schema.py
+++ b/dvc/config_schema.py
@@ -156,6 +156,8 @@ class RelPath(str):
                     "grant_write_acp": str,
                     "grant_full_control": str,
                     "cache_regions": bool,
+                    "read_timeout": Coerce(int),
+                    "connect_timeout": Coerce(int),
                     **REMOTE_COMMON,
                 },
                 "gs": {
diff --git a/dvc/fs/s3.py b/dvc/fs/s3.py
--- a/dvc/fs/s3.py
+++ b/dvc/fs/s3.py
@@ -99,6 +99,11 @@ def _prepare_credentials(self, **config):
         client["endpoint_url"] = config.get("endpointurl")
         client["verify"] = config.get("ssl_verify")
 
+        # timeout configuration
+        config_kwargs = login_info["config_kwargs"]
+        config_kwargs["read_timeout"] = config.get("read_timeout")
+        config_kwargs["connect_timeout"] = config.get("connect_timeout")
+
         # encryptions
         additional = login_info["s3_additional_kwargs"]
         additional["ServerSideEncryption"] = config.get("sse")


Test patch: