Repository: iterative/dvc

Issue:
Support additional configuration parameters for S3 transfer
**Problem**
When using S3 as remote for DVC, boto3 reads also the configuration file `~/.aws/config` typically used by the original AWS CLI tool. Not all configuration parameters from this file have an effect in DVC at the moment. For instance, the last line `multipart_threshold = 500MB` will be skipped by DVC while the previous line `signature_version = s3` will be applied when pushing data into S3:      
```ini
[default]
output = json
s3 =
  signature_version = s3
  multipart_threshold = 500MB
```
**Suggestion**
Suggestion is to support optional yet useful parameters from `~/.aws/config` as per [AWS CLI S3 documentation](https://docs.aws.amazon.com/cli/latest/topic/s3-config.html) to control S3 transfers for performance reasons or to unlock the advanced configuration capabilities. 

These parameters should be passed to TransferConfig:
- `max_concurrent_requests` - The maximum number of concurrent requests.
- `max_queue_size` - The maximum number of tasks in the task queue.
- `multipart_threshold` - The size threshold the CLI uses for multipart transfers of individual files.
- `multipart_chunksize` - When using multipart transfers, this is the chunk size that the CLI uses for multipart transfers of individual files.
- `max_bandwidth` - The maximum bandwidth that will be consumed for uploading and downloading data to and from Amazon S3.

If it is reasonable and doable - support also the following set of parameters:
- `use_accelerate_endpoint` - Use the Amazon S3 Accelerate endpoint for all s3 and s3api commands. You must first enable S3 Accelerate on your bucket before attempting to use the endpoint. This is mutually exclusive with the use_dualstack_endpoint option.
- `use_dualstack_endpoint` - Use the Amazon S3 dual IPv4 / IPv6 endpoint for all s3 and s3api commands. This is mutually exclusive with the use_accelerate_endpoint option.
- `addressing_style` - Specifies which addressing style to use. This controls if the bucket name is in the hostname or part of the URL. Value values are: path, virtual, and auto. The default value is auto.
- `payload_signing_enabled` - Refers to whether or not to SHA256 sign sigv4 payloads. By default, this is disabled for streaming uploads (UploadPart and PutObject) when using https.

These values must be set under the top level `s3` key in the AWS Config File, which has a default location of `~/.aws/config`. Below is an example configuration:
```ini
[profile development]
aws_access_key_id=foo
aws_secret_access_key=bar
s3 =
  max_concurrent_requests = 20
  max_queue_size = 10000
  multipart_threshold = 64MB
  multipart_chunksize = 16MB
  max_bandwidth = 50MB/s
  use_accelerate_endpoint = true
  addressing_style = path
```


Comments:
For the record: seems like those options are handled by aws-cli specifically, and not by the boto3 itself: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html . This will require parsing the config ourselves and applying the options where needed, inspired by aws-cli(need to look into aws-cli source to make it match). All the code related to s3 and s3 configs is in https://github.com/iterative/dvc/blob/master/dvc/tree/s3.py.
Another case where having this would help: https://discord.com/channels/485586884165107732/485596304961962003/804679936886571019 (when the individual files are well over 10GB, and boto3 tries to chunk them in little parts)
@efiop I think at least we can support a subset of these variables, the stuff that goes into a `TransferConfig` (multipart_chunk_size, multipart_threshold etc) which is the real deal since it is causing people trouble when trying to upload huge files. What do you think? 
According to the issue from discord. I'm using S3 compatibile API, so could You also include those parameters (multipart_chunk_size, multipart_threshold) in .dvc/config? In that way .dvc/config would override the values saved in ~/.aws/config.
> According to the issue from discord. I'm using S3 compatibile API, so could You also include those parameters (multipart_chunk_size, multipart_threshold) in .dvc/config? In that way .dvc/config would override the values saved in ~/.aws/config.

@efiop what do you about this? Should we only support `~/.aws/config` to specific configurations for `s3` or have the exact same configuration options in both `~/.dvc/config` and `~/.aws/config`? 

Patch:
diff --git a/dvc/config_schema.py b/dvc/config_schema.py
--- a/dvc/config_schema.py
+++ b/dvc/config_schema.py
@@ -138,6 +138,7 @@ class RelPath(str):
                     "region": str,
                     "profile": str,
                     "credentialpath": str,
+                    "configpath": str,
                     "endpointurl": str,
                     "access_key_id": str,
                     "secret_access_key": str,
diff --git a/dvc/tree/s3.py b/dvc/tree/s3.py
--- a/dvc/tree/s3.py
+++ b/dvc/tree/s3.py
@@ -11,12 +11,14 @@
 from dvc.path_info import CloudURLInfo
 from dvc.progress import Tqdm
 from dvc.scheme import Schemes
-from dvc.utils import error_link
+from dvc.utils import conversions, error_link
 
 from .base import BaseTree
 
 logger = logging.getLogger(__name__)
 
+_AWS_CONFIG_PATH = os.path.join(os.path.expanduser("~"), ".aws", "config")
+
 
 class S3Tree(BaseTree):
     scheme = Schemes.S3
@@ -60,6 +62,56 @@ def __init__(self, repo, config):
         if shared_creds:
             os.environ.setdefault("AWS_SHARED_CREDENTIALS_FILE", shared_creds)
 
+        config_path = config.get("configpath")
+        if config_path:
+            os.environ.setdefault("AWS_CONFIG_FILE", config_path)
+        self._transfer_config = None
+
+    # https://github.com/aws/aws-cli/blob/0376c6262d6b15dc36c82e6da6e1aad10249cc8c/awscli/customizations/s3/transferconfig.py#L107-L113
+    _TRANSFER_CONFIG_ALIASES = {
+        "max_queue_size": "max_io_queue",
+        "max_concurrent_requests": "max_concurrency",
+        "multipart_threshold": "multipart_threshold",
+        "multipart_chunksize": "multipart_chunksize",
+    }
+
+    def _transform_config(self, s3_config):
+        """Splits the general s3 config into 2 different config
+        objects, one for transfer.TransferConfig and other is the
+        general session config"""
+
+        config, transfer_config = {}, {}
+        for key, value in s3_config.items():
+            if key in self._TRANSFER_CONFIG_ALIASES:
+                if key in {"multipart_chunksize", "multipart_threshold"}:
+                    # cast human readable sizes (like 24MiB) to integers
+                    value = conversions.human_readable_to_bytes(value)
+                else:
+                    value = int(value)
+                transfer_config[self._TRANSFER_CONFIG_ALIASES[key]] = value
+            else:
+                config[key] = value
+
+        return config, transfer_config
+
+    def _process_config(self):
+        from boto3.s3.transfer import TransferConfig
+        from botocore.configloader import load_config
+
+        config_path = os.environ.get("AWS_CONFIG_FILE", _AWS_CONFIG_PATH)
+        if not os.path.exists(config_path):
+            return None
+
+        config = load_config(config_path)
+        profile = config["profiles"].get(self.profile or "default")
+        if not profile:
+            return None
+
+        s3_config = profile.get("s3", {})
+        s3_config, transfer_config = self._transform_config(s3_config)
+        self._transfer_config = TransferConfig(**transfer_config)
+        return s3_config
+
     @wrap_prop(threading.Lock())
     @cached_property
     def s3(self):
@@ -78,12 +130,15 @@ def s3(self):
             session_opts["aws_session_token"] = self.session_token
 
         session = boto3.session.Session(**session_opts)
+        s3_config = self._process_config()
 
         return session.resource(
             "s3",
             endpoint_url=self.endpoint_url,
             use_ssl=self.use_ssl,
-            config=boto3.session.Config(signature_version="s3v4"),
+            config=boto3.session.Config(
+                signature_version="s3v4", s3=s3_config
+            ),
         )
 
     @contextmanager
@@ -355,7 +410,7 @@ def get_file_hash(self, path_info):
 
     def _upload_fobj(self, fobj, to_info):
         with self._get_obj(to_info) as obj:
-            obj.upload_fileobj(fobj)
+            obj.upload_fileobj(fobj, Config=self._transfer_config)
 
     def _upload(
         self, from_file, to_info, name=None, no_progress_bar=False, **_kwargs
@@ -366,7 +421,10 @@ def _upload(
                 disable=no_progress_bar, total=total, bytes=True, desc=name
             ) as pbar:
                 obj.upload_file(
-                    from_file, Callback=pbar.update, ExtraArgs=self.extra_args,
+                    from_file,
+                    Callback=pbar.update,
+                    ExtraArgs=self.extra_args,
+                    Config=self._transfer_config,
                 )
 
     def _download(self, from_info, to_file, name=None, no_progress_bar=False):
@@ -377,4 +435,6 @@ def _download(self, from_info, to_file, name=None, no_progress_bar=False):
                 bytes=True,
                 desc=name,
             ) as pbar:
-                obj.download_file(to_file, Callback=pbar.update)
+                obj.download_file(
+                    to_file, Callback=pbar.update, Config=self._transfer_config
+                )
diff --git a/dvc/utils/conversions.py b/dvc/utils/conversions.py
new file mode 100644
--- /dev/null
+++ b/dvc/utils/conversions.py
@@ -0,0 +1,24 @@
+# https://github.com/aws/aws-cli/blob/5aa599949f60b6af554fd5714d7161aa272716f7/awscli/customizations/s3/utils.py
+
+MULTIPLIERS = {
+    "kb": 1024,
+    "mb": 1024 ** 2,
+    "gb": 1024 ** 3,
+    "tb": 1024 ** 4,
+    "kib": 1024,
+    "mib": 1024 ** 2,
+    "gib": 1024 ** 3,
+    "tib": 1024 ** 4,
+}
+
+
+def human_readable_to_bytes(value):
+    value = value.lower()
+    suffix = None
+    if value.endswith(tuple(MULTIPLIERS.keys())):
+        size = 2
+        size += value[-2] == "i"  # KiB, MiB etc
+        value, suffix = value[:-size], value[-size:]
+
+    multiplier = MULTIPLIERS.get(suffix, 1)
+    return int(value) * multiplier


Test patch:
diff --git a/tests/func/test_s3.py b/tests/func/test_s3.py
--- a/tests/func/test_s3.py
+++ b/tests/func/test_s3.py
@@ -1,3 +1,6 @@
+import importlib
+import sys
+import textwrap
 from functools import wraps
 
 import boto3
@@ -130,3 +133,82 @@ def test_s3_upload_fobj(tmp_dir, dvc, s3):
         tree.upload_fobj(stream, to_info, 1)
 
     assert to_info.read_text() == "foo"
+
+
+KB = 1024
+MB = KB ** 2
+GB = KB ** 3
+
+
+def test_s3_aws_config(tmp_dir, dvc, s3, monkeypatch):
+    config_directory = tmp_dir / ".aws"
+    config_directory.mkdir()
+    (config_directory / "config").write_text(
+        textwrap.dedent(
+            """\
+    [default]
+    s3 =
+      max_concurrent_requests = 20000
+      max_queue_size = 1000
+      multipart_threshold = 1000KiB
+      multipart_chunksize = 64MB
+      use_accelerate_endpoint = true
+      addressing_style = path
+    """
+        )
+    )
+
+    if sys.platform == "win32":
+        var = "USERPROFILE"
+    else:
+        var = "HOME"
+    monkeypatch.setenv(var, str(tmp_dir))
+
+    # Fresh import to see the effects of changing HOME variable
+    s3_mod = importlib.reload(sys.modules[S3Tree.__module__])
+    tree = s3_mod.S3Tree(dvc, s3.config)
+    assert tree._transfer_config is None
+
+    with tree._get_s3() as s3:
+        s3_config = s3.meta.client.meta.config.s3
+        assert s3_config["use_accelerate_endpoint"]
+        assert s3_config["addressing_style"] == "path"
+
+    transfer_config = tree._transfer_config
+    assert transfer_config.max_io_queue_size == 1000
+    assert transfer_config.multipart_chunksize == 64 * MB
+    assert transfer_config.multipart_threshold == 1000 * KB
+    assert transfer_config.max_request_concurrency == 20000
+
+
+def test_s3_aws_config_different_profile(tmp_dir, dvc, s3, monkeypatch):
+    config_file = tmp_dir / "aws_config.ini"
+    config_file.write_text(
+        textwrap.dedent(
+            """\
+    [default]
+    extra = keys
+    s3 =
+      addressing_style = auto
+      use_accelerate_endpoint = true
+      multipart_threshold = ThisIsNotGoingToBeCasted!
+    [profile dev]
+    some_extra = keys
+    s3 =
+      addresing_style = virtual
+      multipart_threshold = 2GiB
+    """
+        )
+    )
+    monkeypatch.setenv("AWS_CONFIG_FILE", config_file)
+
+    tree = S3Tree(dvc, {**s3.config, "profile": "dev"})
+    assert tree._transfer_config is None
+
+    with tree._get_s3() as s3:
+        s3_config = s3.meta.client.meta.config.s3
+        assert s3_config["addresing_style"] == "virtual"
+        assert "use_accelerate_endpoint" not in s3_config
+
+    transfer_config = tree._transfer_config
+    assert transfer_config.multipart_threshold == 2 * GB
diff --git a/tests/unit/utils/test_conversions.py b/tests/unit/utils/test_conversions.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/utils/test_conversions.py
@@ -0,0 +1,30 @@
+import pytest
+
+from dvc.utils.conversions import human_readable_to_bytes
+
+KB = 1024
+MB = KB ** 2
+GB = KB ** 3
+TB = KB ** 4
+
+
+@pytest.mark.parametrize(
+    "test_input, expected",
+    [
+        ("10", 10),
+        ("10   ", 10),
+        ("1kb", 1 * KB),
+        ("2kb", 2 * KB),
+        ("1000mib", 1000 * MB),
+        ("20gB", 20 * GB),
+        ("10Tib", 10 * TB),
+    ],
+)
+def test_conversions_human_readable_to_bytes(test_input, expected):
+    assert human_readable_to_bytes(test_input) == expected
+
+
+@pytest.mark.parametrize("invalid_input", ["foo", "10XB", "1000Pb", "fooMiB"])
+def test_conversions_human_readable_to_bytes_invalid(invalid_input):
+    with pytest.raises(ValueError):
+        human_readable_to_bytes(invalid_input)