Repository: jschneier/django-storages

Issue:
Threaded parallel uploads on S3
I'm uploading a lot of small files. I can thread the uploads using `ThreadPool` and it works great.
If I use more than 10 threads, I have the warning `Connection pool is full, discarding connection: my.s3.server.url. Connection pool size: 10 `.
I found 2 issues about this problem (https://github.com/jschneier/django-storages/issues/1106 and https://github.com/jschneier/django-storages/issues/610).

I tried to use `TransferConfig`, without success.
```python
AWS_S3_TRANSFER_CONFIG = TransferConfig(
    use_threads=True,
    max_concurrency=70,
)
```
I tried with `use_threads=False` and  `use_threads=True`.

What should I do to increase the max number of threads?


Discussion:
Thanks, clearly my answer in the other thread is wrong. The correct `config` that needs to be overriden is the one on the class level currently, [here](https://github.com/jschneier/django-storages/blob/master/storages/backends/s3.py#L299).
Would definitely take documentation (or a feature!) for this one. It's session config vs transfer config.
Thanks!
I managed to make it work :)

The best way to override the `config` is not very clear.

```python
class MyS3Storage(S3Storage):
    def __init__(self, **settings):
        # I don't have an easy access to settings before the init
        super().__init__(**settings)
        # Is there any side effect if I change the config after its first definition 
        self.config = Config(
            s3={"addressing_style": self.addressing_style},
            signature_version=self.signature_version,
            proxies=self.proxies,
            max_pool_connections=70,
        )


class MyS3Storage(S3Storage):
    # No easy access to settings here
    config = Config(
        s3={"addressing_style": "virtual"},
        signature_version=5,
        proxies=None,
        max_pool_connections=70,
    )
```

What do you think about something like:
```python
class S3Storage(CompressStorageMixin, BaseStorage):
    """
    Amazon Simple Storage Service using Boto3

    This storage backend supports opening files in read or write
    mode and supports streaming(buffering) data in chunks to S3
    when writing.
    """

    def __init__(self, **settings):
        omitted = object()

        super().__init__(**settings)

        ...

        self.config = self.get_config()

        ...

    def get_config(self):
        # If config provided in subclass, signature_version and addressing_style
        # settings/args are ignored.
        if hasattr(self, 'config'):
            # Keep retro-compatibility
            return self.config

        return Config(
            s3={"addressing_style": self.addressing_style},
            signature_version=self.signature_version,
            proxies=self.proxies,
        )
```


Or the same API than `TransertConfig`
am I right in understanding that using a thread pool TransferConfig will benefit upload of large files (they get turned into concurrent multipart upload) but not really for "lots of small files" case?

seems like underlying Django storage implementation wants to upload each file one-by-one, though there is a workaround for threading multiple files outlined in comments to this old issue: https://code.djangoproject.com/ticket/23517

Resolving patch:
diff --git a/storages/backends/s3.py b/storages/backends/s3.py
--- a/storages/backends/s3.py
+++ b/storages/backends/s3.py
@@ -35,7 +35,7 @@
     import boto3.session
     import s3transfer.constants
     from boto3.s3.transfer import TransferConfig
-    from botocore.client import Config
+    from botocore.config import Config
     from botocore.exceptions import ClientError
     from botocore.signers import CloudFrontSigner
 except ImportError as e:
@@ -318,8 +318,17 @@ def __init__(self, **settings):
         self._bucket = None
         self._connections = threading.local()
 
-        if not self.config:
-            self.config = Config(
+        if self.config is not None:
+            warnings.warn(
+                "The 'config' class property is deprecated and will be "
+                "removed in a future version. Use AWS_S3_CLIENT_CONFIG "
+                "to customize any of the botocore.config.Config parameters.",
+                DeprecationWarning,
+            )
+            self.client_config = self.config
+
+        if self.client_config is None:
+            self.client_config = Config(
                 s3={"addressing_style": self.addressing_style},
                 signature_version=self.signature_version,
                 proxies=self.proxies,
@@ -407,6 +416,7 @@ def get_default_settings(self):
             "default_acl": setting("AWS_DEFAULT_ACL", None),
             "use_threads": setting("AWS_S3_USE_THREADS", True),
             "transfer_config": setting("AWS_S3_TRANSFER_CONFIG", None),
+            "client_config": setting("AWS_S3_CLIENT_CONFIG", None),
         }
 
     def __getstate__(self):
@@ -430,7 +440,7 @@ def connection(self):
                 region_name=self.region_name,
                 use_ssl=self.use_ssl,
                 endpoint_url=self.endpoint_url,
-                config=self.config,
+                config=self.client_config,
                 verify=self.verify,
             )
         return self._connections.connection


Test patch:
diff --git a/tests/test_s3.py b/tests/test_s3.py
--- a/tests/test_s3.py
+++ b/tests/test_s3.py
@@ -11,6 +11,7 @@
 
 import boto3
 import boto3.s3.transfer
+from botocore.config import Config as ClientConfig
 from botocore.exceptions import ClientError
 from django.conf import settings
 from django.core.exceptions import ImproperlyConfigured
@@ -42,6 +43,18 @@ def test_s3_session(self):
                 _ = storage.connection
                 mock_session.assert_called_once_with(profile_name="test_profile")
 
+    def test_client_config(self):
+        with override_settings(
+            AWS_S3_CLIENT_CONFIG=ClientConfig(max_pool_connections=30)
+        ):
+            storage = s3.S3Storage()
+            with mock.patch("boto3.Session.resource") as mock_resource:
+                _ = storage.connection
+                mock_resource.assert_called_once()
+                self.assertEqual(
+                    30, mock_resource.call_args[1]["config"].max_pool_connections
+                )
+
     def test_pickle_with_bucket(self):
         """
         Test that the storage can be pickled with a bucket attached