Repository: jschneier/django-storages

Issue:
S3 provider SSE-C not working since parameters (key) is not passed to Boto3
When we enable SSE-C for the S3 provider using the following settings:
```python3
AWS_S3_OBJECT_PARAMETERS = {
    "SSECustomerAlgorithm": "AES256",
    "SSECustomerKey": "<redacted key>",
    "SSECustomerKeyMD5": "<redacted md5 key>"
}
```

We can successfully save a file to S3 (Minio) but when we try to retrieve it, we get a generic error.
The file however is normally accessible using the aws-cli [when supplied with the correct encryption keys](https://docs.minio.io/docs/how-to-use-minio-s-server-side-encryption-with-aws-cli).
Meaning django-storages handles saving files using encryption just fine, but doesn't read them correctly.

I dived into the code and found 2 issues when reading files from S3.
1. The first HEAD request (https://github.com/jschneier/django-storages/blob/master/storages/backends/s3boto3.py#L74) does not contain the necessary parameters (i.e. `AWS_S3_OBJECT_PARAMETERS` that contain the necessary headers). 
Adding the encryption keys to the HEAD request is [required as per AWS S3 documentation](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html):
> If you encrypt an object by using server-side encryption with customer-provided encryption keys (SSE-C) when you store the object in Amazon S3, then when you retrieve the metadata from the object, you must use the following headers: 
> * x-amz-server-side​-encryption​-customer-algorithm
> * x-amz-server-side​-encryption​-customer-key
> * x-amz-server-side​-encryption​-customer-key-MD5



2. Once 1. is fixed, we still get an error because when we fetch the contents of the file (https://github.com/jschneier/django-storages/blob/master/storages/backends/s3boto3.py#L100), we do so again without passing params (`AWS_S3_OBJECT_PARAMETERS`) to boto3


Discussion:

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -34,6 +34,7 @@
 
 try:
     import boto3.session
+    import s3transfer.constants
     from boto3.s3.transfer import TransferConfig
     from botocore.client import Config
     from botocore.exceptions import ClientError
@@ -88,6 +89,13 @@ def _cloud_front_signer_from_pem(key_id, pem):
             'Supported backends are packages: cryptography and rsa.')
 
 
+def _filter_download_params(params):
+    return {
+        key: value for (key, value) in params.items()
+        if key in s3transfer.constants.ALLOWED_DOWNLOAD_ARGS
+    }
+
+
 @deconstructible
 class S3Boto3StorageFile(CompressedFileMixin, File):
     """
@@ -117,7 +125,8 @@ def __init__(self, name, mode, storage, buffer_size=None):
         self.obj = storage.bucket.Object(name)
         if 'w' not in mode:
             # Force early RAII-style exception if object does not exist
-            self.obj.load()
+            params = _filter_download_params(self._storage.get_object_parameters(self.name))
+            self.obj.load(**params)
         self._is_dirty = False
         self._raw_bytes_written = 0
         self._file = None
@@ -147,7 +156,12 @@ def _get_file(self):
             )
             if 'r' in self._mode:
                 self._is_dirty = False
-                self.obj.download_fileobj(self._file, Config=self._storage.transfer_config)
+                params = _filter_download_params(self._storage.get_object_parameters(self.name))
+                self.obj.download_fileobj(
+                    self._file,
+                    ExtraArgs=params,
+                    Config=self._storage.transfer_config
+                )
                 self._file.seek(0)
                 if self._storage.gzip and self.obj.content_encoding == 'gzip':
                     self._file = self._decompress_file(mode=self._mode, file=self._file)


Test patch:
diff --git a/tests/test_s3boto3.py b/tests/test_s3boto3.py
--- a/tests/test_s3boto3.py
+++ b/tests/test_s3boto3.py
@@ -854,6 +854,19 @@ def setUp(self) -> None:
         self.storage = s3boto3.S3Boto3Storage()
         self.storage._connections.connection = mock.MagicMock()
 
+    def test_loading_ssec(self):
+        params = {"SSECustomerKey": "xyz", "CacheControl": "never"}
+        self.storage.get_object_parameters = lambda name: params
+
+        filtered = {"SSECustomerKey": "xyz"}
+        f = s3boto3.S3Boto3StorageFile('test', 'r', self.storage)
+        f.obj.load.assert_called_once_with(**filtered)
+
+        f.file
+        f.obj.download_fileobj.assert_called_once_with(
+            mock.ANY, ExtraArgs=filtered, Config=self.storage.transfer_config
+        )
+
     def test_closed(self):
         f = s3boto3.S3Boto3StorageFile('test', 'wb', self.storage)