Repository: jschneier/django-storages

Issue:
s3boto: saving media leads to infinite loop since 1.12 
We discovered an issue with django-storages 1.12 where saving new files to S3 ends up in an infinite loop.  
1.11.1 still works fine. A bisect between the releases pinpoints the problem to 3222c235748de6ad6d4e1e3d92601ee0714ee81f reverting this commit on master fixes the issue for us. 

What could be the issue here? It looks like `Storage.save` in Django uses `get_available_name`, which uses `exists`, which is why our new file-save calls this method. 

The error S3 returns here is: 
```python
{'Error': {'Code': '403', 'Message': 'Forbidden'},
 'ResponseMetadata': {'HTTPHeaders': {'content-type': 'application/xml',
                                      'date': 'Mon, 25 Oct 2021 15:25:12 GMT',
                                      'server': 'AmazonS3',
                                      'x-amz-id-2': '(redacted)',
                                      'x-amz-request-id': '(redacted)'},
                      'HTTPStatusCode': 403,
                      'HostId': '(redacted)',
                      'RequestId': '(redacted)',
                      'RetryAttempts': 1}}
```

Which ends up leading to `True` for `exists()`, so Django searches forever for a new filename. 

I don't know enough about the reasoning behind 3222c235748de6ad6d4e1e3d92601ee0714ee81f to know what the solution here is. What information do you need? More about the config? Is our bucket configuration the problem? 

I'm happy to provide a PR with the fix, when I know what the possible solution could be. 


Discussion:
403 is some kind of permissions error so it seems like a misconfigurartion.

In general it doesn’t seem like returning either is the correct thing to do, maybe the error needs to be re-raised.
> 403 is some kind of permissions error so it seems like a misconfigurartion.

I digged further and I tracked it down to [this part of the `boto` documentation](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object) 

> A `HEAD` request has the same options as a `GET` action on an object. The response is identical to the `GET` response except that there is no response body. Because of this, if the `HEAD` request generates an error, it returns a generic `404 Not Found` or `403 Forbidden code`. It is not possible to retrieve the exact exception beyond these error codes.

and 

> You need the relevant read object (or version) permission for this operation. For more information, see Specifying Permissions in a Policy . If the object you request does not exist, the error Amazon S3 returns depends on whether you also have the `s3:ListBucket` permission.
> - If you have the `s3:ListBucket` permission on the bucket, Amazon S3 returns an HTTP status code 404 ("no such key") error.
> - If you don’t have the `s3:ListBucket` permission, Amazon S3 returns an HTTP status code 403 ("access denied") error.

So adding `s3:ListBucket` to the policies would solve the issue. Is there already a page in the docs about needed permissions in the access policies? This would be important to add to prevent the same question coming up again. 

> In general it doesn’t seem like returning either is the correct thing to do, maybe the error needs to be re-raised.

There I agree, this would have been the better experience. 
Ah wait, I already found the docs about this: https://django-storages.readthedocs.io/en/latest/backends/amazon-S3.html#iam-policy

So what remains is making this an actual error

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -469,18 +469,15 @@ def exists(self, name):
             self.connection.meta.client.head_object(Bucket=self.bucket_name, Key=name)
             return True
         except ClientError as error:
-            if error.response.get('Error', {}).get('Code') == '404':
+            if error.response['ResponseMetadata']['HTTPStatusCode'] == 404:
                 return False
 
-            # Some other error was encountered. As `get_available_name` calls this,
-            # we have to assume the filename is unavailable. If we return true due to some
-            # other error, we'd overwrite a file.
-            return True
+            # Some other error was encountered. Re-raise it.
+            raise
 
     def listdir(self, name):
         path = self._normalize_name(self._clean_name(name))
-        # The path needs to end with a slash, but if the root is empty, leave
-        # it.
+        # The path needs to end with a slash, but if the root is empty, leave it.
         if path and not path.endswith('/'):
             path += '/'
 


Test patch:
diff --git a/tests/test_s3boto3.py b/tests/test_s3boto3.py
--- a/tests/test_s3boto3.py
+++ b/tests/test_s3boto3.py
@@ -488,15 +488,25 @@ def test_storage_exists(self):
 
     def test_storage_exists_false(self):
         self.storage.connection.meta.client.head_object.side_effect = ClientError(
-            {'Error': {'Code': '404', 'Message': 'Not Found'}},
+            {'Error': {}, 'ResponseMetadata': {'HTTPStatusCode': 404}},
             'HeadObject',
         )
-        self.assertFalse(self.storage.exists("file.txt"))
+        self.assertFalse(self.storage.exists('file.txt'))
         self.storage.connection.meta.client.head_object.assert_called_with(
             Bucket=self.storage.bucket_name,
             Key='file.txt',
         )
 
+    def test_storage_exists_other_error_reraise(self):
+        self.storage.connection.meta.client.head_object.side_effect = ClientError(
+            {'Error': {}, 'ResponseMetadata': {'HTTPStatusCode': 403}},
+            'HeadObject',
+        )
+        with self.assertRaises(ClientError) as cm:
+            self.storage.exists('file.txt')
+
+        self.assertEqual(cm.exception.response['ResponseMetadata']['HTTPStatusCode'], 403)
+
     def test_storage_delete(self):
         self.storage.delete("path/to/file.txt")
         self.storage.bucket.Object.assert_called_with('path/to/file.txt')