Repository: jschneier/django-storages

Issue:
S3Boto3Storage raises a misleading exception on bucket creation
S3Boto3Storage assumes any HTTP response besides a 301 or a 404 indicates that the bucket could not be created and AWS_AUTO_CREATE_BUCKET is `False` and raises an exception saying this. However, if AWS_S3_ENDPOINT_URL is set to point to a custom S3 host that happens to return some other HTTP error code, this exception is misleading.

Example (using Minio S3 instance running at `http://localhost:9000` with no buckets initially configured):

#### settings.py
```python
...
AWS_ACCESS_KEY_ID = '****'
AWS_SECRET_ACCESS_KEY = '****'
AWS_STORAGE_BUCKET_NAME = 'bucket'

AWS_S3_ENDPOINT_URL = 'http://localhost:9000'

AWS_AUTO_CREATE_BUCKET = True
AWS_S3_FILE_OVERWRITE = True
AWS_S3_USE_SSL = False

STATIC_URL = '%s/minio/%s/' % (AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME)
STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'

DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
...
```

#### models.py
```python
from django.db import models

class Document(models.Model):
    upload = models.FileField(upload_to='documents')
```

#### error
```bash
(env) ainterr@dev ~/g/test> python manage.py shell
Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from app import models
>>> from django.core.files import File
>>> d = models.Document()
>>> d.upload.save('test.txt', File(open('test.txt')))
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 95, in save
    self.name = self.storage.save(name, content, max_length=self.field.max_length)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/django/core/files/storage.py", line 54, in save
    return self._save(name, content)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 436, in _save
    obj = self.bucket.Object(encoded_name)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 279, in bucket
    self._bucket = self._get_or_create_bucket(self.bucket_name)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 353, in _get_or_create_bucket
    "``True``." % name)
ImproperlyConfigured: Bucket bucket does not exist. Buckets can be automatically created by setting AWS_AUTO_CREATE_BUCKET to ``True``.
>>> from django.conf import settings
>>> settings.AWS_AUTO_CREATE_BUCKET
True
```

#### further
```bash
>>> r.upload.storage.auto_create_bucket
True
>>> s = r.upload.storage
>>> bucket = s.connection.Bucket(s.bucket_name)
>>> bucket.meta.client.head_bucket(Bucket=s.bucket_name)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (502) when calling the HeadBucket operation (reached max retries: 4): Bad Gateway
```

In this case a bad proxy configuration caused the Minio instance at `http://localhost:9000` to return a 502 but I had to dig through the django-storages source code to make sure I wasn't going crazy and actually had set AWS_AUTO_CREATE_BUCKET to `True`.


Discussion:
Yep, it does not make much sense.

While you’re here: can you expand a bit on why you are using AWS_AUTO_CREATE_BUCKET? I know it’s convenient but given the additional permissions required I’m considering deprecating and removing or further discouraging it.

> On Aug 31, 2018, at 2:14 PM, Alex Interrante-Grant <notifications@github.com> wrote:
> 
> S3Boto3Storage assumes any HTTP response besides a 301 or a 404 indicates that the bucket could not be created and AWS_AUTO_CREATE_BUCKET is False and raises an exception saying this. However, if AWS_S3_ENDPOINT_URL is set to point to a custom S3 host that happens to return some other HTTP error code, this exception is misleading.
> 
> Example (using Minio S3 instance running at http://localhost:9000 with no buckets initially configured):
> 
> settings.py
> 
> ...
> AWS_ACCESS_KEY_ID = '****'
> AWS_SECRET_ACCESS_KEY = '****'
> AWS_STORAGE_BUCKET_NAME = 'bucket'
> 
> AWS_S3_ENDPOINT_URL = 'http://localhost:9000'
> 
> AWS_AUTO_CREATE_BUCKET = True
> AWS_S3_FILE_OVERWRITE = True
> AWS_S3_USE_SSL = False
> 
> STATIC_URL = '%s/minio/%s/' % (AWS_S3_ENDPOINT_URL, AWS_STORAGE_BUCKET_NAME)
> STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
> 
> DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
> ...
> models.py
> 
> from django.db import models
> 
> class Document(models.Model):
>     upload = models.FileField(upload_to='documents')
> error
> 
> (env) ainterr@dev ~/g/test> python manage.py shell
> Python 2.7.12 (default, Dec  4 2017, 14:50:18) 
> [GCC 5.4.0 20160609] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> (InteractiveConsole)
> >>> from app import models
> >>> from django.core.files import File
> >>> d = models.Document()
> >>> d.upload.save('test.txt', File(open('test.txt')))
> Traceback (most recent call last):
>   File "<console>", line 1, in <module>
>   File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/django/db/models/fields/files.py", line 95, in save
>     self.name = self.storage.save(name, content, max_length=self.field.max_length)
>   File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/django/core/files/storage.py", line 54, in save
>     return self._save(name, content)
>   File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 436, in _save
>     obj = self.bucket.Object(encoded_name)
>   File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 279, in bucket
>     self._bucket = self._get_or_create_bucket(self.bucket_name)
>   File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/storages/backends/s3boto3.py", line 353, in _get_or_create_bucket
>     "``True``." % name)
> ImproperlyConfigured: Bucket bucket does not exist. Buckets can be automatically created by setting AWS_AUTO_CREATE_BUCKET to ``True``.
> >>> from django.conf import settings
> >>> settings.AWS_AUTO_CREATE_BUCKET
> True
> further
> 
> >>> r.upload.storage.auto_create_bucket
> True
> >>> s = r.upload.storage
> >>> bucket = s.connection.Bucket(s.bucket_name)
> >>> bucket.meta.client.head_bucket(Bucket=s.bucket_name)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
>     return self._make_api_call(operation_name, kwargs)
>   File "/home/ainterr/git/test/env/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
>     raise error_class(parsed_response, operation_name)
> botocore.exceptions.ClientError: An error occurred (502) when calling the HeadBucket operation (reached max retries: 4): Bad Gateway
> In this case a bad proxy configuration caused the Minio instance at http://localhost:9000 to return a 502 but I had to dig through the django-storages source code to make sure I wasn't going crazy and actually had set AWS_AUTO_CREATE_BUCKET to True.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub <https://github.com/jschneier/django-storages/issues/574>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJB2Jx1S_7yf15rBLHqbyMSe2g5DeIAks5uWXzvgaJpZM4WVpsU>.
> 


Purely convenience in our case. Though it looks like to me like a 502 from the configured S3 endpoint would raise this exception regardless.
Right. When the Boto3 backend was written a lot of language was taken directly from the older Boto backend which handles auto-creation slightly differently. Since they are different the language from the exception should have been changed.

> On Aug 31, 2018, at 2:58 PM, Alex Interrante-Grant <notifications@github.com> wrote:
> 
> Purely convenience in our case. Though it looks like to me like a 502 from the configured S3 endpoint would raise this exception regardless.
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub <https://github.com/jschneier/django-storages/issues/574#issuecomment-417760269>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACJB2PcFpgH8j2THw1v6cIsZVD1nPTFvks5uWYdrgaJpZM4WVpsU>.
> 



Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -389,10 +389,7 @@ def _get_or_create_bucket(self, name):
                             'LocationConstraint': region_name}
                     bucket.create(**bucket_params)
                 else:
-                    raise ImproperlyConfigured("Bucket %s does not exist. Buckets "
-                                               "can be automatically created by "
-                                               "setting AWS_AUTO_CREATE_BUCKET to "
-                                               "``True``." % name)
+                    raise
         return bucket
 
     def _clean_name(self, name):


Test patch: