internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: jschneier/django-storages

Issue:
S3Boto3Storage: bucket_params not used
On [line 363](https://github.com/jschneier/django-storages/blob/master/storages/backends/s3boto3.py#L363) of the `_get_or_create_bucket()` method of `S3Boto3Storage`, `bucket_params` is defined and includes the bucket's ACL setting. However, when `bucket.create()` is called, `bucket_params` isn't used and `ACL` is set explicitly.

Additionally, if a region is specified, it's added to `bucket_params` but not set when `bucket.create()` is called. This makes all created buckets default to us-standard.

Here's the offending lines:
```python
bucket_params = {'ACL': self.bucket_acl}
region_name = self.connection.meta.client.meta.region_name
if region_name != 'us-east-1':
    bucket_params['CreateBucketConfiguration'] = {'LocationConstraint': region_name}
bucket.create(ACL=self.bucket_acl)
```

I believe the `bucket.create()` call should be `bucket.create(**bucket_params)`. I'll submit a PR for this in the next five minutes or so.


Discussion:

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -365,7 +365,7 @@ def _get_or_create_bucket(self, name):
                     if region_name != 'us-east-1':
                         bucket_params['CreateBucketConfiguration'] = {
                             'LocationConstraint': region_name}
-                    bucket.create(ACL=self.bucket_acl)
+                    bucket.create(**bucket_params)
                 else:
                     raise ImproperlyConfigured("Bucket %s does not exist. Buckets "
                                                "can be automatically created by "


Test patch:
Repository: jschneier/django-storages

Issue:
Update django-storages to 1.11
Just wanted to create a ticket for updating django-storages to be compatible with / tested on Django 1.11, since right now there is no list of officially supported versions outside of the tox.ini file.

(This issue is mostly here for people looking to figure out if this package is 1.11 compatible yet.)


Discussion:
Thanks. I'm aiming to push out a new version by Friday and will update Trove classifiers then. I still need to add the last `get_modified_time` vs `modified_time`.
Ugh the release is today, okay.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -28,6 +28,7 @@ def get_requirements_tests():
         'Framework :: Django :: 1.8',
         'Framework :: Django :: 1.9',
         'Framework :: Django :: 1.10',
+        'Framework :: Django :: 1.11',
         'Intended Audience :: Developers',
         'License :: OSI Approved :: BSD License',
         'Operating System :: OS Independent',
@@ -38,6 +39,7 @@ def get_requirements_tests():
         'Programming Language :: Python :: 3.3',
         'Programming Language :: Python :: 3.4',
         'Programming Language :: Python :: 3.5',
+        'Programming Language :: Python :: 3.6',
     ],
     tests_require=get_requirements_tests(),
     test_suite='tests',


Test patch:
Repository: jschneier/django-storages

Issue:
Dropbox: "'GetTemporaryLinkResult' object is not subscriptable" 
I'm running release 1.6.3 with dropbox==8.0.0 and Django==1.11.1. The ['link'] access on line 140 of /storages/backends/dropbox.py is giving a "'GetTemporaryLinkResult' object is not subscriptable" exception:

    def url(self, name):
        media = self.client.files_get_temporary_link(self._full_path(name))
        return media['link'] 

I've found that this will work as intended:

    d.files_get_temporary_link(path).link

I'm assuming there's something wrong with my install but I've tried several different versions of this package and the Dropbox package and get the same issue.


Discussion:
Unfortunately I don't have anyone who is willing to take on testing and the initial contributor of the Dropbox backend has gone quiet. I'd like to rewrite the tests altogether to make live API calls to prevent these sorts of breakages.



Resolving patch:
diff --git a/storages/backends/dropbox.py b/storages/backends/dropbox.py
--- a/storages/backends/dropbox.py
+++ b/storages/backends/dropbox.py
@@ -101,7 +101,7 @@ def accessed_time(self, name):
 
     def url(self, name):
         media = self.client.files_get_temporary_link(self._full_path(name))
-        return media['link']
+        return media.link
 
     def _open(self, name, mode='rb'):
         remote_file = DropBoxFile(self._full_path(name), self)


Test patch:
diff --git a/tests/test_dropbox.py b/tests/test_dropbox.py
--- a/tests/test_dropbox.py
+++ b/tests/test_dropbox.py
@@ -14,6 +14,10 @@
     import mock
 
 
+class F(object):
+    pass
+
+
 FILE_DATE = datetime(2015, 8, 24, 15, 6, 41)
 FILE_FIXTURE = {
     'bytes': 4,
@@ -52,10 +56,8 @@
     'size': '0 bytes',
     'thumb_exists': False
 }
-FILE_MEDIA_FIXTURE = {
-    'link': 'https://dl.dropboxusercontent.com/1/view/foo',
-    'expires': 'Fri, 16 Sep 2011 01:01:25 +0000',
-}
+FILE_MEDIA_FIXTURE = F()
+FILE_MEDIA_FIXTURE.link = 'https://dl.dropboxusercontent.com/1/view/foo'
 
 
 class DropBoxTest(TestCase):
@@ -123,7 +125,7 @@ def test_save(self, *args):
                 return_value=FILE_MEDIA_FIXTURE)
     def test_url(self, *args):
         url = self.storage.url('foo')
-        self.assertEqual(url, FILE_MEDIA_FIXTURE['link'])
+        self.assertEqual(url, FILE_MEDIA_FIXTURE.link)
 
     def test_formats(self, *args):
         self.storage = dropbox.DropBoxStorage('foo')
Repository: jschneier/django-storages

Issue:
Document extra dependancy when using Google Cloud Storage backend
Nowhere in the docs is mentioned that when using the Google Cloud Storage backend, you actually need to install an extra dependancy: `google-cloud-storage`

When running, an exception is raised:
```
django.core.exceptions.ImproperlyConfigured: Could not load Google Cloud Storage bindings.
See https://github.com/GoogleCloudPlatform/gcloud-python
```

So when surfing to the mentioned url, I thought I had to install the all of google python client, but that install a whole suite of dependancies. `google-cloud-storaga` is enough it seems.

I believe there is a way to install extra's to a python package, something along the lines of `pip install django-storages[google-cloud]` but I'm not familiar if this still works or how involved that is.




Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -17,6 +17,15 @@ def get_requirements_tests():
     name='django-storages',
     version=storages.__version__,
     packages=['storages', 'storages.backends'],
+    extras_require={
+        'azure': ['azure'],
+        'boto': ['boto>=2.32.0'],
+        'boto3': ['boto3>=1.2.3'],
+        'dropbox': ['dropbox>=7.2.1'],
+        'google': ['google-cloud-storage>=0.22.0'],
+        'libcloud': ['apache-libcloud'],
+        'sftp': ['paramiko'],
+    },
     author='Josh Schneier',
     author_email='josh.schneier@gmail.com',
     license='BSD',
diff --git a/storages/backends/ftp.py b/storages/backends/ftp.py
--- a/storages/backends/ftp.py
+++ b/storages/backends/ftp.py
@@ -109,7 +109,7 @@ def _mkremdirs(self, path):
         for path_part in path_splitted:
             try:
                 self._connection.cwd(path_part)
-            except:
+            except Exception:
                 try:
                     self._connection.mkd(path_part)
                     self._connection.cwd(path_part)


Test patch:
Repository: jschneier/django-storages

Issue:
Cannot pickle S3Boto3



Discussion:

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -276,6 +276,17 @@ def __init__(self, acl=None, bucket=None, **settings):
                 "set AWS_DEFAULT_ACL."
             )
 
+    def __getstate__(self):
+        state = self.__dict__.copy()
+        state.pop('_connections', None)
+        state.pop('_bucket', None)
+        return state
+
+    def __setstate__(self, state):
+        state['_connections'] = threading.local()
+        state['_bucket'] = None
+        self.__dict__ = state
+
     @property
     def connection(self):
         # TODO: Support host, port like in s3boto


Test patch:
diff --git a/tests/test_s3boto3.py b/tests/test_s3boto3.py
--- a/tests/test_s3boto3.py
+++ b/tests/test_s3boto3.py
@@ -2,6 +2,7 @@
 from __future__ import unicode_literals
 
 import gzip
+import pickle
 import threading
 import warnings
 from datetime import datetime
@@ -59,6 +60,37 @@ def test_clean_name_windows(self):
         path = self.storage._clean_name("path\\to\\somewhere")
         self.assertEqual(path, "path/to/somewhere")
 
+    def test_pickle_with_bucket(self):
+        """
+        Test that the storage can be pickled with a bucket attached
+        """
+        # Ensure the bucket has been used
+        self.storage.bucket
+        self.assertIsNotNone(self.storage._bucket)
+
+        # Can't pickle MagicMock, but you can't pickle a real Bucket object either
+        p = pickle.dumps(self.storage)
+        new_storage = pickle.loads(p)
+
+        self.assertIsInstance(new_storage._connections, threading.local)
+        # Put the mock connection back in
+        new_storage._connections.connection = mock.MagicMock()
+
+        self.assertIsNone(new_storage._bucket)
+        new_storage.bucket
+        self.assertIsNotNone(new_storage._bucket)
+
+    def test_pickle_without_bucket(self):
+        """
+        Test that the storage can be pickled, without a bucket instance
+        """
+
+        # Can't pickle a threadlocal
+        p = pickle.dumps(self.storage)
+        new_storage = pickle.loads(p)
+
+        self.assertIsInstance(new_storage._connections, threading.local)
+
     def test_storage_url_slashes(self):
         """
         Test URL generation.
Repository: jschneier/django-storages

Issue:
S3Boto3: switch to method GET Bucket (List Objects) Version 2
Hi,

Following the release of the version `1.7`, the implementation of `listdir` in `S3Boto3Backend` no longer use the GET Bucket (List Objects) Version 1 (https://docs.aws.amazon.com/AmazonS3/latest/API/RESTBucketGET.html) but use the new API Version 2 instead (https://docs.aws.amazon.com/cli/latest/reference/s3api/list-objects-v2.html).

Although the Version 1 has been deprecated for a few months already, not every independently developed storage servers implementing the S3 protocol support the version 2 of the API yet.
For instance, one such server that I use: https://github.com/scality/cloudserver

So, the new way the API is used may introduce a breaking change for some users of django-storages. Maybe forcing the use of the new API so fast is a little bit too aggressive without any transition period, deprecation warning or optional settings.

Since the old API (Version 1) is still supported by AWS as a legacy and probably for a few years, maybe it could be useful to implement a fallback strategy to the old API in case the call to the new API (Version 2) fails? 

Please, let me know if you think it could be a good idea?

Thanks for your work


Discussion:
Digital Ocean Spaces does similarly not support the v2 operation.
I would be okay with adding a fallback but would need a decent heuristic.

Although I want to say we only officially support Amazon we have now grown docs for Digital Ocean so that isn’t exactly true.

Suggestions (and pull requests) are welcome on implementation details.

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -543,7 +543,7 @@ def listdir(self, name):
 
         directories = []
         files = []
-        paginator = self.connection.meta.client.get_paginator('list_objects_v2')
+        paginator = self.connection.meta.client.get_paginator('list_objects')
         pages = paginator.paginate(Bucket=self.bucket_name, Delimiter='/', Prefix=path)
         for page in pages:
             for entry in page.get('CommonPrefixes', ()):


Test patch:
Repository: jschneier/django-storages

Issue:
Azure backend pulls broad and outdated dependencies
I am struggling getting the Azure backend to work with my other code, due to its dependencies. It pulls in the "azure" package, which is a wrapper to pull in a wide range of Azure SDKs. It hasn't been updated in quite a bit and thus conflicts with some of my code which has newer Azure SDK dependencies. It also pulls in much more than seems to be needed for the storage backend.
Could this be limited to use the "azure-common" package instead? Aside from "azure-storage-blob" this seems to be the only dependency which is really required.



Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -15,7 +15,7 @@ def read(filename):
     python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*",
     install_requires=['Django>=1.11'],
     extras_require={
-        'azure': ['azure>=3.0.0', 'azure-storage-blob>=1.3.1'],
+        'azure': ['azure-storage-blob>=1.3.1'],
         'boto': ['boto>=2.32.0'],
         'boto3': ['boto3>=1.4.4'],
         'dropbox': ['dropbox>=7.2.1'],


Test patch:
Repository: jschneier/django-storages

Issue:
Invalid sas_token was generated
https://github.com/jschneier/django-storages/blob/b441b74a17a46eb87ee4b10f60774b9a080c0fe1/storages/backends/azure_storage.py#L260

I think this should be corrected as follows.

sas_token = self.service.generate_blob_shared_access_signature(
                self.azure_container, name, permission=BlobPermissions.READ, expiry=self._expire_at(expire))


This problem is caused by
https://github.com/Azure/azure-storage-python/issues/595



Discussion:
I'm sorry to have overlooked the previous issue.

Resolving patch:
diff --git a/storages/backends/azure_storage.py b/storages/backends/azure_storage.py
--- a/storages/backends/azure_storage.py
+++ b/storages/backends/azure_storage.py
@@ -257,7 +257,7 @@ def url(self, name, expire=None):
         make_blob_url_kwargs = {}
         if expire:
             sas_token = self.service.generate_blob_shared_access_signature(
-                self.azure_container, name, BlobPermissions.READ, expiry=self._expire_at(expire))
+                self.azure_container, name, permission=BlobPermissions.READ, expiry=self._expire_at(expire))
             make_blob_url_kwargs['sas_token'] = sas_token
 
         if self.azure_protocol:


Test patch:
diff --git a/tests/test_azure.py b/tests/test_azure.py
--- a/tests/test_azure.py
+++ b/tests/test_azure.py
@@ -139,7 +139,7 @@ def test_url_expire(self):
             self.storage._service.generate_blob_shared_access_signature.assert_called_once_with(
                 self.container_name,
                 'some_blob',
-                BlobPermissions.READ,
+                permission=BlobPermissions.READ,
                 expiry=fixed_time + timedelta(seconds=100))
             self.storage._service.make_blob_url.assert_called_once_with(
                 container_name=self.container_name,
Repository: jschneier/django-storages

Issue:
Make Dropbox timeout configurable
Dropbox receive a optional parameter `timeout`. The default is 30 seconds, but in some cases it is not sufficient to upload large files.

Django Storages can read a setting and define a custom `timeout` for Dropbox.

If it make sense I can open a Pull Request.


Discussion:

Resolving patch:
diff --git a/storages/backends/dropbox.py b/storages/backends/dropbox.py
--- a/storages/backends/dropbox.py
+++ b/storages/backends/dropbox.py
@@ -25,6 +25,8 @@
 
 from storages.utils import setting
 
+_DEFAULT_TIMEOUT = 100
+
 
 class DropBoxStorageException(Exception):
     pass
@@ -67,13 +69,15 @@ class DropBoxStorage(Storage):
 
     CHUNK_SIZE = 4 * 1024 * 1024
 
-    def __init__(self, oauth2_access_token=None, root_path=None):
+    def __init__(self, oauth2_access_token=None, root_path=None, timeout=None):
         oauth2_access_token = oauth2_access_token or setting('DROPBOX_OAUTH2_TOKEN')
-        self.root_path = root_path or setting('DROPBOX_ROOT_PATH', '/')
         if oauth2_access_token is None:
-            raise ImproperlyConfigured("You must configure a token auth at"
+            raise ImproperlyConfigured("You must configure an auth token at"
                                        "'settings.DROPBOX_OAUTH2_TOKEN'.")
-        self.client = Dropbox(oauth2_access_token)
+
+        self.root_path = root_path or setting('DROPBOX_ROOT_PATH', '/')
+        timeout = timeout or setting('DROPBOX_TIMEOUT', _DEFAULT_TIMEOUT)
+        self.client = Dropbox(oauth2_access_token, timeout=timeout)
 
     def _full_path(self, name):
         if name == '/':


Test patch:
Repository: jschneier/django-storages

Issue:
Stale data in S3 entries after file is deleted
If a file is deleted, the entry persists in `entries`, resulting in stale information about files that may be deleted or replaced.

I came across this behavior when looking at the last modified time of assets I have replaced.


Discussion:

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -521,6 +521,9 @@ def delete(self, name):
         name = self._normalize_name(self._clean_name(name))
         self.bucket.Object(self._encode_name(name)).delete()
 
+        if name in self._entries:
+            del self._entries[name]
+
     def exists(self, name):
         name = self._normalize_name(self._clean_name(name))
         if self.entries:


Test patch:
Repository: jschneier/django-storages

Issue:
Dropbox base path / should be an empty string
Using `django-storages` `1.7.2`, a base path (`'/`) is provided, the Dropbox API is rejecting it:

```
  File "/usr/local/lib/python3.7/site-packages/dbbackup/storage.py", line 78, in list_directory
    return self.storage.listdir(path)[1]
  File "/usr/local/lib/python3.7/site-packages/storages/backends/dropbox.py", line 99, in listdir
    metadata = self.client.files_list_folder(full_path)
  File "/usr/local/lib/python3.7/site-packages/dropbox/base.py", line 1744, in files_list_folder
    None,
  File "/usr/local/lib/python3.7/site-packages/dropbox/dropbox.py", line 274, in request
    timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/dropbox/dropbox.py", line 365, in request_json_string_with_retry
    timeout=timeout)
  File "/usr/local/lib/python3.7/site-packages/dropbox/dropbox.py", line 456, in request_json_string
    raise BadInputError(request_id, r.text)
dropbox.exceptions.BadInputError: BadInputError('XXXXXXXXXXXXXXXXXXXX', 'Error in call to API function "files/list_folder": request body: path: Specify the root folder as an empty string rather than as "/".')
```

As the error says it should be an empty string rather than as '/'.


Discussion:

Resolving patch:
diff --git a/storages/backends/dropbox.py b/storages/backends/dropbox.py
--- a/storages/backends/dropbox.py
+++ b/storages/backends/dropbox.py
@@ -96,6 +96,10 @@ def exists(self, name):
     def listdir(self, path):
         directories, files = [], []
         full_path = self._full_path(path)
+
+        if full_path == '/':
+            full_path = ''
+
         metadata = self.client.files_list_folder(full_path)
         for entry in metadata.entries:
             if isinstance(entry, FolderMetadata):


Test patch:
diff --git a/tests/test_dropbox.py b/tests/test_dropbox.py
--- a/tests/test_dropbox.py
+++ b/tests/test_dropbox.py
@@ -80,6 +80,10 @@ def test_not_exists(self, *args):
                 return_value=FILES_MOCK)
     def test_listdir(self, *args):
         dirs, files = self.storage.listdir('/')
+        dirs2, files2 = self.storage.listdir('')
+        self.assertEqual(dirs, dirs2)
+        self.assertEqual(files2, files2)
+
         self.assertGreater(len(dirs), 0)
         self.assertGreater(len(files), 0)
         self.assertEqual(dirs[0], 'bar')
Repository: jschneier/django-storages

Issue:
Include ImportError information in ImproperlyConfigured error raised in s3boto3.py
I'm currently debugging an environment where we're seeing the `ImproperlyConfigured` error raised when importing the various boto libraries that `s3boto3.py` needs to function. It would be easier to debug based on the traceback if the `ImproperlyConfigured` error included some information about the `ImportError` that caused it.

(As it is now, we have to enter the environment in question and do some manual importing in a Python shell to work out which one it is. Not the worst, but certainly not as good as including it in my logs.)


Discussion:

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -33,9 +33,8 @@
     from boto3 import __version__ as boto3_version
     from botocore.client import Config
     from botocore.exceptions import ClientError
-except ImportError:
-    raise ImproperlyConfigured("Could not load Boto3's S3 bindings.\n"
-                               "See https://github.com/boto/boto3")
+except ImportError as e:
+    raise ImproperlyConfigured("Could not load Boto3's S3 bindings. %s" % e)
 
 
 boto3_version_info = tuple([int(i) for i in boto3_version.split('.')])


Test patch:
Repository: jschneier/django-storages

Issue:
S3Boto3StorageFile's readlines always return bytes even if mode is set to `rt`
Sample code
```
with default_storage.open(path, 'rt') as file:
     print(file.readlines())
```

you get something like
```
[b'"Number","Name","Date","Datetime"']
```



Discussion:

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -167,6 +167,9 @@ def readline(self, *args, **kwargs):
             raise AttributeError("File was not opened in read mode.")
         return self._force_mode(super().readline(*args, **kwargs))
 
+    def readlines(self):
+        return list(self)
+
     def write(self, content):
         if 'w' not in self._mode:
             raise AttributeError("File was not opened in write mode.")


Test patch:
diff --git a/tests/test_s3boto3.py b/tests/test_s3boto3.py
--- a/tests/test_s3boto3.py
+++ b/tests/test_s3boto3.py
@@ -1,5 +1,6 @@
 import datetime
 import gzip
+import io
 import pickle
 import threading
 from textwrap import dedent
@@ -300,6 +301,25 @@ def test_storage_open_read_string(self):
         self.assertEqual(content_str, "")
         file.close()
 
+    def test_storage_open_readlines(self):
+        """
+        Test readlines with file opened in "r" and "rb" modes
+        """
+        name = 'test_open_readlines_string.txt'
+        with io.BytesIO() as temp_file:
+            temp_file.write(b"line1\nline2")
+            file = self.storage.open(name, "r")
+            file._file = temp_file
+
+            content_lines = file.readlines()
+            self.assertEqual(content_lines, ["line1\n", "line2"])
+
+            temp_file.seek(0)
+            file = self.storage.open(name, "rb")
+            file._file = temp_file
+            content_lines = file.readlines()
+            self.assertEqual(content_lines, [b"line1\n", b"line2"])
+
     def test_storage_open_write(self):
         """
         Test opening a file in write mode