Repository: jschneier/django-storages

Issue:
S3Boto3Storage raises ValueError: I/O operation on closed file.
When running `python manage.py collectstatic` we get the following exception:

```
Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/core/management/__init__.py", line 363, in execute_from_command_line
    utility.execute()
  File "items()venvs/app_root/lib/python3.5/site-packages/django/core/management/__init__.py", line 355, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 199, in handle
    collected = self.collect()
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 139, in collect
    for original_path, processed_path, processed in processor:
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 246, in post_process
    for name, hashed_name, processed, _ in self._post_process(paths, adjustable_paths, hashed_files):
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 312, in _post_process
    hashed_name = self.hashed_name(name, content_file)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 109, in hashed_name
    file_hash = self.file_hash(clean_name, content)
  File "~/venvs/app_root/lib/python3.5/site-packages/django/contrib/staticfiles/storage.py", line 86, in file_hash
    for chunk in content.chunks():
  File "~/venvs/app_root/lib/python3.5/site-packages/django/core/files/base.py", line 76, in chunks
    self.seek(0)
ValueError: I/O operation on closed file.
```

This only happens when using django-storages 1.6.4 or above. 
Versions 1.6.3 and lower work fine.

We're using Django 1.11.4, python 3.5.2, boto3 1.4.6.


Discussion:
@tsifrer thanks for the report. Just to clarify -- this happens on both 1.6.4 and 1.6.5?
@jschneier Correct, happens on both, 1.6.4 and 1.6.5.
I'm also having this issue, any update on it ?

It looks like the save of S3Boto3Storage now closes the file. I think it's a side effect of https://github.com/jschneier/django-storages/commit/c73680e7d7f906b841f77dd1aa4f5c3cfa8fb3a2 but haven't  tried with the commit reverted.
I am also facing this issue but not with collectstatic. But rather with uploading a file. I just moved from boto 2.40.0 to boto3 1.4.7 and django-storages from 1.1.8 to 1.6.5. After reading the comments above, I changed django-storages to 1.6.3 but the issue still occurs. Any help?
I just tried going back versions even further and at 1.5.0, this issue doesn't occur. It occurs on every version later than 1.5.0.
Moreover this issue only arises when I am trying to upload an image file after cropping it. For all other cases, there is no problem at all. 

Here is a snippet for uploading a cropped image file:-

```
def decodeImage(dataurl):
    from django.core.files.temp import NamedTemporaryFile
    from django.core.files import File
    imgstr = re.search(r'base64,(.*)', dataurl).group(1)
    img_temp = NamedTemporaryFile(delete=True)
    img_temp.write(imgstr.decode('base64'))
    img_temp.flush()
    return File(img_temp)

def post(self, request, *args, **kwargs):
    dataurl = request.POST['cropped']
    new_pic = decodeImage(dataurl)
    from django.utils.timezone import now
    self.myobject.profile_picture.save(str(now()), new_pic)
```


I experience this issue with version 1.6.5, but not with 1.6.3.
Same as @abhinavnair for me. Uploading a file results in a crash, but `collectstatic` is perfectly fine.

The bug seems to be filetype independent, The same issue occurs on image and text files.

The 1.5 version works well, but the bug shows up on every version greater than it.
Has anyone manged to work around this?
@WhyNotHugo yes, you don't need to assume that the saved file handler is open https://github.com/jschneier/django-storages/issues/382#issuecomment-327761800
@abhinavnair 
thx, downgrade to 1.5.0 helps


@xrmx It's not an assumption I'm making; it's an assumption `ManifestFilesMixin` makes (and that's part of Django itself). I've pushed #437 to work around this (I'm working on fixing a few tests right now).

Note that, if you're using `ManifestFilesMixin`, you also need to set `AWS_QUERYSTRING_AUTH = False`; otherwise there's an infinite loop rewriting file path URLs.
I had to lock the Pipfile to this:

```
"boto3" = "==1.5.0"
django-storages = "==1.6.3"
```
I had this issue too with django-storages == "1.6.5" but after downgrading to "1.5.0" with suggestion by @abhinavnair problem doesn't occured.
I get this issue with django-storages version 1.6.6 too.
Using a custom StorageClass fix the issue : 

```python
from storages.backends.s3boto3 import S3Boto3Storage, SpooledTemporaryFile
import os


class CustomS3Boto3Storage(S3Boto3Storage):

    def _save_content(self, obj, content, parameters):
        """
        We create a clone of the content file as when this is passed to boto3 it wrongly closes
        the file upon upload where as the storage backend expects it to still be open
        """
        # Seek our content back to the start
        content.seek(0, os.SEEK_SET)

        # Create a temporary file that will write to disk after a specified size
        content_autoclose = SpooledTemporaryFile()

        # Write our original content into our copy that will be closed by boto3
        content_autoclose.write(content.read())

        # Upload the object which will auto close the content_autoclose instance
        super(CustomS3Boto3Storage, self)._save_content(obj, content_autoclose, parameters)

        # Cleanup if this is fixed upstream our duplicate should always close
        if not content_autoclose.closed:
            content_autoclose.close()

```
I tried suggestion by @coderberry and it didn't work.

I tried @WhyNotHugo pull request (did `pip install git+git://github.com/WhyNotHugo/django-storages.git@fix-valueerror-boto3`) but it didn't help either.

One thing I noticed was that when I tried to upload an image the first time it crashed (but uploaded), but second attempt worked. **Every image worked in second attempt but not in first.**


I managed to get it working with example by @charlesthk 
the PR from @WhyNotHugo worked with Boto3==1.5.36 and it seems @charlesthk's fix works with the latest version of everything `django-storages[boto3]==1.6.6` and `boto3==1.7.3`
@charlesthk can you make a pull request with this to the master?
I tried the solution by @charlesthk but it gives me an error: cant' start new thread.
Can anyone help me with this?

Edit: The error occurs once in a while but not every time. I have been searching for the solution but found nothing.
I confirm that `collectstatic` works downgrading django-storages to 1.6.3. (I'm using the Storage class used in #151)
And I also confirm that @charlesthk solution works with django-storages 1.6.6 (currently the lastest)

Thanks @charlesthk! 🎉 
#481 fixes similar issue in Google Cloud Storage. All the storages need to act like the Django file storage, which calls `content.seek(0)` before doing anything else. 
I still experience this issue in django-storages 1.7.1 .
Also having this issue. The custom class in @charlesthk's comment solves it.
What needs to happen to get this fix enabled by default?
@charlesthk Do you know if your custom save content method will work with the ManifestFilesMixin? I'm haven't had a problem running this until I added the mixin and now I'm seeing this exact error. 
@selected-pixel-jameson - I also wasn't having an issue until I started using `ManifestFilesMixin` and charlesthk's solution fixed it.
Guys, Any updates? When the updated version is going to be released?
@charlesthk Did you submit a pull request with your solution?
+1 on this, I also used **charlesthk** solution temporarily and it worked with:
```
django-storages==1.7.1
boto3==1.9.168
botocore==1.12.168
Django==1.11.21
```
Pull request with @charlesthk's solution is here: https://github.com/jschneier/django-storages/pull/717
Would it be possible to get a new release out?

Resolving patch:
diff --git a/storages/backends/s3boto3.py b/storages/backends/s3boto3.py
--- a/storages/backends/s3boto3.py
+++ b/storages/backends/s3boto3.py
@@ -488,18 +488,6 @@ def _save(self, name, content):
         if self.preload_metadata:
             self._entries[encoded_name] = obj
 
-        # If both `name` and `content.name` are empty or None, your request
-        # can be rejected with `XAmzContentSHA256Mismatch` error, because in
-        # `django.core.files.storage.Storage.save` method your file-like object
-        # will be wrapped in `django.core.files.File` if no `chunks` method
-        # provided. `File.__bool__`  method is Django-specific and depends on
-        # file name, for this reason`botocore.handlers.calculate_md5` can fail
-        # even if wrapped file-like object exists. To avoid Django-specific
-        # logic, pass internal file-like object if `content` is `File`
-        # class instance.
-        if isinstance(content, File):
-            content = content.file
-
         self._save_content(obj, content, parameters=parameters)
         # Note: In boto3, after a put, last_modified is automatically reloaded
         # the next time it is accessed; no need to specifically reload it.


Test patch:
diff --git a/tests/test_s3boto3.py b/tests/test_s3boto3.py
--- a/tests/test_s3boto3.py
+++ b/tests/test_s3boto3.py
@@ -116,7 +116,7 @@ def test_storage_save(self):
 
         obj = self.storage.bucket.Object.return_value
         obj.upload_fileobj.assert_called_with(
-            content.file,
+            content,
             ExtraArgs={
                 'ContentType': 'text/plain',
                 'ACL': self.storage.default_acl,
@@ -135,7 +135,7 @@ def test_storage_save_with_acl(self):
 
         obj = self.storage.bucket.Object.return_value
         obj.upload_fileobj.assert_called_with(
-            content.file,
+            content,
             ExtraArgs={
                 'ContentType': 'text/plain',
                 'ACL': 'private',
@@ -154,7 +154,7 @@ def test_content_type(self):
 
         obj = self.storage.bucket.Object.return_value
         obj.upload_fileobj.assert_called_with(
-            content.file,
+            content,
             ExtraArgs={
                 'ContentType': 'image/jpeg',
                 'ACL': self.storage.default_acl,
@@ -170,7 +170,7 @@ def test_storage_save_gzipped(self):
         self.storage.save(name, content)
         obj = self.storage.bucket.Object.return_value
         obj.upload_fileobj.assert_called_with(
-            content.file,
+            content,
             ExtraArgs={
                 'ContentType': 'application/octet-stream',
                 'ContentEncoding': 'gzip',