Repository: wagtail/wagtail

Issue:
Error Uploading an Image
When I click the "Upload" button to upload an image from a page, I get `ValueError: The file cannot be reopened.`

Here is the full stack trace:

```
[11/Jun/2015 13:05:29] "POST /cms/images/chooser/upload/ HTTP/1.1" 500 23947
Internal Server Error: /cms/images/chooser/upload/
Traceback (most recent call last):
  File "/Users/john/.virtualenvs/website/lib/python2.7/site-packages/django/core/handlers/base.py", line 111, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/john/.virtualenvs/website/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 21, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/Users/john/.virtualenvs/website/lib/python2.7/site-packages/wagtail/wagtailimages/views/chooser.py", line 141, in     chooser_upload
    {'image_json': get_image_json(image)}
  File "/Users/john/.virtualenvs/website/lib/python2.7/site-packages/wagtail/wagtailimages/views/chooser.py", line 22, in     get_image_json
    preview_image = image.get_rendition('max-130x100')
  File "/Users/john/.virtualenvs/website/lib/python2.7/site-packages/wagtail/wagtailimages/models.py", line 196, in get_rendition
    generated_image = filter.process_image(image_file, backend_name=backend_name, focal_point=self.get_focal_point())
  File "/Users/john/.virtualenvs/website/lib/python2.7/site-packages/wagtail/wagtailimages/models.py", line 371, in process_image
    input_file.open('rb')
  File "/Users/john/.virtualenvs/website/lib/python2.7/site-packages/django/core/files/base.py", line 128, in open
    raise ValueError("The file cannot be reopened.")
ValueError: The file cannot be reopened.
```



Discussion:
UPDATE:

So when this error happened, the image did actually upload just fine.  After the error I went over to "Images" in the left menu and it showed up and I could select it for the page and everything.  So this error is happening _after_ the image is uploaded successfully. 

This error only happens using the modal window for uploading images while editing a page.  When I upload an image using the tool under "Images" it works great without errors.

Looks like Wagtail is trying to generate a rendition from an image that has just been uploaded (in the same view). This is probably an easy fix, just need to refresh the image object before generating a rendition from it.

Hi @chadsaun, can you confirm which version of Wagtail you're using? Please post the output of `pip freeze`.

What version of Django are you using? Are you storing image files locally?

Here's my `pip freeze`

```
Babel==1.3
BabelDjango==0.2.2
beautifulsoup4==4.3.2
boto==2.38.0
braintree==3.12.0
cairocffi==0.6
CairoSVG==1.0.13
cffi==0.9.2
click==4.0
cryptography==0.8.2
Django==1.7.8
django-appconf==1.0.1
django-compressor==1.5
django-datetime-widget==0.9.3
django-debug-toolbar==1.3.0
django-images==0.4.1
django-libsass==0.3
django-livefield==2.1.0
django-model-utils==2.0.3
django-modelcluster==0.6.2
django-mptt==0.7.4
django-offsite-storage==0.0.5
django-overextends==0.3.2
django-payments==0.6.5
django-prices==0.3.4
django-recaptcha==1.0.3
django-selectable==0.8.0
django-taggit==0.12.3
django-treebeard==2.0
djangorestframework==3.0.3
djrill==1.3.0
docopt==0.4.0
enum34==1.0.4
fake-factory==0.4.2
feedparser==5.2.0
google-measurement-protocol==0.1.3
html5lib==0.999
jsonfield==1.0.0
libsass==0.8.2
mailchimp==2.0.9
Markdown==2.5.2
mysqlclient==1.3.6
ndg-httpsclient==0.3.3
Pillow==2.8.2
prices==0.5.1
psycopg2==2.5.4
pyasn1==0.1.7
pycparser==2.13
pycrypto==2.6.1
PyJWT==1.0.1
pyOpenSSL==0.15.1
python-dotenv==0.1.2
pytz==2015.4
reportlab==2.7
requests==2.7.0
saleor==0.1.0a0
satchless==1.1.3
six==1.9.0
South==1.0.2
sphinx-me==0.3
sqlparse==0.1.15
stripe==1.22.1
suds-jurko==0.6
unicodecsv==0.13.0
Unidecode==0.4.18
Unipath==1.0
wagtail==0.8.7
XlsxWriter==0.7.2
```

These images are being stored in an S3 bucket.  Everything seems to work perfectly through the "Images" menu page/uploader though.  Just seems to be an issue with the modal window uploader tool thingymabob.

I take it you're using [django-offsite-storage](https://github.com/mirumee/django-offsite-storage) for that, and you have `DEFAULT_FILE_STORAGE` set to `'offsite_storage.storages.S3MediaStorage'`...?

Yes
`DEFAULT_FILE_STORAGE = 'offsite_storage.storages.S3MediaStorage'`

It might be the fact that we're calling `image.file.file.open()`. In wagtail 1.0+ we call `image.file.open()` which is probably a bit higher level and may be able to handle this.

I can confirm that this issue appears in the current master as well


Resolving patch:
diff --git a/wagtail/wagtailimages/models.py b/wagtail/wagtailimages/models.py
--- a/wagtail/wagtailimages/models.py
+++ b/wagtail/wagtailimages/models.py
@@ -75,6 +75,17 @@ class AbstractImage(models.Model, TagSearchable):
 
     file_size = models.PositiveIntegerField(null=True, editable=False)
 
+    def is_stored_locally(self):
+        """
+        Returns True if the image is hosted on the local filesystem
+        """
+        try:
+            self.file.path
+
+            return True
+        except NotImplementedError:
+            return False
+
     def get_file_size(self):
         if self.file_size is None:
             try:
@@ -107,8 +118,18 @@ def get_willow_image(self):
         # Open file if it is closed
         close_file = False
         try:
+            image_file = self.file
+
             if self.file.closed:
-                self.file.open('rb')
+                # Reopen the file
+                if self.is_stored_locally():
+                    self.file.open('rb')
+                else:
+                    # Some external storage backends don't allow reopening
+                    # the file. Get a fresh file instance. #1397
+                    storage = self._meta.get_field('file').storage
+                    image_file = storage.open(self.file.name, 'rb')
+
                 close_file = True
         except IOError as e:
             # re-throw this as a SourceImageIOError so that calling code can distinguish
@@ -116,13 +137,13 @@ def get_willow_image(self):
             raise SourceImageIOError(text_type(e))
 
         # Seek to beginning
-        self.file.seek(0)
+        image_file.seek(0)
 
         try:
-            yield WillowImage.open(self.file)
+            yield WillowImage.open(image_file)
         finally:
             if close_file:
-                self.file.close()
+                image_file.close()
 
     def get_rect(self):
         return Rect(0, 0, self.width, self.height)
diff --git a/wagtail/wagtailimages/views/images.py b/wagtail/wagtailimages/views/images.py
--- a/wagtail/wagtailimages/views/images.py
+++ b/wagtail/wagtailimages/views/images.py
@@ -123,15 +123,9 @@ def edit(request, image_id):
     except NoReverseMatch:
         url_generator_enabled = False
 
-    try:
-        local_path = image.file.path
-    except NotImplementedError:
-        # Image is hosted externally (eg, S3)
-        local_path = None
-
-    if local_path:
+    if image.is_stored_locally():
         # Give error if image file doesn't exist
-        if not os.path.isfile(local_path):
+        if not os.path.isfile(image.file.path):
             messages.error(request, _("The source image file could not be found. Please change the source or delete the image.").format(image.title), buttons=[
                 messages.button(reverse('wagtailimages:delete', args=(image.id,)), _('Delete'))
             ])


Test patch:
diff --git a/wagtail/tests/dummy_external_storage.py b/wagtail/tests/dummy_external_storage.py
new file mode 100644
--- /dev/null
+++ b/wagtail/tests/dummy_external_storage.py
@@ -0,0 +1,72 @@
+# This file contains a file storage backend that imitates behaviours of
+# common external storage backends (S3 boto, libcloud, etc).
+
+# The following behaviours have been added to this backend:
+#  - Calling .path on the storage or image file raises NotImplementedError
+#  - File.open() after the file has been closed raises an error
+
+from django.core.files.storage import Storage, FileSystemStorage
+from django.core.files.base import File
+from django.utils.deconstruct import deconstructible
+
+
+@deconstructible
+class DummyExternalStorage(Storage):
+    def __init__(self, *args, **kwargs):
+        self.wrapped = FileSystemStorage(*args, **kwargs)
+
+    def path(self, name):
+        # Overridden to give it the behaviour of the base Storage class
+        # This is what an external storage backend would have
+        raise NotImplementedError("This backend doesn't support absolute paths.")
+
+    def _open(self, name, mode='rb'):
+        # Overridden to return a DummyExternalStorageFile instead of a normal
+        # File object
+        return DummyExternalStorageFile(open(self.wrapped.path(name), mode))
+
+
+    # Wrap all other functions
+
+    def _save(self, name, content):
+        return self.wrapped._save(name, content)
+
+    def delete(self, name):
+        self.wrapped.delete(name)
+
+    def exists(self, name):
+        return self.wrapped.exists(name)
+
+    def listdir(self, path):
+        return self.wrapped.listdir(path)
+
+    def size(self, name):
+        return self.wrapped.size(name)
+
+    def url(self, name):
+        return self.wrapped.url(name)
+
+    def accessed_time(self, name):
+        return self.wrapped.accessed_time(name)
+
+    def created_time(self, name):
+        return self.wrapped.created_time(name)
+
+    def modified_time(self, name):
+        return self.wrapped.modified_time(name)
+
+
+class DummyExternalStorageFile(File):
+    def open(self, mode=None):
+        # Based on: https://github.com/django/django/blob/2c39f282b8389f47fee4b24e785a58567c6c3629/django/core/files/base.py#L135-L141
+
+        # I've commented out two lines of this function which stops it checking
+        # the filesystem for the file. Making it behave as if it is using an
+        # external file storage.
+
+        if not self.closed:
+            self.seek(0)
+        # elif self.name and os.path.exists(self.name):
+        #    self.file = open(self.name, mode or self.mode)
+        else:
+            raise ValueError("The file cannot be reopened.")
diff --git a/wagtail/wagtailimages/tests/test_admin_views.py b/wagtail/wagtailimages/tests/test_admin_views.py
--- a/wagtail/wagtailimages/tests/test_admin_views.py
+++ b/wagtail/wagtailimages/tests/test_admin_views.py
@@ -89,6 +89,19 @@ def test_add(self):
         # Test that the file_size field was set
         self.assertTrue(image.file_size)
 
+    @override_settings(DEFAULT_FILE_STORAGE='wagtail.tests.dummy_external_storage.DummyExternalStorage')
+    def test_add_with_external_file_storage(self):
+        response = self.post({
+            'title': "Test image",
+            'file': SimpleUploadedFile('test.png', get_test_image_file().file.getvalue()),
+        })
+
+        # Should redirect back to index
+        self.assertRedirects(response, reverse('wagtailimages:index'))
+
+        # Check that the image was created
+        self.assertTrue(Image.objects.filter(title="Test image").exists())
+
     def test_add_no_file_selected(self):
         response = self.post({
             'title': "Test image",
@@ -142,6 +155,19 @@ def test_simple(self):
         self.assertEqual(response.status_code, 200)
         self.assertTemplateUsed(response, 'wagtailimages/images/edit.html')
 
+    @override_settings(DEFAULT_FILE_STORAGE='wagtail.tests.dummy_external_storage.DummyExternalStorage')
+    def test_simple_with_external_storage(self):
+        # The view calls get_file_size on the image that closes the file if
+        # file_size wasn't prevously populated.
+
+        # The view then attempts to reopen the file when rendering the template
+        # which caused crashes when certian storage backends were in use.
+        # See #1397
+
+        response = self.get()
+        self.assertEqual(response.status_code, 200)
+        self.assertTemplateUsed(response, 'wagtailimages/images/edit.html')
+
     def test_edit(self):
         response = self.post({
             'title': "Edited",
@@ -173,6 +199,26 @@ def test_edit_with_new_image_file(self):
         image = Image.objects.get(id=self.image.id)
         self.assertNotEqual(image.file_size, 100000)
 
+    @override_settings(DEFAULT_FILE_STORAGE='wagtail.tests.dummy_external_storage.DummyExternalStorage')
+    def test_edit_with_new_image_file_and_external_storage(self):
+        file_content = get_test_image_file().file.getvalue()
+
+        # Change the file size of the image
+        self.image.file_size = 100000
+        self.image.save()
+
+        response = self.post({
+            'title': "Edited",
+            'file': SimpleUploadedFile('new.png', file_content),
+        })
+
+        # Should redirect back to index
+        self.assertRedirects(response, reverse('wagtailimages:index'))
+
+        # Check that the image file size changed (assume it changed to the correct value)
+        image = Image.objects.get(id=self.image.id)
+        self.assertNotEqual(image.file_size, 100000)
+
     def test_with_missing_image_file(self):
         self.image.file.delete(False)
 
@@ -304,6 +350,19 @@ def test_upload_no_file_selected(self):
         # The form should have an error
         self.assertFormError(response, 'uploadform', 'file', "This field is required.")
 
+    @override_settings(DEFAULT_FILE_STORAGE='wagtail.tests.dummy_external_storage.DummyExternalStorage')
+    def test_upload_with_external_storage(self):
+        response = self.client.post(reverse('wagtailimages:chooser_upload'), {
+            'title': "Test image",
+            'file': SimpleUploadedFile('test.png', get_test_image_file().file.getvalue()),
+        })
+
+        # Check response
+        self.assertEqual(response.status_code, 200)
+
+        # Check that the image was created
+        self.assertTrue(Image.objects.filter(title="Test image").exists())
+
 
 class TestMultipleImageUploader(TestCase, WagtailTestUtils):
     """
diff --git a/wagtail/wagtailimages/tests/test_models.py b/wagtail/wagtailimages/tests/test_models.py
--- a/wagtail/wagtailimages/tests/test_models.py
+++ b/wagtail/wagtailimages/tests/test_models.py
@@ -79,6 +79,13 @@ def test_set_focal_point(self):
         self.assertEqual(self.image.focal_point_width, None)
         self.assertEqual(self.image.focal_point_height, None)
 
+    def test_is_stored_locally(self):
+        self.assertTrue(self.image.is_stored_locally())
+
+    @override_settings(DEFAULT_FILE_STORAGE='wagtail.tests.dummy_external_storage.DummyExternalStorage')
+    def test_is_stored_locally_with_external_storage(self):
+        self.assertFalse(self.image.is_stored_locally())
+
 
 class TestImagePermissions(TestCase):
     def setUp(self):