Repository: python-pillow/Pillow

Issue:
Pillow 3.0 raises errors on open when using tempfile.TemporaryFile
Here is a Travis job which starts to fail because of Pillow update: https://travis-ci.org/scrapy/scrapy/jobs/83354354. Sorry, I haven't checked what happens and why, so this bug report is very vague. Maybe someone with more detailed knowledge can spot an issue immediately. If not, I'll take a deeper look later.

This line fails: https://github.com/scrapy/scrapy/blob/fe15f93e533be36e81e0385691fe5571c88b0b31/tests/test_pipeline_images.py#L81. This is `_create_image` function: https://github.com/scrapy/scrapy/blob/fe15f93e533be36e81e0385691fe5571c88b0b31/tests/test_pipeline_images.py#L208.

There are also some errors printed to stdout in this job; they can be a different Pillow issue.



Discussion:
How are you installing pillow?  

It is in tox.ini deps; we're running tox on Travis. I believe Travis uses Ubuntu 12.04 and tox just runs 'pip install'.

Yes, Travis is Ubuntu 12.04. If you pin Pillow to 2.9.0, what do you get?

Just tried it on OS X locally: Scrapy testing suite fails with Pillow 3.0, but passes if I pin Pillow to 2.9.0. So with Pillow 2.9.0 there are no exceptions. I haven't seen these error messages in stdout before either.

The command to get a test failure is `tox -r -e py27 tests/test_pipeline_images.py` executed from https://github.com/scrapy/scrapy checkout.

I wonder if Travis has libjpeg-dev and libz-dev packages installed.

So, on OS X it should be installing a binary wheel, so that shouldn't have the jpeg or zlib build issue.

These packages should be installed: Travis tests used to work before the upgrade, they started to fail when Pilow 3.0 was released. Or are there new dependencies in Pillow 3.0?

right, and it also fails in OS X, so this is not OS-specific.

Pillow 3.0.0 now requires libjpeg and zlib, before they were optional: https://github.com/python-pillow/Pillow/issues/1412

They were formerly optional, but commonly used and if not used, indicative of installing pillow without a working tool chain.  Now they are required unless disabled. 

But it should fail during installation, not at runtime, right? Travis tox log shows that Pillow==3.0.0 is installed.

Yes..  It should fail on install. So, then not that problem.  Also ruled out by the OS X failure.

What image is failing in this test?   It's looking like it's just not getting recognized, and it's falling through to the we can't identify this image error.

Could be bad exif info borking a jpeg, could be an mpo change. That's what I remember touching in JpegImagePlugin this time through.

I think stdout log errors are not related to the test failure, they are different.

Ok, a short example to reproduce the issue. This was previously supported, but not now:

``` python
from PIL import Image
from tempfile import TemporaryFile

buf = TemporaryFile()
im = Image.new('RGB', (100, 100), (0, 127, 255))
im.save(buf, 'JPEG')

buf.seek(0)
im = Image.open(buf)
```

I'm not sure why is the code written this way though.

Previously we used BytesIO instead of TemporaryFile, but as I recall BytesIO buffers were not supported by Pillow.

So, I just ran this:

```
from helper import PillowTestCase
from PIL import Image

class test_scrapy(PillowTestCase):
    def _create_image(self, format, *a, **kw):
        buf = open(self.tempfile('temp.jpg'), 'w+b')
        Image.new(*a, **kw).save(buf, format)
        buf.seek(0)
        return Image.open(buf)

    def test_create_image(self):
        SIZE = (100, 100)
        # straigh forward case: RGB and JPEG
        COLOUR = (0, 127, 255)
        im = self._create_image('JPEG', 'RGB', SIZE, COLOUR)

        self.assert_image_equal(Image.new('RGB', (100,100), COLOUR), im)

```

and it passes. 

Do you use Python 2 or Python 3? My example with TemporaryFile fails in Python 2.7 but passes in Python 3.3.

BytesIO are fine. They're all over our test cases. 

This was 2.7

I think the only difference here is the tempfile implementation. I'll try the core python one. 

hm, the difference between open(self.tempfile(...), 'w+b') and TemporaryFile is that TemporaryFile uses os.fdopen

Here's two builds of scrapy. The only change is Python 2.7 fails with Pillow 3.0.0 but passes with Pillow 2.9.0. (Python 3.3 passes for both.)

Pillow 3.0.0: https://travis-ci.org/hugovk/scrapy/builds/83371654
Pillow 2.9.0: https://travis-ci.org/hugovk/scrapy/builds/83373252

```
from helper import PillowTestCase
from PIL import Image
import tempfile

class test_scrapy(PillowTestCase):
    def _create_image(self, format, *a, **kw):
        buf = open(self.tempfile('temp.jpg'), 'w+b')
        Image.new(*a, **kw).save(buf, format)
        buf.seek(0)
        return Image.open(buf)

    def _create_image_tempfile(self, format, *a, **kw):
        buf = tempfile.TemporaryFile()
        Image.new(*a, **kw).save(buf, format)
        buf.seek(0)
        return Image.open(buf)    

    def test_create_image(self):
        SIZE = (100, 100)
        # straigh forward case: RGB and JPEG
        COLOUR = (0, 127, 255)
        im = self._create_image('JPEG', 'RGB', SIZE, COLOUR)
        self.assert_image_equal(Image.new('RGB', (100,100), COLOUR), im)

        im = self._create_image_tempfile('JPEG', 'RGB', SIZE, COLOUR)
        self.assert_image_equal(Image.new('RGB', (100,100), COLOUR), im)

```

I'm failing at the call to _create_image_tempfile.   It's not scrapy, It appears to be file handling somewhere. I'm wondering if we're not getting a flush happening somewhere. 

> BytesIO are fine. They're all over our test cases. 

Yeah, right, but they were not fine in PIL or Pillow < 2.0, and this is what Ubuntu Precise users get by apt-get. Scrapy tries to support system packages from Ubuntu Precise, so we had to avoid using BytesIO. Test failure: https://travis-ci.org/scrapy/scrapy/jobs/29975641 and a comment: https://github.com/scrapy/scrapy/pull/803#issuecomment-49028433

Right. Old distro versions. 

I'm going to bisect this one while running on battery. 

and watching the rugby. 

Thanks!

Also [bisecting](https://robots.thoughtbot.com/git-bisect):

```
$ git bisect start
$ git bisect good 80672b6
$ git bisect bad 0177cce
$ cat doit.sh
python setup.py develop build_ext --disable-zlib && python Tests/1460.py
$ git bisect run ./doit.sh
```

I get:

```
457d39832d4f433a59c78598849eef3d073cf7e5 is the first bad commit
commit 457d39832d4f433a59c78598849eef3d073cf7e5
Author: radarhere
Date:   Wed Aug 5 20:54:33 2015 +1000

    Added support for pathlib Path objects to open and save

:040000 040000 77908caa2a5443c957478713e8b97dce3834cb87 ed58750db298c338e3475385c1928f2de1a7bc66 M  PIL
```

From PR https://github.com/python-pillow/Pillow/pull/1372: "Added support for pathlib Path objects to open and save"

Yep, that's the one I got. 

Also, Georgia fell apart in the last 10 minutes against NZ. Not that they had a snowball's chance.

One thing that's happening is this `if ... elif` used to be an `if ... else`, and neither of the `if` or `elif` match anything so `filename == ""`.

https://github.com/radarhere/Pillow/commit/457d39832d4f433a59c78598849eef3d073cf7e5#diff-3fef5f960d5bd8bdfb9c6fadcf96af22R2283

Further, adding this:

```
    elif hasattr(fp, "name") and isPath(fp.name):
        filename = fp.name
```

passes the test. 

This snippet then matches the `if ... elif ... elif` by line 1659.

Looking at the first part of the diff:

```
index 861599b..87d77ef 100644
--- a/PIL/Image.py
+++ b/PIL/Image.py
@@ -1653,13 +1653,15 @@ class Image(object):
            may have been created, and may contain partial data.
         """

+        filename = ""
         if isPath(fp):
             filename = fp
-        else:
-            if hasattr(fp, "name") and isPath(fp.name):
-                filename = fp.name
-            else:
-                filename = ""
+        elif sys.version_info >= (3, 4):
+            from pathlib import Path
+            if isinstance(fp, Path):
+                filename = str(fp.resolve())
+        elif hasattr(fp, "name") and isPath(fp.name):
+            filename = fp.name
```

Filename is the original fp, '', or fp.name. The new version adds the possibility of fp.resolve(). 

The next part:

```

-        if isPath(fp):
-            fp = builtins.open(fp, "wb")
+        if filename:
+            fp = builtins.open(filename, "wb")
             close = 1
```

Looks suspect to me, because now we're opening the filename, and not the file pointer. I think that the file pointer should always point to something, but it might not be named. 

e.g.:

```
(vpy27)erics-air:Pillow erics$ python 
Python 2.7.10 (default, Jul 13 2015, 12:18:59) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from tempfile import TemporaryFile
>>> f = TemporaryFile()
>>> f
<open file '<fdopen>', mode 'w+b' at 0x109feb9c0>
>>> hasattr(f, 'name')
True
>>> f.name
'<fdopen>'
>>> 
```

So, in this case, we have a valid fp, but the name is not openable. And for the TemporaryFile, we'd never be able to reopen it from anything other than the file pointer, because it's been deleted. (and isPath is essentially a check for a string type, so this would pass muster)

Basically, I'm thinking that when the fp has a name attribute, the filename should only be passed into the save for metadata purposes and that name should never be used to reopen the file pointer. 

Here's an example of the issue in a flask application: https://gist.github.com/pawl/ee7c76bb066b7d1cbe5d

2.9.0 is able to open the file, but 3.0.0 gives this error: `IOError: cannot identify image file <FileStorage: u'copyleft.tiff' ('image/tiff')>`

Edit: Looks like this is only happening with TIFF files.

It might be better to start a separate issue and close this one?

The title needs to be something like "Pillow 3.0 issues with opening temporary files". This issue starts too vague and it's starting to get too long.

I've edited the title, context is good so I'd prefer to have it here.


Resolving patch:
diff --git a/PIL/Image.py b/PIL/Image.py
--- a/PIL/Image.py
+++ b/PIL/Image.py
@@ -1620,13 +1620,17 @@ def save(self, fp, format=None, **params):
         """
 
         filename = ""
+        open_fp = False
         if isPath(fp):
             filename = fp
+            open_fp = True
         elif sys.version_info >= (3, 4):
             from pathlib import Path
             if isinstance(fp, Path):
                 filename = str(fp.resolve())
+                open_fp = True
         elif hasattr(fp, "name") and isPath(fp.name):
+            # only set the name for metadata purposes
             filename = fp.name
 
         # may mutate self!
@@ -1655,17 +1659,14 @@ def save(self, fp, format=None, **params):
         else:
             save_handler = SAVE[format.upper()]
 
-        if filename:
+        if open_fp:
             fp = builtins.open(filename, "wb")
-            close = 1
-        else:
-            close = 0
 
         try:
             save_handler(self, fp, filename)
         finally:
             # do what we can to clean up
-            if close:
+            if open_fp:
                 fp.close()
 
     def seek(self, frame):


Test patch:
diff --git a/Tests/test_image.py b/Tests/test_image.py
--- a/Tests/test_image.py
+++ b/Tests/test_image.py
@@ -57,6 +57,17 @@ def test_pathlib(self):
         self.assertEqual(im.mode, "RGB")
         self.assertEqual(im.size, (128, 128))
 
+    def test_tempfile(self):
+        # see #1460, pathlib support breaks tempfile.TemporaryFile on py27
+        # Will error out on save on 3.0.0
+        import tempfile
+        im = hopper()
+        fp = tempfile.TemporaryFile()
+        im.save(fp, 'JPEG')
+        fp.seek(0)
+        reloaded = Image.open(fp)
+        self.assert_image_similar(im, reloaded, 20)
+
     def test_internals(self):
 
         im = Image.new("L", (100, 100))