Repository: python-pillow/Pillow

Issue:
Open file handles after processing a multipage tiff
When processing _some_ TIFF files with multiple pages Pillow keeps some open files handles around so that the file can neither be moved nor deleted anymore on Windows. The following script shows the error (tested here with 64-bit Python 3.4.3 on Windows). The file handle leak does _not_ happen with the `multipage.tiff` test file from Pillow's repository. However, the example file [g4-multi.tiff.gz](https://github.com/python-pillow/Pillow/files/500686/g4-multi.tiff.gz) can reproduce this problem. The main difference between the two files is that my file uses the G4 compression codec.

``` python
import os
import shutil
import tempfile
from PIL import Image

TESTFILE = r'g4-multi.tiff'

def create_tempfile():
    with open(TESTFILE, 'rb') as f, tempfile.NamedTemporaryFile('wb', delete=False) as t:
        shutil.copyfileobj(f, t)
        return t.name

def process(filename):
    with open(filename, 'rb') as f:
        im = Image.open(f)
        count = im.n_frames
        im.close()
    print('File is closed: %s' % f.closed) # True
    return count

def main():
    tmp = create_tempfile()
    count = process(tmp)
    print('Number of pages: %d' % count) # 3
    os.remove(tmp) # Windows PermissionError here!

main()
```



Discussion:
I am unable to reproduce on OS X 10.11 with Pillow 3.1.0 and multiple Python versions.

You may be interested in #1144

You won't be able to reproduce this on OS X (or any Unix system) because there it's always allowed to delete files that are still open somewhere. So `os.remove(tmp)` will never fail on OS X even if you also have leaked file handlers, but on Windows it's not possible to delete open files.

Ah, didn't realise that it was only Windows. Thanks for clarifying.

Also reproducible on Windows with a simpler `process()`:

``` python
def process(filename):
    im = Image.open(filename)
    count = im.n_frames
    im.close()
    return count
```

Problem also occurs on OS X.
I did an explicit `del im` after im.close().

The following issue may have the same root cause: http://stackoverflow.com/questions/38541735/pillownumpyunittest-gives-a-resourcewarning-does-this-indicate-a-leak/38542329 .  There, when I try to close a TIFF `Image` object after use, the `.fp` member already has been set to `None` and thus the `Image.close()` method fails to close the file descriptor.  Maybe the line `self.fp = None` in `TiffImageFile._load_libtiff()` is to blame?  Not sure ...

I think I found the cause of the issue. The file descriptor of the image is duplicated here and stored in `fp`: https://github.com/python-pillow/Pillow/blob/d3d29ab2e31b3c647f5355482c2e7e4c552dc055/PIL/TiffImagePlugin.py#L1200

At least one of the multiple file descriptors is not closed correctly, especially when `im.load()` (and `_load_libtiff`) are never even executed as in the example script above.

So here is the actual problem in `TiffImageFile`:
- The file descriptor [is duplicated](https://github.com/python-pillow/Pillow/blob/d3d29ab2e31b3c647f5355482c2e7e4c552dc055/PIL/TiffImagePlugin.py#L1200) each time `TiffImageFile._setup()` is called.
- `TiffImageFile._setup()` is called [every time](https://github.com/python-pillow/Pillow/blob/d3d29ab2e31b3c647f5355482c2e7e4c552dc055/PIL/TiffImagePlugin.py#L981) you call `TiffImageFile._seek()`. That happens 4 times in the 3-page long multipage tiff above: Once when the file is opened and three times from n_frames. So we end up with four additional file descriptors that prevent the file from being deleted on Windows.
- libtiff closes the file descriptor passed to it in `TiffImageFile._load_libtiff()` when [decoder.decode()](https://github.com/python-pillow/Pillow/blob/d3d29ab2e31b3c647f5355482c2e7e4c552dc055/PIL/TiffImagePlugin.py#L1066) is called. That's why os.dup() is actually used. But even when `im.load()` is called and libtiff closes the `fd` passed to it, we still remain with 3 out of 4 created file descriptors still open.

Now I'm not sure if I understand the code correctly, but wouldn't it make more sense to only duplicate the file descriptor when it's actually passed to libtiff (in `_load_libtiff()`) instead of every time `_setup()` is called? I've created the following patch which does just that and it fixes the bug for me, but someone with more insight to Pillow should probably take a look at it and see if it causes any problems elsewhere: https://gist.github.com/sekrause/584aa2845c9a55c1f668a476999b77a4


Resolving patch:
diff --git a/PIL/TiffImagePlugin.py b/PIL/TiffImagePlugin.py
--- a/PIL/TiffImagePlugin.py
+++ b/PIL/TiffImagePlugin.py
@@ -1036,8 +1036,28 @@ def _load_libtiff(self):
         # (self._compression, (extents tuple),
         #   0, (rawmode, self._compression, fp))
         extents = self.tile[0][1]
-        args = self.tile[0][3] + (self.tag_v2.offset,)
-        decoder = Image._getdecoder(self.mode, 'libtiff', args,
+        args = list(self.tile[0][3]) + [self.tag_v2.offset]
+
+        # To be nice on memory footprint, if there's a
+        # file descriptor, use that instead of reading
+        # into a string in python.
+        # libtiff closes the file descriptor, so pass in a dup.
+        try:
+            fp = hasattr(self.fp, "fileno") and os.dup(self.fp.fileno())
+            # flush the file descriptor, prevents error on pypy 2.4+
+            # should also eliminate the need for fp.tell for py3
+            # in _seek
+            if hasattr(self.fp, "flush"):
+                self.fp.flush()
+        except IOError:
+            # io.BytesIO have a fileno, but returns an IOError if
+            # it doesn't use a file descriptor.
+            fp = False
+        
+        if fp:
+            args[2] = fp
+
+        decoder = Image._getdecoder(self.mode, 'libtiff', tuple(args),
                                     self.decoderconfig)
         try:
             decoder.setimage(self.im, extents)
@@ -1190,24 +1210,6 @@ def _setup(self):
 
                 self.use_load_libtiff = True
 
-                # To be nice on memory footprint, if there's a
-                # file descriptor, use that instead of reading
-                # into a string in python.
-
-                # libtiff closes the file descriptor, so pass in a dup.
-                try:
-                    fp = hasattr(self.fp, "fileno") and \
-                        os.dup(self.fp.fileno())
-                    # flush the file descriptor, prevents error on pypy 2.4+
-                    # should also eliminate the need for fp.tell for py3
-                    # in _seek
-                    if hasattr(self.fp, "flush"):
-                        self.fp.flush()
-                except IOError:
-                    # io.BytesIO have a fileno, but returns an IOError if
-                    # it doesn't use a file descriptor.
-                    fp = False
-
                 # libtiff handles the fillmode for us, so 1;IR should
                 # actually be 1;I. Including the R double reverses the
                 # bits, so stripes of the image are reversed.  See
@@ -1233,7 +1235,7 @@ def _setup(self):
 
                 # Offset in the tile tuple is 0, we go from 0,0 to
                 # w,h, and we only do this once -- eds
-                a = (rawmode, self._compression, fp)
+                a = (rawmode, self._compression, False)
                 self.tile.append(
                     (self._compression,
                      (0, 0, w, ysize),


Test patch:
diff --git a/Tests/images/g4-multi.tiff b/Tests/images/g4-multi.tiff
new file mode 100644
Binary files /dev/null and b/Tests/images/g4-multi.tiff differ
diff --git a/Tests/test_file_libtiff.py b/Tests/test_file_libtiff.py
--- a/Tests/test_file_libtiff.py
+++ b/Tests/test_file_libtiff.py
@@ -505,6 +505,24 @@ def test_page_number_x_0(self):
         # Should not divide by zero
         im.save(outfile)
 
+    def test_fd_duplication(self):
+        # https://github.com/python-pillow/Pillow/issues/1651
+
+        tmpfile = self.tempfile("temp.tif")
+        with open(tmpfile, 'wb') as f:
+            with open("Tests/images/g4-multi.tiff", 'rb') as src:
+                f.write(src.read())
+
+        im = Image.open(tmpfile)
+        count = im.n_frames
+        im.close()
+        try:
+            os.remove(tmpfile) # Windows PermissionError here!
+        except:
+            self.fail("Should not get permission error here")
+
+    
+
 
 if __name__ == '__main__':
     unittest.main()