Repository: python-pillow/Pillow

Problem statement:
JPEG 2000 memory leak
When JPEG 2000 images are decoded, they are read using the Incremental decoding adapter.
On Linux, this uses pthreads to do the decoding in a separate thread.

However, after the image is loaded, the associated resources are never actually released.
The functionality for this is present in `ImagingIncrementalCodecDestroy()`, which is called by the cleanup function. But it looks like this is only called if there was an issue reading the image.

Because the call to `pthread_join` is never called, the memory allocated for the stack for the created thread is not freed, and will accumulate for each image loaded.

An example, attempting reading a test image 1000 times, on a 32-bit OS:

``` python
#!/usr/bin/env python

from PIL import Image
from commands import getoutput
from os import getpid

test_file = "Pillow/Tests/images/rgb_trns_ycbc.jp2"

for count in range(1000):
    with Image.open(test_file) as im:
        im.load()
        im.close()
        vsz = getoutput("ps -o vsz -p {}".format(getpid())).split("\n")[1]
        print("Count {}: {} Kb".format(count, vsz))
```

will return:

``` python
[root@dorapp02 test]# python jp2k_test.py
Count 0:  25008 Kb
Count 1:  35256 Kb
Count 2:  45500 Kb
Count 3:  55744 Kb
Count 4:  65988 Kb
Count 5:  76232 Kb
Count 6:  86476 Kb
...
Count 293: 3035088 Kb
Count 294: 3045332 Kb
Count 295: 3055576 Kb
Count 296: 3065820 Kb
Count 297: 3076064 Kb
Count 298: 3086308 Kb
Traceback (most recent call last):
  File "jp2k_test.py", line 10, in <module>
    im.load()
  File "build/bdist.linux-i686/egg/PIL/Jpeg2KImagePlugin.py", line 207, in load
  File "build/bdist.linux-i686/egg/PIL/ImageFile.py", line 234, in load
  File "build/bdist.linux-i686/egg/PIL/ImageFile.py", line 56, in raise_ioerror
IOError: broken data stream when reading image file

```

This will also happen on a 64-bit Linux OS, but the virtual memory usage will be allowed to get MUCH higher before failing.

Something like this will easily fix this:

``` diff
diff --git a/decode.c b/decode.c
index d5e3293..f728ceb 100644
--- a/decode.c
+++ b/decode.c
@@ -103,6 +103,8 @@ PyImaging_DecoderNew(int contextsize)
 static void
 _dealloc(ImagingDecoderObject* decoder)
 {
+    if (decoder->cleanup)
+        decoder->cleanup(&decoder->state);
     free(decoder->state.buffer);
     free(decoder->state.context);
     Py_XDECREF(decoder->lock);
```

The cleanup function is called from `_dealloc` in `encoder.c`, so I am unsure of why it was not done here in the decoder, it does not appear to break any of the tests.

Thanks all :)



Hints:
Would you like to make a pull request?

It'd be useful to have `jp2k_test.py` in Tests/ as well (and update the `test_file` path), but call it something like `check_jp2k.py` as if has  `test` in the name it'll be picked up and run on the CI.

(Unless you can think of a useful unit test to show this broken before and working after.)

Ouch, I'll take a look at that. 


Patch:
diff --git a/PIL/ImageFile.py b/PIL/ImageFile.py
--- a/PIL/ImageFile.py
+++ b/PIL/ImageFile.py
@@ -227,6 +227,8 @@ def load(self):
                         break
                     b = b[n:]
                     t = t + n
+                # Need to cleanup here to prevent leaks in PyPy
+                d.cleanup()
 
         self.tile = []
         self.readonly = readonly
@@ -471,6 +473,7 @@ def _save(im, fp, tile, bufsize=0):
                     break
             if s < 0:
                 raise IOError("encoder error %d when writing image file" % s)
+            e.cleanup()
     else:
         # slight speedup: compress to real file object
         for e, b, o, a in tile:
@@ -481,6 +484,7 @@ def _save(im, fp, tile, bufsize=0):
             s = e.encode_to_file(fh, bufsize)
             if s < 0:
                 raise IOError("encoder error %d when writing image file" % s)
+            e.cleanup()
     try:
         fp.flush()
     except: pass


Test patch:
diff --git a/Tests/check_j2k_leaks.py b/Tests/check_j2k_leaks.py
new file mode 100755
--- /dev/null
+++ b/Tests/check_j2k_leaks.py
@@ -0,0 +1,42 @@
+from helper import unittest, PillowTestCase
+import sys
+from PIL import Image
+from io import BytesIO
+
+# Limits for testing the leak
+mem_limit = 1024*1048576
+stack_size = 8*1048576
+iterations = int((mem_limit/stack_size)*2)
+codecs = dir(Image.core)
+test_file = "Tests/images/rgb_trns_ycbc.jp2"
+
+
+@unittest.skipIf(sys.platform.startswith('win32'), "requires Unix or MacOS")
+class TestJpegLeaks(PillowTestCase):
+    def setUp(self):
+        if "jpeg2k_encoder" not in codecs or "jpeg2k_decoder" not in codecs:
+            self.skipTest('JPEG 2000 support not available')
+
+    def test_leak_load(self):
+        from resource import setrlimit, RLIMIT_AS, RLIMIT_STACK
+        setrlimit(RLIMIT_STACK, (stack_size, stack_size))
+        setrlimit(RLIMIT_AS, (mem_limit, mem_limit))
+        for count in range(iterations):
+            with Image.open(test_file) as im:
+                im.load()
+
+    def test_leak_save(self):
+        from resource import setrlimit, RLIMIT_AS, RLIMIT_STACK
+        setrlimit(RLIMIT_STACK, (stack_size, stack_size))
+        setrlimit(RLIMIT_AS, (mem_limit, mem_limit))
+        for count in range(iterations):
+            with Image.open(test_file) as im:
+                im.load()
+                test_output = BytesIO()
+                im.save(test_output, "JPEG2000")
+                test_output.seek(0)
+                output = test_output.read()
+
+
+if __name__ == '__main__':
+    unittest.main()