Repository: python-pillow/Pillow

Issue:
Automatic scaling of Image.MAXBLOCK
There's a venerable old PIL bug which caused `Image.save(optimize=True)` to fail with JPEGs when the file size is over a certain threshold. The earliest reference I could find dates back to 1999:

http://mail.python.org/pipermail/image-sig/1999-December/000942.html

The workaround commonly used is to either hardcode something like this:

```
ImageFile.MAXBLOCK = <empirically determined constant (i.e. wild guess)>
```

or the more dynamic approach used in e.g. jdriscoll/django-imagekit#50:

```
ImageFile.MAXBLOCK = max(img.size) ** 2
```

This seems like exactly the kind of thing Pillow could just fix once and for all.



Discussion:
Do you have a preference? And/or do you want to send a pull request? @cgohlke @wiredfool any thoughts?

I'd really like to fix a 14 year old bug. (Sounds like a fine whiskey actually. older than some decent ones.) 

I'd have to dig to see what MAXBLOCK affects. I'd be especially wary if maxblock ends up affecting buffer sizes that get filled and passed to the lib\* decoders. 

Good (or at least some) test coverage of this would be nice, and go a long way toward mitigating any bad effects.

So, it looks like there's a comment in RawEncode.c

``` c
 /* FIXME: This encoder will fail if the buffer is not large enough to
 * hold one full line of data.  There's a workaround for this problem
 * in ImageFile.py, but it should be solved here instead.
*/
```

and in ImageFile.py

``` python
    # FIXME: make MAXBLOCK a configuration parameter
    bufsize = max(MAXBLOCK, im.size[0] * 4) # see RawEncode.c
```

My Wild Ass Guess is that when libjpeg is optimizing, it passes back the whole image in one big chunk. Otherwise, worst case it can do scanline by scanline, and the bufsize guarantees that it's big enough for one scan line. 

If we bump up the MAXBLOCK, or the bufsize here, we're going to allocate that much memory each time we do an encode. The brute force method would be size[0]_size[1] \* 4, but that's going to be suboptimal for large images. max(size)_*2 would be less than that, but it may not actually fix all the cases. 

I'd be willing to dig more, assuming I can get a reproducable test case.  

Strong +1 on expanding tests. I strongly suspect your guess is correct and this could have ripple effects with a codebase this old and squirrelly.

If you want to reproduce it, here's a transcript:

``` python

Python 2.7.2 (default, Oct 11 2012, 20:14:37) 
[GCC 4.2.1 Compatible Apple Clang 4.0 (tags/Apple/clang-418.0.60)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image, ImageFile
>>> i = Image.new("RGB", (4096, 4096), 0xff3333)
>>> i.save("/dev/null", format="JPEG")
>>> i.save("/dev/null", format="JPEG", optimize=True)
Suspension not allowed here
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cadams/.virtualenvs/project/lib/python2.7/site-packages/PIL/Image.py", line 1437, in save
    save_handler(self, fp, filename)
  File "/Users/cadams/.virtualenvs/project/lib/python2.7/site-packages/PIL/JpegImagePlugin.py", line 471, in _save
    ImageFile._save(im, fp, [("jpeg", (0,0)+im.size, 0, rawmode)])
  File "/Users/cadams/.virtualenvs/project/lib/python2.7/site-packages/PIL/ImageFile.py", line 501, in _save
    raise IOError("encoder error %d when writing image file" % s)
IOError: encoder error -2 when writing image file
>>> ImageFile.MAXBLOCK = max(i.size) ** 2
>>> i.save("/dev/null", format="JPEG", optimize=True)
>>>
```

Ideally, this feels like something which should just work. Perhaps something as simple as having `save()` check the format and optimize flags and increase MAXBLOCK for that operation along with documenting that this can increase RAM consumption? The part which feels dirtiest is changing this by setting a variable on a class – simply moving the change closer to the function call seems like a good idea.

So, having poked around a bit, I can confirm a few things. 
1. It does appear that the jpeg code is writing out optimized images in one push to the buffer. 
2. The "Suspension not allowed here" message basically means that it tried to write, and there wasn't enough space. 
3. That really boring image takes 8k/megapixel to save, one from my camera has a different constant there. In each case, MAXBLOCK had to be > than the final image file size. 
4. MAXBLOCK appears to be allocated for _every_ image encode, so just randomly upping it is a bad idea. 
5. Passing in an open file instead of a filename doesn't change anything, because it's just doing the shuffling to and from the buffer in c instead of python, not pushing the fp all the way down into the library. (which I did with the tiff lib, in sort of an aggressive manner. )
6. It's probably not threadsafe to just change maxblock for the one operation, since it's a module level global variable.
7. I'm still thinking on this, but I think pushing the bufsize into an optional parameter on _save, initialized to MAXBLOCK, and then figuring out some sort of guess for the final jpeg size would be an appropriate fix. 

``` python
from PIL import Image, ImageFile

def search(i):
  ImageFile.MAXBLOCK = 65536
  while True:
    try:
      i.save("/tmp/jpg", format="JPEG", optimize=True)
      print "Save succeded at MAXBLOCK=%d" % ImageFile.MAXBLOCK
      break
    except: 
      print "Save failed at MAXBLOCK=%d" % ImageFile.MAXBLOCK
      ImageFile.MAXBLOCK *= 2

def im(size=(1024,1024)):
  i = Image.new("RGB", size, 0xff3333)
  return i


for x in [128,512,1024,2048,4096,8192, 16384]:
  print "%d wide" %x
  search(im((x,2048)))

for y in [128,512,1024,2048,4096,8192, 16384]:
  print "%d tall" %y
  search(im((2048,y)))
```

It looks like the two plugins need special handling with maxblock are the JpegImagePlugin and the RawEncode.c. Unfortunately, the RawEncodes go straight into the save instead of through an image plugin, so there's no place to hang the hook to up the buffer size if necessary. So, there's still a bit of a special case in there, but at least the JpegImagePlugin can override as necessary. 

The only problem I've had with maxblock so far was an image withh an exif header >64k.
Will check the fix on monday,

This probably won't fix that case, as it's pretty specifically targeted at the optimized case. We might be able check the size of the chunks that we have and make sure MAXBLOCK is big enough to handle that. 

@d-schmidt did you ever check the jpeg image plugin with a >64k exif size?

What do you mean by "check"? We are using it, I'm increasing maxblock manually in the image converter. The converter is running 24/7.


Resolving patch:
diff --git a/PIL/JpegImagePlugin.py b/PIL/JpegImagePlugin.py
--- a/PIL/JpegImagePlugin.py
+++ b/PIL/JpegImagePlugin.py
@@ -554,6 +554,7 @@ def validate_qtables(qtables):
         info.get("exif", b"")
         )
 
+
     # if we optimize, libjpeg needs a buffer big enough to hold the whole image in a shot.
     # Guessing on the size, at im.size bytes. (raw pizel size is channels*size, this
     # is a value that's been used in a django patch.
@@ -561,6 +562,10 @@ def validate_qtables(qtables):
     bufsize=0
     if "optimize" in info:
         bufsize = im.size[0]*im.size[1]
+        
+    # The exif info needs to be written as one block, + APP1, + one spare byte.
+	# Ensure that our buffer is big enough
+    bufsize = max(ImageFile.MAXBLOCK, bufsize, len(info.get("exif",b"")) + 5 )
 
     ImageFile._save(im, fp, [("jpeg", (0,0)+im.size, 0, rawmode)], bufsize)
 


Test patch:
diff --git a/Tests/test_file_jpeg.py b/Tests/test_file_jpeg.py
--- a/Tests/test_file_jpeg.py
+++ b/Tests/test_file_jpeg.py
@@ -120,6 +120,12 @@ def test_optimize_large_buffer():
 	im = Image.new("RGB", (4096,4096), 0xff3333)
 	im.save(f, format="JPEG", optimize=True)
 
+def test_large_exif():
+	#https://github.com/python-imaging/Pillow/issues/148
+	f = tempfile('temp.jpg')
+	im = lena()
+	im.save(f,'JPEG', quality=90, exif=b"1"*65532)
+
 def test_progressive():
     im1 = roundtrip(lena())
     im2 = roundtrip(lena(), progressive=1)