Repository: python-pillow/Pillow

Issue:
Saving GIF uses huge amount of memory
I'm encountering limits processing GIF's.  I've noticed that saving GIF's uses a massive amount of memory.  

The example below opens a 6.5M GIF, rotates it and saves it to a new file, 4.5M GIF.  Somehow the function Image.save uses ~1G to get it done!

Doing the same thing with a jpeg uses less than 1M.
## Demonstration

The entire script is shown in the profiler output.
### Profiler results
#### GIF

``` bash
$ python -m memory_profiler pillow_issue_gif.py
Filename: pillow_issue_gif.py

Line #    Mem usage    Increment   Line Contents
================================================
     3   18.934 MiB    0.000 MiB   @profile
     4                             def f():
     5   20.012 MiB    1.078 MiB       im = Image.open('MountWilson.gif')
     6                                 #im.rotate(45).show()
     7  129.625 MiB  109.613 MiB       im2 = im.rotate(45)
     8 1153.125 MiB 1023.500 MiB       im2.save('Test-new.gif', "GIF")
```
#### JPEG

``` bash
$ python -m memory_profiler pillow_issue_jpeg.py
Filename: pillow_issue_jpeg.py

Line #    Mem usage    Increment   Line Contents
================================================
     3   18.980 MiB    0.000 MiB   @profile
     4                             def f():
     5   20.055 MiB    1.074 MiB       im = Image.open('MountWilson.jpeg')
     6                                 #im.rotate(45).show()
     7  308.359 MiB  288.305 MiB       im2 = im.rotate(45)
     8  308.555 MiB    0.195 MiB       im2.save('Test-new.jpeg', "JPEG")
```

I'm kinda surprised the rotate operation uses so much memory too.
### Input files

``` bash
$ llg MountWilson
-rw-r--r--@  1 minmac  staff   6.5M 28 Aug 09:40 MountWilson.gif
-rw-r--r--@  1 minmac  staff   2.2M 28 Aug 09:43 MountWilson.jpeg
```
### Output files

``` bash
$ llg Test
-rw-r--r--   1 minmac  staff   1.7M 28 Aug 16:21 Test-new.jpeg
-rw-r--r--   1 minmac  staff   4.9M 28 Aug 16:24 Test-new.gif
```
## Environment Information

OSX 10.11.6
python 2.7.12
pillow 3.1.1

I haven't profiled it on linux but i know its happening because I'm reaching preset 1GB memory limit on AWS lambda which is running:
- Linux kernel version – 4.1.27-25.49.amzn1.x86_64
- python 2.7
- pillow 3.3.0



Discussion:
How big are the images in pixels?

FWIW, there are a couple of possibilities for the GIF taking more memory, but it's hard to tell at this granularity, since are several things that are happening at the save command. I'd have to guess that it's quantization, but that's with no data. 

The images are 9566 × 3909.

~1e4 times is quite a lot _more_ though isn't it.

With that pixel size, the raw size of the image is h x w x 4 bytes, which is ... about 150 megs. So your peak memory usage on save is O(5x) uncompressed image size.

Stepping through this, when you open it, you're loading the header. Which is small. 
Once you rotate it, you're loading the data into `im`, rotating it, and saving that copy into `im2`. That explains the 300 megs for the JPEG. The GIF is apparently taking a lot less memory in that step, which may mean that the interpolation in rotate is happening in the `P` mode, which would be... potentially bad for quality. 

Here is where the processes diverge.  The JPEG save is essentially a streamable function that compresses blocks as they come out of the image. It's got overhead, but (for the most part) it's not related to the entire image size.  The GIF makes at least one copy, and potentially more in the quantization step. 

There's probably some inefficiency in there, but I'd have to dig a little deeper to see exactly what modes you're getting and where the copies are coming from. 

So, with a 36Mpx gif, I'm not seeing the same results.  I've got a 140 Meg uncompressed size, both in P mode. It looks like it's taking the fast path, and I'm not even certain why it's apparently not copying the image in the _save unless the memory profiler just isn't picking that up. 

```
Filename: test_large_gif.py

Line #    Mem usage    Increment   Line Contents
================================================
     3   15.598 MiB    0.000 MiB   @profile
     4                             def main():
     5   15.766 MiB    0.168 MiB       im = Image.open('lg.gif')
     6   15.773 MiB    0.008 MiB       print im.mode, im.tile
     7   15.773 MiB    0.000 MiB       print im.size, (im.width * im.height * 4) / (1024 *1024)
     8   85.840 MiB   70.066 MiB       im.load()
     9  120.875 MiB   35.035 MiB       im2 = im.rotate(45)
    10  120.875 MiB    0.000 MiB       print im2.mode
    11  156.105 MiB   35.230 MiB       im2.save('lg_out.gif')
```

Valgrind is not showing anything remarkably different either:

```
--------------------------------------------------------------------------------
Command:            python test_large_gif.py
Massif arguments:   (none)
ms_print arguments: massif.out.20111
--------------------------------------------------------------------------------


    MB
149.3^                                                               ::       
     |                                           ####:::::::::@::::::: @::::::
     |                                           #   : : ::: :@:: :::: @: : ::
     |                                           #   : : ::: :@:: :::: @: : ::
     |                                           #   : : ::: :@:: :::: @: : ::
     |                                 @@@@@@@@@@#   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |                                 @         #   : : ::: :@:: :::: @: : ::
     |    @:::::::::@:::::::::::@::::::@         #   : : ::: :@:: :::: @: : ::
     |    @:: :::: :@::::::: :::@: ::: @         #   : : ::: :@:: :::: @: : ::
     |    @:: :::: :@::::::: :::@: ::: @         #   : : ::: :@:: :::: @: : ::
     |    @:: :::: :@::::::: :::@: ::: @         #   : : ::: :@:: :::: @: : ::
     |   :@:: :::: :@::::::: :::@: ::: @         #   : : ::: :@:: :::: @: : ::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   5.627

Number of snapshots: 56
 Detailed snapshots: [4, 12, 23, 28, 30 (peak), 37, 44]
```

Thanks @wiredfool.  For now I'm happy with using JPEG but I suppose I would like to know how my GIF differs to yours.

I've attached the input GIF here if you're interested.

![mount_wilson](https://cloud.githubusercontent.com/assets/4349566/18111542/d7d62a30-6f63-11e6-8f07-b76578938dff.gif)

Ok, so there's something different about that gif (and consistent with what you're seeing) -- I'm seeing this in valgrind:

```
--------------------------------------------------------------------------------
Command:            python test_large_gif.py
Massif arguments:   (none)
ms_print arguments: massif.out.23666
--------------------------------------------------------------------------------


    GB
1.311^                                                                     #  
     |        @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@#  
     |        @                                                            #  
     |        @                                                            #  
     |        @                                                            #  
     |       :@                                                            #::
     |       :@                                                            #::
     |       :@                                                            #::
     |       :@                                                            #::
     |       :@                                                            #::
     |      ::@                                                            #::
     |      ::@                                                            #::
     |      ::@                                                            #::
     |      ::@                                                            #::
     |      ::@                                                            #::
     |      ::@                                                            #::
     |      ::@                                                            #::
     |      ::@                                                            #::
     |     @::@                                                            #::
     |   @@@::@                                                            #::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   29.88

```

Looking at the memory allocations, it's certainly different. At the top level, there's:

```
->64.28% (904,546,272B) 0x572C27: ??? (in /home/erics/vpy27/bin/python)
->21.26% (299,168,952B) 0x4FA112: PyList_New (in /home/erics/vpy27/bin/python)
->10.63% (149,573,976B) 0x8020C0F: ImagingNewArray (Storage.c:315)
```

 (and that one is two copies and two new allocations): 

For reference, the allocations for my previous one looked more like this, which is 93% image storage including two new allocations and two copies.:

```
->93.92% (147,000,000B) 0x8020C0F: ImagingNewArray (Storage.c:315)
```

As to why it's happening...  I still suspect quantization. I'm wondering if there's a different code path when the palette is full vs when the palette is empty. 

Right. This is suboptimal.   Remapping the image palette in python by looping over every byte of the image, using a range, when the number of pixels is large. 

https://github.com/python-pillow/Pillow/blob/master/PIL/GifImagePlugin.py#L591

```
    if _get_optimize(im, info):
        used_palette_colors = _get_used_palette_colors(im)

        # create the new palette if not every color is used
        if len(used_palette_colors) < 256:
            palette_bytes = b""
            new_positions = {}

            i = 0
            # pick only the used colors from the palette
            for oldPosition in used_palette_colors:
                palette_bytes += source_palette[oldPosition*3:oldPosition*3+3]
                new_positions[oldPosition] = i
                i += 1

            # replace the palette color id of all pixel with the new id
            image_bytes = bytearray(im.tobytes())
            for i in range(len(image_bytes)):
                image_bytes[i] = new_positions[image_bytes[i]]
            im.frombytes(bytes(image_bytes))
            new_palette_bytes = (palette_bytes +
                                 (768 - len(palette_bytes)) * b'\x00')
            im.putpalette(new_palette_bytes)
            im.palette = ImagePalette.ImagePalette("RGB",
                                                   palette=palette_bytes,
                                                   size=len(palette_bytes))
```

It dates to this commit, to fix #211 https://github.com/python-pillow/Pillow/commit/a466b3e09982ebcf5aee2cbe957ce90e6ddfd0ae

So, for a workaround, when saving a largish gif:

```
im.save('foo.gif', optimize=False)
```

Future possible fixes:
1) Don't optimize palettes for anything bigger than N mpix.
2) Don't optimize palettes if the palette doesn't change
3) Don't do it in python by looping over every pixel.
4) There should be a palette mapping function in the c layer of the palette code.

Also, FWIW, that image is probably best off as a PNG. 

Right. Wow! Thanks @wiredfool .

Excuse my ignorance (and perhaps a link is better than an answer) but what makes PNG the best format in this case?

Roughly, if you've got continuous tone, like a photograph, JPEG is your best bet. There's lossy compression that is tripped up with high contrast edges, like text.

If have images which aren't continuous, PNG is generally the best choice. It can do full color or palette images, (though, lossless compression of photos is a lot larger than JPEG's lossy) and it handles most sorts of transparency. Its compression is basically gzip.

GIF predates both of them, and is a 8 bit (or less) palette image.  It's got 1 bit of transparency. I'd say the biggest use of GIF these days is for multiframe images. 


Resolving patch:
diff --git a/PIL/GifImagePlugin.py b/PIL/GifImagePlugin.py
--- a/PIL/GifImagePlugin.py
+++ b/PIL/GifImagePlugin.py
@@ -586,6 +586,10 @@ def _get_header_palette(palette_bytes):
         palette_bytes += o8(0) * 3 * actual_target_size_diff
     return palette_bytes
 
+# Force optimization so that we can test performance against
+# cases where it took lots of memory and time previously. 
+_FORCE_OPTIMIZE = False
+
 def _get_palette_bytes(im, palette, info):
     if im.mode == "P":
         if palette and isinstance(palette, bytes):
@@ -603,32 +607,78 @@ def _get_palette_bytes(im, palette, info):
     if _get_optimize(im, info):
         used_palette_colors = _get_used_palette_colors(im)
 
+        # Potentially expensive operation.
+
+        # The palette saves 3 bytes per color not used, but palette
+        # lengths are restricted to 3*(2**N) bytes. Max saving would
+        # be 768 -> 6 bytes if we went all the way down to 2 colors.
+        # * If we're over 128 colors, we can't save any space.
+        # * If there aren't any holes, it's not worth collapsing. 
+        # * If we have a 'large' image, the palette is in the noise.
+
         # create the new palette if not every color is used
-        if len(used_palette_colors) < 256:
+        if _FORCE_OPTIMIZE or im.mode == 'L' or \
+               (len(used_palette_colors) <= 128 and  
+                max(used_palette_colors) > len(used_palette_colors) and
+                im.width * im.height < 512 * 512):
             palette_bytes = b""
-            new_positions = {}
+            new_positions = [0]*256
 
-            i = 0
             # pick only the used colors from the palette
-            for oldPosition in used_palette_colors:
+            for i, oldPosition in enumerate(used_palette_colors):
                 palette_bytes += source_palette[oldPosition*3:oldPosition*3+3]
                 new_positions[oldPosition] = i
-                i += 1
 
             # replace the palette color id of all pixel with the new id
-            image_bytes = bytearray(im.tobytes())
-            for i in range(len(image_bytes)):
-                image_bytes[i] = new_positions[image_bytes[i]]
-            im.frombytes(bytes(image_bytes))
+
+            # Palette images are [0..255], mapped through a 1 or 3
+            # byte/color map.  We need to remap the whole image
+            # from palette 1 to palette 2. New_positions is
+            # an array of indexes into palette 1.  Palette 2 is
+            # palette 1 with any holes removed.
+
+            # We're going to leverage the convert mechanism to use the
+            # C code to remap the image from palette 1 to palette 2,
+            # by forcing the source image into 'L' mode and adding a
+            # mapping 'L' mode palette, then converting back to 'L'
+            # sans palette thus converting the image bytes, then
+            # assigning the optimized RGB palette.
+
+            # perf reference, 9500x4000 gif, w/~135 colors
+            # 14 sec prepatch, 1 sec postpatch with optimization forced.
+
+            mapping_palette = bytearray(new_positions)
+
+            m_im = im.copy()
+            m_im.mode = 'P'
+
+            m_im.palette = ImagePalette.ImagePalette("RGB",
+                                                   palette=mapping_palette*3,
+                                                   size=768)
+            #possibly set palette dirty, then 
+            #m_im.putpalette(mapping_palette, 'L')  # converts to 'P'
+            # or just force it.
+            # UNDONE -- this is part of the general issue with palettes
+            m_im.im.putpalette(*m_im.palette.getdata())
+            
+            m_im = m_im.convert('L')
+         
+            # Internally, we require 768 bytes for a palette. 
             new_palette_bytes = (palette_bytes +
                                  (768 - len(palette_bytes)) * b'\x00')
-            im.putpalette(new_palette_bytes)
-            im.palette = ImagePalette.ImagePalette("RGB",
+            m_im.putpalette(new_palette_bytes)
+            m_im.palette = ImagePalette.ImagePalette("RGB",
                                                    palette=palette_bytes,
                                                    size=len(palette_bytes))
 
+            # oh gawd, this is modifying the image in place so I can pass by ref.
+            # REFACTOR SOONEST 
+            im.frombytes(m_im.tobytes())
+            
     if not palette_bytes:
         palette_bytes = source_palette
+
+    # returning palette, _not_ padded to 768 bytes like our internal ones.
     return palette_bytes, used_palette_colors
 
 def getheader(im, palette=None, info=None):


Test patch:
diff --git a/Tests/test_file_gif.py b/Tests/test_file_gif.py
--- a/Tests/test_file_gif.py
+++ b/Tests/test_file_gif.py
@@ -3,6 +3,8 @@
 from PIL import Image
 from PIL import GifImagePlugin
 
+from io import BytesIO
+
 codecs = dir(Image.core)
 
 # sample gif stream
@@ -33,8 +35,6 @@ def test_invalid_file(self):
                           lambda: GifImagePlugin.GifImageFile(invalid_file))
 
     def test_optimize(self):
-        from io import BytesIO
-
         def test_grayscale(optimize):
             im = Image.new("L", (1, 1), 0)
             filename = BytesIO()
@@ -52,6 +52,43 @@ def test_bilevel(optimize):
         self.assertEqual(test_bilevel(0), 800)
         self.assertEqual(test_bilevel(1), 800)
 
+    def test_optimize_correctness(self):
+        # 256 color Palette image, posterize to > 128 and < 128 levels
+        # Size bigger and smaller than 512x512
+        # Check the palette for number of colors allocated.
+        # Check for correctness after conversion back to RGB        
+        def check(colors, size, expected_palette_length):
+            # make an image with empty colors in the start of the palette range
+            im = Image.frombytes('P', (colors,colors),
+                                 bytes(bytearray(list(range(256-colors,256))*colors)))
+            im = im.resize((size,size))
+            outfile = BytesIO()
+            im.save(outfile, 'GIF')
+            outfile.seek(0)
+            reloaded = Image.open(outfile)
+
+            # check palette length
+            palette_length = max(i+1 for i,v in enumerate(reloaded.histogram()) if v)
+            self.assertEqual(expected_palette_length, palette_length)
+            
+            self.assert_image_equal(im.convert('RGB'), reloaded.convert('RGB'))
+
+
+        # These do optimize the palette
+        check(128, 511, 128)
+        check(64, 511, 64)
+        check(4, 511, 4)
+
+        # These don't optimize the palette
+        check(128, 513, 256)
+        check(64, 513, 256)
+        check(4, 513, 256)
+
+        # other limits that don't optimize the palette
+        check(129, 511, 256)
+        check(255, 511, 256)
+        check(256, 511, 256)
+
     def test_optimize_full_l(self):
         from io import BytesIO