Repository: python-pillow/Pillow

Issue:
Add support for saving multipage images, esp. TIFF and PDF
While reading multiple frames/pages is supported, I couldn't find a way to write them, despite some mentions in the code and in the changelog of version 1.1.7c1.



Discussion:
If anyone needs it, I have solved the need for myself. PDFs by calling tiff2pdf, but more interestingly, TIFFs by creating a "file object wrapper class" that is able to append a TIF to an existing one (it simulates a new empty file for Pillow to write into, while actually appending to an existing TIF and fixing the IFDs and IFD pointer). Kind of a hackish solution that I'm not very proud of but if anyone needs it, feel free to ask.

Thinking about this, would it be good for Pillow to open a multipage/frame image and then save it again with all those frames by default - to be able to roundtrip the image without losing all but the first frame?

Or would that be considered too much of a change from the existing behaviour?

I would have liked that if that was the declared behavior from the start, but as you say, I think this would now be too much of a change. I would prefer adding a save_all method or an optional parameter to save or something like that.

Fair enough, and others are highly likely to be thinking the same way. Does anyone else have any thoughts as to what form this new api should take? My current thoughts are of copying `register_save` into `register_save_all`, and that users would pass in `save_all` as part of the params variable in Image's existing `save` method.

Considering that Pillow doesn't presently read or write PDFs of any kind, I think that multipage PDFs are perhaps a bit beyond the current state. GIF multipage saving has been supported more or less by Pillow for a long while, and it has just been a question of merging that functionality into the main system.

You mention TIFFs. Are there any other formats that would be considered useful?

PDF writing is supported (in a basic way), is it not?
The only three formats that I can think of are GIF, TIFF and PDF, but then again I am not an expert on the multitude of other possible formats.

Ah, yes, you're right, PDF writing is supported. Thanks for pointing that out.

What about TIFF multipage saving? Is it supported by now? Or what would have to be done in order to support it in a similar API like PDF multipage saving?

@jkfindeisen Multipage saving was added #1445 which was released in the latest Pillow 3.0.0.

I'll close this issue, if there's something still pending, we can re-open it.

@hugovk Does #1445 really implement multipage writing for TIFFs?

In the commit in #1445 is no reference to TIFF. Also in the current master https://github.com/python-pillow/Pillow/blob/master/PIL/TiffImagePlugin.py there is no save_all method which I would expect if multipage writing for TIFFs would be implemented.

@jkfindeisen Apologies, that's only for PDF. Sorry for the confusion!

Hi. To add support for TIFF multipage saving, it's a matter of reading the TIFF specification and making the necessary changes to TiffImagePlugin. I have tried previously, but without success.

If you would like to have a shot at it, you're welcome to do so.

Hi, I will try it. I might come back when I have something.

@hugovk, what about TIFF multipage saving? Still not done?

@Koudy Still not done. PRs welcome!

@hugovk, unfortunately I not a python developer.
Just need that function.

Any advance on that? With some guidance I would be able to take a look on this...

@vashek Do you still have your "hackish code" for multi-page TIFFs?  The approach doesn't sound too hackish at all, but somewhat appropriate, and I think we should try get it cleaned up and merged.

You wrote: "TIFFs by creating a "file object wrapper class" that is able to append a TIF to an existing one (it simulates a new empty file for Pillow to write into, while actually appending to an existing TIF and fixing the IFDs and IFD pointer)."

Yes, and it's been working for me without problems so far. Here you are.
Sorry about the lack of comments.
[multiPageTiff.zip](https://github.com/python-pillow/Pillow/files/491263/multiPageTiff.zip)

@vashek, could you be able to provide a example of how to use it? Do you mind to upload somewhere on GitHub or Gist so I could be able to add the comments, or is it fine if I upload them here?

I have clean it and uploaded to https://gist.github.com/piranna/4c0787a12cf5885eb36a5b561e3347fc if you are interested in merge it upstream or anybody wants to use it. It seems it can be used both opening a TIFF file and write the content of another "by hand" and by opening a TIFF file and using the returned file descriptor as the one where Pillow will automatically write the new TIFF pages, isn't it?

@piranna Thanks for the cleanup (I reviewed it and can confirm it's equivalent to vashek's code). I don't know where logging.trace is coming from, and I had to add an UTF-8 marker.  Finally I added support for filehandles instead of filenames (so you can use BytesIO, for example) and a simple test routine.

Running this on Pillow/Tests/images/multipage.tiff shows corruption in the third image, though, so it's debug time :cactus: 

https://gist.github.com/lambdafu/fc3ef0863d6b986bc74030f92bb6154f

Now I am eating my words, the original code works fine, so they are not 100% equivalent.

Hum, maybe I added a typo during clean-up stage? Good you added tests! :-D

`tag in self.Tags` on Enum is always False, so I changed it to `tag in set(self.Tags)` and IntEnum.  Now it's working: https://gist.github.com/lambdafu/fc3ef0863d6b986bc74030f92bb6154f

I guess a save_all function is now easy to implement.  But there is no interface in PIL to create multi-frame Image objects...

> I added support for filehandles instead of filenames (so you can use BytesIO, for example)

That's a great adition :-D

> tag in self.Tags on Enum is always False, so I changed it to tag in set(self.Tags) and IntEnum.

That's curious, it was working for me in other places, I'll need to review them... Good idea to use IntEnum instead.

> I guess a save_all function is now easy to implement. But there is no interface in PIL to create multi-frame Image objects...

Maybe is it done using extra dimensions on the image?

`Image.new("RGB", (3, 100, 100))` doesn't work.  But more importantly, this would mean all frames must have the same dimension, something that is not true for multipage-TIFF files.

Maybe PIL should just export the AppendingTiffWriter object for now.

> But more importantly, this would mean all frames must have the same dimension, something that is not true for multipage-TIFF files

I'm not sure if this is really needed, dimensions are just defined as a tuple... Are you sure that all the fields need to maintain their range? If so, maybe they are filled with null data?

P.D.: I've fixed two minor PEP8 things.

Well, fact is that PIL right now can load existing multiframe TIFF files but not create or save them.  With the code from the gist, we can implement save_all to save them.  But it will still not be able to create them from scratch.  The AppendingTiffWriter object can be used to create multiframe TIFFs from scratch, but that's still not a generic API for multiframe Image objects that can be used for GIF and PDF as well.

What I was pointing out is that a multiframe Image should be able to have frames with different xy-sizes, so a single (nrframes, x, y) argument to Image.new to create arbitrary multiframe images would not be sufficient. For example, the multipage.tiff from the test suite has image sizes (10,10), (10,10), (20,20), so  it could not be created with such a simple interface.

However, there are plenty of ways to create an appropriately powerful interface, and I don't think we should wait for that to be written before working on save_all and exporting the class to at least allow users to create multiframe TIFF files (even if it is TIFF specific).  With only save_all and not exporting AppendingTiffWriter, everybody who needs more than save_all will have to duplicate the code in the gist.


Resolving patch:
diff --git a/PIL/Image.py b/PIL/Image.py
--- a/PIL/Image.py
+++ b/PIL/Image.py
@@ -1686,7 +1686,9 @@ def save(self, fp, format=None, **params):
             save_handler = SAVE[format.upper()]
 
         if open_fp:
-            fp = builtins.open(filename, "wb")
+            # Open also for reading ("+"), because TIFF save_all
+            # writer needs to go back and edit the written data.
+            fp = builtins.open(filename, "w+b")
 
         try:
             save_handler(self, fp, filename)
diff --git a/PIL/TiffImagePlugin.py b/PIL/TiffImagePlugin.py
--- a/PIL/TiffImagePlugin.py
+++ b/PIL/TiffImagePlugin.py
@@ -1404,7 +1404,6 @@ def _save(im, fp, filename):
     if im.mode == "P":
         lut = im.im.getpalette("RGB", "RGB;L")
         ifd[COLORMAP] = tuple(i8(v) * 256 for v in lut)
-
     # data orientation
     stride = len(bits) * ((im.size[0]*bits[0]+7)//8)
     ifd[ROWSPERSTRIP] = im.size[1]
@@ -1491,12 +1490,273 @@ def _save(im, fp, filename):
         # just to access o32 and o16 (using correct byte order)
         im._debug_multipage = ifd
 
+class AppendingTiffWriter:
+    fieldSizes = [
+        0,  # None
+        1,  # byte
+        1,  # ascii
+        2,  # short
+        4,  # long
+        8,  # rational
+        1,  # sbyte
+        1,  # undefined
+        2,  # sshort
+        4,  # slong
+        8,  # srational
+        4,  # float
+        8,  # double
+    ]
+
+    #    StripOffsets = 273
+    #    FreeOffsets = 288
+    #    TileOffsets = 324
+    #    JPEGQTables = 519
+    #    JPEGDCTables = 520
+    #    JPEGACTables = 521
+    Tags = set((273, 288, 324, 519, 520, 521))
+
+    def __init__(self, fn, new=False):
+        if hasattr(fn, 'read'):
+            self.f = fn
+            self.close_fp = False
+        else:
+            self.name = fn
+            self.close_fp = True
+            try:
+                self.f = io.open(fn, "w+b" if new else "r+b")
+            except IOError:
+                self.f = io.open(fn, "w+b")
+        self.beginning = self.f.tell()
+        self.setup()
+
+    def setup(self):
+        # Reset everything.
+        self.f.seek(self.beginning, os.SEEK_SET)
+
+        self.whereToWriteNewIFDOffset = None
+        self.offsetOfNewPage = 0
+
+        self.IIMM = IIMM = self.f.read(4)
+        if not IIMM:
+            # empty file - first page
+            self.isFirst = True
+            return
+
+        self.isFirst = False
+        if IIMM == b"II\x2a\x00":
+            self.setEndian("<")
+        elif IIMM == b"MM\x00\x2a":
+            self.setEndian(">")
+        else:
+            raise RuntimeError("Invalid TIFF file header")
+
+        self.skipIFDs()
+        self.goToEnd()
+
+    def finalize(self):
+        if self.isFirst:
+            return
+
+        # fix offsets
+        self.f.seek(self.offsetOfNewPage)
+
+        IIMM = self.f.read(4)
+        if not IIMM:
+            # raise RuntimeError("nothing written into new page")
+            # Make it easy to finish a frame without committing to a new one.
+            return
+
+        if IIMM != self.IIMM:
+            raise RuntimeError("IIMM of new page doesn't match IIMM of "
+                               "first page")
+
+        IFDoffset = self.readLong()
+        IFDoffset += self.offsetOfNewPage
+        self.f.seek(self.whereToWriteNewIFDOffset)
+        self.writeLong(IFDoffset)
+        self.f.seek(IFDoffset)
+        self.fixIFD()
+
+    def newFrame(self):
+        # Call this to finish a frame.
+        self.finalize()
+        self.setup()
+
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        if self.close_fp:
+            self.close()
+        return False
+
+    def tell(self):
+        return self.f.tell() - self.offsetOfNewPage
+
+    def seek(self, offset, whence):
+        if whence == os.SEEK_SET:
+            offset += self.offsetOfNewPage
+
+        self.f.seek(offset, whence)
+        return self.tell()
+
+    def goToEnd(self):
+        self.f.seek(0, os.SEEK_END)
+        pos = self.f.tell()
+
+        # pad to 16 byte boundary
+        padBytes = 16 - pos % 16
+        if 0 < padBytes < 16:
+            self.f.write(bytes(bytearray(padBytes)))
+        self.offsetOfNewPage = self.f.tell()
+
+    def setEndian(self, endian):
+        self.endian = endian
+        self.longFmt = self.endian + "L"
+        self.shortFmt = self.endian + "H"
+        self.tagFormat = self.endian + "HHL"
+
+    def skipIFDs(self):
+        while True:
+            IFDoffset = self.readLong()
+            if IFDoffset == 0:
+                self.whereToWriteNewIFDOffset = self.f.tell() - 4
+                break
+
+            self.f.seek(IFDoffset)
+            numTags = self.readShort()
+            self.f.seek(numTags * 12, os.SEEK_CUR)
+
+    def write(self, data):
+        return self.f.write(data)
+
+    def readShort(self):
+        value, = struct.unpack(self.shortFmt, self.f.read(2))
+        return value
+
+    def readLong(self):
+        value, = struct.unpack(self.longFmt, self.f.read(4))
+        return value
+
+    def rewriteLastShortToLong(self, value):
+        self.f.seek(-2, os.SEEK_CUR)
+        bytesWritten = self.f.write(struct.pack(self.longFmt, value))
+        if bytesWritten is not None and bytesWritten != 4:
+            raise RuntimeError("wrote only %u bytes but wanted 4" %
+                               bytesWritten)
+
+    def rewriteLastShort(self, value):
+        self.f.seek(-2, os.SEEK_CUR)
+        bytesWritten = self.f.write(struct.pack(self.shortFmt, value))
+        if bytesWritten is not None and bytesWritten != 2:
+            raise RuntimeError("wrote only %u bytes but wanted 2" %
+                               bytesWritten)
+
+    def rewriteLastLong(self, value):
+        self.f.seek(-4, os.SEEK_CUR)
+        bytesWritten = self.f.write(struct.pack(self.longFmt, value))
+        if bytesWritten is not None and bytesWritten != 4:
+            raise RuntimeError("wrote only %u bytes but wanted 4" %
+                               bytesWritten)
+
+    def writeShort(self, value):
+        bytesWritten = self.f.write(struct.pack(self.shortFmt, value))
+        if bytesWritten is not None and bytesWritten != 2:
+            raise RuntimeError("wrote only %u bytes but wanted 2" %
+                               bytesWritten)
+
+    def writeLong(self, value):
+        bytesWritten = self.f.write(struct.pack(self.longFmt, value))
+        if bytesWritten is not None and bytesWritten != 4:
+            raise RuntimeError("wrote only %u bytes but wanted 4" %
+                               bytesWritten)
+
+    def close(self):
+        self.finalize()
+        self.f.close()
+
+    def fixIFD(self):
+        numTags = self.readShort()
+        #trace("fixing IFD at %X; number of tags: %u (0x%X)", self.f.tell()-2,
+        #      numTags, numTags)
+
+        for i in range(numTags):
+            tag, fieldType, count = struct.unpack(self.tagFormat, self.f.read(8))
+            #trace("  at %X: tag %u (0x%X), type %u, count %u", self.f.tell()-8,
+            #      tag, tag, fieldType, count)
+
+            fieldSize = self.fieldSizes[fieldType]
+            totalSize = fieldSize * count
+            isLocal = (totalSize <= 4)
+            if not isLocal:
+                offset = self.readLong()
+                offset += self.offsetOfNewPage
+                self.rewriteLastLong(offset)
+
+            if tag in self.Tags:
+                curPos = self.f.tell()
+
+                if isLocal:
+                    self.fixOffsets(count, isShort=(fieldSize == 2),
+                                    isLong=(fieldSize == 4))
+                    self.f.seek(curPos + 4)
+                else:
+                    self.f.seek(offset)
+                    self.fixOffsets(count, isShort=(fieldSize == 2),
+                                    isLong=(fieldSize == 4))
+                    self.f.seek(curPos)
+
+                offset = curPos = None
+
+            elif isLocal:
+                # skip the locally stored value that is not an offset
+                self.f.seek(4, os.SEEK_CUR)
+
+    def fixOffsets(self, count, isShort=False, isLong=False):
+        if not isShort and not isLong:
+            raise RuntimeError("offset is neither short nor long")
+
+        for i in range(count):
+            offset = self.readShort() if isShort else self.readLong()
+            offset += self.offsetOfNewPage
+            if isShort and offset >= 65536:
+                # offset is now too large - we must convert shorts to longs
+                if count != 1:
+                    raise RuntimeError("not implemented")  # XXX TODO
+
+                # simple case - the offset is just one and therefore it is
+                # local (not referenced with another offset)
+                self.rewriteLastShortToLong(offset)
+                self.f.seek(-10, os.SEEK_CUR)
+                self.writeShort(4)  # rewrite the type to LONG
+                self.f.seek(8, os.SEEK_CUR)
+            elif isShort:
+                self.rewriteLastShort(offset)
+            else:
+                self.rewriteLastLong(offset)
+
+def _save_all(im, fp, filename):
+    if not hasattr(im, "n_frames"):
+        return _save(im, fp, filename)
+
+    cur_idx = im.tell()
+    try:
+        with AppendingTiffWriter(fp) as tf:
+            for idx in range(im.n_frames):
+                im.seek(idx)
+                im.load()
+                _save(im, tf, filename)
+                tf.newFrame()
+    finally:
+        im.seek(cur_idx)
+
 #
 # --------------------------------------------------------------------
 # Register
 
 Image.register_open(TiffImageFile.format, TiffImageFile, _accept)
 Image.register_save(TiffImageFile.format, _save)
+Image.register_save_all(TiffImageFile.format, _save_all)
 
 Image.register_extension(TiffImageFile.format, ".tif")
 Image.register_extension(TiffImageFile.format, ".tiff")


Test patch:
diff --git a/Tests/test_file_tiff.py b/Tests/test_file_tiff.py
--- a/Tests/test_file_tiff.py
+++ b/Tests/test_file_tiff.py
@@ -487,6 +487,17 @@ def test_roundtrip_tiff_uint16(self):
 
         self.assert_image_equal(im, reloaded)
 
+    def test_tiff_save_all(self):
+        import io
+        import os
+
+        mp = io.BytesIO()
+        with Image.open("Tests/images/multipage.tiff") as im:
+            im.save(mp, format="tiff", save_all=True)
+
+        mp.seek(0, os.SEEK_SET)
+        with Image.open(mp) as im:
+            self.assertEqual(im.n_frames, 3)
 
 if __name__ == '__main__':
     unittest.main()