Repository: python-pillow/Pillow

Issue:
Saving TIFF with compression throws Segmentation Fault 11 in 5.4.1, but not 5.3.0
### What did you do?
Opened a bitonal TIFF created with Adobe Photoshop CC (version 20.0.2) then saved it with Group4 compression using Pillow.

TIFF Image should be downloadable from Github repository here: [https://github.com/photosbyjeremy/images/blob/master/test_bitonal_text.tif.zip](https://github.com/photosbyjeremy/images/blob/master/test_bitonal_text.tif.zip)

### What did you expect to happen?
I expected the image to save with compression.

### What actually happened?
In Pillow 5.4.1 there was a Segmentation Fault 11, but in Pillow 5.3.0 it works just fine.

<img width="624" alt="screen shot 2019-02-25 at 2 09 47 pm" src="https://user-images.githubusercontent.com/5132339/53362031-406dc200-3907-11e9-95b3-d9ae4724db9a.png">

<img width="640" alt="screen shot 2019-02-25 at 2 09 50 pm" src="https://user-images.githubusercontent.com/5132339/53362045-495e9380-3907-11e9-8334-9523cc340dde.png">

### What are your OS, Python and Pillow versions?

* OS: macOS 10.14.3
* Python: 3.6.8 (and I actually tried 3.7.1 first, but didn't screenshot those attempts)
* Pillow: 5.4.1 and 5.3.0

Please include **code** that reproduces the issue and whenever possible, an **image** that demonstrates the issue. Please upload images to GitHub, not to third-party file hosting sites. If necessary, add the image to a zip or tar archive.

The best reproductions are self-contained scripts with minimal dependencies. If you are using a framework such as plone, Django, or buildout, try to replicate the issue just using Pillow. 

```python
from PIL import Image
image = Image.open('test_bitonal_text.tif')
image.save('test_pillow_save.tif', compression='group4')
```



Discussion:
<img width="668" alt="screen shot 2019-02-25 at 2 43 28 pm" src="https://user-images.githubusercontent.com/5132339/53363987-ce4bac00-390b-11e9-836e-c71b450beb85.png">
Git bisect points to https://github.com/python-pillow/Pillow/commit/6ead422e91bec1facc644f02bc25a561d0a02383 from PR https://github.com/python-pillow/Pillow/pull/3513.

```console
$ python3 3677.py
Python(31119,0x7fffb3bf1380) malloc: *** mach_vm_map(size=18446744072426508288) failed (error code=3)
*** error: can't allocate region
*** set a breakpoint in malloc_error_break to debug
test_pillow_save.tif: Failed to allocate memory for custom tag binary object (-1283043808 elements of 1 bytes each).
Traceback (most recent call last):
  File "3677.py", line 3, in <module>
    image.save('test_pillow_save.tif', compression='group4')
  File "/Users/hugo/github/Pillow/src/PIL/Image.py", line 1988, in save
    save_handler(self, fp, filename)
  File "/Users/hugo/github/Pillow/src/PIL/TiffImagePlugin.py", line 1552, in _save
    e = Image._getencoder(im.mode, 'libtiff', a, im.encoderconfig)
  File "/Users/hugo/github/Pillow/src/PIL/Image.py", line 477, in _getencoder
    return encoder(mode, *args + extra)
RuntimeError: Error setting from dictionary
```

* OS: macOS High Sierra 10.13.6
* Python: 3.7.2

If I replace the code in PIL.TiffImagePlugin.py changed by [6ead422](
https://github.com/python-pillow/Pillow/commit/6ead422e91bec1facc644f02bc25a561d0a02383) with that from 5.3.0 the save succeeds in 5.4.1

LEFT: TiffImagePlugin.py from 5.4.1 that works RIGHT: TiffImagePlugin.py from 5.3.0 that works
<img width="1151" alt="screen shot 2019-02-25 at 4 01 25 pm" src="https://user-images.githubusercontent.com/5132339/53368528-10c6b600-3917-11e9-80b0-11acf9bcc6fa.png">

I have found the problematic tags. The following code runs without error.

```python
from PIL import Image
image = Image.open('test_bitonal_text.tif')
del image.tag_v2[34377]
del image.tag_v2[33723]
del image.tag_v2[700]
image.save('test_pillow_save.tif', compression='group4')
```

The following code is all that is needed to reproduce the segmentation fault.

```python
from PIL import Image, TiffImagePlugin
TiffImagePlugin.WRITE_LIBTIFF = True
image = Image.new("RGB", (100, 100))
image.save('out.tif', tiffinfo={700: b"a"})
```

The test suite is able to save byte data for other tag numbers, so the problem is specific to these numbers. The seg fault is occuring at https://github.com/python-pillow/Pillow/blob/d167f9e0bd8a71f7d0a5a1098dc9d51eceb67be2/src/libImaging/TiffDecode.c#L432
While an error still occurs for this issue, the segfault has been fixed in #4033
The error now received here is
```python
from PIL import Image
image = Image.open('test_bitonal_text.tif')
image.save('test_pillow_save.tif', compression='group4')
```

```
TypeError: an integer is required (got type bytes)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "code.py", line 3, in <module>
    image.save('test_pillow_save.tif', compression='group4')
  File "PIL/Image.py", line 2054, in save
  File "PIL/TiffImagePlugin.py", line 1598, in _save
  File "PIL/Image.py", line 428, in _getencoder
SystemError: <built-in function libtiff_encoder> returned a result with an error set
```
Also getting the `SystemError: <built-in function libtiff_encoder> returned a result with an error set` when daving a tif file through `tesserocr`. Anyone with a solution yet?
I'm actually working on a solution at the moment. To be sure that your problem is solved though, in case your error is triggered by a different case than the one in this issue, could we have a simple self-contained example of your situation?
> I'm actually working on a solution at the moment. To be sure that your problem is solved though, in case your error is triggered by a different case than the one in this issue, could we have a simple self-contained example of your situation?

it gets triggered when performing OCR on a tiff PIL image via the `tesserocr` library. SOmething like:

```python
import tesserocr
from tesserocr import PSM
tesserocr.image_to_text(scanned_tiff_img, psm=PSM.SPARSE_TEXT_OSD)
```
Then I get `SystemError: <built-in function libtiff_encoder> returned a result with an error set`. From what I remember from looking at the stacktrace, it jammed when the library tried to save the tiff.
My version of PIL is 7.1.2

@zoj613 Please could you post the full stacktrace? 
FWIW, I am getting the same error on the following snippet:

```python
from PIL import Image, TiffImagePlugin                                                                                     
image = Image.open('mytiff.tif')
image.save('test_pillow_save.tif')
```

with the following stracktrace:
```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
TypeError: an integer is required (got type bytes)

The above exception was the direct cause of the following exception:

SystemError                               Traceback (most recent call last)
<ipython-input-27-3b2a40a8e19e> in <module>
----> 1 image.save('test_pillow_save.tif')

~/miniconda3/envs/fw/lib/python3.7/site-packages/PIL/Image.py in save(self, fp, format, **params)
   2100 
   2101         try:
-> 2102             save_handler(self, fp, filename)
   2103         finally:
   2104             # do what we can to clean up

~/miniconda3/envs/fw/lib/python3.7/site-packages/PIL/TiffImagePlugin.py in _save(im, fp, filename)
   1614         tags.sort()
   1615         a = (rawmode, compression, _fp, filename, tags, types)
-> 1616         e = Image._getencoder(im.mode, "libtiff", a, im.encoderconfig)
   1617         e.setimage(im.im, (0, 0) + im.size)
   1618         while True:

~/miniconda3/envs/fw/lib/python3.7/site-packages/PIL/Image.py in _getencoder(mode, encoder_name, args, extra)
    429         # get encoder
    430         encoder = getattr(core, encoder_name + "_encoder")
--> 431         return encoder(mode, *args + extra)
    432     except AttributeError:
    433         raise OSError("encoder %s not available" % encoder_name)

SystemError: <built-in function libtiff_encoder> returned a result with an error set
```

OS: macOS 10.14.6
Python: 3.7.5
Pillow: 7.0.0 and 7.1.2
I've created PR #4605 to resolve this.

@zoj613 @NPann if you could check and confirm that this also resolves your situations. If it does not, please provide images to demonstrate the problem.
@radarhere, it does resolve my situation. Thanks a lot for figuring this out!

Resolving patch:
diff --git a/src/PIL/TiffImagePlugin.py b/src/PIL/TiffImagePlugin.py
--- a/src/PIL/TiffImagePlugin.py
+++ b/src/PIL/TiffImagePlugin.py
@@ -553,9 +553,10 @@ def _setitem(self, tag, value, legacy_api):
                         )
                 elif all(isinstance(v, float) for v in values):
                     self.tagtype[tag] = TiffTags.DOUBLE
-                else:
-                    if all(isinstance(v, str) for v in values):
-                        self.tagtype[tag] = TiffTags.ASCII
+                elif all(isinstance(v, str) for v in values):
+                    self.tagtype[tag] = TiffTags.ASCII
+                elif all(isinstance(v, bytes) for v in values):
+                    self.tagtype[tag] = TiffTags.BYTE
 
         if self.tagtype[tag] == TiffTags.UNDEFINED:
             values = [
@@ -573,8 +574,10 @@ def _setitem(self, tag, value, legacy_api):
         # Spec'd length == 1, Actual > 1, Warn and truncate. Formerly barfed.
         # No Spec, Actual length 1, Formerly (<4.2) returned a 1 element tuple.
         # Don't mess with the legacy api, since it's frozen.
-        if (info.length == 1) or (
-            info.length is None and len(values) == 1 and not legacy_api
+        if (
+            (info.length == 1)
+            or self.tagtype[tag] == TiffTags.BYTE
+            or (info.length is None and len(values) == 1 and not legacy_api)
         ):
             # Don't mess with the legacy api, since it's frozen.
             if legacy_api and self.tagtype[tag] in [
@@ -1546,16 +1549,17 @@ def _save(im, fp, filename):
             # Custom items are supported for int, float, unicode, string and byte
             # values. Other types and tuples require a tagtype.
             if tag not in TiffTags.LIBTIFF_CORE:
-                if (
-                    TiffTags.lookup(tag).type == TiffTags.UNDEFINED
-                    or not Image.core.libtiff_support_custom_tags
-                ):
+                if not Image.core.libtiff_support_custom_tags:
                     continue
 
                 if tag in ifd.tagtype:
                     types[tag] = ifd.tagtype[tag]
                 elif not (isinstance(value, (int, float, str, bytes))):
                     continue
+                else:
+                    type = TiffTags.lookup(tag).type
+                    if type:
+                        types[tag] = type
             if tag not in atts and tag not in blocklist:
                 if isinstance(value, str):
                     atts[tag] = value.encode("ascii", "replace") + b"\0"


Test patch:
diff --git a/Tests/test_file_libtiff.py b/Tests/test_file_libtiff.py
--- a/Tests/test_file_libtiff.py
+++ b/Tests/test_file_libtiff.py
@@ -299,9 +299,6 @@ def check_tags(tiffinfo):
                             )
                             continue
 
-                        if libtiff and isinstance(value, bytes):
-                            value = value.decode()
-
                         assert reloaded_value == value
 
             # Test with types
@@ -322,6 +319,17 @@ def check_tags(tiffinfo):
             )
         TiffImagePlugin.WRITE_LIBTIFF = False
 
+    def test_xmlpacket_tag(self, tmp_path):
+        TiffImagePlugin.WRITE_LIBTIFF = True
+
+        out = str(tmp_path / "temp.tif")
+        hopper().save(out, tiffinfo={700: b"xmlpacket tag"})
+        TiffImagePlugin.WRITE_LIBTIFF = False
+
+        with Image.open(out) as reloaded:
+            if 700 in reloaded.tag_v2:
+                assert reloaded.tag_v2[700] == b"xmlpacket tag"
+
     def test_int_dpi(self, tmp_path):
         # issue #1765
         im = hopper("RGB")
@@ -667,6 +675,26 @@ def test_read_icc(self):
         TiffImagePlugin.READ_LIBTIFF = False
         assert icc == icc_libtiff
 
+    def test_write_icc(self, tmp_path):
+        def check_write(libtiff):
+            TiffImagePlugin.WRITE_LIBTIFF = libtiff
+
+            with Image.open("Tests/images/hopper.iccprofile.tif") as img:
+                icc_profile = img.info["icc_profile"]
+
+                out = str(tmp_path / "temp.tif")
+                img.save(out, icc_profile=icc_profile)
+            with Image.open(out) as reloaded:
+                assert icc_profile == reloaded.info["icc_profile"]
+
+        libtiffs = []
+        if Image.core.libtiff_support_custom_tags:
+            libtiffs.append(True)
+        libtiffs.append(False)
+
+        for libtiff in libtiffs:
+            check_write(libtiff)
+
     def test_multipage_compression(self):
         with Image.open("Tests/images/compression.tif") as im:
 
diff --git a/Tests/test_file_tiff_metadata.py b/Tests/test_file_tiff_metadata.py
--- a/Tests/test_file_tiff_metadata.py
+++ b/Tests/test_file_tiff_metadata.py
@@ -319,13 +319,13 @@ def test_empty_values():
 
 def test_PhotoshopInfo(tmp_path):
     with Image.open("Tests/images/issue_2278.tif") as im:
-        assert len(im.tag_v2[34377]) == 1
-        assert isinstance(im.tag_v2[34377][0], bytes)
+        assert len(im.tag_v2[34377]) == 70
+        assert isinstance(im.tag_v2[34377], bytes)
         out = str(tmp_path / "temp.tiff")
         im.save(out)
     with Image.open(out) as reloaded:
-        assert len(reloaded.tag_v2[34377]) == 1
-        assert isinstance(reloaded.tag_v2[34377][0], bytes)
+        assert len(reloaded.tag_v2[34377]) == 70
+        assert isinstance(reloaded.tag_v2[34377], bytes)
 
 
 def test_too_many_entries():