Repository: scrapy/scrapy

Issue:
[bug] pillow will always recode images in imagepipieline
https://github.com/scrapy/scrapy/blob/aa83e159c97b441167d0510064204681bbc93f21/scrapy/pipelines/images.py#L151

this line will always recode images silently and damage the image quality.

please add an option to avoid this.
[MRG+1] Do not degrade JPEG files.
Fixes #3055 by not converting a JPEG file again.


Discussion:
Well, seems reasonable to me, but you can simply extend the default image pipeline and override the method to do what you want.
I don't know image pipeline of scrapy will always damage the quality of image, even after have read scrapy's documents.
Hi @NewUserHa - I'm sorry this has been a problem for you! The documents do say, at the higher-level overview, that [the images pipeline will convert all images to a common format](https://github.com/scrapy/scrapy/blob/1.5/docs/topics/media-pipeline.rst#L23), namely JPG with RGBA colourspace.

I think similarly to you, that unless the user explicitly requests recoding, the images would be better left alone. But, that's one of the big differences between simply using the Files pipeline and the Images pipeline, I guess. I'd suggest using the Files pipeline for now, if you want the unaltered images to be exactly as they occur online.
I meant images pipeline will still convert a jpg file to a less quality jpg file. is that a problem?
I'd need to read the code again to see whether quality is 'expected' to change under normal conditions.. but I do think there's a problem with the docs if this behaviour is surprising.
it's a problem. jpg encoding is lossy. pillow shouldn't reencode jpg to jpg at least.
but I noticed not all jpgs will not be re-encoded by pillow. maybe there's difference within the jpgs(I checked both they are jpgs, not only the extensions.)
I believe the documentation is OK.

> The Images Pipeline uses `Pillow`_ for thumbnailing and normalizing images to JPEG/RGB format.

It says `JPEG/RGB`. The `RGB` part must be the culprit of the quality change @NewUserHa is experiencing.

I’ve checked the code and `JPEG/RGB` images are not converted.

@NewUserHa If you believe they are, please share an example image that gets converted and should not. Otherwise, I believe we should close this issue.
Did you try yourself?
`Image.open('a.jpg').save('b.jpg', 'JPEG')`
I just pick a jpg randomly and run it with the above code, and it was re-encoded just like my last try.

DID you really double check it ?
picture info:
```
JPEG, quality: 77, subsampling ON (2x2)
16,7 Million   (24 BitsPerPixel)
```
If that is so, please provide the smallest image possible that allows to reproduce the issue. It can be used to test a fix.
![Q20161115204613](https://user-images.githubusercontent.com/32261870/54168432-6bc8e300-44a9-11e9-92d6-222201c18165.jpg)

Although I could not visually appreciate any difference, `Image.open('original.jpg').save('modified.jpg', 'JPEG')` does indeed slightly reduce the image size. `Image.open('original.jpg').save('modified.jpg', 'JPEG', quality='keep')` does not maintain the size, it increases it instead.

I think we could consider modifying `convert_image` to accept `BytesIO(response.body)` as a second parameter, and return it as `buf` when no conversion is necessary. @cathalgarvey I think this could be a good first issue.


Resolving patch:
diff --git a/scrapy/pipelines/images.py b/scrapy/pipelines/images.py
--- a/scrapy/pipelines/images.py
+++ b/scrapy/pipelines/images.py
@@ -17,7 +17,7 @@
 # TODO: from scrapy.pipelines.media import MediaPipeline
 from scrapy.settings import Settings
 from scrapy.utils.misc import md5sum
-from scrapy.utils.python import to_bytes
+from scrapy.utils.python import get_func_args, to_bytes
 
 
 class NoimagesDrop(DropItem):
@@ -92,6 +92,8 @@ def __init__(self, store_uri, download_func=None, settings=None):
             resolve('IMAGES_THUMBS'), self.THUMBS
         )
 
+        self._deprecated_convert_image = None
+
     @classmethod
     def from_settings(cls, settings):
         s3store = cls.STORE_SCHEMES['s3']
@@ -142,15 +144,33 @@ def get_images(self, response, request, info, *, item=None):
                                  f"({width}x{height} < "
                                  f"{self.min_width}x{self.min_height})")
 
-        image, buf = self.convert_image(orig_image)
+        if self._deprecated_convert_image is None:
+            self._deprecated_convert_image = 'response_body' not in get_func_args(self.convert_image)
+            if self._deprecated_convert_image:
+                warnings.warn(f'{self.__class__.__name__}.convert_image() method overriden in a deprecated way, '
+                              'overriden method does not accept response_body argument.',
+                              category=ScrapyDeprecationWarning)
+
+        if self._deprecated_convert_image:
+            image, buf = self.convert_image(orig_image)
+        else:
+            image, buf = self.convert_image(orig_image, response_body=BytesIO(response.body))
         yield path, image, buf
 
         for thumb_id, size in self.thumbs.items():
             thumb_path = self.thumb_path(request, thumb_id, response=response, info=info, item=item)
-            thumb_image, thumb_buf = self.convert_image(image, size)
+            if self._deprecated_convert_image:
+                thumb_image, thumb_buf = self.convert_image(image, size)
+            else:
+                thumb_image, thumb_buf = self.convert_image(image, size, buf)
             yield thumb_path, thumb_image, thumb_buf
 
-    def convert_image(self, image, size=None):
+    def convert_image(self, image, size=None, response_body=None):
+        if response_body is None:
+            warnings.warn(f'{self.__class__.__name__}.convert_image() method called in a deprecated way, '
+                          'method called without response_body argument.',
+                          category=ScrapyDeprecationWarning, stacklevel=2)
+
         if image.format == 'PNG' and image.mode == 'RGBA':
             background = self._Image.new('RGBA', image.size, (255, 255, 255))
             background.paste(image, image)
@@ -173,6 +193,8 @@ def convert_image(self, image, size=None):
             except AttributeError:
                 resampling_filter = self._Image.ANTIALIAS
             image.thumbnail(size, resampling_filter)
+        elif response_body is not None and image.format == 'JPEG':
+            return image, response_body
 
         buf = BytesIO()
         image.save(buf, 'JPEG')


Test patch:
diff --git a/tests/test_pipeline_images.py b/tests/test_pipeline_images.py
--- a/tests/test_pipeline_images.py
+++ b/tests/test_pipeline_images.py
@@ -2,9 +2,10 @@
 import hashlib
 import io
 import random
+import warnings
 from shutil import rmtree
 from tempfile import mkdtemp
-from warnings import catch_warnings
+from unittest.mock import patch
 
 import attr
 from itemadapter import ItemAdapter
@@ -13,7 +14,7 @@
 from scrapy.exceptions import ScrapyDeprecationWarning
 from scrapy.http import Request, Response
 from scrapy.item import Field, Item
-from scrapy.pipelines.images import ImagesPipeline, NoimagesDrop
+from scrapy.pipelines.images import ImageException, ImagesPipeline, NoimagesDrop
 from scrapy.settings import Settings
 from scrapy.utils.python import to_bytes
 
@@ -104,32 +105,143 @@ def thumb_path(self, request, thumb_id, response=None, info=None, item=None):
         request = Request("http://example.com")
         self.assertEqual(thumb_path(request, 'small', item=item), 'thumb/small/path-to-store-file')
 
-    def test_convert_image(self):
+    def test_get_images_exception(self):
+        self.pipeline.min_width = 100
+        self.pipeline.min_height = 100
+
+        _, buf1 = _create_image('JPEG', 'RGB', (50, 50), (0, 0, 0))
+        _, buf2 = _create_image('JPEG', 'RGB', (150, 50), (0, 0, 0))
+        _, buf3 = _create_image('JPEG', 'RGB', (50, 150), (0, 0, 0))
+
+        resp1 = Response(url="https://dev.mydeco.com/mydeco.gif", body=buf1.getvalue())
+        resp2 = Response(url="https://dev.mydeco.com/mydeco.gif", body=buf2.getvalue())
+        resp3 = Response(url="https://dev.mydeco.com/mydeco.gif", body=buf3.getvalue())
+        req = Request(url="https://dev.mydeco.com/mydeco.gif")
+
+        with self.assertRaises(ImageException):
+            next(self.pipeline.get_images(response=resp1, request=req, info=object()))
+        with self.assertRaises(ImageException):
+            next(self.pipeline.get_images(response=resp2, request=req, info=object()))
+        with self.assertRaises(ImageException):
+            next(self.pipeline.get_images(response=resp3, request=req, info=object()))
+
+    def test_get_images_new(self):
+        self.pipeline.min_width = 0
+        self.pipeline.min_height = 0
+        self.pipeline.thumbs = {'small': (20, 20)}
+
+        orig_im, buf = _create_image('JPEG', 'RGB', (50, 50), (0, 0, 0))
+        orig_thumb, orig_thumb_buf = _create_image('JPEG', 'RGB', (20, 20), (0, 0, 0))
+        resp = Response(url="https://dev.mydeco.com/mydeco.gif", body=buf.getvalue())
+        req = Request(url="https://dev.mydeco.com/mydeco.gif")
+
+        get_images_gen = self.pipeline.get_images(response=resp, request=req, info=object())
+
+        path, new_im, new_buf = next(get_images_gen)
+        self.assertEqual(path, 'full/3fd165099d8e71b8a48b2683946e64dbfad8b52d.jpg')
+        self.assertEqual(orig_im, new_im)
+        self.assertEqual(buf.getvalue(), new_buf.getvalue())
+
+        thumb_path, thumb_img, thumb_buf = next(get_images_gen)
+        self.assertEqual(thumb_path, 'thumbs/small/3fd165099d8e71b8a48b2683946e64dbfad8b52d.jpg')
+        self.assertEqual(thumb_img, thumb_img)
+        self.assertEqual(orig_thumb_buf.getvalue(), thumb_buf.getvalue())
+
+    def test_get_images_old(self):
+        self.pipeline.thumbs = {'small': (20, 20)}
+        orig_im, buf = _create_image('JPEG', 'RGB', (50, 50), (0, 0, 0))
+        resp = Response(url="https://dev.mydeco.com/mydeco.gif", body=buf.getvalue())
+        req = Request(url="https://dev.mydeco.com/mydeco.gif")
+
+        def overridden_convert_image(image, size=None):
+            im, buf = _create_image('JPEG', 'RGB', (50, 50), (0, 0, 0))
+            return im, buf
+
+        with patch.object(self.pipeline, 'convert_image', overridden_convert_image):
+            with warnings.catch_warnings(record=True) as w:
+                warnings.simplefilter('always')
+                get_images_gen = self.pipeline.get_images(response=resp, request=req, info=object())
+                path, new_im, new_buf = next(get_images_gen)
+                self.assertEqual(path, 'full/3fd165099d8e71b8a48b2683946e64dbfad8b52d.jpg')
+                self.assertEqual(orig_im.mode, new_im.mode)
+                self.assertEqual(orig_im.getcolors(), new_im.getcolors())
+                self.assertEqual(buf.getvalue(), new_buf.getvalue())
+
+                thumb_path, thumb_img, thumb_buf = next(get_images_gen)
+                self.assertEqual(thumb_path, 'thumbs/small/3fd165099d8e71b8a48b2683946e64dbfad8b52d.jpg')
+                self.assertEqual(orig_im.mode, thumb_img.mode)
+                self.assertEqual(orig_im.getcolors(), thumb_img.getcolors())
+                self.assertEqual(buf.getvalue(), thumb_buf.getvalue())
+
+                expected_warning_msg = ('.convert_image() method overriden in a deprecated way, '
+                                        'overriden method does not accept response_body argument.')
+                self.assertEqual(len([warning for warning in w if expected_warning_msg in str(warning.message)]), 1)
+
+    def test_convert_image_old(self):
+        # tests for old API
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            SIZE = (100, 100)
+            # straigh forward case: RGB and JPEG
+            COLOUR = (0, 127, 255)
+            im, _ = _create_image('JPEG', 'RGB', SIZE, COLOUR)
+            converted, _ = self.pipeline.convert_image(im)
+            self.assertEqual(converted.mode, 'RGB')
+            self.assertEqual(converted.getcolors(), [(10000, COLOUR)])
+
+            # check that thumbnail keep image ratio
+            thumbnail, _ = self.pipeline.convert_image(converted, size=(10, 25))
+            self.assertEqual(thumbnail.mode, 'RGB')
+            self.assertEqual(thumbnail.size, (10, 10))
+
+            # transparency case: RGBA and PNG
+            COLOUR = (0, 127, 255, 50)
+            im, _ = _create_image('PNG', 'RGBA', SIZE, COLOUR)
+            converted, _ = self.pipeline.convert_image(im)
+            self.assertEqual(converted.mode, 'RGB')
+            self.assertEqual(converted.getcolors(), [(10000, (205, 230, 255))])
+
+            # transparency case with palette: P and PNG
+            COLOUR = (0, 127, 255, 50)
+            im, _ = _create_image('PNG', 'RGBA', SIZE, COLOUR)
+            im = im.convert('P')
+            converted, _ = self.pipeline.convert_image(im)
+            self.assertEqual(converted.mode, 'RGB')
+            self.assertEqual(converted.getcolors(), [(10000, (205, 230, 255))])
+
+            # ensure that we recieved deprecation warnings
+            expected_warning_msg = '.convert_image() method called in a deprecated way'
+            self.assertTrue(len([warning for warning in w if expected_warning_msg in str(warning.message)]) == 4)
+
+    def test_convert_image_new(self):
+        # tests for new API
         SIZE = (100, 100)
         # straigh forward case: RGB and JPEG
         COLOUR = (0, 127, 255)
-        im = _create_image('JPEG', 'RGB', SIZE, COLOUR)
-        converted, _ = self.pipeline.convert_image(im)
+        im, buf = _create_image('JPEG', 'RGB', SIZE, COLOUR)
+        converted, converted_buf = self.pipeline.convert_image(im, response_body=buf)
         self.assertEqual(converted.mode, 'RGB')
         self.assertEqual(converted.getcolors(), [(10000, COLOUR)])
+        # check that we don't convert JPEGs again
+        self.assertEqual(converted_buf, buf)
 
         # check that thumbnail keep image ratio
-        thumbnail, _ = self.pipeline.convert_image(converted, size=(10, 25))
+        thumbnail, _ = self.pipeline.convert_image(converted, size=(10, 25), response_body=converted_buf)
         self.assertEqual(thumbnail.mode, 'RGB')
         self.assertEqual(thumbnail.size, (10, 10))
 
         # transparency case: RGBA and PNG
         COLOUR = (0, 127, 255, 50)
-        im = _create_image('PNG', 'RGBA', SIZE, COLOUR)
-        converted, _ = self.pipeline.convert_image(im)
+        im, buf = _create_image('PNG', 'RGBA', SIZE, COLOUR)
+        converted, _ = self.pipeline.convert_image(im, response_body=buf)
         self.assertEqual(converted.mode, 'RGB')
         self.assertEqual(converted.getcolors(), [(10000, (205, 230, 255))])
 
         # transparency case with palette: P and PNG
         COLOUR = (0, 127, 255, 50)
-        im = _create_image('PNG', 'RGBA', SIZE, COLOUR)
+        im, buf = _create_image('PNG', 'RGBA', SIZE, COLOUR)
         im = im.convert('P')
-        converted, _ = self.pipeline.convert_image(im)
+        converted, _ = self.pipeline.convert_image(im, response_body=buf)
         self.assertEqual(converted.mode, 'RGB')
         self.assertEqual(converted.getcolors(), [(10000, (205, 230, 255))])
 
@@ -422,23 +534,23 @@ class NoimagesDropTestCase(unittest.TestCase):
 
     def test_deprecation_warning(self):
         arg = str()
-        with catch_warnings(record=True) as warnings:
+        with warnings.catch_warnings(record=True) as w:
             NoimagesDrop(arg)
-            self.assertEqual(len(warnings), 1)
-            self.assertEqual(warnings[0].category, ScrapyDeprecationWarning)
-        with catch_warnings(record=True) as warnings:
+            self.assertEqual(len(w), 1)
+            self.assertEqual(w[0].category, ScrapyDeprecationWarning)
+        with warnings.catch_warnings(record=True) as w:
             class SubclassedNoimagesDrop(NoimagesDrop):
                 pass
             SubclassedNoimagesDrop(arg)
-            self.assertEqual(len(warnings), 1)
-            self.assertEqual(warnings[0].category, ScrapyDeprecationWarning)
+            self.assertEqual(len(w), 1)
+            self.assertEqual(w[0].category, ScrapyDeprecationWarning)
 
 
 def _create_image(format, *a, **kw):
     buf = io.BytesIO()
     Image.new(*a, **kw).save(buf, format)
     buf.seek(0)
-    return Image.open(buf)
+    return Image.open(buf), buf
 
 
 if __name__ == "__main__":