Repository: scrapy/scrapy

Issue:
Custom ImagesPipeline doesn't generate thumbnails
For some reason, when using a customised ImagesPipeline thumbnails are no longer generated. This seems to be the case for 1.1.1 and 1.1.2 but not for 1.0.0 so somewhere between those releases something must have changed.

This [example](https://gist.github.com/briehanlombaard/b19885f99b9ee26363070fd9883695b9) should reproduce the problem.



Discussion:
@briehanlombaard , did you mean _"but not for 1.1.0"_ ?

@redapple I actually didn't try that version until now, but it does seem to work as well. So the problem must be somewhere between 1.1.0 and 1.1.1 

I used [git bisect](https://gist.github.com/briehanlombaard/02d2ff5c6d590de621be1b60640ae66f) to track down the 'bad' commit at https://github.com/scrapy/scrapy/commit/c6277c9963af24d1a61fa201693fafdd46426ee8. I'll see if I can get a fix for the issue if one is needed. I might just be doing something wrong.

@briehanlombaard , thank you for continuing the investigation.
I can confirm that thumbnails are not generated with your example starting from scrapy 1.1.1 (worked on 1.1.0)
I'll also have a look in more details.

@briehanlombaard , so the trouble is indeed with that commit you linked.
Current implementation is looking for `ARTICLEIMAGESPIPELINE_IMAGES_THUMBS` without looking for `IMAGES_THUMBS`, contrary to what is done for `IMAGES_STORE`, and contrary to what the docs currently say.
@pawelmhm , it looks like we still have a backward incompatibility issue with https://github.com/scrapy/scrapy/pull/1891 and the subsequent fix in https://github.com/scrapy/scrapy/pull/1989
@briehanlombaard's use-case looks very reasonable:
- custom ImagesPipeline but only for converting, no class attribute change
- use of custom settings for `IMAGES_STORE` and `IMAGES_THUMBS`, like the docs recommend

@redapple , I am not sure it is a bug. It was coded to work just like that. See [this test](https://github.com/scrapy/scrapy/blob/master/tests/test_pipeline_images.py#L324). I don't remember details, but if I am not wrong it was coded in this way to accommodate class attributes in custom ImagesPipelines.

I think that in order to do this given example work it would be necessary some backward incompatibility with class attributes. And, as far as we know, there is no backward incompatibility with class attributes now.

Maybe the way to go now should be just improve the docs and better explain how to create/use custom pipelines. (And accept this little backward incompatibility as it is.)

@briehanlombaard , you should precede `IMAGES_THUMBS` with your custom ImagesPipeline class name:

``` python
custom_settings = {
    'ITEM_PIPELINES': {
        'spider.ArticleImagesPipeline': 100,
    },
    'IMAGES_STORE': 'articles',
    #'IMAGES_THUMBS': {
    'ARTICLEIMAGESPIPELINE_IMAGES_THUMBS': {
        'small': (290, 150),
        'medium': (610, 320),
    },
    'AUTOTHROTTLE_ENABLED': True,
}
```


Resolving patch:
diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.py
--- a/scrapy/pipelines/files.py
+++ b/scrapy/pipelines/files.py
@@ -233,7 +233,8 @@ def __init__(self, store_uri, download_func=None, settings=None):
         cls_name = "FilesPipeline"
         self.store = self._get_store(store_uri)
         resolve = functools.partial(self._key_for_pipe,
-                                    base_class_name=cls_name)
+                                    base_class_name=cls_name,
+                                    settings=settings)
         self.expires = settings.getint(
             resolve('FILES_EXPIRES'), self.EXPIRES
         )
diff --git a/scrapy/pipelines/images.py b/scrapy/pipelines/images.py
--- a/scrapy/pipelines/images.py
+++ b/scrapy/pipelines/images.py
@@ -55,7 +55,8 @@ def __init__(self, store_uri, download_func=None, settings=None):
             settings = Settings(settings)
 
         resolve = functools.partial(self._key_for_pipe,
-                                    base_class_name="ImagesPipeline")
+                                    base_class_name="ImagesPipeline",
+                                    settings=settings)
         self.expires = settings.getint(
             resolve("IMAGES_EXPIRES"), self.EXPIRES
         )
diff --git a/scrapy/pipelines/media.py b/scrapy/pipelines/media.py
--- a/scrapy/pipelines/media.py
+++ b/scrapy/pipelines/media.py
@@ -28,7 +28,8 @@ def __init__(self, download_func=None):
         self.download_func = download_func
 
 
-    def _key_for_pipe(self, key, base_class_name=None):
+    def _key_for_pipe(self, key, base_class_name=None,
+                      settings=None):
         """
         >>> MediaPipeline()._key_for_pipe("IMAGES")
         'IMAGES'
@@ -38,9 +39,11 @@ def _key_for_pipe(self, key, base_class_name=None):
         'MYPIPE_IMAGES'
         """
         class_name = self.__class__.__name__
-        if class_name == base_class_name or not base_class_name:
+        formatted_key = "{}_{}".format(class_name.upper(), key)
+        if class_name == base_class_name or not base_class_name \
+            or (settings and not settings.get(formatted_key)):
             return key
-        return "{}_{}".format(class_name.upper(), key)
+        return formatted_key
 
     @classmethod
     def from_crawler(cls, crawler):


Test patch:
diff --git a/tests/test_pipeline_files.py b/tests/test_pipeline_files.py
--- a/tests/test_pipeline_files.py
+++ b/tests/test_pipeline_files.py
@@ -255,16 +255,17 @@ def test_subclass_attributes_preserved_if_no_settings(self):
 
     def test_subclass_attrs_preserved_custom_settings(self):
         """
-        If file settings are defined but they are not defined for subclass class attributes
-        should be preserved.
+        If file settings are defined but they are not defined for subclass
+        settings should be preserved.
         """
         pipeline_cls = self._generate_fake_pipeline()
         settings = self._generate_fake_settings()
         pipeline = pipeline_cls.from_settings(Settings(settings))
         for pipe_attr, settings_attr, pipe_ins_attr in self.file_cls_attr_settings_map:
             value = getattr(pipeline, pipe_ins_attr)
+            setting_value = settings.get(settings_attr)
             self.assertNotEqual(value, self.default_cls_settings[pipe_attr])
-            self.assertEqual(value, getattr(pipeline, pipe_attr))
+            self.assertEqual(value, setting_value)
 
     def test_no_custom_settings_for_subclasses(self):
         """
@@ -321,6 +322,24 @@ class UserDefinedFilesPipeline(FilesPipeline):
         self.assertEqual(pipeline.files_urls_field, "that")
 
 
+    def test_user_defined_subclass_default_key_names(self):
+        """Test situation when user defines subclass of FilesPipeline,
+        but uses attribute names for default pipeline (without prefixing
+        them with pipeline class name).
+        """
+        settings = self._generate_fake_settings()
+
+        class UserPipe(FilesPipeline):
+            pass
+
+        pipeline_cls = UserPipe.from_settings(Settings(settings))
+
+        for pipe_attr, settings_attr, pipe_inst_attr in self.file_cls_attr_settings_map:
+            expected_value = settings.get(settings_attr)
+            self.assertEqual(getattr(pipeline_cls, pipe_inst_attr),
+                             expected_value)
+
+
 class TestS3FilesStore(unittest.TestCase):
     @defer.inlineCallbacks
     def test_persist(self):
diff --git a/tests/test_pipeline_images.py b/tests/test_pipeline_images.py
--- a/tests/test_pipeline_images.py
+++ b/tests/test_pipeline_images.py
@@ -309,17 +309,19 @@ def test_subclass_attrs_preserved_default_settings(self):
 
     def test_subclass_attrs_preserved_custom_settings(self):
         """
-        If image settings are defined but they are not defined for subclass class attributes
-        should be preserved.
+        If image settings are defined but they are not defined for subclass default
+        values taken from settings should be preserved.
         """
         pipeline_cls = self._generate_fake_pipeline_subclass()
         settings = self._generate_fake_settings()
         pipeline = pipeline_cls.from_settings(Settings(settings))
         for pipe_attr, settings_attr in self.img_cls_attribute_names:
-            # Instance attribute (lowercase) must be equal to class attribute (uppercase).
+            # Instance attribute (lowercase) must be equal to
+            # value defined in settings.
             value = getattr(pipeline, pipe_attr.lower())
             self.assertNotEqual(value, self.default_pipeline_settings[pipe_attr])
-            self.assertEqual(value, getattr(pipeline, pipe_attr))
+            setings_value = settings.get(settings_attr)
+            self.assertEqual(value, setings_value)
 
     def test_no_custom_settings_for_subclasses(self):
         """
@@ -370,11 +372,26 @@ def test_cls_attrs_with_DEFAULT_prefix(self):
         class UserDefinedImagePipeline(ImagesPipeline):
             DEFAULT_IMAGES_URLS_FIELD = "something"
             DEFAULT_IMAGES_RESULT_FIELD = "something_else"
-
         pipeline = UserDefinedImagePipeline.from_settings(Settings({"IMAGES_STORE": self.tempdir}))
         self.assertEqual(pipeline.images_result_field, "something_else")
         self.assertEqual(pipeline.images_urls_field, "something")
 
+    def test_user_defined_subclass_default_key_names(self):
+        """Test situation when user defines subclass of ImagePipeline,
+        but uses attribute names for default pipeline (without prefixing
+        them with pipeline class name).
+        """
+        settings = self._generate_fake_settings()
+
+        class UserPipe(ImagesPipeline):
+            pass
+
+        pipeline_cls = UserPipe.from_settings(Settings(settings))
+
+        for pipe_attr, settings_attr in self.img_cls_attribute_names:
+            expected_value = settings.get(settings_attr)
+            self.assertEqual(getattr(pipeline_cls, pipe_attr.lower()),
+                             expected_value)
 
 def _create_image(format, *a, **kw):
     buf = TemporaryFile()