Repository: scrapy/scrapy

Issue:
MediaPipeline (and ImagesPipeline/FilesPipeline) does not handle HTTP redirections
Basically, what's happened is that my spider is unable to download the files because the file_urls provided are actually redirected to the final download link. However, because of the following code, the redirect download middleware is effectively disabled, which makes the download fail.

```
def _check_media_to_download(self, result, request, info):
        if result is not None:
            return result
        if self.download_func:
            # this ugly code was left only to support tests. TODO: remove
            dfd = mustbe_deferred(self.download_func, request, info.spider)
            dfd.addCallbacks(
                callback=self.media_downloaded, callbackArgs=(request, info),
                errback=self.media_failed, errbackArgs=(request, info))
        else:
            request.meta['handle_httpstatus_all'] = True
            dfd = self.crawler.engine.download(request, info.spider)
            dfd.addCallbacks(
                callback=self.media_downloaded, callbackArgs=(request, info),
                errback=self.media_failed, errbackArgs=(request, info))
        return dfd
```

And here is the check in the redirect middleware that disables it:

```
if (request.meta.get('dont_redirect', False) or
                response.status in getattr(spider, 'handle_httpstatus_list', []) or
                response.status in request.meta.get('handle_httpstatus_list', []) or
                request.meta.get('handle_httpstatus_all', False)):
            return response

```

My question is: What is the point of enabling the handling of httpstatus_all, when it effectively disables all of the checks?  



Discussion:
Also, here is the error that is caused:

`[scrapy] WARNING: File (code: 302): Error downloading file from <GET <url> referred in <None>
`

That [line is quite old](https://github.com/scrapy/scrapy/commit/03751749a80526f3d97f60f0f63892501ef5de19#diff-f37ac01f1c3a8cc5f87456da118d7d1cR77).
I don't know the exact reason.
It may be to catch all non-200 responses. But then 301 and 302 are interesting to have handled by redirect mw indeed. So `handle_httpstatus_all` may very well be too-broad scope.

ditto. I overrode the function setting it to True but I'm nervous it'll bite me.
What does a request do when there's an error? will it raise an exception or something so  I don't end up treating an error or empty body as my file?

I think a 301 should modify the corresponding element in file_urls to the new location, although that might be surprising to the user. Adding a redirect trail to the item might be excessive?

A 302 should probably just follow through to the final destination?

This should all respect the allowed domains, right? Is that automatically taken care of by the downloader?

So correct behaviour might look like:
1. remove the override of `handle_httpstatus_all`
2. verify that any http errors result in helpful error logs and storing anything or setting the files field
3. logging a helpful error
4. updating the relevant original url to the last location of the first chain of 301s, e.g. `scraped url > 301 a > 301 b > 302 c > 301 d > 200 e` should see b placed in file_urls

I think 2 and 3 are taken care of by the current code.

I think 4 is technically correct but it isn't easy since redirect status codes aren't in `redirect_urls`. Do you think it's necessary or just over-complicating? 

@jbothma To answer your question:  

> This should all respect the allowed domains, right? Is that automatically taken care of by the downloader?

Yes since all requests scheduled go through download middlewares it will go through [OffsiteMiddleware](http://doc.scrapy.org/en/latest/topics/spider-middleware.html#scrapy.spidermiddlewares.offsite.OffsiteMiddleware), so it will do the usual match to spider.allowed_domains.

Regarding the issue I think `handle_httpstatus_all` should be replaced with a `handle_httpstatus_list` that exclused some 300 codes. 
For `FilesPipeline` the callback `media_downloaded()` by default raises an exception on anything that is not 200 or has an empty body, so this pipeline doesn't benefit from this setting at all other than wrapping the error. Excluding that from 300 codes wouldn't break anything other than enabling the redirection, which might be unwanted for some reason but I bet it's the other way around more often than not.

I think implementing a pipeline setting to manage this would be an ideal solution.

Thanks for the response.

A pipeline setting to opt in to allow 300 redirect codes?

On 22 August 2016 at 16:59, Bernardas Ališauskas notifications@github.com
wrote:

> @jbothma https://github.com/jbothma To answer your question:
> 
> This should all respect the allowed domains, right? Is that automatically
> taken care of by the downloader?
> 
> Yes since all requests scheduled go through download middlewares it will
> go through OffsiteMiddleware
> http://doc.scrapy.org/en/latest/topics/spider-middleware.html#scrapy.spidermiddlewares.offsite.OffsiteMiddleware,
> so it will do the usual match to spider.allowed_domains.
> 
> Regarding the issue I think handle_httpstatus_all should be replaced with
> a handle_httpstatus_list that exclused some 300 codes.
> For FilesPipeline the callback media_downloaded() by default raises an
> exception on anything that is not 200 or has an empty body, so this
> pipeline doesn't benefit from this setting at all other than wrapping the
> error. Excluding that from 300 codes wouldn't break anything other than
> enabling the redirection, which might be unwanted for some reason but I bet
> it's the other way around more often than not.
> 
> I think implementing a pipeline setting to manage this would be an ideal
> solution.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> https://github.com/scrapy/scrapy/issues/2004#issuecomment-241440597, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AAOZGeEAoQt-j7fuR1_pBU5sLupDkF7xks5qiblJgaJpZM4IkC4v
> .


Resolving patch:
diff --git a/scrapy/pipelines/files.py b/scrapy/pipelines/files.py
--- a/scrapy/pipelines/files.py
+++ b/scrapy/pipelines/files.py
@@ -249,7 +249,7 @@ def __init__(self, store_uri, download_func=None, settings=None):
             resolve('FILES_RESULT_FIELD'), self.FILES_RESULT_FIELD
         )
 
-        super(FilesPipeline, self).__init__(download_func=download_func)
+        super(FilesPipeline, self).__init__(download_func=download_func, settings=settings)
 
     @classmethod
     def from_settings(cls, settings):
diff --git a/scrapy/pipelines/media.py b/scrapy/pipelines/media.py
--- a/scrapy/pipelines/media.py
+++ b/scrapy/pipelines/media.py
@@ -1,10 +1,13 @@
 from __future__ import print_function
 
+import functools
 import logging
 from collections import defaultdict
 from twisted.internet.defer import Deferred, DeferredList
 from twisted.python.failure import Failure
 
+from scrapy.settings import Settings
+from scrapy.utils.datatypes import SequenceExclude
 from scrapy.utils.defer import mustbe_deferred, defer_result
 from scrapy.utils.request import request_fingerprint
 from scrapy.utils.misc import arg_to_iter
@@ -24,9 +27,23 @@ def __init__(self, spider):
             self.downloaded = {}
             self.waiting = defaultdict(list)
 
-    def __init__(self, download_func=None):
+    def __init__(self, download_func=None, settings=None):
         self.download_func = download_func
 
+        if isinstance(settings, dict) or settings is None:
+            settings = Settings(settings)
+        resolve = functools.partial(self._key_for_pipe,
+                                    base_class_name="MediaPipeline",
+                                    settings=settings)
+        self.allow_redirects = settings.getbool(
+            resolve('MEDIA_ALLOW_REDIRECTS'), False
+        )
+        self._handle_statuses(self.allow_redirects)
+
+    def _handle_statuses(self, allow_redirects):
+        self.handle_httpstatus_list = None
+        if allow_redirects:
+            self.handle_httpstatus_list = SequenceExclude(range(300, 400))
 
     def _key_for_pipe(self, key, base_class_name=None,
                       settings=None):
@@ -93,6 +110,12 @@ def _process_request(self, request, info):
         )
         return dfd.addBoth(lambda _: wad)  # it must return wad at last
 
+    def _modify_media_request(self, request):
+        if self.handle_httpstatus_list:
+            request.meta['handle_httpstatus_list'] = self.handle_httpstatus_list
+        else:
+            request.meta['handle_httpstatus_all'] = True
+
     def _check_media_to_download(self, result, request, info):
         if result is not None:
             return result
@@ -103,7 +126,7 @@ def _check_media_to_download(self, result, request, info):
                 callback=self.media_downloaded, callbackArgs=(request, info),
                 errback=self.media_failed, errbackArgs=(request, info))
         else:
-            request.meta['handle_httpstatus_all'] = True
+            self._modify_media_request(request)
             dfd = self.crawler.engine.download(request, info.spider)
             dfd.addCallbacks(
                 callback=self.media_downloaded, callbackArgs=(request, info),


Test patch:
diff --git a/tests/mockserver.py b/tests/mockserver.py
--- a/tests/mockserver.py
+++ b/tests/mockserver.py
@@ -5,13 +5,17 @@
 
 from twisted.web.server import Site, NOT_DONE_YET
 from twisted.web.resource import Resource
+from twisted.web.static import File
 from twisted.web.test.test_webclient import PayloadResource
 from twisted.web.server import GzipEncoderFactory
 from twisted.web.resource import EncodingResourceWrapper
+from twisted.web.util import redirectTo
 from twisted.internet import reactor, ssl
 from twisted.internet.task import deferLater
 
+
 from scrapy.utils.python import to_bytes, to_unicode
+from tests import tests_datadir
 
 
 def getarg(request, name, default=None, type=None):
@@ -120,6 +124,16 @@ def render_GET(self, request):
         return to_bytes(json.dumps(output))
 
 
+class RedirectTo(LeafResource):
+
+    def render(self, request):
+        goto = getarg(request, b'goto', b'/')
+        # we force the body content, otherwise Twisted redirectTo()
+        # returns HTML with <meta http-equiv="refresh"
+        redirectTo(goto, request)
+        return b'redirecting...'
+
+
 class Partial(LeafResource):
 
     def render_GET(self, request):
@@ -160,6 +174,8 @@ def __init__(self):
         self.putChild(b"echo", Echo())
         self.putChild(b"payload", PayloadResource())
         self.putChild(b"xpayload", EncodingResourceWrapper(PayloadResource(), [GzipEncoderFactory()]))
+        self.putChild(b"files", File(os.path.join(tests_datadir, 'test_site/files/')))
+        self.putChild(b"redirect-to", RedirectTo())
 
     def getChild(self, name, request):
         return self
diff --git a/tests/sample_data/test_site/files/images/python-logo-master-v3-TM-flattened.png b/tests/sample_data/test_site/files/images/python-logo-master-v3-TM-flattened.png
new file mode 100644
Binary files /dev/null and b/tests/sample_data/test_site/files/images/python-logo-master-v3-TM-flattened.png differ
diff --git a/tests/sample_data/test_site/files/images/python-powered-h-50x65.png b/tests/sample_data/test_site/files/images/python-powered-h-50x65.png
new file mode 100644
Binary files /dev/null and b/tests/sample_data/test_site/files/images/python-powered-h-50x65.png differ
diff --git a/tests/sample_data/test_site/files/images/scrapy.png b/tests/sample_data/test_site/files/images/scrapy.png
new file mode 100644
Binary files /dev/null and b/tests/sample_data/test_site/files/images/scrapy.png differ
diff --git a/tests/test_pipeline_crawl.py b/tests/test_pipeline_crawl.py
new file mode 100644
--- /dev/null
+++ b/tests/test_pipeline_crawl.py
@@ -0,0 +1,182 @@
+# -*- coding: utf-8 -*-
+import os
+import shutil
+
+from testfixtures import LogCapture
+from twisted.internet import defer
+from twisted.trial.unittest import TestCase
+from w3lib.url import add_or_replace_parameter
+
+from scrapy.crawler import CrawlerRunner
+from scrapy import signals
+from tests.mockserver import MockServer
+from tests.spiders import SimpleSpider
+
+
+class MediaDownloadSpider(SimpleSpider):
+    name = 'mediadownload'
+
+    def _process_url(self, url):
+        return url
+
+    def parse(self, response):
+        self.logger.info(response.headers)
+        self.logger.info(response.text)
+        item = {
+            self.media_key: [],
+            self.media_urls_key: [
+                self._process_url(response.urljoin(href))
+                    for href in response.xpath('''
+                        //table[thead/tr/th="Filename"]
+                            /tbody//a/@href
+                        ''').extract()],
+        }
+        yield item
+
+
+class BrokenLinksMediaDownloadSpider(MediaDownloadSpider):
+    name = 'brokenmedia'
+
+    def _process_url(self, url):
+        return url + '.foo'
+
+
+class RedirectedMediaDownloadSpider(MediaDownloadSpider):
+    name = 'redirectedmedia'
+
+    def _process_url(self, url):
+        return add_or_replace_parameter(
+                    'http://localhost:8998/redirect-to',
+                    'goto', url)
+
+
+class FileDownloadCrawlTestCase(TestCase):
+    pipeline_class = 'scrapy.pipelines.files.FilesPipeline'
+    store_setting_key = 'FILES_STORE'
+    media_key = 'files'
+    media_urls_key = 'file_urls'
+    expected_checksums = set([
+        '5547178b89448faf0015a13f904c936e',
+        'c2281c83670e31d8aaab7cb642b824db',
+        'ed3f6538dc15d4d9179dae57319edc5f'])
+
+    def setUp(self):
+        self.mockserver = MockServer()
+        self.mockserver.__enter__()
+
+        # prepare a directory for storing files
+        self.tmpmediastore = self.mktemp()
+        os.mkdir(self.tmpmediastore)
+        self.settings = {
+            'ITEM_PIPELINES': {self.pipeline_class: 1},
+            self.store_setting_key: self.tmpmediastore,
+        }
+        self.runner = CrawlerRunner(self.settings)
+        self.items = []
+
+    def tearDown(self):
+        shutil.rmtree(self.tmpmediastore)
+        self.items = []
+        self.mockserver.__exit__(None, None, None)
+
+    def _on_item_scraped(self, item):
+        self.items.append(item)
+
+    def _create_crawler(self, spider_class, **kwargs):
+        crawler = self.runner.create_crawler(spider_class, **kwargs)
+        crawler.signals.connect(self._on_item_scraped, signals.item_scraped)
+        return crawler
+
+    def _assert_files_downloaded(self, items, logs):
+        self.assertEqual(len(items), 1)
+        self.assertIn(self.media_key, items[0])
+
+        # check that logs show the expected number of successful file downloads
+        file_dl_success = 'File (downloaded): Downloaded file from'
+        self.assertEqual(logs.count(file_dl_success), 3)
+
+        # check that the images/files checksums are what we know they should be
+        if self.expected_checksums is not None:
+            checksums = set(
+                i['checksum']
+                    for item in items
+                        for i in item[self.media_key])
+            self.assertEqual(checksums, self.expected_checksums)
+
+        # check that the image files where actually written to the media store
+        for item in items:
+            for i in item[self.media_key]:
+                self.assertTrue(
+                    os.path.exists(
+                        os.path.join(self.tmpmediastore, i['path'])))
+
+    def _assert_files_download_failure(self, crawler, items, code, logs):
+
+        # check that the item does NOT have the "images/files" field populated
+        self.assertEqual(len(items), 1)
+        self.assertIn(self.media_key, items[0])
+        self.assertFalse(items[0][self.media_key])
+
+        # check that there was 1 successful fetch and 3 other responses with non-200 code
+        self.assertEqual(crawler.stats.get_value('downloader/request_method_count/GET'), 4)
+        self.assertEqual(crawler.stats.get_value('downloader/response_count'), 4)
+        self.assertEqual(crawler.stats.get_value('downloader/response_status_count/200'), 1)
+        self.assertEqual(crawler.stats.get_value('downloader/response_status_count/%d' % code), 3)
+
+        # check that logs do show the failure on the file downloads
+        file_dl_failure = 'File (code: %d): Error downloading file from' % code
+        self.assertEqual(logs.count(file_dl_failure), 3)
+
+        # check that no files were written to the media store
+        self.assertEqual(os.listdir(self.tmpmediastore), [])
+
+    @defer.inlineCallbacks
+    def test_download_media(self):
+        crawler = self._create_crawler(MediaDownloadSpider)
+        with LogCapture() as log:
+            yield crawler.crawl("http://localhost:8998/files/images/",
+                media_key=self.media_key,
+                media_urls_key=self.media_urls_key)
+        self._assert_files_downloaded(self.items, str(log))
+
+    @defer.inlineCallbacks
+    def test_download_media_wrong_urls(self):
+        crawler = self._create_crawler(BrokenLinksMediaDownloadSpider)
+        with LogCapture() as log:
+            yield crawler.crawl("http://localhost:8998/files/images/",
+                media_key=self.media_key,
+                media_urls_key=self.media_urls_key)
+        self._assert_files_download_failure(crawler, self.items, 404, str(log))
+
+    @defer.inlineCallbacks
+    def test_download_media_redirected_default_failure(self):
+        crawler = self._create_crawler(RedirectedMediaDownloadSpider)
+        with LogCapture() as log:
+            yield crawler.crawl("http://localhost:8998/files/images/",
+                media_key=self.media_key,
+                media_urls_key=self.media_urls_key)
+        self._assert_files_download_failure(crawler, self.items, 302, str(log))
+
+    @defer.inlineCallbacks
+    def test_download_media_redirected_allowed(self):
+        settings = dict(self.settings)
+        settings.update({'MEDIA_ALLOW_REDIRECTS': True})
+        self.runner = CrawlerRunner(settings)
+
+        crawler = self._create_crawler(RedirectedMediaDownloadSpider)
+        with LogCapture() as log:
+            yield crawler.crawl("http://localhost:8998/files/images/",
+                media_key=self.media_key,
+                media_urls_key=self.media_urls_key)
+        self._assert_files_downloaded(self.items, str(log))
+        self.assertEqual(crawler.stats.get_value('downloader/response_status_count/302'), 3)
+
+
+class ImageDownloadCrawlTestCase(FileDownloadCrawlTestCase):
+    pipeline_class = 'scrapy.pipelines.images.ImagesPipeline'
+    store_setting_key = 'IMAGES_STORE'
+    media_key = 'images'
+    media_urls_key = 'image_urls'
+
+    # somehow checksums for images are different for Python 3.3
+    expected_checksums = None
diff --git a/tests/test_pipeline_media.py b/tests/test_pipeline_media.py
--- a/tests/test_pipeline_media.py
+++ b/tests/test_pipeline_media.py
@@ -6,6 +6,7 @@
 from twisted.internet.defer import Deferred, inlineCallbacks
 
 from scrapy.http import Request, Response
+from scrapy.settings import Settings
 from scrapy.spiders import Spider
 from scrapy.utils.request import request_fingerprint
 from scrapy.pipelines.media import MediaPipeline
@@ -22,10 +23,12 @@ def _mocked_download_func(request, info):
 class BaseMediaPipelineTestCase(unittest.TestCase):
 
     pipeline_class = MediaPipeline
+    settings = None
 
     def setUp(self):
         self.spider = Spider('media.com')
-        self.pipe = self.pipeline_class(download_func=_mocked_download_func)
+        self.pipe = self.pipeline_class(download_func=_mocked_download_func,
+                                        settings=Settings(self.settings))
         self.pipe.open_spider(self.spider)
         self.info = self.pipe.spiderinfo
 
@@ -82,6 +85,11 @@ def test_default_process_item(self):
         new_item = yield self.pipe.process_item(item, self.spider)
         assert new_item is item
 
+    def test_modify_media_request(self):
+        request = Request('http://url')
+        self.pipe._modify_media_request(request)
+        assert request.meta == {'handle_httpstatus_all': True}
+
 
 class MockedMediaPipeline(MediaPipeline):
 
@@ -249,3 +257,61 @@ def test_use_media_to_download_result(self):
         self.assertEqual(new_item['results'], [(True, 'ITSME')])
         self.assertEqual(self.pipe._mockcalled, \
                 ['get_media_requests', 'media_to_download', 'item_completed'])
+
+
+class MediaPipelineAllowRedirectSettingsTestCase(unittest.TestCase):
+
+    def _assert_request_no3xx(self, pipeline_class, settings):
+        pipe = pipeline_class(settings=Settings(settings))
+        request = Request('http://url')
+        pipe._modify_media_request(request)
+
+        self.assertIn('handle_httpstatus_list', request.meta)
+        for status, check in [
+                (200, True),
+
+                # These are the status codes we want
+                # the downloader to handle itself
+                (301, False),
+                (302, False),
+                (302, False),
+                (307, False),
+                (308, False),
+
+                # we still want to get 4xx and 5xx
+                (400, True),
+                (404, True),
+                (500, True)]:
+            if check:
+                self.assertIn(status, request.meta['handle_httpstatus_list'])
+            else:
+                self.assertNotIn(status, request.meta['handle_httpstatus_list'])
+
+    def test_standard_setting(self):
+        self._assert_request_no3xx(
+            MediaPipeline,
+            {
+                'MEDIA_ALLOW_REDIRECTS': True
+            })
+
+    def test_subclass_standard_setting(self):
+
+        class UserDefinedPipeline(MediaPipeline):
+            pass
+
+        self._assert_request_no3xx(
+            UserDefinedPipeline,
+            {
+                'MEDIA_ALLOW_REDIRECTS': True
+            })
+
+    def test_subclass_specific_setting(self):
+
+        class UserDefinedPipeline(MediaPipeline):
+            pass
+
+        self._assert_request_no3xx(
+            UserDefinedPipeline,
+            {
+                'USERDEFINEDPIPELINE_MEDIA_ALLOW_REDIRECTS': True
+            })