Repository: scrapy/scrapy

Issue:
More possibilities to cancel downloads inside HTTP downloader handler
Hello,

Currently, a download is cancelled in the HTTP downloader if the expected size of the response is greater than the `DOWNLOAD_MAXSIZE` setting. However, there is no way to cancel a download after the headers are received and before the body is downloaded based on other conditions, such as the value of a specific header, and I see some cases where it could be useful.  

For instance, one cannot rely on LinkExtractor to filter out media links (images, videos, etc...) since a link without a media extension could still be a media. Thus, by having a way to obtain the headers of a response when they are received, one can check the value of the `Content-Type` header and trigger the cancellation of the download if necessary.  

I thought about an implementation and came up with [this](https://github.com/Djayb6/scrapy/commit/f8080748db04daf922f48650d3396ac7732d8bde) . The main idea is when the headers of the response are received, the downloader handler sends a signal `headers_received` with the `txresponse` and the `request`, and cancels the download based on the return value of the first receiver's callback. It is a quick hack but is not very intrusive. The main drawback is that when connecting to this signal in a spider, one must specifies `sender=Any` as the crawler's signal manager is not available in the downloader handler.

I'm waiting for your remarks and ideas.

Many thanks,

JB.



Discussion:
Yeah, we need a public & documented way to cancel downloads. That's not good users can't implement an alternative to DOWNLOAD_WARNSIZE / DOWNLOAD_MAXSIZE themselves.

Signals look like a reasonable way to do that. The current signal dispatching implementation is slow though, so we need to check that this new signal won't slow down anything. I'm not sure it is good to expose twisted response directly; it is better to design Scrapy API in a way Twisted is hidden. `sender=Any` is not good because there may be several spiders in the same process, and user may want to filter out requests from one spider but not from another.

IMO, triggering signals on every http request is very expensive. It would be nice to have an option for disabling that.

@kmike @sibiryakov I took into account what you said about signal dispatching and I came up with [this](https://github.com/scrapy/scrapy/compare/master...Djayb6:cancel-request-downloaderhandler)

This approach does not use signal dispatching. Instead, an optional callback `on_headers_received` has been added to `scrapy.http.Request` that is called in the dowloader handler if the callback was set. 
A `scrapy.http.Response` is created with the url and the headers in order to hide Twisted's API, and is passed to the callback along with the original `request`. Then as before if the callback returns `True` the request is cancelled.  
I think this approach is way less expensive and more user-friendly. What do you think ?  

JB

Hey @Djayb6,

With `on_headers_received` callback it becomes hard to implement reusable extensions which use this feature - e.g. an extension to limit max response size or an extension to filter out responses based on their content-type. I'd love to see an extension point which allows to implement e.g. `DOWNLOAD_WARNSIZE` feature without hardcoding it in a downloader handler.

I think signals are preferrable, but we should measure their impact. I'd say a single signal per response should be fine; a signal per downloaded chunk is much more risky. But maybe the slowness can be fixed by moving to a different signal implementation (see #8), or maybe overhead is not that big.

@kmike see my new implementation [here](https://github.com/scrapy/scrapy/compare/master...Djayb6:headers-received-event) . It is based on signals (one signal per download), hides Twisted API, and it is possible to disable the signal based on a setting (for @sibiryakov).  
You can find an extension that cancels downloads based on pre-defined conditions [here](https://github.com/Djayb6/received_headers_spider/blob/master/received_headers_spider/headersreceived.py)

FYI, regarding the signal dispatching implementation, the celery project extracted django pydispatcher's fork and [modified it](https://github.com/celery/celery/tree/master/celery/utils/dispatch) for its needs.  
JB

//cc @elacuesta - is it fixed by the signals you introduced?
Hey @kmike. Interesting, I wasn't aware of this thread, thanks for pointing this out. #4205 adds a way to stop downloads, but this issue still seems valid to me because [headers are not sent as arguments in the signal](https://github.com/scrapy/scrapy/blob/2.3.0/scrapy/core/downloader/handlers/http11.py#L513-L515). They are available though, it would be a matter of doing something like:
```diff
diff --git scrapy/core/downloader/handlers/http11.py scrapy/core/downloader/handlers/http11.py
index fb04d1fb..c419b195 100644
--- scrapy/core/downloader/handlers/http11.py
+++ scrapy/core/downloader/handlers/http11.py
@@ -513,6 +513,7 @@ class _ResponseReader(protocol.Protocol):
             data=bodyBytes,
             request=self._request,
             spider=self._crawler.spider,
+            headers=Headers(self._txresponse.headers.getAllRawHeaders()),
         )
         for handler, result in bytes_received_result:
             if isinstance(result, Failure) and isinstance(result.value, StopDownload):
```
Or adding a new signal as originally proposed, or both; I don't really have a strong preference either way. A good thing is that the sender issue should not be a problem anymore, now that the download handler has access to the crawler instance since #4205.

Resolving patch:
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -382,6 +382,29 @@ def _cb_latency(self, result, request, start_time):
         return result
 
     def _cb_bodyready(self, txresponse, request):
+        headers_received_result = self._crawler.signals.send_catch_log(
+            signal=signals.headers_received,
+            headers=Headers(txresponse.headers.getAllRawHeaders()),
+            body_length=txresponse.length,
+            request=request,
+            spider=self._crawler.spider,
+        )
+        for handler, result in headers_received_result:
+            if isinstance(result, Failure) and isinstance(result.value, StopDownload):
+                logger.debug("Download stopped for %(request)s from signal handler %(handler)s",
+                             {"request": request, "handler": handler.__qualname__})
+                txresponse._transport.stopProducing()
+                with suppress(AttributeError):
+                    txresponse._transport._producer.loseConnection()
+                return {
+                    "txresponse": txresponse,
+                    "body": b"",
+                    "flags": ["download_stopped"],
+                    "certificate": None,
+                    "ip_address": None,
+                    "failure": result if result.value.fail else None,
+                }
+
         # deliverBody hangs for responses without body
         if txresponse.length == 0:
             return {
@@ -529,6 +552,7 @@ def dataReceived(self, bodyBytes):
             if isinstance(result, Failure) and isinstance(result.value, StopDownload):
                 logger.debug("Download stopped for %(request)s from signal handler %(handler)s",
                              {"request": self._request, "handler": handler.__qualname__})
+                self.transport.stopProducing()
                 self.transport._producer.loseConnection()
                 failure = result if result.value.fail else None
                 self._finish_response(flags=["download_stopped"], failure=failure)
diff --git a/scrapy/signals.py b/scrapy/signals.py
--- a/scrapy/signals.py
+++ b/scrapy/signals.py
@@ -17,6 +17,7 @@
 request_left_downloader = object()
 response_received = object()
 response_downloaded = object()
+headers_received = object()
 bytes_received = object()
 item_scraped = object()
 item_dropped = object()


Test patch:
diff --git a/tests/spiders.py b/tests/spiders.py
--- a/tests/spiders.py
+++ b/tests/spiders.py
@@ -390,3 +390,32 @@ class BytesReceivedErrbackSpider(BytesReceivedCallbackSpider):
     def bytes_received(self, data, request, spider):
         self.meta["bytes_received"] = data
         raise StopDownload(fail=True)
+
+
+class HeadersReceivedCallbackSpider(MetaSpider):
+
+    @classmethod
+    def from_crawler(cls, crawler, *args, **kwargs):
+        spider = super().from_crawler(crawler, *args, **kwargs)
+        crawler.signals.connect(spider.headers_received, signals.headers_received)
+        return spider
+
+    def start_requests(self):
+        yield Request(self.mockserver.url("/status"), errback=self.errback)
+
+    def parse(self, response):
+        self.meta["response"] = response
+
+    def errback(self, failure):
+        self.meta["failure"] = failure
+
+    def headers_received(self, headers, body_length, request, spider):
+        self.meta["headers_received"] = headers
+        raise StopDownload(fail=False)
+
+
+class HeadersReceivedErrbackSpider(HeadersReceivedCallbackSpider):
+
+    def headers_received(self, headers, body_length, request, spider):
+        self.meta["headers_received"] = headers
+        raise StopDownload(fail=True)
diff --git a/tests/test_crawl.py b/tests/test_crawl.py
--- a/tests/test_crawl.py
+++ b/tests/test_crawl.py
@@ -35,6 +35,8 @@
     DelaySpider,
     DuplicateStartRequestsSpider,
     FollowAllSpider,
+    HeadersReceivedCallbackSpider,
+    HeadersReceivedErrbackSpider,
     SimpleSpider,
     SingleRequestSpider,
 )
@@ -496,7 +498,7 @@ def test_dns_server_ip_address(self):
         self.assertEqual(str(ip_address), gethostbyname(expected_netloc))
 
     @defer.inlineCallbacks
-    def test_stop_download_callback(self):
+    def test_bytes_received_stop_download_callback(self):
         crawler = self.runner.create_crawler(BytesReceivedCallbackSpider)
         yield crawler.crawl(mockserver=self.mockserver)
         self.assertIsNone(crawler.spider.meta.get("failure"))
@@ -505,7 +507,7 @@ def test_stop_download_callback(self):
         self.assertLess(len(crawler.spider.meta["response"].body), crawler.spider.full_response_length)
 
     @defer.inlineCallbacks
-    def test_stop_download_errback(self):
+    def test_bytes_received_stop_download_errback(self):
         crawler = self.runner.create_crawler(BytesReceivedErrbackSpider)
         yield crawler.crawl(mockserver=self.mockserver)
         self.assertIsNone(crawler.spider.meta.get("response"))
@@ -518,3 +520,23 @@ def test_stop_download_errback(self):
         self.assertLess(
             len(crawler.spider.meta["failure"].value.response.body),
             crawler.spider.full_response_length)
+
+    @defer.inlineCallbacks
+    def test_headers_received_stop_download_callback(self):
+        crawler = self.runner.create_crawler(HeadersReceivedCallbackSpider)
+        yield crawler.crawl(mockserver=self.mockserver)
+        self.assertIsNone(crawler.spider.meta.get("failure"))
+        self.assertIsInstance(crawler.spider.meta["response"], Response)
+        self.assertEqual(crawler.spider.meta["response"].headers, crawler.spider.meta.get("headers_received"))
+
+    @defer.inlineCallbacks
+    def test_headers_received_stop_download_errback(self):
+        crawler = self.runner.create_crawler(HeadersReceivedErrbackSpider)
+        yield crawler.crawl(mockserver=self.mockserver)
+        self.assertIsNone(crawler.spider.meta.get("response"))
+        self.assertIsInstance(crawler.spider.meta["failure"], Failure)
+        self.assertIsInstance(crawler.spider.meta["failure"].value, StopDownload)
+        self.assertIsInstance(crawler.spider.meta["failure"].value.response, Response)
+        self.assertEqual(
+            crawler.spider.meta["failure"].value.response.headers,
+            crawler.spider.meta.get("headers_received"))
diff --git a/tests/test_engine.py b/tests/test_engine.py
--- a/tests/test_engine.py
+++ b/tests/test_engine.py
@@ -19,14 +19,12 @@
 import attr
 from itemadapter import ItemAdapter
 from pydispatch import dispatcher
-from testfixtures import LogCapture
 from twisted.internet import defer, reactor
 from twisted.trial import unittest
 from twisted.web import server, static, util
 
 from scrapy import signals
 from scrapy.core.engine import ExecutionEngine
-from scrapy.exceptions import StopDownload
 from scrapy.http import Request
 from scrapy.item import Item, Field
 from scrapy.linkextractors import LinkExtractor
@@ -143,6 +141,7 @@ def __init__(self, spider_class):
         self.reqreached = []
         self.itemerror = []
         self.itemresp = []
+        self.headers = {}
         self.bytes = defaultdict(lambda: list())
         self.signals_caught = {}
         self.spider_class = spider_class
@@ -165,6 +164,7 @@ def run(self):
         self.crawler = get_crawler(self.spider_class)
         self.crawler.signals.connect(self.item_scraped, signals.item_scraped)
         self.crawler.signals.connect(self.item_error, signals.item_error)
+        self.crawler.signals.connect(self.headers_received, signals.headers_received)
         self.crawler.signals.connect(self.bytes_received, signals.bytes_received)
         self.crawler.signals.connect(self.request_scheduled, signals.request_scheduled)
         self.crawler.signals.connect(self.request_dropped, signals.request_dropped)
@@ -183,6 +183,7 @@ def stop(self):
             if not name.startswith('_'):
                 disconnect_all(signal)
         self.deferred.callback(None)
+        return self.crawler.stop()
 
     def geturl(self, path):
         return f"http://localhost:{self.portno}{path}"
@@ -197,6 +198,9 @@ def item_error(self, item, response, spider, failure):
     def item_scraped(self, item, spider, response):
         self.itemresp.append((item, response))
 
+    def headers_received(self, headers, body_length, request, spider):
+        self.headers[request] = headers
+
     def bytes_received(self, data, request, spider):
         self.bytes[request].append(data)
 
@@ -220,18 +224,7 @@ def record_signal(self, *args, **kwargs):
         self.signals_caught[sig] = signalargs
 
 
-class StopDownloadCrawlerRun(CrawlerRun):
-    """
-    Make sure raising the StopDownload exception stops the download of the response body
-    """
-
-    def bytes_received(self, data, request, spider):
-        super().bytes_received(data, request, spider)
-        raise StopDownload(fail=False)
-
-
 class EngineTest(unittest.TestCase):
-
     @defer.inlineCallbacks
     def test_crawler(self):
 
@@ -241,8 +234,8 @@ def test_crawler(self):
             self.run = CrawlerRun(spider)
             yield self.run.run()
             self._assert_visited_urls()
-            self._assert_scheduled_requests(urls_to_visit=9)
-            self._assert_downloaded_responses()
+            self._assert_scheduled_requests(count=9)
+            self._assert_downloaded_responses(count=9)
             self._assert_scraped_items()
             self._assert_signals_caught()
             self._assert_bytes_received()
@@ -251,7 +244,7 @@ def test_crawler(self):
     def test_crawler_dupefilter(self):
         self.run = CrawlerRun(TestDupeFilterSpider)
         yield self.run.run()
-        self._assert_scheduled_requests(urls_to_visit=8)
+        self._assert_scheduled_requests(count=8)
         self._assert_dropped_requests()
 
     @defer.inlineCallbacks
@@ -267,8 +260,8 @@ def _assert_visited_urls(self):
         urls_expected = {self.run.geturl(p) for p in must_be_visited}
         assert urls_expected <= urls_visited, f"URLs not visited: {list(urls_expected - urls_visited)}"
 
-    def _assert_scheduled_requests(self, urls_to_visit=None):
-        self.assertEqual(urls_to_visit, len(self.run.reqplug))
+    def _assert_scheduled_requests(self, count=None):
+        self.assertEqual(count, len(self.run.reqplug))
 
         paths_expected = ['/item999.html', '/item2.html', '/item1.html']
 
@@ -286,10 +279,10 @@ def _assert_scheduled_requests(self, urls_to_visit=None):
     def _assert_dropped_requests(self):
         self.assertEqual(len(self.run.reqdropped), 1)
 
-    def _assert_downloaded_responses(self):
+    def _assert_downloaded_responses(self, count):
         # response tests
-        self.assertEqual(9, len(self.run.respplug))
-        self.assertEqual(9, len(self.run.reqreached))
+        self.assertEqual(count, len(self.run.respplug))
+        self.assertEqual(count, len(self.run.reqreached))
 
         for response, _ in self.run.respplug:
             if self.run.getpath(response.url) == '/item999.html':
@@ -323,6 +316,13 @@ def _assert_scraped_items(self):
                 self.assertEqual('Item 2 name', item['name'])
                 self.assertEqual('200', item['price'])
 
+    def _assert_headers_received(self):
+        for headers in self.run.headers.values():
+            self.assertIn(b"Server", headers)
+            self.assertIn(b"TwistedWeb", headers[b"Server"])
+            self.assertIn(b"Date", headers)
+            self.assertIn(b"Content-Type", headers)
+
     def _assert_bytes_received(self):
         self.assertEqual(9, len(self.run.bytes))
         for request, data in self.run.bytes.items():
@@ -371,6 +371,7 @@ def _assert_signals_caught(self):
         assert signals.spider_opened in self.run.signals_caught
         assert signals.spider_idle in self.run.signals_caught
         assert signals.spider_closed in self.run.signals_caught
+        assert signals.headers_received in self.run.signals_caught
 
         self.assertEqual({'spider': self.run.spider},
                          self.run.signals_caught[signals.spider_opened])
@@ -403,48 +404,6 @@ def test_close_engine_spiders_downloader(self):
         self.assertEqual(len(e.open_spiders), 0)
 
 
-class StopDownloadEngineTest(EngineTest):
-
-    @defer.inlineCallbacks
-    def test_crawler(self):
-        for spider in TestSpider, DictItemsSpider:
-            self.run = StopDownloadCrawlerRun(spider)
-            with LogCapture() as log:
-                yield self.run.run()
-                log.check_present(("scrapy.core.downloader.handlers.http11",
-                                   "DEBUG",
-                                   f"Download stopped for <GET http://localhost:{self.run.portno}/redirected> "
-                                   "from signal handler"
-                                   " StopDownloadCrawlerRun.bytes_received"))
-                log.check_present(("scrapy.core.downloader.handlers.http11",
-                                   "DEBUG",
-                                   f"Download stopped for <GET http://localhost:{self.run.portno}/> "
-                                   "from signal handler"
-                                   " StopDownloadCrawlerRun.bytes_received"))
-                log.check_present(("scrapy.core.downloader.handlers.http11",
-                                   "DEBUG",
-                                   f"Download stopped for <GET http://localhost:{self.run.portno}/numbers> "
-                                   "from signal handler"
-                                   " StopDownloadCrawlerRun.bytes_received"))
-            self._assert_visited_urls()
-            self._assert_scheduled_requests(urls_to_visit=9)
-            self._assert_downloaded_responses()
-            self._assert_signals_caught()
-            self._assert_bytes_received()
-
-    def _assert_bytes_received(self):
-        self.assertEqual(9, len(self.run.bytes))
-        for request, data in self.run.bytes.items():
-            joined_data = b"".join(data)
-            self.assertTrue(len(data) == 1)  # signal was fired only once
-            if self.run.getpath(request.url) == "/numbers":
-                # Received bytes are not the complete response. The exact amount depends
-                # on the buffer size, which can vary, so we only check that the amount
-                # of received bytes is strictly less than the full response.
-                numbers = [str(x).encode("utf8") for x in range(2**18)]
-                self.assertTrue(len(joined_data) < len(b"".join(numbers)))
-
-
 if __name__ == "__main__":
     if len(sys.argv) > 1 and sys.argv[1] == 'runserver':
         start_test_site(debug=True)
diff --git a/tests/test_engine_stop_download_bytes.py b/tests/test_engine_stop_download_bytes.py
new file mode 100644
--- /dev/null
+++ b/tests/test_engine_stop_download_bytes.py
@@ -0,0 +1,60 @@
+from testfixtures import LogCapture
+from twisted.internet import defer
+
+from scrapy.exceptions import StopDownload
+
+from tests.test_engine import (
+    AttrsItemsSpider,
+    DataClassItemsSpider,
+    DictItemsSpider,
+    TestSpider,
+    CrawlerRun,
+    EngineTest,
+)
+
+
+class BytesReceivedCrawlerRun(CrawlerRun):
+    def bytes_received(self, data, request, spider):
+        super().bytes_received(data, request, spider)
+        raise StopDownload(fail=False)
+
+
+class BytesReceivedEngineTest(EngineTest):
+    @defer.inlineCallbacks
+    def test_crawler(self):
+        for spider in (TestSpider, DictItemsSpider, AttrsItemsSpider, DataClassItemsSpider):
+            if spider is None:
+                continue
+            self.run = BytesReceivedCrawlerRun(spider)
+            with LogCapture() as log:
+                yield self.run.run()
+                log.check_present(("scrapy.core.downloader.handlers.http11",
+                                   "DEBUG",
+                                   f"Download stopped for <GET http://localhost:{self.run.portno}/redirected> "
+                                   "from signal handler BytesReceivedCrawlerRun.bytes_received"))
+                log.check_present(("scrapy.core.downloader.handlers.http11",
+                                   "DEBUG",
+                                   f"Download stopped for <GET http://localhost:{self.run.portno}/> "
+                                   "from signal handler BytesReceivedCrawlerRun.bytes_received"))
+                log.check_present(("scrapy.core.downloader.handlers.http11",
+                                   "DEBUG",
+                                   f"Download stopped for <GET http://localhost:{self.run.portno}/numbers> "
+                                   "from signal handler BytesReceivedCrawlerRun.bytes_received"))
+            self._assert_visited_urls()
+            self._assert_scheduled_requests(count=9)
+            self._assert_downloaded_responses(count=9)
+            self._assert_signals_caught()
+            self._assert_headers_received()
+            self._assert_bytes_received()
+
+    def _assert_bytes_received(self):
+        self.assertEqual(9, len(self.run.bytes))
+        for request, data in self.run.bytes.items():
+            joined_data = b"".join(data)
+            self.assertTrue(len(data) == 1)  # signal was fired only once
+            if self.run.getpath(request.url) == "/numbers":
+                # Received bytes are not the complete response. The exact amount depends
+                # on the buffer size, which can vary, so we only check that the amount
+                # of received bytes is strictly less than the full response.
+                numbers = [str(x).encode("utf8") for x in range(2**18)]
+                self.assertTrue(len(joined_data) < len(b"".join(numbers)))
diff --git a/tests/test_engine_stop_download_headers.py b/tests/test_engine_stop_download_headers.py
new file mode 100644
--- /dev/null
+++ b/tests/test_engine_stop_download_headers.py
@@ -0,0 +1,56 @@
+from testfixtures import LogCapture
+from twisted.internet import defer
+
+from scrapy.exceptions import StopDownload
+
+from tests.test_engine import (
+    AttrsItemsSpider,
+    DataClassItemsSpider,
+    DictItemsSpider,
+    TestSpider,
+    CrawlerRun,
+    EngineTest,
+)
+
+
+class HeadersReceivedCrawlerRun(CrawlerRun):
+    def headers_received(self, headers, body_length, request, spider):
+        super().headers_received(headers, body_length, request, spider)
+        raise StopDownload(fail=False)
+
+
+class HeadersReceivedEngineTest(EngineTest):
+    @defer.inlineCallbacks
+    def test_crawler(self):
+        for spider in (TestSpider, DictItemsSpider, AttrsItemsSpider, DataClassItemsSpider):
+            if spider is None:
+                continue
+            self.run = HeadersReceivedCrawlerRun(spider)
+            with LogCapture() as log:
+                yield self.run.run()
+                log.check_present(("scrapy.core.downloader.handlers.http11",
+                                   "DEBUG",
+                                   f"Download stopped for <GET http://localhost:{self.run.portno}/redirected> from"
+                                   " signal handler HeadersReceivedCrawlerRun.headers_received"))
+                log.check_present(("scrapy.core.downloader.handlers.http11",
+                                   "DEBUG",
+                                   f"Download stopped for <GET http://localhost:{self.run.portno}/> from signal"
+                                   " handler HeadersReceivedCrawlerRun.headers_received"))
+                log.check_present(("scrapy.core.downloader.handlers.http11",
+                                   "DEBUG",
+                                   f"Download stopped for <GET http://localhost:{self.run.portno}/numbers> from"
+                                   " signal handler HeadersReceivedCrawlerRun.headers_received"))
+            self._assert_visited_urls()
+            self._assert_downloaded_responses(count=6)
+            self._assert_signals_caught()
+            self._assert_bytes_received()
+            self._assert_headers_received()
+
+    def _assert_bytes_received(self):
+        self.assertEqual(0, len(self.run.bytes))
+
+    def _assert_visited_urls(self):
+        must_be_visited = ["/", "/redirect", "/redirected"]
+        urls_visited = {rp[0].url for rp in self.run.respplug}
+        urls_expected = {self.run.geturl(p) for p in must_be_visited}
+        assert urls_expected <= urls_visited, f"URLs not visited: {list(urls_expected - urls_visited)}"