Repository: scrapy/scrapy

Issue:
HttpErrorMiddleware log level should be at least INFO, not DEBUG
I want to spot immediately if one of my spiders get 403 or 404 response codes. But the current implementation of HttpErrorMiddleware, sets log as DEBUG... debug?! It really doesn't matter if a spider is not able to scrape a page? It should matter! It should be a very important log to read!



Discussion:
Are you looking at the log while the spider is running?, because at the end of the spider it prints some stats and there is a count of the status codes.

Yes, I'm reading the console output while the spider is running

Then you are debugging and log DEBUG is right. You can also implement a middleware that mails you when one of these happen.

Not a long time ago these errors were not logged at all :)

I'm +0 to change log level to INFO or WARNING because 403/404 errors are more important than HTTP 200 responses. These errors are not something a developer can reliably fix - a website can change, so it is important to look for these errors in production, not only during development.

Yes, I know that I can implement my own middlewares and this is cool, but I don't agree with you... yes, I'm currently debugging, but what does it mean? A 403 or 404 response deserves an higher log level importance like WARNING, even when an item is discarded due to a DropItem esception it gets logged as WARNING... so why an http error should be considered in a different way? This is at least incoherent... isn't it?

> It really doesn't matter if a spider is not able to scrape a page?

No, often it does not.. The internet is made of people and people make mistakes, so dead links will from time to time exists, and the crawler will follow em, it is not the end of the world. If you are scraping  30000 pages for products I really do not care if I get a couple of 40x. Also one should note that 40x is not mistakes. It is just scrapy and the websever communicating. But nothing unexpected is happening here.

scrapy: Hay webserver do you have this page?
webserver: no i dont (404)
scrapy Okay, so what about this one?
webserver: yep, here it is (200)

This flow is fine, it works exactly as expected, so no need to warn anybody in the generel case.
I can agree that INFO would seem more appropriate though. 

@daveoncode In regard to `DropItem` it is different. That is your program dropping the item. So that is something you have control over and coded your self. (internal vs external errors) but I also think that this one should use INFO to be honest ;)

@brunsgaard I agree with you ;)


Resolving patch:
diff --git a/scrapy/spidermiddlewares/httperror.py b/scrapy/spidermiddlewares/httperror.py
--- a/scrapy/spidermiddlewares/httperror.py
+++ b/scrapy/spidermiddlewares/httperror.py
@@ -46,7 +46,7 @@ def process_spider_input(self, response, spider):
 
     def process_spider_exception(self, response, exception, spider):
         if isinstance(exception, HttpError):
-            logger.debug(
+            logger.info(
                 "Ignoring response %(response)r: HTTP status code is not handled or not allowed",
                 {'response': response}, extra={'spider': spider},
             )


Test patch:
diff --git a/tests/test_spidermiddleware_httperror.py b/tests/test_spidermiddleware_httperror.py
--- a/tests/test_spidermiddleware_httperror.py
+++ b/tests/test_spidermiddleware_httperror.py
@@ -1,3 +1,4 @@
+import logging
 from unittest import TestCase
 
 from testfixtures import LogCapture
@@ -185,3 +186,29 @@ def test_logging(self):
         self.assertIn('Ignoring response <500', str(log))
         self.assertNotIn('Ignoring response <200', str(log))
         self.assertNotIn('Ignoring response <402', str(log))
+
+    @defer.inlineCallbacks
+    def test_logging_level(self):
+        # HttpError logs ignored responses with level INFO
+        crawler = get_crawler(_HttpErrorSpider)
+        with LogCapture(level=logging.INFO) as log:
+            yield crawler.crawl()
+        self.assertEqual(crawler.spider.parsed, {'200'})
+        self.assertEqual(crawler.spider.failed, {'404', '402', '500'})
+
+        self.assertIn('Ignoring response <402', str(log))
+        self.assertIn('Ignoring response <404', str(log))
+        self.assertIn('Ignoring response <500', str(log))
+        self.assertNotIn('Ignoring response <200', str(log))
+
+        # with level WARNING, we shouldn't capture anything from HttpError
+        crawler = get_crawler(_HttpErrorSpider)
+        with LogCapture(level=logging.WARNING) as log:
+            yield crawler.crawl()
+        self.assertEqual(crawler.spider.parsed, {'200'})
+        self.assertEqual(crawler.spider.failed, {'404', '402', '500'})
+
+        self.assertNotIn('Ignoring response <402', str(log))
+        self.assertNotIn('Ignoring response <404', str(log))
+        self.assertNotIn('Ignoring response <500', str(log))
+        self.assertNotIn('Ignoring response <200', str(log))