scrapy/scrapy — pull request #553 by redapple (2014-01-21 17:36:12+00:00)
Title: DupeFilter: add settings for verbose logging and filtered requests stats

Too many duplicate requests is often not a good sign (buggy spider or misbehaving website)

Here I propose adding 2 settings for the default `RFPDupeFilter`class:

* `DUPEFILTER_VERBOSE_LOG`: to log all duplicate requests and not only the first one
* `DUPEFILTER_VERBOSE_STATS`: to add a stats counter for filtered duplicate requests

Question: is `spider.crawler.stats` the proper way to access the crawler's stats collector? is it safe?