scrapy/scrapy — pull request #991 by adon-at-work (2014-12-23 11:04:27+00:00)
Title: dupefilter should record requests even with dont_filter

dupefilter must see and record all requests even with dont_filter, and
thus a subsequent duplicate request happened to have no dont_filter can
be filtered out.

Example: consider having some start_urls marked as dont_filter, any
self-referencing urls will not be de-duplicated.

Apparently this problem was introduced since Jun 2011 after a refactoring at
https://github.com/scrapy/scrapy/commit/03751749a80526f3d97f60f0f63892501ef5de19