scrapy-redis-expiredupefilter (pypi): A distributed crawler component based on scrapy_redis which can specify the expiration time of fingerprints in dupefilter.
Keywords: scrapy