GitHub repository: mayli/scrapy-deltafetch-redis
Primary language: Python
Description: Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls