GitHub repository: thsheep/scrapy_redis_cluster
Primary language: Python
Created: 2018-05-23T02:55:56Z
Description: scrapy-redis的集群版，可以借助Redis集群实现海量网站的独立去重，避免单机内存不足的尴尬