GitHub repository: quixey/scrapy-cluster
Primary language: Python
Description: Forked for @dpb. This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.