GitHub repository: maxdata/common_crawl_with_scrapy
Primary language: Python
Description: Parsing Huge Web Archive files from Common Crawl data index to fetch any required domain's data concurrently with Python and Scrapy.