GitHub repository: scrapinghub/crawl-frontier
Primary language: Python
Description: A flexible frontier for web crawlers