crawl-frontier (pypi): A flexible frontier for web crawlers
Keywords: crawler|frontier|scrapy|web|requests