frontera (pypi): A scalable frontier for web crawlers
Keywords: crawler