scrapy-dynamic-spiders (pypi): Dynamically generate spider subclasses. Run crawls sequentially with crochet. Do both.
Keywords: scrapy|spiders|web-scraping