final commit before rewriting this in Scrapy