scrapy/scrapyd — pull request #17 by xaqq (2013-05-06 12:41:39+00:00)
Title: check if a spider exists before schedule it (with sqlite cache)

I pulled https://github.com/scrapy/scrapyd/pull/8 to my local repo and added sqlite caching like pablohoffman suggested. I ran some performance tests using apache bench, and its ok.