mozilla/spade — pull request #12 by samliu (2012-07-12 19:19:43+00:00)
Title: Bugfixing

* Fixed bug where sites containing two links to the same page emit two identical requests (thus saving duplicates)
* Replaced the default scrapy depth middleware with a modified one that allows JS and CSS to be saved 1 level deeper, so a page scanned at depth 2 might have a stylesheet on depth 3 but that will be saved with this modification.
* Modified the offsite middleware to allow remote JS and CSS because many sites use CDNs.