mozilla/spade — pull request #14 by samliu (2012-07-18 23:23:08+00:00)
Title: Testing

* Tested all added middlewares (we only added spidermiddlewares) to make sure only the correct requests get filtered.
- Depth 2 requests are not filtered (linked from base 1, a.k.a base links)
- Depth 3 requests are filtered if they're html
- Depth 3 requests are not filtered if they're css/js
- Offsite requests are filtered

* Tested the spider's crawling mechanism to ensure
- The correct items are emitted
- Different ua strings are used correctly
- Reading from a text file works

* Tested model constraints