make Crawler.start() return a deferred that is fired when the crawl is finished
initial version of crawl tests using a mock HTTP server (in separate process). This can also be used to benchmark scrapy performance, although a script (specially suited for that task) would be more convenient
replaced use of depricated module scrapy.settings with the method get_project_settings()
Merge pull request #274 from brunsgaard/master

Updated settings import in contrib/feedexport.py class S3FeedStorage
Revert "replaced use of depricated module scrapy.settings with the method get_project_settings()"

This reverts commit 1b4d14c8f635b28d5f72d37f924c4e71d71520ca.

Calling `get_project_settings()` generates a new independent Settings
object that doesn't contain the overrides passed by command line
arguments, for example.

Proper port would require implementing the from_crawler() class method
and making sure the settings object is passed to all internal objects
(probably breaking some minor backwards compatibility).
Spider Arguments support for parse command and test
Merge pull request #277 from nramirezuy/cmd-parse-args

Spider Arguments support for parse command
fix inaccurate downloader middleware documentation. refs #280