Add tests for OffsiteMiddleware() + use re.escape() in domains regexp
AjaxCrawlableMiddleware
add an undocumented setting for lookup body size
Fail-fast path.

For me middleware now can process about 2-3k ajax crawlable pages/sec and 50k+ regular pages/sec (if they don’t contain «fragment» or «content» words).
extra test
AjaxCrawlableMiddleware in Broad Crawl docs
kill AjaxCrawlableMiddlewar.enabled_setting
add a remark about 1%
DOWNLOAD_DELAY docs clarification:

* delay is enforced per website, not per spider;
* document download_delay attribute (it was previously documented only in FAQ about 999 error codes);
* document how CONCURRENT_REQUESTS_PER_IP affects download delays.
Clarify return value from extract_links
Promote startup info on settings and middleware to INFO level
Merge pull request #520 from redapple/startupinfo

Promote startup info on settings and middleware to INFO level
warn on XPathItemLoader subclassing
documentation code example correction corrections per pablohoffman
support showing subclassing deprecation warning only once
revert warning filter on deprecated class instantiation
only warn on direct subclasses of the deprecated class
split subclassing tests and default to warning only once per deprecated class
allow deprecation of a class with custom metaclass
allow overriding old and new class paths