AjaxCrawlableMiddleware
add an undocumented setting for lookup body size
Fail-fast path.

For me middleware now can process about 2-3k ajax crawlable pages/sec and 50k+ regular pages/sec (if they don’t contain «fragment» or «content» words).
extra test
AjaxCrawlableMiddleware in Broad Crawl docs
kill AjaxCrawlableMiddlewar.enabled_setting
add a remark about 1%
Merge pull request #343 from kmike/ajax-crawlable

[MRG] AjaxCrawlableMiddleware