add an undocumented setting for lookup body size
Fail-fast path.

For me middleware now can process about 2-3k ajax crawlable pages/sec and 50k+ regular pages/sec (if they don’t contain «fragment» or «content» words).
extra test
AjaxCrawlableMiddleware in Broad Crawl docs