[MRG] AjaxCrawlableMiddleware

What do you think about adding support for https://developers.google.com/webmasters/ajax-crawling/docs/getting-started, part 3 ("Handle pages without hash fragments")? In this pull request it is implemented using downloader middleware. 

I don't know if such middleware should be enabled by default. On one hand, it is almost unuseful for 'focused' crawls (because usually only main/index pages are marked as AJAX crawlable via meta tag), and it has a performance impact (about 1/2000s in average according to my unscientific tests). On the other hand, MetaRefreshMiddleware (that also parses HTML) is enabled by default, and Scrapy handles 'ajax crawlable' pages with `'#!'` in URLs by default. 

Another totally unscientific test shows that index pages of about 0.85% US small business websites indicate themselves as 'AJAX crawlable' using meta tag.

TODO:

* [x] mention it in 'broad crawling' docs;
* [x] add an undocumented option for changing the size of parsed contents;
* [x] investigate if fail-fast methods help performance.