scrapy/scrapy — pull request #343 by kmike (2013-07-09 22:44:35+00:00)
Title: AjaxCrawlableMiddleware

What do you think about adding support for https://developers.google.com/webmasters/ajax-crawling/docs/getting-started, part 3 ("Handle pages without hash fragments")? In this pull request it is implemented using downloader middleware.

I don't know if such middleware should be enabled by default. On one hand, it is almost unuseful for 'focused' crawls (because usually only main/index pages are marked as AJAX crawlable via meta tag), and it has a performance impact (about 1/2000s in average according to my unscientific tests). On the other hand, MetaRefreshMiddleware (that also parses HTML) is enabled by default, and Scrapy handles 'ajax crawlable' pages with `'#!'` in URLs by default.

Another totally unscientific test shows that index pages of about 0.85% US small business websites indicate themselves as 'AJAX crawlable' using meta tag.