reimund/ntlm-middleware — pull request #2 by tobalsgithub (2013-12-03 05:08:24+00:00)
Title: Returning HtmlResponse, rather than TextResponse

When using the scrapy crawl spider, rather than just the base spider,
we check to see if the response object is of type HtmlResponse, and if
it is not, we quit out without following any links.

Specifically, this happens in _requests_to_follow in the CrawlSpider
class here:

https://github.com/scrapy/scrapy/blob/master/scrapy/contrib/spiders/crawl.py

Looks like HtmlResponse is just a shell class that inherits from
TextResponse, so I wouldn't think it would cause any issues to change
it.