tenderlove/mechanize — pull request #221 by GarthSnyder (2012-04-11 23:22:16+00:00)
Title: Mechanize::Page content-type checking should be relaxed in light of pluggable parsers

I have a use case that involves dealing with HTML fragments that are normally retrieved by a page through AJAX requests. Since Mechanize doesn't process JavaScript, I'm accessing the fragments directly by passing their URLs to Agent.get.

Unfortunately, the server in this case doesn't serve these fragments with a text/html content type, or in fact, with any content-type header at all, even though they are in fact valid HTML. By default, Mechanize represents the fragments as Mechanize::File objects because of their unknown type.

I can set the default pluggable parser for the agent to Mechanize::Page to force HTML parsing, but Mechanize::Page peeks at the response headers and enforces content-type checking of its own, raising a ContentTypeError if the content-type does not look HTMLish.

I suggest that this latter behavior may be unnecessary. As a sanity check on automatic parser selection, it seems superfluous. As a barrier to accessing servers that don't set their content-type headers correctly, it's actively counterproductive. I'm having a hard time thinking of a case in which it would actually be useful. Would it be possible to remove this check entirely?

This clone implements a more conservative change in that it still validates the content-type header, but only if the serve actually supplies a content type.