kennethreitz/requests — pull request #1589 by jdennis (2013-09-10 19:46:09+00:00)
Title: Make sure content is checked when setting encoding

HTML pages which declared their charset encoding only in their content
had the wrong encoding applied because the content was never checked.

Currently only the request headers are checked for the charset
encoding, if absent the apparent_encoding heuristic is applied. But
the W3.org doc says one should check the header first for a charset
declaration, then if that's absent check the meta tags in the content
for a charset encoding declaration. It also says if no charset
encoding declaration is found one should assume UTF-8, not ISO-8859-1
(a bad recommendation from the early days of the web).

This patch does the following:

* Removes the default ISO-8859-1 from get_encoding_from_headers(),
it's wrong for 2 reasons, 1) get_encoding_from_headers() should
return None if no encoding was found in the header, otherwise how
can you tell it was absent in the headers? 2) It's the wrong default
for the contemporary web. Also because get_encoding_from_headers()
always returned an encoding any subsequent logic failed to execute.

* The Request text property now does a 4 stage check for encoding in
this priority order:

1) encoding declared in the headers
2) encoding declared in the content (selects the highest priority
encoding if more than one encoding is declared in the content)
3) apply apparent_encoding hueristic (includes BOM)
4) if none of the above default to UTF-8