Wrong enconding detection for text/* content types (ISO-8859-1)

Hi.

I've read the docs which refer to RFC 2616:

> The "charset" parameter is used with some media types to define the character set (section 3.4) of the data. When no explicit charset parameter is provided by the sender, media subtypes of the "text" type are defined to have a default charset value of "ISO-8859-1" when received via HTTP. Data in character sets other than "ISO-8859-1" or its subsets MUST be labeled with an appropriate charset value. See section 3.4.1 for compatibility problems.

So, it's no longer relevant.
Why?

Firstly, I can read facebook (which returns content type 'text/html' and sets `<meta charset="utf-8">`, see https://www.facebook.com) in Russian (utf-8, indeed) in any browser. It proves only that all browsers violate this RFC or understand it other way.

Secondly, [it's said that](http://www.w3.org/Protocols/rfc2616/rfc2616.html) "in 2014, RFC2616 was replaced by multiple RFCs (7230-7237)". And there is no mention of default charsets in them.

I also found [RFC 6657](https://tools.ietf.org/html/rfc6657):

> all new "text/*" registrations
>    MUST clearly specify how the charset is determined; relying on the
>    default defined in Section 4.1.2 of [RFC2046] is no longer permitted

Look at the [registration for text/html](http://www.w3.org/TR/html5/iana.html#text/html):

> Optional parameters:
> charset
> The charset parameter may be provided to definitively specify the document's character encoding, overriding any character encoding declarations in the document. The parameter's value must be one of the labels of the character encoding used to serialize the file. [ENCODING]

So now there is not default charset for text/html specified.

Finally, even in HTML 4.0 specification they [suggested to ignore this rule](http://www.w3.org/TR/html4/charset.html#h-5.2.2) about default charset:

> The HTTP protocol ([RFC2616], section 3.7.1) mentions ISO-8859-1 as a default character encoding when the "charset" parameter is absent from the "Content-Type" header field. In practice, this recommendation has proved useless because some servers don't allow a "charset" parameter to be sent, and others may not be configured to send the parameter. Therefore, user agents must not assume any default value for the "charset" parameter.

My suggestion: use default charset ("US-ASCII") only for text/plain only, so it is said in [RFC 6657](https://tools.ietf.org/html/rfc6657#page-4).

> The default "charset" parameter value for "text/plain" is unchanged
>    from [RFC2046] and remains as "US-ASCII".


Hi there! Thanks for this issue, you've done some exhaustive research!

This is a known issue: see #2086. The upshot of this is that we agree that the current behaviour is wrong. It should be noted that browsers aren't ignoring the heuristic. The heuristic explicitly points out that information from the body (e.g. the `<meta>` tag) can provide more specific information than the header. However, Requests explicitly _never_ parses HTML, so we cannot take advantage of that approach. Similarly, we do not follow HTML specifications because we are not a HTML implementation.

If you are parsing HTML we recommend using `response.content`: a good HTML parser should be able to do the decoding appropriately.

I'm closing this to center the discussion on #2086. Thanks again!


Got it. Thanks.