Incorrect encoding detected for UTF-8 XML response.

Requests decodes the UTF-8 response as TIS-620:

``` python
>>> response = requests.get('http://lyrics.wikia.com/api.php', params={'artist': 'Javiera Mena', 'song': 'Al Siguiente Nivel', 'fmt': 'xml'})
>>> print response.text
<?xml version="1.0" encoding="UTF-8"?>
<LyricsResult>
[...]
    <lyrics>Esa visiรณn estรก
[...]
</LyricsResult>
```

Forcing UTF-8 yields the correct value, "Esa visión está", instead of "Esa visiรณn estรก":

``` python
>>> print response.content.decode('utf8')
<?xml version="1.0" encoding="UTF-8"?>
[...]
    <lyrics>Esa visión está
[...]
</LyricsResult>
```


Requests is a HTTP library: we only consider HTTP headers when attempting to work out text encodings. We think it sets a dangerous precedent to attempt to parse body content. If you want to do something specific for a given content type, you have to do it yourself.


@Lukasa Absolutely agree: I wouldn't expect Requests to parse the XML declaration to read the encoding, that completely out of scope for a HTTP library.

I wonder, though, what made Requests autodetect a Thai encoding for an XML document with some Spanish content. It's latin alphabet after all.

If the autodetection feature is not reliable enough, maybe the documentation should discourage the users from using `response.text` at all and use `response.content` instead. Or made an opt-in feature instead of default?


Ah, good spot. We have mixed feelings about auto detection. It's hard to do right, and no matter how well we do there will always be edge cases which cause problems.

Realistically, there's not a lot we can do about that beyond warn people. Auto detection _usually_ works well enough for people. I'm interested to see what @sigmavirus24 thinks. 


Could you print `r.headers` and `r.encoding`? Charade is the fallback when the content-type has no charset specified and charade can actually (typically) detect utf-8 very reliably. That said, it seems like charade and python-chardet will be merging soon and have a real maintainer after that (which is more than I can say for charade right now).


Sure! Here's the output:

``` python
>>> print response.headers
CaseInsensitiveDict({'content-length': '319', 'x-served-by': 'cache-s24-SJC, cache-mia1327-MIA', 'x-cache': 'MISS, MISS', 'x-content-type-options': 'nosniff', 'content-encoding': 'gzip', 'set-cookie': 'wikia_beacon_id=7PuM__S5b5; domain=.wikia.com; path=/; expires=Thu, 12 Jun 2014 03:21:31 GMT;, Geo={%22city%22:%22FIXME%22%2C%22country%22:%22AR%22%2C%22continent%22:%22SA%22}; path=/', 'accept-ranges': 'bytes', 'x-timer': 'S1386991290.966581345,VS0,VS38,VE123,VE162', 'vary': 'Accept-Encoding', 'x-cacheable': 'YES', 'server': 'Apache', 'connection': 'keep-alive', 'x-cache-hits': '0, 0', 'cache-control': 'max-age=3600, s-maxage=3600, public', 'date': 'Sat, 14 Dec 2013 03:21:31 GMT', 'x-age': '0', 'content-type': 'application/xml'})
>>> print response.encoding
None
```


I think here's the interesting part:

``` python
>>> charade.detect(response.content)
{'confidence': 0.99, 'encoding': 'TIS-620'}
```

`charade` is detecting TIS-620 with a rather high confidence. It's something I've seen before when using `charade` outside of `requests`:
- https://github.com/sigmavirus24/charade/issues/24
- https://github.com/sigmavirus24/charade/issues/25


Does it make sense to at least expose the confidence value for the autodetected encoding, as in https://github.com/kennethreitz/requests/pull/1796?
It won't be of much help in this particular case because the confidence value is unusually high, but I think it's a good thing to have as a user of the library.


I guess charade is in worse shape than I thought and that's mostly my fault. I don't have the interest or the time to maintain it and frankly there's data that's needed that just doesn't exist. This is why I'm hopeful a merge with python-chardet will fix some things.

> Does it make sense to at least expose the confidence value for the autodetected encoding, as in #1796?

I don't think so. _Most_ users don't run into charade frankly. That said, if others are interested, I'd rather we only calculate the encoding once and store the result of `charade.detect` once and just reference that. It's unfortunately computationally expensive.


To recap:
- When there's no explicit charset in the response headers, `Response.text` falls back silently to an auto detected charset that in many cases is not the right one.
- From a user's perspective, AFAIK there's no obvious way to tell whether an explicit or an automatic charset was used to decode the contents.
- Exposing the charset detection confidence is [not a good idea](https://github.com/kennethreitz/requests/pull/1796#issuecomment-30563831).
- Issuing or logging a warning is [not either](https://github.com/kennethreitz/requests/pull/1799).


@martinblech Point 2 isn't correct. If `response.encoding` is `None`, an automatic charset is used.

The biggest problem here is that Requests can only do so much to protect users. Text encoding is a horrible minefield of pain, and `charade`/`chardet` usually helps us solve _some_ of that pain. However, users who are in a position to know better should take a more active role in choosing text encodings. For instance, on HTML/XML pages, people should look for an `encoding` directive in the content.


@Lukasa Got it. It wasn't obvious to me but it's [right there](http://requests.readthedocs.org/en/latest/api/?highlight=text#requests.Response.text) in the docs: "If Response.encoding is None, encoding will be guessed using charade." My bad.

My point is, I'm not sure if falling back to `charade`/`chardet` solves more pain than it creates. It didn't for me, but maybe I just hit an unusual number of edge cases that most users of Requests won't. It's hard to estimate how well my experience extrapolates to the average Requests user.

I would love to see a shorter version of this part of your answer in the docs for `Response.text`, just in case:

> […] users who are in a position to know better should take a more active role in choosing text encodings. For instance, on HTML/XML pages, people should look for an encoding directive in the content.


I would not be adverse to adding that to the documentation.

The average user of requests receives responses that either indicate the encoding correctly or vaguely indicates the encoding. For example, we've received complaints that we follow the RFC when we receive a `text/*` Content-Type with no charset provided. In that case we use what the RFC tells us to: `ISO-8859-1`. Encodings and detection thereof are hard and vastly inaccurate. One other thing we might add to the documentation is to tell users to check `response.encoding` if they know what they should be expecting. If they do not receive what they expect, they can override it to ensure they get the results they need. Overriding it, however, should be discouraged.


Docs update was made as part of #1823.