response.text converting to utf-8 poorly

with `response = requests.get('some_xml_api_endpoint')`
The xml response has the header:     <?xml version="1.0" encoding="UTF-8"?>
I'm using ubuntu, python3, requests 2.24.0, and saving that file in utf-8 on a ext-4 partition.

response.text gives a lot of broken characters.
response.content.decode('utf-8') works well.

The response is rendered:

Reptileâs Tail   **response.text broke the curly quote character**
Reptiles’s Tail   **content.decode('utf-8') handled the curly-quote well**

Could be described as:   print(bytes([226, 128, 153]).decode('utf-8'))




I expected `response.text` to behave the same as `response.content.decode('utf-8')`.  I've never read any warnings in the guides or docs to anticipate this behavior.  Some prominent warnings would help.

Unless .decode('utf-8') is doing some magic heavy-lifting.




I believe what you're missing is likely that the endpoint is giving us the wrong charset or we're trying to detect it with chardet. You should look at `response.encoding` and `response.apparent_encoding` (if `.encoding` is `None`). If you know the encoding and `response.encoding` is either unset or _wrong_ you should override this. And yes, this is well documented, and there are many old issues about the fact that chardet is a bit imprecise at times when `encoding` is unset.

You're right -- this discussed is noted in the docs at https://requests.readthedocs.io/en/master/user/quickstart/#response-content

It's surprising behavior still.

The response's XML header is
`<?xml version="1.0" encoding="UTF-8"?>`

Requests gets response.headers:
{'Date': 'Fri, 25 Sep 2020 18:00:22 GMT', 'Server': 'Apache', 'Content-Encoding': 'gzip', 'Content-Type': 'text/xml', 'Content-Length': '49447', 'X-Robots-Tag': 'noindex, nofollow'}

curl gets response headers:
< HTTP/1.1 200 
< Date: Fri, 25 Sep 2020 18:01:13 GMT
< Server: Apache
< Content-Type: text/xml
< Content-Length: 71293
< Vary: Accept-Encoding
< X-Robots-Tag: noindex, nofollow

However, from requests I get:
response.encoding = ISO-8859-1
response.apparent-encoding = {ascii or utf-8 depending on the file}

Where could Requests be getting that response.encoding = ISO-8859-1?


Taken from [RFC 6657 Section 3](https://tools.ietf.org/html/rfc6657#section-3):

> In accordance with option (a) above, registrations for "text/*" media
   types that can transport charset information inside the corresponding
   payloads (such as "text/html" and "text/xml") SHOULD NOT specify the
   use of a "charset" parameter, nor any default value, in order to
   avoid conflicting interpretations should the "charset" parameter
   value and the value specified in the payload disagree.

Since text/xml defines it's charset within the payload we can't make an assumption of a charset from an HTTP perspective. This is up to the requester (ie, users of the client) to parse the XML prolog for the encoding of the document. A potential route for you is:

- Detect a `Content-Type: text/xml`
- Pull the raw binary body of the response
- Read and parse the prolog from the binary body (ie `<?xml version = "1.0" encoding = "UTF-8" ...>`)
- Using the `encoding` parameter from the document decode the binary body
- Parse the rest of the XML body as normal

Also unrelated because we're not doing the "wrong" thing here by guessing but here's the code that decides iso-8859-1 for any `text/*` responses by default: https://github.com/psf/requests/blob/1ca1c52e698b13d3d5cc0755a6450306d880b933/requests/utils.py#L497

Is my problem that the http header should include:
`Content-Type: text/xml; charset=UTF-8`  ?



@GarrettArm If you're in control of the server and plan on only using the UTF-8 encoding for your XML documents that should work fine despite being contrary to the RFC.

I like the process you've described earlier.  My immediate problem is solved by it.

But would you consider adding something like this to the docs at:
https://requests.readthedocs.io/en/master/user/quickstart/#response-content

"""The default encoding is ISO-8859-1 for text/* Content-Types.  A Content-Type charset attribute will override that, if present.  `response.text` follows this pattern.  If you wish to specify a different encoding..."""

Maybe a nudge will prevent another neophyte from making the same mistake as me?

We can't possibly document every possible use-case in an HTTP client. That said, I think it's fair that for cases where we have defaults like this we add a table:

```reStructuredText
If the Content-Type of the response is one of these well-known types, we'll assume an encoding if none is provided:

================= ================
Content-Type      Default Encoding
================= ================
text/*            ISO-8859-1
application/json  UTF-8
================= ================
```