Proper encoding for application/json Content-Type

Hello,

I've looked through open and closed issues but haven't seen this raised before. Sorry if I'm bringing up something that's already been addressed. That said, here we go:

I'm running a flask-restful app and returning JSON responses to requests. Flask-restful helpfully encodes the response body as UTF-8 and sets the Content-Type to "application/json". When i make a request to my server using requests, requests can't properly detect the encoding and it looks like this is because I haven't set the charset on the Content-Type header. However, according to [RFC 4627](http://www.ietf.org/rfc/rfc4627.txt) the encoding for application/json responses is either UTF-8, UTF-16, or UTF-32. The default is UTF-8, but you can detect which one of the three encodings is being used by looking at the first four bytes as mentioned in [RFC 4627](http://www.ietf.org/rfc/rfc4627.txt).

When requests receives a response with a Content-Type header of application/json shouldn't it try to detect the encoding as UTF-8, UTF-16, or UTF-32 and set the encoding on the response object accordingly? This would help greatly with using `response.text`. Right now I'm setting the encoding on the response object manually. 

Just to note: `response.json()` works perfectly so I realize this is a bit of a corner case since users will probably just deserialize the response anyway. 

Thanks for your time, and all your awesome work. I absolutely love requests.
Mike

Possibly Related Issues: #1665 #1467 
Reference to @mitsuhiko's opinion on the matter: https://github.com/mitsuhiko/flask/issues/454#issuecomment-4578200


> requests can't properly detect the encoding

What does `response.apparent_encoding` return? This sounds l like a bug in chardet.

> The default is UTF-8, but you can detect which one of the three encodings is being used by looking at the first four bytes

requests _could_ do this, but won't because this is delegated to chardet.

> When requests receives a response with a Content-Type header of application/json shouldn't it try to detect the encoding as UTF-8, UTF-16, or UTF-32 and set the encoding on the response object accordingly?

We are an _HTTP_ library, not JSON RPC, nor are we a library that specifically conforms to RFC 4627. It's easy to see how having `r.json()` could make that less obvious _but_ it's a convenience method, like `r.encoding` and `r.apparent_encoding` are there as conveniences.

If you look at RFCs 7230-7235, you'll note that we should only be using what's provided by the server in the headers for `r.encoding`. The fact that 4627 doesn't allow for the server to set an explicit charset is a flaw in the design in my opinion. Users of JSON transmitted over HTTP shouldn't have to inspect the first 4 bytes if there are (and have been) other mechanisms for determining the encoding.

Does this explain it well enough?


Thanks for your fast response.

`response.apparent_encoding` returns`'ISO-8859-2'`

``` python
In [1]: import requests

In [3]: resp = requests.get(my_api_endpoint)

In [4]: resp.headers
Out[4]: CaseInsensitiveDict({'content-length': '213', 'server': 'Werkzeug/0.9.4 Python/2.7.8', 'date': 'Fri, 03 Oct 2014 05:01:51 GMT', 'x-server-time': '0.003', 'content-type': 'application/json'})

In [5]: resp.encoding

In [6]: resp.apparent_encoding
Out[6]: 'ISO-8859-2'
```

chardet doesn't have a problem with the utf-16 or utf-32 encoding however (probably due to the byte order marks):

``` python
In [7]: from chardet import detect

In [8]: detect(resp.content.decode('utf-8').encode('utf-16'))
Out[8]: {'confidence': 1.0, 'encoding': 'UTF-16LE'}

In [9]: detect(resp.content.decode('utf-8').encode('utf-32'))
Out[9]: {'confidence': 1.0, 'encoding': 'UTF-32LE'}
```

I just noticed as well that RFC 4627 states that the encoding for application/json will be UTF-8 and if UTF-16 or UTF-32 are being used the Content-Transfer-Encoding header must be set to binary.

> Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32
> 
>    JSON may be represented using UTF-8, UTF-16, or UTF-32.  When JSON is written in UTF-8, JSON is 8bit compatible.  When JSON is written in UTF-16 or UTF-32, the binary content-transfer-encoding must be used.

Maybe requests could check for Content-Type == application/json and set `response.encoding = 'utf-8'` iff the Content-Transfer-Encoding header isn't set to binary? `response.encoding` could be left as None in the case of the binary Content-Transfer-Encoding and chardet should be easily able to differentiate between UTF-16 and UTF-32?

> ... nor are we a library that specifically conforms to RFC 4627

I understand. Your documentation says:

> Requests is intended to be compliant with all relevant specifications and RFCs where that compliance will not cause difficulties for users

So I guess RFC 4627 is deemed irrelevant? The documentation also says this regarding `response.text`:

> The encoding of the response content is determined based solely on HTTP headers, following RFC 2616 to the letter. If you can take advantage of non-HTTP knowledge to make a better guess at the encoding, you should set r.encoding appropriately before accessing this property.

So with all that said, I completely understand your position. It does feel a little gross to have requests inspect response content, and chardet should be handling this. And yeah, it's a major bummer that RFC 4627 doesn't allow for charset declarations. It would certainly make things a lot easier. Clients shouldn't have to try to interpret encodings. A better spec might say that UTF-8 is to be assumed unless UTF-16 or UTF-32 is specified in a charset parameter.

It'd be nice to have the convenience property `response.encoding` handle the most common case of UTF-8, but I can understand your apprehension. Thanks again for reviewing this.


> Just to note: response.json() works perfectly so I realize this is a bit of a corner case since users will probably just deserialize the response anyway.

This is the relevant bit of your message. We have conformed to the relevant RFCs where you provide us out-of-band information on what you are expecting. Basically, when you call `response.json()` you are saying to us "yes, this is definitely JSON, please apply JSON-specific logic here". When you _don't_ call that, we aren't going to apply JSON-specific logic, we're going to restrict ourselves to HTTP.


This should be fixed in chardet now (I just remembered why this seemed familiar, it's because of https://github.com/chardet/chardet/issues/30). That said, this really is a chardet issue, as we've already said and as @Lukasa said, we do the right thing when you tell us you want JSON from the body. Other than that, we have to assume it falls under the domain of HTTP and that explicitly requires a `charset` parameter in `Content-Type`. We have no intention of every doing anything more than using that or trying to guess the encoding with chardet. We already do not introspect the body automatically (although we provide that as a convenience function) for meta tags or anything else that would hint the character encoding and we provide you with the ability to tell us what encoding to use.

There's enough degrees of freedom to do what you want to do. Further, if you're only ever using a session to talk to your API, you could (if you desired) do something like:

``` python
def force_utf8(response, *args):
    response.encoding = 'utf8'
    return None

s.hooks['response'].append(force_utf8)
r = s.get('http://10.0.0.1:5000/my/endpoint')
assert r.encoding == 'utf8'
```

If you're not using a session, you can do:

``` python
requests.get(my_api_endpoint, hooks={'response': [force_utf8]})
```


Good to know it's fixed in chardet, thanks @sigmavirus24. Yeah, I have ways of detecting the encoding in my client code. I'll close this now.

Thanks for your time, guys.


@Opethil Our pleasure, thanks for taking the time to talk this through with us. =) Users like you are a pleasure to work with, keep it up!