Error in JSON decoding for Python 3

In the unit tests for a server project, I'm using requests to pull down data from the server and verify that it's correct. This data is in JSON format, so it seems only natural to use r.json() to decode the data.

Unfortunately, while this works for Python 2.X, it doesn't seem to work for Python 3, giving the following error:

```
Traceback (most recent call last):
  File "/home/philip/projects/python-djdns/djdns/tests/test_script.py", line 129, in test_get_user
    r.json(),
  File "/home/philip/projects/python-djdns/.tox/py32/lib/python3.2/site-packages/requests/models.py", line 690, in json
    return json.loads(self.text or self.content, **kwargs)
  File "/usr/lib/python3.2/json/__init__.py", line 309, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.2/json/decoder.py", line 353, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: can't use a string pattern on a bytes-like object
```

This happens whenever you try to feed a bytestring into the JSON decoder. For a simple test case (and remember, this is only in Python 3):

``` python
>>> import json
>>> json.loads(b"{}")
```

It seems like as long as requests just dumps the loaded bytestrings directly into the decoder, without turning them into proper string first, this will be a problem. It is entirely possible that this error can (in this case) be patched server-side by specifying the encoding as an HTTP header - at present, the server is extremely basic, and doesn't have any encoding header at all. But this is still probably an edge case that you folks will want to patch for.


Hi there!

Can you print `response.encoding` for me?


Wow, that is a seriously impressive response time!

I tried this with and without encoding data in the header. If my server uses Content-type "application/json", then the response encoding is None. If I send it as "application/json; charset=utf-8", then the response encoding is "utf-8".

The crazy thing is, I get the decoding error either way - that's right, even when r.encoding = "utf-8".


Uh...odd. Is the stack trace the same for both cases? (And you timed your issue asking well, I happened to be sitting at my laptop.) =)


Yup. I suspect that the trouble is mostly involved in the `return json.loads(self.text or self.content, **kwargs)` line of the stack trace, based on what I've read in the requests documentation about response properties.


Yeah, it's perplexing. `Response.json()` works fine for me using `http://httpbin.org/get', so I suspect there's something particular in the flow.


Do you have a reproducible test case?


Nothing small at the moment, just the actual server I'm working on. I'll see if I can get a pared-down version working.

In the same Python environment, by the way, it turns out that I can get r.json() from http://httpbin.org/get, so whatever the problem is, it's definitely not general - it's being specifically triggered by some aspect of my code.


I've narrowed down the error a bit. Turns out there was some kind of silent server-side error, where there was _no output of actual data._ Headers, yes. Payload, no. Because the json.dump(data, self.wfile) was failing.

So I still don't have a useful test case, but I think I've got the problem narrowed down. It _is_ that `or` that I was feeling suspicious about earlier. If the payload is zero-length, then it will end up using self.content instead of self.text, and self.content is a blank bytestring. But it's a bytestring, and that's enough to make the json decoder throw a fit about encodings.


Yup, that doesn't look great. We should aim to catch that case (zero-length content) and throw a useful Requests exception.


Sounds like a plan. You do want to be hardened against that. And in the meantime, I'm going to beat my server with a wrench until it actually works properly in Python 3 :)


Sounds good. Thanks so much for reporting this!


No problem, happy to help. I'm just astounded how fast and helpful you guys are! Will definitely be using requests in all my projects (or at least, all the ones that download things) from now on.


Glad to hear it, and glad to have been of assistance. =)


Here's a question:

If we use `self.content`, `self.text` is populated and in fact `self.text` is really a [method](https://github.com/kennethreitz/requests/blob/master/requests/models.py#L646..L677) being called on the Response object. Why do we need the `or` there? It doesn't prevent an exception caused in `self.text`, and the only (major) version of Python on which using `self.content` to parse JSON with is 2. Would it not simply be better to remove the `or self.content`? What purpose does it really serve there?

Let's assume that `self.text` is an empty string again, if on Python 3 you do: `json.loads('')` you receive an exception but it is a `ValueError`. This is ostensibly a bit less confusing than what @campadrenalin but it also prevents breaking the API in 2.1 with #1666. Technically this is the right behaviour (for `json.loads` to raise a `ValueError` when there is no JSON to be loaded).


> Will definitely be using requests in all my projects (or at least, all the ones that download things) from now on.

Go ahead and use it in every other project too ;)


@sigmavirus24 Yeah, that's a much better way. Let's do that instead.


:heart: @Lukasa 


Resolved by #1667. Thanks again for raising this!