Response Content not read properly. _content_consumed: False

Summary.

We are observing some weird behaviour with our requests lib. While making GET/POST calls we observe most of the time Headers are read and status=200, but fails to read _content.

May I know what are the possible reasons for this? This is happening way too many times.

The Response we get is -

```
"response":"{'cookies': , '_content': False, 'headers': {'Content-Length': '939', 'request-id': 'ZLnsodTJ', 'Server': 'nginx', 'Connection': 'keep-alive', 'Date': 'Mon, 24 May 2021 14:12:30 GMT', 'Content-Type': 'application\/json'}, 'url': u'http:\/\/xxx:yyyy\/', 'status_code': 200, '_content_consumed': False, 'encoding': 'utf-8', 'request': , 'connection': , 'elapsed': datetime.timedelta(0, 0, 9880), 'raw': , 'reason': 'OK', '_next': None, 'history': []}
```

This result in error in decoding response content.

## Expected Result



## Actual Result

Exception while reading response.

What happened instead.

## Reproduction Steps

```python
import requests

```

## System Information

    $ python -m requests.help

```
{
  "chardet": {
    "version": "3.0.4"
  }, 
  "cryptography": {
    "version": "3.2.1"
  }, 
  "idna": {
    "version": ""
  }, 
  "implementation": {
    "name": "CPython", 
    "version": "2.7.17"
  }, 
  "platform": {
    "release": "4.4.0-1119-aws", 
    "system": "Linux"
  }, 
  "pyOpenSSL": {
    "openssl_version": "1010108f", 
    "version": "20.0.1"
  }, 
  "requests": {
    "version": "2.25.1"
  }, 
  "system_ssl": {
    "version": "1000207f"
  }, 
  "urllib3": {
    "version": "1.26.2"
  }, 
  "using_pyopenssl": true
}
```

This command is only available on Requests v2.16.4 and greater. Otherwise,
please provide some basic information about your system (Python version,
operating system, &c).


Hi @tandonraghav,

This issue doesn't make it clear what problem you're actually encountering. `_content` is a private variable, so there should never be a case where you're touching this directly. Are you saying the [`content`](https://docs.python-requests.org/en/master/user/quickstart/#binary-response-content) (note the lack of `_`) is producing a decoding error?

If you feel there is actually a bug here, please provide a complete example of how you're interacting with the response and the stack trace you're receiving. Thanks.

@nateprewitt Most of the time we get 200 in status but the `content`  is not present, hence json.load fails.
I am not accessing `_content` but referred because that of what is see if i print `response.__dict__`

I am seeing this behaviour when we do `requests.get` or `requests.post` call.

This is the logger which we are getting on printing `response.__dict__`

```
"response":"{'cookies': , '_content': False, 'headers': {'Content-Length': '939', 'request-id': 'ZLnsodTJ', 'Server': 'nginx', 'Connection': 'keep-alive', 'Date': 'Mon, 24 May 2021 14:12:30 GMT', 'Content-Type': 'application\/json'}, 'url': u'http:\/\/xxx:yyyy\/', 'status_code': 200, '_content_consumed': False, 'encoding': 'utf-8', 'request': , 'connection': , 'elapsed': datetime.timedelta(0, 0, 9880), 'raw': , 'reason': 'OK', '_next': None, 'history': []}
```

How is it possible that thousands of times status is 200 and no content in response? 
Also, there is `Content-Length` present but still `_content_consumed: False` is coming.

I do not have conclusive evidence that it is requests/urllib3 bug only. But this is our most likely suspect.

Also, I am not able to reproduce the exact behaviour but this is happening very frequently in our production systems.
Can you please give some more pointers which can help in debugging this issue.


```
try:
 resp = requests.get(url=url, **requests_kwargs)
 if self.is_successful(resp):                    
    json_dict = json.loads(resp.content) # Getting Error here!!!
except Exception as e:                
    print resp.__dict__ 
```


Given how much information you've redacted, it's not exactly plausible to debug this magically for you. I would suggest looking at `request_kwargs` in these cases. I suspect you'll find `stream=True`.

Beyond that, `json.loads` expects `text` not `bytes` and `resp.content` will be bytes. You should be using `resp.text`.


> How is it possible that thousands of times status is 200 and no content in response?

Servers do the wrong thing often. This isn't surprising, especially seeing that your `Content-Type` appears to have a `\` in it.

Also if you're on a reasonably modern version of `requests` you could also more directly do `json_dict = resp.json()`. I don't believe this is a bug in the library either

@sigmavirus24 `\` you are seeing because of the Log line we are printing.
`request_kwargs` just contains some `auth` headers and are you saying `json.loads` is the issue which is not the case here.
Also, I am not setting `stream=True` which due to which it is reading half response only.

What do you suspect server would have done wrong which can cause such a behaviour?

Also, i can agree might be this is not a bug at requests lib at end, but not sure of it.



> are you saying `json.loads` is the issue which is not the case here

I'm not saying it's the cause, I'm saying it's not necessary nor is the way you're using `resp.content` here correct. The stdlib expects text and you are providing bytes.

>  Also, I am not setting `stream=True` which due to which it is reading half response only.

I don't understand what you're trying to say here.

> What do you suspect server would have done wrong which can cause such a behaviour?

There's plenty. The best way to 

> Also, i can agree might be this is not a bug at requests lib at end, but not sure of it.

Except I'm fairly certain it isn't a bug.

You haven't provided the exception you're seeing and what you're pointing out `_content` is `False` is the case when `stream=True` is provided and the content hasn't been consumed from the underlying socket. There's so much information missing we can't possibly help you and you seem to be insisting that this should be kept open as a potential bug without any supporting evidence.

At this point, you've provided no way to reproduce and you haven't provided what you're actually seeing, just what you seem to think are relevant minimal details from which we must magically discern the cause. Sadly we're not magicians, nor are we network mind readers. We can't figure this out for you. I strongly suggest you invest in observability for your software. It will pay off far more than just whatever problem you're running into here

@sigmavirus24 Thanks for response.. I was trying to say we are not setting `stream=True` then too we are getting `_content= False`. 
Currently, I am trying to find out where the bug lies. Our servers are returning response properly and I can see that in the logs.

But somehow client is not getting the response body.

Anyways, I will investigate more and get back to you.