Error decoding chunked response in _update_chunk_length: ValueError: invalid literal for int() with base 16: b'HTTP/1.1 200 OK\r\n'

Decoding a response with chunked encoding fails because it seems to be reading from the HTTP status line instead of the start of the body.

## Expected Result

Successfully return a 200 response object.

## Actual Result

```
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 535, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b'HTTP/1.1 200 OK\r\n'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 298, in _error_catcher
    yield
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 590, in read_chunked
    self._update_chunk_length()
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 539, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(17 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 719, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 428, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 618, in read_chunked
    self._original_response.close()
  File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 316, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(17 bytes read)', IncompleteRead(17 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 70, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/api.py", line 56, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 488, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 641, in send
    r.content
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 797, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/usr/lib/python3.6/site-packages/requests/models.py", line 722, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(17 bytes read)', IncompleteRead(17 bytes read))
```

## Reproduction Steps

```python
import requests
requests.get('https://my.virginmoney.com.au/AUVMA/JSO/signon/DisplayUsernameSignon.do')
```

## System Information

requests 2.13.0
urllib3 1.20
Python 3.6.2
Fedora 26

This works fine with requests 2.10.0 so something regressed between then and 2.13.0. Also tested with 2.18.4 and the bug exists as above.

I can't spot anything defective or invalid in the response from that server when I look at it manually.

One thing I do notice is that it is using both `Content-Encoding: gzip` and `Transfer-Encoding: chunked` which might be a little unusual. Could this bug be caused by some bad interaction between the gzip decoding and chunk decoding?

I can't reproduce this with Requests 2.18.4 on either Python 3 or Python 2. Have you tried updating urllib3 to check if this resolves the problem?

Okay sorry, my comment above about 2.10.0 vs. 2.13.0 was incorrect. I did not thoroughly test all the combinations. It turns out this works fine with all versions/combinations of requests and urllib3 that I tested on Fedora 25 but fails with all versions/combinations of requests and urllib3 that I tested on Fedora 26.

Which most likely means: it's something specific to Python 3.6?



Or at least Fedora 36, as it works fine for me with my Python 3.6.2 on macOS. I recommend reporting a bug with Fedora. 

I get this error on Windows with the latest Python 3.7 alpha and latest requests / urllib3. When I use Python 3.5 then it works fine with the same requests / urllib3. I'm suspecting a python version issue as well. 

I haven't dug into this in a while. Back when I was looking at it, I could not find any substantial differences between Fedora 25 and 26 in Python land at all (same Python version, same version of all the relevant Python libraries I could see, same code for urllib3 and requests) *but* one big difference did stand out: Fedora 25 has openssl 1.0.2 whereas Fedora 26+ has openssl 1.1.0. Not sure if that is related.

i'm facing same kind of issue. any idea how to bypass this?
```
<Response [200]>
Traceback (most recent call last):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 539, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 302, in _error_catcher
    yield
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 594, in read_chunked
    self._update_chunk_length()
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 543, in _update_chunk_length
    raise httplib.IncompleteRead(line)
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/models.py", line 738, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 432, in stream
    for line in self.read_chunked(amt, decode_content=decode_content):
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 622, in read_chunked
    self._original_response.close()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/contextlib.py", line 100, in __exit__
    self.gen.throw(type, value, traceback)
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/packages/urllib3/response.py", line 320, in _error_catcher
    raise ProtocolError('Connection broken: %r' % e, e)
requests.packages.urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "loginscheck.py", line 49, in <module>
    f.write(req.text)
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/models.py", line 840, in text
    if not self.content:
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/models.py", line 816, in content
    self._content = bytes().join(self.iter_content(CONTENT_CHUNK_SIZE)) or bytes()
  File "/Users/officeuse/environs/py_elt_service/lib/python3.6/site-packages/requests/models.py", line 741, in generate
    raise ChunkedEncodingError(e)
requests.exceptions.ChunkedEncodingError: ('Connection broken: IncompleteRead(0 bytes read)', IncompleteRead(0 bytes read))
```


I solved this problem by adding one line code.
# 
python3 on Mac
urllib3 == 1.23
requests == 2.19.1
# 
> Traceback (most recent call last):
  File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/response.py", line 574, in _update_chunk_length
    self.chunk_left = int(line, 16)
ValueError: invalid literal for int() with base 16: b''

so, I open the file "......./urllib3/response.py", add `line = (len(line)>0 and line or "0")` in 571, like below:
```
    def _update_chunk_length(self):
        # First, we'll figure out length of a chunk and then
        # we'll try to read it from socket.
        if self.chunk_left is not None:
            return
        line = self._fp.fp.readline()
        line = line.split(b';', 1)[0]
        line = (len(line)>0 and line or "0")     # added this line
        try:
            self.chunk_left = int(line, 16)
        except ValueError:
            # Invalid chunked protocol response, abort.
            self.close()
            raise httplib.IncompleteRead(line)
```




> I solved this problem by adding one line code.
> 
> python3 on Mac
> urllib3 == 1.23
> requests == 2.19.1
> 
> > Traceback (most recent call last):
> > File "/usr/local/Cellar/python3/3.6.4/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/response.py", line 574, in _update_chunk_length
> > self.chunk_left = int(line, 16)
> > ValueError: invalid literal for int() with base 16: b''
> 
> so, I open the file "......./urllib3/response.py", add `line = (len(line)>0 and line or "0")` in 571, like below:
> 
> ```
>     def _update_chunk_length(self):
>         # First, we'll figure out length of a chunk and then
>         # we'll try to read it from socket.
>         if self.chunk_left is not None:
>             return
>         line = self._fp.fp.readline()
>         line = line.split(b';', 1)[0]
>         line = (len(line)>0 and line or "0")     # added this line
>         try:
>             self.chunk_left = int(line, 16)
>         except ValueError:
>             # Invalid chunked protocol response, abort.
>             self.close()
>             raise httplib.IncompleteRead(line)
> ```

It looks that ths line fixes the problem for me. Can you please explain what this line do?

Shouldn't this then be fixed in urllib3? @Lukasa 

@KonMann yes it should. 

@cosminnastasie I am a little forgotten. It seems like that without this line, there should be occur problem at the last chunk data. when the value of 'line' is Null, the error arise.

@cosminnastasie 

```
>>> line = b''
>>> line = line.split(b';', 1)[0]
>>> int(line,16)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 16: b''
```

but if we add that line:
```
>>> line = b''
>>> line = line.split(b';', 1)[0]
>>> line = (len(line)>0 and line or "0")     # added this line
>>> int(line,16)
0
```

problem solved!!!

@sigmavirus24 ok, as far as I can see there is no open issue about this in the [Urllib3 Issue Tracker](https://github.com/urllib3/urllib3/issues?utf8=%E2%9C%93&q=is%3Aissue+open) at the moment => Let's open an issue there?

@KonMann you seem to have more time than me. Go for it.

@odboy Do you have some code to reproduce?

We temporarily repaired it in this way. I wonder if there will be any problem.


    def _update_chunk_length(self):
        # First, we'll figure out length of a chunk and then
        # we'll try to read it from socket.
        if self.chunk_left is not None:
            return
        line = self._fp.fp.readline()
        line = line.split(b';', 1)[0]
        try:
            if len(line) == 0:
                self.chunk_left = 0
            else:
                self.chunk_left = int(line, 16)
        except ValueError:
            # Invalid chunked protocol response, abort.
            self.close()
            raise httplib.IncompleteRead(line)

I'm running into this exact issue as well, with an identical traceback as https://github.com/requests/requests/issues/4248#issuecomment-398007524. I've tried reverting requests as far back as 2.9.1, and as far forward as the current release (2.20.1), with a few different combinations of urllib3 versions in each case - they all give the exact same traceback.

Is there a release which we can use until this is fixed, or do we have to patch our requests with the above corrected def?

I am having this issue as well, and I agree the problem is at the urllib3 level.

Can we agree on where we want this issue to be fixed? Here in requests or in urllib3?

The underlying exception comes from urllib3, but as @dulitz explained in urllib3/urllib3#1516, this isn't necessarily always a "bug" in requests or urllib3. So "fixing" it might not be easy or possible. Regardless, I'm closing this issue here.

I am able to reproduce the same bug by using this code

import requests
requests.get('https://www.telecreditobcp.com/tlcnp/index.do')

I think this is a bug in the server side, but maybe urllib3 could do a better job workarounding this bug in the server, as other libraries / applications do for example (same URL works ok in a web browser like chrome or firefox ...)

As far as I could see, the server (in the example above) sends chunked transfer, but it does not send properly the last 0-length chunk and returns the response code. That's where urllib3 raises exception when trying to decode the length of the chunk from the response status code line. Some insights I could find so far:

* the same request works in a browser without complain (I have tried sending same headers than the browser but didn't see a different for the sample code reproducing the bug)

* sometimes the request works perfectly with the sample code for a while, and then stops working again. So as our local side has not changed really, I assume there is some load balancing or something and we are served by another server maybe with possibly a slightly different version of software.

* the following code makes it deliver part of the content but still not the full content:

r = requests.get('https://www.telecreditobcp.com/tlcnp/index.do', stream=True)
for line in r.iter_lines():
    print(line)

* the bug in my local environment happens with python 3.6.8, 3.7.3 and 2.7.16 versions in Linux. Have tried same kernel though (5.1.16-300.fc30.x86_64).

* Have tried with Ubuntu and Fedora, both seem to fail same way.

* I have a server in AWS running Linux with python 3.5.2, and it works always, no bugs

* I've been told the bug seems not reproducible in Windows environment with any version.

* I've been told enabling traffic through a VPN makes it work in a Linux environment that used to fail.

* same bug seems to happen with curl
curl -X GET https://www.telecreditobcp.com/tlcnp/index.do

curl: (56) Illegal or missing hexadecimal sequence in chunked-encoding


> line = (len(line)>0 and line or "0")     # added this line

Solved my problem. Man is a LEGEND.