Repository: urllib3/urllib3

Issue:
Keep-alive broken in 1.10.3 with chunked transfer-encoding
Per kennethreitz/requests#2568, starting in urllib3 1.10.3 an exception is thrown if a connection with chunked transfer-encoding is reused.

Keep-alive broken in 1.10.3 with chunked transfer-encoding
Per kennethreitz/requests#2568, starting in urllib3 1.10.3 an exception is thrown if a connection with chunked transfer-encoding is reused.



Discussion:
Thanks @lmikkelsen 

Yeh I am also having same issue is there a solution for this?

We're working on it. =) Keep an eye on this issue for more.

> On 24 Apr 2015, at 18:09, ne0ark notifications@github.com wrote:
> 
> Yeh I am also having same issue is there a solution for this?
> 
> —
> Reply to this email directly or view it on GitHub.

It seems like this is broken in mechanize as well.

I get a similar exception with the following script, but the traceback is different:

```
import requests

s = requests.session()

r = s.get('http://httpbin.org/stream/20')
s.get('http://httpbin.org/stream/20')
```

Start of traceback ([full](https://gist.github.com/untitaker/5b3ff800951747701ebb)):

```
Traceback (most recent call last):
  File "/home/untitaker/projects/requests/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:
```

Happening with Python 3.4.3

Please don't print the start of the traceback, it's irrelevant. Print the _end_ of it. =D

To be clear @untitaker, your traceback is different because you've got Python 3 printing the chained exceptions. If the others users did, they'd show the same traceback as you.

Here is my traceback using urllib2:

https://gist.github.com/ne0ark/a451c13de9e6643a751e

So using _just_ urllib3, I'm having trouble reproducing this. The following is more-or-less what an adapter does in requests

``` py
import urllib3

url = 'http://httpbin.org/stream/20'
http = urllib3.PoolManager(
    block=False,
    num_pools=10,
    maxsize=10,
    strict=True,
)
conn = http.connection_from_url(url)
response = conn.urlopen(
    method='GET',
    url=url,
    headers={
        'Connection': 'keep-alive',
        'Accept-Encoding': 'gzip, compress',
    },
    preload_content=False,
    decode_content=False,
)
list(response.stream())
response = conn.urlopen(
    method='GET',
    url=url,
    headers={
        'Connection': 'keep-alive',
        'Accept-Encoding': 'gzip, compress',
    },
    preload_content=False,
    decode_content=False,
)
list(response.stream())
```

53a1e6a is the culprit.

@t-8ch do we really think closing the connection is the right thing to do here?

_Edit_ that _does_ fix the issue though

@sigmavirus24 I don't think so. But somehow httplib has to be conviced, that we are done with this response.

Doing

``` py
if self._original_response is not None:
    self._original_response.fp = None
self.release_conn()
```

fixes the bug but is missing testcoverage

@sigmavirus24 It seems even weirder. Running your example from upthread shows me _three_ requests on two sockets in wireshark.
The second request is duplicated and sent on two sockets. I assume the exception is caught somewhere.

Using `requests` gives me two requests on one socket (as expected, plus the exception).

Yeah, ofc retries... So requests disables retries which explains the differences.

Also:
testing against a local httpbin works (gunicorn does not use keepalives...). And reading the documentation for the `release_con` parameter to `ConnectionPool.__init__` we should _not_ try to automatically return the connection back to the pool, if we set `preload_content` (which we do).

@lmikkelsen Did you run into this problem using httpbin?

Thanks @lmikkelsen 

Yeh I am also having same issue is there a solution for this?

We're working on it. =) Keep an eye on this issue for more.

> On 24 Apr 2015, at 18:09, ne0ark notifications@github.com wrote:
> 
> Yeh I am also having same issue is there a solution for this?
> 
> —
> Reply to this email directly or view it on GitHub.

It seems like this is broken in mechanize as well.

I get a similar exception with the following script, but the traceback is different:

```
import requests

s = requests.session()

r = s.get('http://httpbin.org/stream/20')
s.get('http://httpbin.org/stream/20')
```

Start of traceback ([full](https://gist.github.com/untitaker/5b3ff800951747701ebb)):

```
Traceback (most recent call last):
  File "/home/untitaker/projects/requests/requests/packages/urllib3/connectionpool.py", line 372, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:
```

Happening with Python 3.4.3

Please don't print the start of the traceback, it's irrelevant. Print the _end_ of it. =D

To be clear @untitaker, your traceback is different because you've got Python 3 printing the chained exceptions. If the others users did, they'd show the same traceback as you.

Here is my traceback using urllib2:

https://gist.github.com/ne0ark/a451c13de9e6643a751e

So using _just_ urllib3, I'm having trouble reproducing this. The following is more-or-less what an adapter does in requests

``` py
import urllib3

url = 'http://httpbin.org/stream/20'
http = urllib3.PoolManager(
    block=False,
    num_pools=10,
    maxsize=10,
    strict=True,
)
conn = http.connection_from_url(url)
response = conn.urlopen(
    method='GET',
    url=url,
    headers={
        'Connection': 'keep-alive',
        'Accept-Encoding': 'gzip, compress',
    },
    preload_content=False,
    decode_content=False,
)
list(response.stream())
response = conn.urlopen(
    method='GET',
    url=url,
    headers={
        'Connection': 'keep-alive',
        'Accept-Encoding': 'gzip, compress',
    },
    preload_content=False,
    decode_content=False,
)
list(response.stream())
```

53a1e6a is the culprit.

@t-8ch do we really think closing the connection is the right thing to do here?

_Edit_ that _does_ fix the issue though

@sigmavirus24 I don't think so. But somehow httplib has to be conviced, that we are done with this response.

Doing

``` py
if self._original_response is not None:
    self._original_response.fp = None
self.release_conn()
```

fixes the bug but is missing testcoverage

@sigmavirus24 It seems even weirder. Running your example from upthread shows me _three_ requests on two sockets in wireshark.
The second request is duplicated and sent on two sockets. I assume the exception is caught somewhere.

Using `requests` gives me two requests on one socket (as expected, plus the exception).

Yeah, ofc retries... So requests disables retries which explains the differences.

Also:
testing against a local httpbin works (gunicorn does not use keepalives...). And reading the documentation for the `release_con` parameter to `ConnectionPool.__init__` we should _not_ try to automatically return the connection back to the pool, if we set `preload_content` (which we do).

@lmikkelsen Did you run into this problem using httpbin?


Resolving patch:
diff --git a/dummyserver/handlers.py b/dummyserver/handlers.py
--- a/dummyserver/handlers.py
+++ b/dummyserver/handlers.py
@@ -21,9 +21,6 @@
 
 class Response(object):
     def __init__(self, body='', status='200 OK', headers=None):
-        if not isinstance(body, bytes):
-            body = body.encode('utf8')
-
         self.body = body
         self.status = status
         self.headers = headers or [("Content-type", "text/plain")]
@@ -34,7 +31,20 @@ def __call__(self, request_handler):
         for header,value in self.headers:
             request_handler.add_header(header,value)
 
-        request_handler.write(self.body)
+        # chunked
+        if isinstance(self.body, list):
+            for item in self.body:
+                if not isinstance(item, bytes):
+                    item = item.encode('utf8')
+                request_handler.write(item)
+                request_handler.flush()
+        else:
+            body = self.body
+            if not isinstance(body, bytes):
+                body = body.encode('utf8')
+
+            request_handler.write(body)
+
 
 RETRY_TEST_NAMES = collections.defaultdict(int)
 
@@ -208,6 +218,9 @@ def successful_retry(self, request):
         else:
             return Response("need to keep retrying!", status="418 I'm A Teapot")
 
+    def chunked(self, request):
+        return Response(['123'] * 4)
+
     def shutdown(self, request):
         sys.exit()
 
diff --git a/urllib3/response.py b/urllib3/response.py
--- a/urllib3/response.py
+++ b/urllib3/response.py
@@ -455,4 +455,6 @@ def read_chunked(self, amt=None, decode_content=None):
                 break
 
         # We read everything; close the "file".
+        if self._original_response:
+            self._original_response.close()
         self.release_conn()


Test patch:
diff --git a/test/with_dummyserver/test_connectionpool.py b/test/with_dummyserver/test_connectionpool.py
--- a/test/with_dummyserver/test_connectionpool.py
+++ b/test/with_dummyserver/test_connectionpool.py
@@ -618,6 +618,25 @@ def test_source_address_error(self):
             self.assertRaises(ProtocolError,
                     pool.request, 'GET', '/source_address')
 
+    def test_stream_keepalive(self):
+        x = 2
+
+        for _ in range(x):
+            response = self.pool.request(
+                    'GET',
+                    '/chunked',
+                    headers={
+                        'Connection': 'keep-alive',
+                        },
+                    preload_content=False,
+                    retries=False,
+                    )
+            for chunk in response.stream():
+                self.assertEqual(chunk, b'123')
+
+        self.assertEqual(self.pool.num_connections, 1)
+        self.assertEqual(self.pool.num_requests, x)
+
 
 class TestRetry(HTTPDummyServerTestCase):
     def setUp(self):