Repository: urllib3/urllib3

Issue:
After read timeout, connection broken for a second timeout period
Using urllib3 1.12 to access etcd with a HTTPConnectionPool.  I do a long-poll read in a retry loop:

```
http = HTTPConnectionPool("localhost", 4001, maxsize=2)
while True:
    try:
        resp = http.request("GET", "http://localhost:4001/v2/keys/calico/v1",
                            fields={"recursive": "true", "wait": "true",
                                    "waitIndex": next_index},
                            timeout=5)
        resp_body = loads(resp.data)
    except ReadTimeoutError:
        _log.info("Watch read timed out, restarting watch at index %s",
                  next_index)
        continue
    else:
        ...
```

I expect some reads to time out because there was no data available.  In that case, I expect to hit the ReadTimeoutError, loop again and retry.  If there's then data waiting, I should get it immediately.

I do see the ReadTimeoutError after the first failure.  However, when my code retries, I see a 5s delay and then this log:

```
2015-10-15 09:38:16,311[DEBUG]273: Incremented Retry for (url='http://localhost:4001/v2/keys/calico/v1?waitIndex=7611342&recursive=true&wait=true'): Retry(total=2, connect=None, read=None, redirect=None)
2015-10-15 09:38:16,311[WARNING]625: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=4001): Read timed out.",)': http://localhost:4001/v2/keys/calico/v1?waitIndex=7611342&recursive=true&wait=true
```

Only then do I receive a response to my retry.



Discussion:
This is using normal Threads on Python 2.7, no gevent monkeypatching going on.

(It also repros 100% in my script.)

So just to clarify, when the 5s delay occurs on the _second_ call to `request()`, before either of those logs is emitted?

Hmm, I'm a little bit interested in where your `ReadTimeoutError` is coming from. My suspicion is that you should be getting some retries before that hits, but I'd like to see proof. Can you show me the traceback _immediately before_ you hit the hang?

Adjusted my script to give better diags:

```
2015-10-15 10:52:03,594 [INFO][46889/140035896854272] __main__ 64: About to call http.request...
2015-10-15 10:52:03,596 [INFO][46889/140035896854272] urllib3.connectionpool 207: Starting new HTTP connection (1): localhost
2015-10-15 10:52:03,598 [DEBUG][46889/140035896854272] urllib3.connectionpool 387: "GET http://localhost:4001/v2/keys/calico/v1?waitIndex=7653237&recursive=true&wait=true HTTP/1.1" 200 None
2015-10-15 10:52:08,604 [ERROR][46889/140035896854272] __main__ 72: Watch read timed out, restarting watch at index 7653237
Traceback (most recent call last):
  File "calico/felix/readetcd.py", line 69, in watch_etcd
    resp_body = loads(resp.data)
  File "/home/gulfstream/git/calico/env27/local/lib/python2.7/site-packages/urllib3/response.py", line 164, in data
    return self.read(cache_content=True)
  File "/home/gulfstream/git/calico/env27/local/lib/python2.7/site-packages/urllib3/response.py", line 292, in read
    flush_decoder = True
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/gulfstream/git/calico/env27/local/lib/python2.7/site-packages/urllib3/response.py", line 219, in _error_catcher
    raise ReadTimeoutError(self._pool, None, 'Read timed out.')
ReadTimeoutError: HTTPConnectionPool(host='localhost', port=4001): Read timed out.
2015-10-15 10:52:08,606 [INFO][46889/140035896854272] __main__ 64: About to call http.request...
2015-10-15 10:52:13,611 [DEBUG][46889/140035896854272] urllib3.util.retry 273: Incremented Retry for (url='http://localhost:4001/v2/keys/calico/v1?waitIndex=7653237&recursive=true&wait=true'): Retry(total=2, connect=None, read=None, redirect=None)
2015-10-15 10:52:13,612 [WARNING][46889/140035896854272] urllib3.connectionpool 625: Retrying (Retry(total=2, connect=None, read=None, redirect=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='localhost', port=4001): Read timed out. (read timeout=5)",)': http://localhost:4001/v2/keys/calico/v1?waitIndex=7653237&recursive=true&wait=true
2015-10-15 10:52:13,612 [INFO][46889/140035896854272] urllib3.connectionpool 207: Starting new HTTP connection (2): localhost
2015-10-15 10:52:13,613 [DEBUG][46889/140035896854272] urllib3.connectionpool 387: "GET http://localhost:4001/v2/keys/calico/v1?waitIndex=7653237&recursive=true&wait=true HTTP/1.1" 200 None
```

Yes, there's a 5s delay after the second call to request.

In case it helps, etcd pushes its headers immediately and then it's the read of the body that times out.

Right, ok, so that was roughly my theory. For people other than me, it should be noted that etcd sends chunked transfer encoding for its responses. This means that urllib3 does _not_ preload the content, which means that a ReadTimeout is not raised from `urlopen` but instead from `.data`.

The problem here is that we try to close the socket but it doesn't work (silently), because we technically don't have access to the connection. This is due to this interesting selection of code:

``` python

# First, in ConnectionPool.urlopen

if release_conn is None:
    release_conn = response_kw.get('preload_content', True)

...

response_conn = not release_conn and conn
response = HTTPResponse.from_httplib(httplib_response,
                                     pool=self,
                                     connection=response_conn,
                                     **response_kw)

# Then, in HTTPResponse.__init__

# We certainly don't want to preload content when the response is chunked.
if not self.chunked and preload_content and not self._body:
    self._body = self.read(decode_content=decode_content)

# Then back in ConnectionPool
finally:
    if release_conn:
        # Put the connection back to be reused. If the connection is
        # expired then it will be None, which will get replaced with a
        # fresh connection during _get_conn.
        self._put_conn(conn)
```

This is a problem: the response is basically overriding the logic of `preload_content`, which is _not_ only local to the response. Essentially, the response is choosing _not_ to preload the content if it's chunked, which is a bad idea.

I believe this is the root of the problem: because the connection still exists (in the pool), our error handling code in the response closes _its_ handle on the socket, but the connection keeps its one open. However, the socket is in an error state, so it's no longer useful.

Removing the (totally unnecessary) preload exemption for chunked transfer encoding will resolve the problem.


Resolving patch:
diff --git a/urllib3/response.py b/urllib3/response.py
--- a/urllib3/response.py
+++ b/urllib3/response.py
@@ -130,8 +130,8 @@ def __init__(self, body='', headers=None, status=0, version=0, reason=None,
         if "chunked" in encodings:
             self.chunked = True
 
-        # We certainly don't want to preload content when the response is chunked.
-        if not self.chunked and preload_content and not self._body:
+        # If requested, preload the body.
+        if preload_content and not self._body:
             self._body = self.read(decode_content=decode_content)
 
     def get_redirect_location(self):


Test patch: