Repository: urllib3/urllib3

Issue:
Connection pool queue can still be exhausted by gevent timeouts
A similar error was discussed here https://github.com/shazow/urllib3/issues/644 already and some fix applied, but the problem still partly exists. Consider this code:

``` python
from gevent import monkey, spawn, sleep, Timeout, joinall
monkey.patch_all()

import sys
from itertools import product
from urllib3.connectionpool import HTTPConnectionPool

def timed_read(cp, i, timeout, release=None, preload_content=True):
    try:
        with Timeout(timeout):
            resp = cp.urlopen('GET', 'http://httpbin.org/redirect/:1',
                              release_conn=release,
                              preload_content=preload)
            print "%s opened" % i
            sys.stdout.flush()
            data = resp.data
    except Timeout:
        print "%s timed out" % i
    else:
        content_len = int(resp.headers['content-length'])
        if len(data) != content_len:
            print "%s incomplete read: %s/%s" % (i, len(data), content_len)
        else:
            print "%s finished" % i
    sys.stdout.flush()


def main(timeout=10, release=None, preload=True, poolsize=2):
    print '-' * 10 + "  timeout=%s preload=%s release=%s " % (timeout, preload, release) + '-' * 10
    cp = HTTPConnectionPool('httpbin.org', block=True, maxsize=poolsize, timeout=None)
    group = []
    for i in range(3):
        group.append(spawn(timed_read, cp, i, timeout, release=release, preload_content=preload))
    joinall(group)

    current_size = cp.pool.qsize()
    if current_size != poolsize:
        print "poolsize decreased: %s" % current_size


for timeout, preload in product([10, 4, 0.1], [True, False]):
    main(timeout=timeout, preload=preload)
```

Setting a timeout that way may seem stupid, as it could be set on the connection itself, but it's just an example for having a timeout running on a longer running task including several calls into the connection pool. What happens is, that the connections are not properly returned back into the queue, and the queue finally runs empty, and any further call to it blocks forever:

<pre>
----------  timeout=10 preload=True release=None ----------
1 opened
1 finished
0 opened
0 finished
2 opened
2 finished
----------  timeout=10 preload=False release=None ----------
0 opened
0 finished
1 opened
1 finished
2 opened
2 finished
----------  timeout=4 preload=True release=None ----------
1 opened
1 finished
0 timed out
2 timed out
----------  timeout=4 preload=False release=None ----------
1 opened
1 finished
0 opened
0 finished
2 timed out
poolsize decreased: 1
----------  timeout=0.1 preload=True release=None ----------
0 timed out
1 timed out
2 timed out
----------  timeout=0.1 preload=False release=None ----------
0 timed out
1 timed out
2 timed out
poolsize decreased: 0
</pre>


Simply setting `preload_content` to `True` is not an option, as e.g. `False` is set by default by requests.

The main problem is in https://github.com/shazow/urllib3/blob/master/urllib3/response.py#L246 . The `Timeout` does not derive from `Exception`, so `self._original_response` is not closed in the except clause, and therefore the connection is not released in the finally clause.



Discussion:
Urgh, so `gevent.Timeout` is an instance of `BaseException`, which is pretty annoying. I understand why they did it (didn't want overbroad `except` clauses preventing the timeout from firing), but it leads to awkward problems like these.

The simplest way to do this might be just to broaden _our_ `except` clause. Because there's no way we can swallow the exception inside our `except` block, we may want to write `except BaseException` instead and then comment why we did it.

Of course, this is a fairly nonsensical arms race, so the other option is to have an `else` clause that sets a boolean on clean exit, and if that boolean isn't present have the `finally` clause close and dispose of the connection. That would move almost all the error handling into `finally`, so it'd be fairly bloody similar to the current code, but might appease those who object to overbroad `except` blocks.

@shazow, I'm happy to do the legwork to fix this: do you have a preference as to how?

@Lukasa Yea I think a cleanup boolean + finally is the way to go for complex cleanups based on how the rest of our code has been evolving.

Looks good to me.

@ml31415 Should be fixed in master, can you confirm? (Let's re-open if it's still an issue.)

Hmmm, unfortunately it looks like  it's still there. Looks like i overlooked something initially :/

<pre>
----------  timeout=0.1 preload=False release=None ----------
0 timed out
1 timed out
2 timed out
poolsize decreased: 0
</pre>

Though I think the changes in #805 are the right thing to do.

Yea we're keeping the changes, just removing the changelog entry that says it fixes a gevent thing since it's not fixed. :)

@ml31415 Do we have a shorter repro scenario than the one above? Ideally one it's not a problem for me to run locally?

Hmm, not shorter, but at least local:

``` python
from gevent import monkey, spawn, sleep, Timeout, joinall
from gevent.server import StreamServer
monkey.patch_all()

import sys
from itertools import product
from urllib3.connectionpool import HTTPConnectionPool


RESPONSE = """
HTTP/1.1 200 OK
Server: gevent
Content-Type: text/html; charset=windows-1251
Content-Encoding: identity
Content-Length: 10

asdfasdf

""".lstrip().replace('\n', '\r\n')
# Do not remove the final empty line!


class RawServer(object):
    def __init__(self, response, host='127.0.0.1'):
        self.response = response
        self.server = StreamServer((host, 0), self.handler)

    def handler(self, sock, addr):
        sleep(1)
        sock.sendall(self.response)

    @property
    def url(self):
        return 'http://%s:%s' % self.server.address

    def __enter__(self):
        self.server.start()
        return self

    def __exit__(self, *args, **kwargs):
        self.server.stop()


def timed_read(cp, i, timeout, host, port, release=None, preload_content=True):
    try:
        with Timeout(timeout):
            resp = cp.urlopen('GET', 'http://%s:%s' % (host, port),
                              release_conn=release,
                              preload_content=preload)
            print "%s opened" % i
            data = resp.data
    except Timeout:
        print "%s timed out" % i
    else:
        print "%s finished" % i


def main(timeout=10, release=None, preload=True, poolsize=2):
    with RawServer(RESPONSE) as s:
        host, port = s.server.address
        cp = HTTPConnectionPool(*s.server.address, block=True, maxsize=poolsize, timeout=None)
        print '-' * 10 + "  timeout=%s preload=%s release=%s " % (timeout, preload, release) + '-' * 10
        group = []
        for i in range(3):
            group.append(spawn(timed_read, cp, i, timeout, host, port, release=release, preload_content=preload))
        joinall(group)

    current_size = cp.pool.qsize()
    if current_size != poolsize:
        print "poolsize decreased: %s" % current_size


for timeout, preload in product([2, 0.1], [True, False]):
    main(timeout=timeout, preload=preload)
```

I'll try to get the whole thing more spot on than this.

There is another scenario, that guaratees an exhausted pool, when the greenlet / thread fails, before the response is read (the timeout stuff is only there, so it doesn't block eternally on the empty loop):

``` python
def timed_read(cp, i, timeout, host, port, release=None, preload_content=True):
    try:
        with Timeout(timeout):
            resp = cp.urlopen('GET', 'http://%s:%s' % (host, port),
                              release_conn=release,
                              preload_content=preload)
            print "%s opened" % i
            1 / 0
            data = resp.data
    except Timeout:
        print "%s timed out" % i
    except ZeroDivisionError:
        print "%s intentionally failed" % i
    else:
        print "%s finished" % i
```

I guess the best solution to this would be, if the response supports the with protocol, so we could do smth like:

``` python

with cp.urlopen('asdf.org') as resp:
    1 / 0
    resp.data
```

And it still cleans itself up afterwards.

Shorter test version:

``` python
def test_response_closed():
    class Raiser(object):
        def read(self):
            raise Timeout()

    with RawServer(RESPONSE) as s:
        cp = HTTPConnectionPool(*s.server.address)
        resp = cp.urlopen('GET', s.url, preload_content=False)
        resp._fp = Raiser()
        try:
            resp.data
        except Timeout:
            print "timed out"
        assert resp._connection is None
        assert resp._original_response.isclosed()

```

@ml31415 Just to be clear, are you testing with the master branch? For me, the master branch passes your smaller test.

```
gevent==1.0.2
greenlet==0.4.9
-e git+git@github.com:Lukasa/urllib3.git@3ecc0cee79971e9cae9412266e07001c7922104b#egg=urllib3
wheel==0.26.0
```

Hmm, yes the smaller test runs now, yes. Still the other one doesn't. And I actually don't really understand how it still fails :(

Ok, so the reason you're still seeing this problem is that, in basically all these cases the timeout is _actually_ occurring in the connection handling code. This is not covered by the fix in the #806, because I didn't touch that other block there. I think it likely needs a similar change.

The change for that block is going to be a bit trickier in part because that block _already_ has a condition variable for releasing the connection, and even worse, that condition variable is passed _into_ the function (so I can't default it to something that will behave sensibly).

This is definitely fix-able, though it's going to be a bit frustrating to have two connection-close metavariables.


Resolving patch:
diff --git a/urllib3/connectionpool.py b/urllib3/connectionpool.py
--- a/urllib3/connectionpool.py
+++ b/urllib3/connectionpool.py
@@ -556,6 +556,10 @@ def urlopen(self, method, url, body=None, headers=None, retries=None,
         # complains about UnboundLocalError.
         err = None
 
+        # Keep track of whether we cleanly exited the except block. This
+        # ensures we do proper cleanup in finally.
+        clean_exit = False
+
         try:
             # Request a connection from the queue.
             timeout_obj = self._get_timeout(timeout)
@@ -585,10 +589,8 @@ def urlopen(self, method, url, body=None, headers=None, retries=None,
                                                  connection=response_conn,
                                                  **response_kw)
 
-            # else:
-            #     The connection will be put back into the pool when
-            #     ``response.release_conn()`` is called (implicitly by
-            #     ``response.read()``)
+            # Everything went great!
+            clean_exit = True
 
         except Empty:
             # Timed out by queue.
@@ -598,22 +600,19 @@ def urlopen(self, method, url, body=None, headers=None, retries=None,
             # Close the connection. If a connection is reused on which there
             # was a Certificate error, the next request will certainly raise
             # another Certificate error.
-            conn = conn and conn.close()
-            release_conn = True
+            clean_exit = False
             raise SSLError(e)
 
         except SSLError:
             # Treat SSLError separately from BaseSSLError to preserve
             # traceback.
-            conn = conn and conn.close()
-            release_conn = True
+            clean_exit = False
             raise
 
         except (TimeoutError, HTTPException, SocketError, ProtocolError) as e:
             # Discard the connection for these exceptions. It will be
             # be replaced during the next _get_conn() call.
-            conn = conn and conn.close()
-            release_conn = True
+            clean_exit = False
 
             if isinstance(e, (SocketError, NewConnectionError)) and self.proxy:
                 e = ProxyError('Cannot connect to proxy.', e)
@@ -628,6 +627,14 @@ def urlopen(self, method, url, body=None, headers=None, retries=None,
             err = e
 
         finally:
+            if not clean_exit:
+                # We hit some kind of exception, handled or otherwise. We need
+                # to throw the connection away unless explicitly told not to.
+                # Close the connection, set the variable to None, and make sure
+                # we put the None back in the pool to avoid leaking it.
+                conn = conn and conn.close()
+                release_conn = True
+
             if release_conn:
                 # Put the connection back to be reused. If the connection is
                 # expired then it will be None, which will get replaced with a


Test patch:
diff --git a/test/test_connectionpool.py b/test/test_connectionpool.py
--- a/test/test_connectionpool.py
+++ b/test/test_connectionpool.py
@@ -283,6 +283,33 @@ def test_ca_certs_default_cert_required(self):
             conn = pool._get_conn()
             self.assertEqual(conn.cert_reqs, 'CERT_REQUIRED')
 
+    def test_cleanup_on_extreme_connection_error(self):
+        """
+        This test validates that we clean up properly even on exceptions that
+        we'd not otherwise catch, i.e. those that inherit from BaseException
+        like KeyboardInterrupt or gevent.Timeout. See #805 for more details.
+        """
+        class RealBad(BaseException):
+            pass
+
+        def kaboom(*args, **kwargs):
+            raise RealBad()
+
+        c = connection_from_url('http://localhost:80')
+        c._make_request = kaboom
+
+        initial_pool_size = c.pool.qsize()
+
+        try:
+            # We need to release_conn this way or we'd put it away regardless.
+            c.urlopen('GET', '/', release_conn=False)
+        except RealBad:
+            pass
+
+        new_pool_size = c.pool.qsize()
+        self.assertEqual(initial_pool_size, new_pool_size)
+
+
 
 if __name__ == '__main__':
     unittest.main()