Repository: urllib3/urllib3

Issue:
Connection pool exhausted when connection failures occur, should refill with empty connections
If preload_content is not specified on a request, connections are implicitly returned to the pool after the request is read. However, when a timeout error occurs or the connection is dropped the connection is closed but not returned in to the pool (since the response is None). This problem is compounded when retries are turned on since the next attempt will grab a new connection from the pool thus depleting it further. With non-blocking pools this is not a problem since we will create a new connection if the pool is empty but when the pool is blocking we have found that eventually the pool size becomes zero (after no.of connections timeout errors) which causes the calling application to hang on it's next request.
This can be fixed in connectionpool.urlopen in the exception handlers by explicitly returning the closed connection to the pool via _put_conn (if release_conn is false), since subsequent calls to _get_conn will check if the connection has been dropped and return a new one.



Discussion:
Hello again!

I think we need to be sure that we close the connection before we return them to the pool. Otherwise we run the risk of attempting to re-use a live connection that has timed out, which will end extremely poorly for us.

We should be putting a `None` into the pool if we want to discard the connection. That will be replaced with a fresh connection when we try to get it out of the pool. That does sound like a bug.

@shazow's idea is even better than mine.

@jlatherfold Is this something you'd be interested in working on?

Producing a failing test would be the first step. Make a ConnectionPool with a small pool (1-2), trigger some connection exceptions, see if the pool gets exhausted.

Hi,

I've already done this, hence this bug report. Locally I've fixed it by putting the [closed] connection back on to the pool in the exception handlers (which do close the connections). Those Nones or closed connections are discarded and a new one is returned in subsequent calls to _get_conn. Sadly I don't know when I will get time to do any more on this right now.

@jlatherfold Would be handy if you can share your code for reproducing the issue. :)

Also if you can confirm that putting `None` rather than the closed connection still works as expected, that would be great.

I can confirm that _put_conn(None) works as expected.

Here's some basic code, which also produces this error. Setting the read-timeout to a low value will cause read timeout exceptions. You'll need to hit a url with some large content - doesn't have to be massive - 500KB or so although you can probably replicate this with much less):

``` python
import os
import sys
import requests

def get(req, url):
    r = req.get(url, stream=True, timeout=(0.5, 0.1)) # set get timeout low to force timeout exceptions
    r.raise_for_status()

    return r.iter_content(chunk_size=4096)

if __name__ == '__main__':

    # if you need to auth the session do so....
    session = requests.Session()

    # setting the poolsize to 1 and 1 retries will result in this program
    # hanging on the first retry attempt (failed connection or read timeout)
    # set the pool_maxsize=4 and set some trace statements in connectionpool.py
    # to print the queue size in _get_conn and _put_conn and watch it slowly
    # decrease as we hit timeouts and retries.....
    adpater = requests.adapters.HTTPAdapter(pool_connections=1, pool_maxsize=1,
                                            pool_block=True, max_retries=1) 
    session.mount('http://', adpater)

    while True:

        try:
            print('request.get......')
            content = get(session, '<url to large content>')

            # consume the response
            for chunk in content:
                pass
        except Exception:
            print('ops.....')

        # after a while when you've hit few timeouts you're not going to get here....
        print('request.done......')

```

Sorry - didn't know how to format for code - but you get the idea....

Updated code formatting.

Bonus points if somebody wants to translate that to plain-urllib3 and double-bonus if you make it into a test. :)

Ok, I'll make the patch myself. Please let me know what procedure I need to follow or point me in the right direction where I can get that info as I have not made any code contributions before.

@jlatherfold Thanks! Fork the project, make a branch, start with a test that shows the failure (we have full test coverage, have a look at other tests for examples), then send a PR for feedback. We'll review the code and give you more pointers.

If you can think of a way to reproduce the scenario without doing a full end-to-end request, then urllib/tests/test_connectionpool.py would be a good place to put it. If you need a server, then take a look at the with_dummyserver subdirectory of suites. We lately prefer the test_socketlevel.py method of testing with a dummyserver when possible.

Might be wise to just write a simple example of the scenario reproducing in urllib3 first, to make sure it's not requests' fault.

Hi,

I now have a fix for this with unit test and am ready to submit a PR. The work has been done on a local branch. Would you like me to merge it to master or push the branch to my fork and use that?

@jlatherfold Whatever you prefer, as long as it ends up as a Pull Request on our end. :)


Resolving patch:
diff --git a/urllib3/connectionpool.py b/urllib3/connectionpool.py
--- a/urllib3/connectionpool.py
+++ b/urllib3/connectionpool.py
@@ -120,7 +120,7 @@ class HTTPConnectionPool(ConnectionPool, RequestMethods):
 
     :param maxsize:
         Number of connections to save that can be reused. More than 1 is useful
-        in multithreaded situations. If ``block`` is set to false, more
+        in multithreaded situations. If ``block`` is set to False, more
         connections will be created but they will not be saved once they've
         been used.
 
@@ -409,7 +409,7 @@ def is_same_host(self, url):
 
         # TODO: Add optional support for socket.gethostbyname checking.
         scheme, host, port = get_host(url)
-
+ 
         # Use explicit default port for comparison when none is given
         if self.port and not port:
             port = port_by_scheme.get(scheme)
@@ -568,25 +568,22 @@ def urlopen(self, method, url, body=None, headers=None, retries=None,
             # Close the connection. If a connection is reused on which there
             # was a Certificate error, the next request will certainly raise
             # another Certificate error.
-            if conn:
-                conn.close()
-                conn = None
+            conn = conn and conn.close()
+            release_conn = True
             raise SSLError(e)
 
         except SSLError:
             # Treat SSLError separately from BaseSSLError to preserve
             # traceback.
-            if conn:
-                conn.close()
-                conn = None
+            conn = conn and conn.close()
+            release_conn = True
             raise
 
         except (TimeoutError, HTTPException, SocketError, ConnectionError) as e:
-            if conn:
-                # Discard the connection for these exceptions. It will be
-                # be replaced during the next _get_conn() call.
-                conn.close()
-                conn = None
+            # Discard the connection for these exceptions. It will be
+            # be replaced during the next _get_conn() call.
+            conn = conn and conn.close()
+            release_conn = True
 
             if isinstance(e, SocketError) and self.proxy:
                 e = ProxyError('Cannot connect to proxy.', e)
@@ -626,6 +623,9 @@ def urlopen(self, method, url, body=None, headers=None, retries=None,
                 retries = retries.increment(method, url, response=response, _pool=self)
             except MaxRetryError:
                 if retries.raise_on_redirect:
+                    # Release the connection for this response, since we're not
+                    # returning it to be released manually.
+                    response.release_conn()
                     raise
                 return response
 


Test patch:
diff --git a/test/test_connectionpool.py b/test/test_connectionpool.py
--- a/test/test_connectionpool.py
+++ b/test/test_connectionpool.py
@@ -217,13 +217,10 @@ def test_contextmanager(self):
             old_pool_queue = pool.pool
 
         self.assertEqual(pool.pool, None)
-
         self.assertRaises(ClosedPoolError, pool._get_conn)
 
         pool._put_conn(conn3)
-
         self.assertRaises(ClosedPoolError, pool._get_conn)
-
         self.assertRaises(Empty, old_pool_queue.get, block=False)
 
 
diff --git a/test/with_dummyserver/test_connectionpool.py b/test/with_dummyserver/test_connectionpool.py
--- a/test/with_dummyserver/test_connectionpool.py
+++ b/test/with_dummyserver/test_connectionpool.py
@@ -647,6 +647,27 @@ def test_chunked_gzip(self):
 
         self.assertEqual(b'123' * 4, response.read())
 
+    def test_cleanup_on_connection_error(self):
+        '''
+        Test that connections are recycled to the pool on 
+        connection errors where no http response is received.
+        '''
+        poolsize = 3
+        with HTTPConnectionPool(self.host, self.port, maxsize=poolsize, block=True) as http:
+            self.assertEqual(http.pool.qsize(), poolsize)
+
+            # force a connection error by supplying a non-existent 
+            # url. We won't get a response for this  and so the 
+            # conn won't be implicitly returned to the pool.
+            self.assertRaises(MaxRetryError,
+                http.request, 'GET', '/redirect', fields={'target': '/'}, release_conn=False, retries=0)
+
+            r = http.request('GET', '/redirect', fields={'target': '/'}, release_conn=False, retries=1)
+            r.release_conn()
+
+            # the pool should still contain poolsize elements
+            self.assertEqual(http.pool.qsize(), http.pool.maxsize)
+
 
 class TestRetry(HTTPDummyServerTestCase):
     def setUp(self):
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -119,8 +119,9 @@ def socket_handler(listener):
     def test_connection_refused(self):
         # Does the pool retry if there is no listener on the port?
         host, port = get_unreachable_address()
-        pool = HTTPConnectionPool(host, port)
-        self.assertRaises(MaxRetryError, pool.request, 'GET', '/', retries=0)
+        http = HTTPConnectionPool(host, port, maxsize=3, block=True)
+        self.assertRaises(MaxRetryError, http.request, 'GET', '/', retries=0, release_conn=False)
+        self.assertEqual(http.pool.qsize(), http.pool.maxsize)
 
     def test_connection_read_timeout(self):
         timed_out = Event()
@@ -133,13 +134,15 @@ def socket_handler(listener):
             sock.close()
 
         self._start_server(socket_handler)
-        pool = HTTPConnectionPool(self.host, self.port, timeout=0.001, retries=False)
+        http = HTTPConnectionPool(self.host, self.port, timeout=0.001, retries=False, maxsize=3, block=True)
 
         try:
-            self.assertRaises(ReadTimeoutError, pool.request, 'GET', '/')
+            self.assertRaises(ReadTimeoutError, http.request, 'GET', '/', release_conn=False)
         finally:
             timed_out.set()
 
+        self.assertEqual(http.pool.qsize(), http.pool.maxsize)
+
     def test_https_connection_read_timeout(self):
         """ Handshake timeouts should fail with a Timeout"""
         timed_out = Event()