Repository: urllib3/urllib3

Issue:
HTTPResponse.close may not close underlying connection.
Found while investigating kennethreitz/requests#2963

The `HTTPResponse` class has a `close` method that rather suggests it will try to close the backing TCP connection behind the given HTTP response. Right now, that's not what happens if the connection is kept alive for any reason (that is, if the server did not send `Connection: close`): instead, the TCP connection will be kept alive and handled as normal.

This seems moderately surprising to me. What it means, in practice, is that calling `HTTPResponse.close()` in both urllib3 and httplib/http.client does not guarantee the closure of the backing TCP connection: instead, in both cases it says "I'm done with the TCP connection, but the underlying connection is free to re-use it". The problems this causes can be see in the `_error_catcher` context manager on the HTTPResponse which does not actually call the class `close` method, presumably because it's too deficient to do the job.

This behaviour affects the chunked transfer encoding decoding logic which calls `self.close()` and therefore may incorrectly keep the connection alive, though it does not itself return the connection to the pool.

I believe it _should_ be safe to have `close` close the underlying connection if it is present. As something of an optimisation, we can then safely assume that `close` can call `release_conn`, which will allow us to keep hold of the `HTTPConnection` object in a situation where otherwise we might lose it.



Discussion:
Reason 89 for dropping httplib.


Resolving patch:
diff --git a/urllib3/response.py b/urllib3/response.py
--- a/urllib3/response.py
+++ b/urllib3/response.py
@@ -387,6 +387,9 @@ def close(self):
         if not self.closed:
             self._fp.close()
 
+        if self._connection is not None:
+            self._connection.close()
+
     @property
     def closed(self):
         if self._fp is None:


Test patch:
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -433,6 +433,53 @@ def socket_handler(listener):
                                     timeout=Timeout(connect=1, read=0.1))
             self.assertEqual(len(response.read()), 8)
 
+    def test_closing_response_actually_closes_connection(self):
+        done_closing = Event()
+        complete = Event()
+        # The insane use of this variable here is to get around the fact that
+        # Python 2.6 does not support returning a value from Event.wait(). This
+        # means we can't tell if an event timed out, so we can't use the timing
+        # out of the 'complete' event to determine the success or failure of
+        # the test. Python 2 also doesn't have the nonlocal statement, so we
+        # can't write directly to this variable, only mutate it. Hence: list.
+        successful = []
+
+        def socket_handler(listener):
+            sock = listener.accept()[0]
+
+            buf = b''
+            while not buf.endswith(b'\r\n\r\n'):
+                buf = sock.recv(65536)
+
+            sock.send(('HTTP/1.1 200 OK\r\n'
+                      'Content-Type: text/plain\r\n'
+                      'Content-Length: 0\r\n'
+                      '\r\n').encode('utf-8'))
+
+            # Wait for the socket to close.
+            done_closing.wait(timeout=1)
+
+            # Look for the empty string to show that the connection got closed.
+            # Don't get stuck in a timeout.
+            sock.settimeout(1)
+            new_data = sock.recv(65536)
+            self.assertFalse(new_data)
+            successful.append(True)
+            sock.close()
+            complete.set()
+
+        self._start_server(socket_handler)
+        pool = HTTPConnectionPool(self.host, self.port)
+
+        response = pool.request('GET', '/', retries=0, preload_content=False)
+        self.assertEqual(response.status, 200)
+        response.close()
+
+        done_closing.set()  # wait until the socket in our pool gets closed
+        complete.wait(timeout=1)
+        if not successful:
+            self.fail("Timed out waiting for connection close")
+
 
 class TestProxyManager(SocketDummyServerTestCase):