Repository: urllib3/urllib3

Issue:
urllib3 hanging inside call to PoolManager request
I have a repeatable issue with urllib3 hanging inside PoolManager request for an unusual edge case.  I have repeated this issue on two separate machines.

I am using Windows 10, 64-bit (fully patched) and Python 3.8.2, 64-bit.  The urllib3 version is 1.25.8 in a fresh venv environment.

The edge case is:
  1. Second call to urllib3 PoolManager request hangs inside request call and never returns
  2. Using streaming to read and write response to file
  3. url has a redirect (http status 301)

The original code used concurrent.futures, but the issue is the same without it using a simple loop.

I have put together a small extract from the code I have been running for about one year.  It was only when I recently started with a new use case that co-incidentally started using urls with redirects that I found this problem where each of concurrent.futures workers in my Python application would hang.  Then I removed concurrent features.

After stepping through the urllib3 source code, I could not figure out the root cause.  

I ask if others can reproduce the issue and then help find find the root cause.

# Code:
```
import logging
from pathlib import Path
import shutil
import urllib3

_logger = logging.getLogger(__name__)
_BUFFER_SIZE = 1024 * 1024   # buffer for downloading remote resource

url_headers = urllib3.make_headers(keep_alive=True, accept_encoding=True)
url_retries = urllib3.Retry(total=2, backoff_factor=1, status_forcelist=[500, 502, 503, 504])
url_client = urllib3.PoolManager(timeout=urllib3.Timeout(total=15.0), retries=url_retries, block=True,
                                 headers=url_headers)


def stream_response(url: str, filepath: Path, id: int):
    # Must call release_conn() after file copied but opening/writing exception is possible
    rsp = None
    try:
        _logger.debug(f'> {id:4d} GET: {url}')
        rsp = url_client.request('GET', url, preload_content=False)
        _logger.debug(f'> {id:4d} resp code: {rsp.status}')
        if rsp.status == 200:
            _logger.debug(f'> {id:4d} writing: "{filepath.name}"')
            with filepath.open('wb', buffering=_BUFFER_SIZE) as rfp:
                shutil.copyfileobj(rsp, rfp, length=_BUFFER_SIZE)
    except Exception as ex:
        _logger.exception(f'> {id:4d} Error')
    finally:
        if rsp:
            rsp.release_conn()

    return rsp.status


logging.basicConfig(level=logging.DEBUG,
            handlers=[logging.FileHandler('output.log', mode='w'), logging.StreamHandler()])
logging.captureWarnings(True)

_logger.info('Started')
items = [
    ['https://fwd.aws/v3mz8', Path('d1.pdf')],
    ['https://fwd.aws/88JKk', Path('d2.pdf')],
    ['https://fwd.aws/D9yDG', Path('d3.pdf')],
    ['https://fwd.aws/NmXza', Path('d4.pdf')],
    ['https://fwd.aws/NmXza', Path('d5.pdf')]
]

for id, it in enumerate(items):
    stream_response(it[0], it[1], id)
```

# Output:
```
D:\Temp\Python issue>
D:\Temp\Python issue>.env\Scripts\activate

(.env) D:\Temp\Python issue>
(.env) D:\Temp\Python issue>python urllib3_locking.py
INFO:__main__:Started
DEBUG:__main__:>    0 GET: https://fwd.aws/v3mz8
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): fwd.aws:443
DEBUG:urllib3.connectionpool:https://fwd.aws:443 "GET /v3mz8 HTTP/1.1" 301 0
DEBUG:urllib3.util.retry:Incremented Retry for (url='https://fwd.aws/v3mz8'): Retry(total=2, connect=None, read=None, redirect=None, status=None)
INFO:urllib3.poolmanager:Redirecting https://fwd.aws/v3mz8 -> https://aws-quickstart.s3.amazonaws.com/quickstart-hashicorp-consul/doc/hashicorp-consul-on-the-aws-cloud.pdf
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): aws-quickstart.s3.amazonaws.com:443
DEBUG:urllib3.connectionpool:https://aws-quickstart.s3.amazonaws.com:443 "GET /quickstart-hashicorp-consul/doc/hashicorp-consul-on-the-aws-cloud.pdf HTTP/1.1" 200 1610783
DEBUG:__main__:>    0 resp code: 200
DEBUG:__main__:>    0 writing: "d1.pdf"
DEBUG:__main__:>    1 GET: https://fwd.aws/88JKk

Process finished with exit code -1
```

I have to manually terminate the application (CPU usage for process is zero) and the output never shows the response http status code.

[Python issue - 200307.zip](https://github.com/urllib3/urllib3/files/4301755/Python.issue.-.200307.zip)



Discussion:
Thanks for the full example! Can you try setting `block=False` when instantiating urllib3.PoolManager to see if it helps?
Hi @pquentin 
I tried setting `block=False` as you suggested, and it made no difference.  The call to urllib3.PoolManager request still hangs and never returns
Well, I reproduce your problem, and setting `block=False` (or `maxsize=10`) fixes it for me. So I still *think* my analysis below is correct, but I don't explain why the `block=False` workaround does not help for you. Are you sure you're using the exact example in your first post?

The problem lies at the interaction between blocking connection pools, redirects and streaming download.

urllib3 uses one connection pool by host, in your example there are three hosts if we include redirects: `fwd.aws`, `aws-quickstart.s3.amazonaws.com` and `aws.amazon.com`. So that's three connection pools. Each connection pool has `block=True` and `maxsize=1` (the default `maxsize`), so you can only perform one request at a time because there can be only one connection per pool.

So what's going on in your example? When you request `https://fwd.aws/v3mz8`, you:

 * open a connection to `fwd.aws`, then get a redirect,
 * so you open a connection to `aws-quickstart.s3.amazonaws.com`, read the file
 * then `release_conn` releases the connection to `aws-quickstart.s3.amazonaws.com` to the pool for further use.

But the connection to `fwd.aws` was never released! urllib3 should have read the `fwd.aws` response and released the connection before performing the redirect. This is a bug that was fixed in connection pools in https://github.com/urllib3/urllib3/pull/1174, but unfortunately pool managers also have the bug and it wasn't fixed there. Congratulations for finding the bug. :)

And indeed, `calling drain_and_release_conn` just before the recursive call in `PoolManager.urlopen()` also allows me to run your example without problems.

Would you be interested in working on a fix? This would involve moving `drain_and_release_conn()` to src/urllib3/response.py, calling it both from connectionpool.py and poolmanager.py, and writing tests for the `PoolManager` case (by copying the connection pool tests from https://github.com/urllib3/urllib3/pull/1174).
Hi @pquentin 
I now also can use the workaround `block=False`.  I had run the code from within PyCharm which hangs regardless, but running the code from the command line has the same result as you (ie does not hang with `block=False`).  I might have a look at this test running from VSCode out-of-interest.
I have an understanding of your explanation and I will have an indepth look at the urrlib3 code.  I will see if I am up to the fix (being refactoring and new unit tests).
@tycarac That's great to hear! I'd be happy to assist if you have any difficulties. I guess the first step is to call `drain_and_release_conn()` before the recursive call/info log here:

https://github.com/urllib3/urllib3/blob/33a29c5e34ee3375cde07addeb979aba56f2ca5a/src/urllib3/poolmanager.py#L365-L369

And see if that fixes your issue locally
@pquentin 
I have a fix (with unit test) ready for review.  The unit test took a long time as my first test was complicated and then I found ways to simplify it.

But, I am at a loss to how to submit it.  This fix has been a lot of firsts for me (first time using nox and second time using pytest, first GitHub contribution, first git pull request).  I have only seriously re-started programming (after a years and years and years break) 3 months ago.  Back then I uses Visual SourceSafe :-)
So could you please say if I should submit to a git branch.  If so, which one?

The unit test hangs without the fix "drain_and_release_conn", but I consider it fine-by-me as a test should never fail.  That OK by you?

I found it useful to put: 
```
import logging
import os
logging.basicConfig(level=logging.DEBUG,
            handlers=[logging.FileHandler(os.path.splitext(__file__)[0] + '.log', mode='w'), logging.StreamHandler()])
logging.captureWarnings(True)
```
at the top of the test file but it generates a log file per test file (and was not part of the original test source code).  But it is useful to see the urllib3 internal logging statements.
Lots of firsts indeed! Congratulations on the progress!

For your next pull request you can try following https://guides.github.com/activities/hello-world/ but for this one can you please upload the modified files on GitHub Gist? I'll also need a name or pseudonym and possibly an email.

I'll create the pull request myself if that's okay with you.

Thanks!

[urllib3 - 200317.zip](https://github.com/urllib3/urllib3/files/4340577/urllib3.-.200317.zip)

@pquentin 
I would have preferred to put the code directly into urllib3 but I see the permission problem [Permission levels](https://help.github.com/en/github/setting-up-and-managing-your-github-user-account/permission-levels-for-a-user-account-repository).  Either you have a LOT of access or very limited access - even for contributors.

I put the patch (3 files) at [Patch on Gist](https://gist.github.com/tycarac/2a61d3169b6ef49ed577eb611fb807eb).  Notes:
1. Without the src\urllib3\poolmanager.py change, the test suite execution hangs.  I did not investigate test timeouts as I did not see timers used elsewhere and it adds complexity
2. The change to handlers.py is nice-to-have as it gives a counter for loops.  I used "page" as it is a typical query key.
3. I looked for the required exceptions in the function drain_and_release_conn().  Most urllib3 errors subclass from HTTPError
4. The unit test loops 5 times - an arbitrary number greater than 3 used for default max retries and redirects
5. I did not see loops used in unit tests but it does not make the unit test more complex or add to test execution time.
Otherwise I tried to follow the style and conventions I found in the code.

> I would have preferred to put the code directly into urllib3 but I see the permission problem [Permission levels](https://help.github.com/en/github/setting-up-and-managing-your-github-user-account/permission-levels-for-a-user-account-repository). Either you have a LOT of access or very limited access - even for contributors.

Ah, I see. You need to fork the urllib3 repository in your own namespace, push your changes to a branch, and submit a pull request. It's not easy when you do that for the first time, but I think GitHub has nice documentation to do this.

Please tell me if you want to go that route, it's the best way to learn really fast but can be demotivating too. :) 
I will give it a go.  Be brave :-).  I know enough about git to know I can make a real mess, but that's the way to learn.

Resolving patch:
diff --git a/src/urllib3/connectionpool.py b/src/urllib3/connectionpool.py
--- a/src/urllib3/connectionpool.py
+++ b/src/urllib3/connectionpool.py
@@ -765,21 +765,6 @@ def urlopen(
                 **response_kw
             )
 
-        def drain_and_release_conn(response):
-            try:
-                # discard any remaining response body, the connection will be
-                # released back to the pool once the entire response is read
-                response.read()
-            except (
-                TimeoutError,
-                HTTPException,
-                SocketError,
-                ProtocolError,
-                BaseSSLError,
-                SSLError,
-            ):
-                pass
-
         # Handle redirect?
         redirect_location = redirect and response.get_redirect_location()
         if redirect_location:
@@ -790,15 +775,11 @@ def drain_and_release_conn(response):
                 retries = retries.increment(method, url, response=response, _pool=self)
             except MaxRetryError:
                 if retries.raise_on_redirect:
-                    # Drain and release the connection for this response, since
-                    # we're not returning it to be released manually.
-                    drain_and_release_conn(response)
+                    response.drain_conn()
                     raise
                 return response
 
-            # drain and return the connection to the pool before recursing
-            drain_and_release_conn(response)
-
+            response.drain_conn()
             retries.sleep_for_retry(response)
             log.debug("Redirecting %s -> %s", url, redirect_location)
             return self.urlopen(
@@ -824,15 +805,11 @@ def drain_and_release_conn(response):
                 retries = retries.increment(method, url, response=response, _pool=self)
             except MaxRetryError:
                 if retries.raise_on_status:
-                    # Drain and release the connection for this response, since
-                    # we're not returning it to be released manually.
-                    drain_and_release_conn(response)
+                    response.drain_conn()
                     raise
                 return response
 
-            # drain and return the connection to the pool before recursing
-            drain_and_release_conn(response)
-
+            response.drain_conn()
             retries.sleep(response)
             log.debug("Retry: %s", url)
             return self.urlopen(
diff --git a/src/urllib3/poolmanager.py b/src/urllib3/poolmanager.py
--- a/src/urllib3/poolmanager.py
+++ b/src/urllib3/poolmanager.py
@@ -7,6 +7,7 @@
 from ._collections import RecentlyUsedContainer
 from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool
 from .connectionpool import port_by_scheme
+
 from .exceptions import (
     HTTPWarning,
     LocationValueError,
@@ -384,6 +385,7 @@ def urlopen(self, method, url, redirect=True, **kw):
             retries = retries.increment(method, url, response=response, _pool=conn)
         except MaxRetryError:
             if retries.raise_on_redirect:
+                response.drain_conn()
                 raise
             return response
 
@@ -391,6 +393,8 @@ def urlopen(self, method, url, redirect=True, **kw):
         kw["redirect"] = redirect
 
         log.info("Redirecting %s -> %s", url, redirect_location)
+
+        response.drain_conn()
         return self.urlopen(method, redirect_location, **kw)
 
 
diff --git a/src/urllib3/response.py b/src/urllib3/response.py
--- a/src/urllib3/response.py
+++ b/src/urllib3/response.py
@@ -20,6 +20,7 @@
     ResponseNotChunked,
     IncompleteRead,
     InvalidHeader,
+    HTTPError,
 )
 from .packages.six import string_types as basestring, PY3
 from .packages.six.moves import http_client as httplib
@@ -277,6 +278,17 @@ def release_conn(self):
         self._pool._put_conn(self._connection)
         self._connection = None
 
+    def drain_conn(self):
+        """
+        Read and discard any remaining HTTP response data in the response connection.
+
+        Unread data in the HTTPResponse connection blocks the connection from being released back to the pool.
+        """
+        try:
+            self.read()
+        except (HTTPError, SocketError, BaseSSLError, HTTPException):
+            pass
+
     @property
     def data(self):
         # For backwords-compat with earlier urllib3 0.4 and earlier.


Test patch:
diff --git a/test/with_dummyserver/test_poolmanager.py b/test/with_dummyserver/test_poolmanager.py
--- a/test/with_dummyserver/test_poolmanager.py
+++ b/test/with_dummyserver/test_poolmanager.py
@@ -115,6 +115,7 @@ def test_too_many_redirects(self):
                         % (self.base_url, self.base_url)
                     },
                     retries=1,
+                    preload_content=False,
                 )
 
             with pytest.raises(MaxRetryError):
@@ -126,8 +127,15 @@ def test_too_many_redirects(self):
                         % (self.base_url, self.base_url)
                     },
                     retries=Retry(total=None, redirect=1),
+                    preload_content=False,
                 )
 
+            # Even with preload_content=False and raise on redirects, we reused the same
+            # connection
+            assert len(http.pools) == 1
+            pool = http.connection_from_host(self.host, self.port)
+            assert pool.num_connections == 1
+
     def test_redirect_cross_host_remove_headers(self):
         with PoolManager() as http:
             r = http.request(
@@ -206,6 +214,15 @@ def test_redirect_cross_host_set_removed_headers(self):
             assert "X-API-Secret" not in data
             assert data["Authorization"] == "bar"
 
+    def test_redirect_without_preload_releases_connection(self):
+        with PoolManager(block=True, maxsize=2) as http:
+            r = http.request(
+                "GET", "%s/redirect" % self.base_url, preload_content=False
+            )
+            assert r._pool.num_requests == 2
+            assert r._pool.num_connections == 1
+            assert len(http.pools) == 1
+
     def test_raise_on_redirect(self):
         with PoolManager() as http:
             r = http.request(