Repository: urllib3/urllib3

Issue:
trying to close a response hangs if a blocking read is happening on another thread
### Subject

Thread A starts Thread B to read a chunked HTTP response. At some point, regardless of whether the server has more data, I want to break the connection from the client side. Thread B is blocked trying to read the socket, so I'm trying to call HTTPResponse.close() from Thread A. This call hangs.

I'm not sure if this is a urllib3 issue, or a limitation of the underlying Python synchronous I/O model, or something I am doing wrong.

### Environment

* Python 3.7.13, 3.10.6
* platform: Darwin-22.2.0-x86_64-i386-64bit, Linux-5.10.104-linuxkit-x86_64-with-debian-10.13
* urllib3 1.26.13

### Steps to Reproduce

Unzip this archive and run `main.py`.

[urllib3-close.zip](https://github.com/urllib3/urllib3/files/10355867/urllib3-close.zip)

This starts a simple HTTP server on a daemon thread; makes a request to it from another thread, which receives a little bit of data and then blocks waiting for more; then, from the main thread, tries to close the response.

### Expected Behavior

I hoped that closing the response would cause the underlying socket to be closed, causing the client thread to get a read error and terminate. For this demo app, the output would look like this:

```
main [MainThread] starting server
  server [Thread-1] listening on port 10000
main [MainThread] starting client
    client [Thread-2] opening connection
main [MainThread] sleeping a while
  server [Thread-1] starting stream
127.0.0.1 - - [05/Jan/2023 13:38:11] "GET / HTTP/1.1" 200 -
  server [Thread-1] wrote 10 bytes
    client [Thread-2] reading data
    client [Thread-2] read 10 bytes
main [MainThread] about to close response
main [MainThread] closed response
    client [Thread-2] failed: <some kind of I/O error>
    client [Thread-2] no longer reading
main [MainThread] about to clear connection pool
main [MainThread] cleared connection pool
main [MainThread] exiting
```

### Actual Behavior

The HTTPResponse.close() call hangs. The output looks like this, and then the process waits indefinitely:

```
main [MainThread] starting server
  server [Thread-1] listening on port 10000
main [MainThread] starting client
    client [Thread-2] opening connection
main [MainThread] sleeping a while
  server [Thread-1] starting stream
127.0.0.1 - - [05/Jan/2023 13:38:11] "GET / HTTP/1.1" 200 -
  server [Thread-1] wrote 10 bytes
    client [Thread-2] reading data
    client [Thread-2] read 10 bytes
main [MainThread] about to close response
```

The only way I have found to make it not hang is to make the server send some more data; then, the blocking read on the client read unblocks, and the client is able to notice that the HTTPResponse is no longer in a valid state. But then that no longer corresponds to the use case I'm talking about, where the server does not have more data yet but I still want to break the connection.



Discussion:
Here's something I just came up with that appears to work, but it's pretty hacky and I'm not sure how portable it is:

```py
import socket

def force_close_http_response(resp: HTTPResponse):
    socket.close(resp._fp.fileno())
```

When I substitute this for `resp.close()`, it has the desired effect of immediately closing the socket so that the current blocking read errors out.
However, on further investigation, that still seems to be a bit wrong. After doing that, the next time I try make an HTTP request with the same PoolManager, it hangs. It seems like there's something more I would need to do to restore the connection pool stuff to a usable state. When I trace into the code, it seems like the problem is not that it can't acquire a connection object, but that the object it does acquire is in a bad state, like it doesn't realize it is no longer a live connection. I tried `resp.connection.close()` and `resp.release_conn()` but neither seems to make a difference.
Have you tried doing the same things with plain sockets? This would probably help understanding the issue better.
@pquentin I'm not sure what you mean by "the same things."
Using the `socket` module to have a thread reading from a socket and another one closing it, while the server is sending chunks of data to it.
@pquentin I already know that a low-level close call will close the socket, as I mentioned in my first follow-up comment. But my second follow-up comment is about a subsequent problem with the state of the connection pool object. That is specific to urllib3 and has no equivalent with plain socket calls.
I would like to raise attention with this issue again and see if any progress can be made. Thank you!
@pquentin sorry for pinging you directly, but I wanted to see if there was more information we can provide you.

I did setup a simple test using sockets as you directed. When doing that, the client thread will raise an exception on the `socket.recv` method (`OSError: [Errno 9] Bad file descriptor`).

If you catch this, all threads are able to exit cleanly. The server process will also have an exception raised once the socket is closed (`BrokenPipeError: [Errno 32] Broken pipe`) but that seems expected.
It seems that you have to call `resp.release_conn()` after the socket error. Though it looks like a bug for me that urllib3 doesn't handle socket errors.
> It seems that you have to call `resp.release_conn()` after the socket error.

I'm not sure what you mean by calling `resp.release_conn()`. In the case I described where I received a socket exception, I am not using urllib3 at all, so there is no response object to operate on.

In the original setup, there wasn't a socket error. [Eli originally stated he had tried using that method](https://github.com/urllib3/urllib3/issues/2868#issuecomment-1372938927) to reset the connection pool to a known good state, but that didn't seem to work either.
Oops, I was replying to https://github.com/urllib3/urllib3/issues/2868#issuecomment-1372880358

Here is how I aborted the request:
https://github.com/eight04/pyWorker/blob/87c2e770b973fbb7a6b3a7219d87c2b560395215/test.py#L510

The second request succeed after aborting the first one. Though I don't know if something is actually broken as described in https://github.com/urllib3/urllib3/issues/2868#issuecomment-1372938927.


Python 3.12.2
Windows 10
urllib3 2.2.1

This is an old issue so maybe something had changed.
@eight04 your example works but isn't quite the same as issue we are experiencing.

In your example, the `t.get()` call is going to block until the `r.stream` throws an exception. And that doesn't occur until the server tries to send some new bytes down the line and then the stream moves on to the NEXT iteration.

If I replace the httpbin example with an SSE server that sends some initial payload but then waits a minute before sending the next payload, your example will block until that minute has lapsed and the new data has been pushed down.

Is there a way we can avoid this blocking behavior?
Any updates on this issue, or suggestions on how to get more eyes on it? 
@eight04, @pquentin  et al.

I did some more testing, and it looks like the problem is the underlying io.BufferReader isn't realizing the socket is closed, so it's waiting on that.

In previous comments, we had mentioned trying

```python
    socket.close(resp._fp.fileno())
```
but that didn't allow us to re-use the connection pool. I think calling

```python
resp._fp.fp.raw._sock.shutdown(socket.SHUT_RDWR)
```
works. At least my local testing seems to indicate it is working. However, I have _no idea_ what sort of ramifications there might be to doing something like this. Also I'm reaching deep into implementation details, so this is also clearly a fragile solution.

Can someone with more expertise on this repository take a look and let me know if this is a viable solution to the problem, and if we can get this fix introduced into the official library somewhere?
I'm moving to [curl_cffi](https://github.com/lexiforest/curl_cffi/tree/main) which has better thread/asyncio support.
I've spent three hours looking into this issue.

My understanding of sockets is that if you have a blocked recv() call, then closing the socket will block until that recv() call finishes. This is what you're observing. The correct fix for this issue, when you really want to close() even when recv() is ongoing, is to call [`socket.shutdown(socket.SHUT_RD)`](https://docs.python.org/3/library/socket.html#socket.socket.shutdown) first.

How can we call shutdown() when we only have a response? `urllib3.HTTPResponse` is a wrapper around `http.client.HTTPResponse`. When initialized, `http.client.HTTPResponse` receives a `socket.socket` object and only calls [self.fp = socket.makefile()](https://docs.python.org/3/library/socket.html#socket.socket.makefile) on it, which is essentially equivalent to `self.fp = io.BufferedReader(socket.SocketIO(sock))`. (Note that [socket.SocketIO](https://github.com/python/cpython/blob/9dabace39d118ec7a204b6970f8a3f475a11522c/Lib/socket.py#L683-L689) is undocumented.)

So, how do we get the socket from the response? Starting from `self` in `TestClient` of your `client.py` file:

 * `self.resp` is an `urllib3.HTTPResponse`
 * `self.resp._fp` is the underlying `http.client.HTTPResponse`.
 * `self.resp._fp.fp` is the BufferedReader created by makefile()
 * `self.resp._fp.fp.raw` is the SocketIO object created by makefile()
 * `self.resp._fp.fp.raw._sock` is the socket!

And indeed, if I call `self.resp._fp.fp.raw._sock.shutdown(socket.SHUT_RD)` in `TestClient.close_response`, closing works and the call no longer hangs, instead it fails with a urllib3.exceptions.ProtocolError "Response ended prematurely".

Is this something that you could test?

If that works for you, we could make this available as `self.resp.shutdown()` with a patch like this one:

```diff
diff --git a/src/urllib3/connection.py b/src/urllib3/connection.py
index 7b83cad6..38c1a461 100644
--- a/src/urllib3/connection.py
+++ b/src/urllib3/connection.py
@@ -506,6 +506,8 @@ class HTTPConnection(_HTTPConnection):
         # This is needed here to avoid circular import errors
         from .response import HTTPResponse
 
+        # Save a reference to the shutdown function
+        _shutdown = self.sock.shutdown
         # Get the response from http.client.HTTPConnection
         httplib_response = super().getresponse()
 
@@ -534,6 +536,7 @@ class HTTPConnection(_HTTPConnection):
             enforce_content_length=resp_options.enforce_content_length,
             request_method=resp_options.request_method,
             request_url=resp_options.request_url,
+            sock_shutdown=_shutdown,
         )
         return response
 
diff --git a/src/urllib3/response.py b/src/urllib3/response.py
index 9cc4ac37..465eb431 100644
--- a/src/urllib3/response.py
+++ b/src/urllib3/response.py
@@ -5,6 +5,7 @@ import io
 import json as _json
 import logging
 import re
+import socket
 import sys
 import typing
 import warnings
@@ -589,6 +590,7 @@ class HTTPResponse(BaseHTTPResponse):
         request_method: str | None = None,
         request_url: str | None = None,
         auto_close: bool = True,
+        sock_shutdown: typing.Callable[[int], None] | None = None,  # TODO typing
     ) -> None:
         super().__init__(
             headers=headers,
@@ -618,6 +620,8 @@ class HTTPResponse(BaseHTTPResponse):
 
         if hasattr(body, "read"):
             self._fp = body  # type: ignore[assignment]
+            assert sock_shutdown is not None
+            self._sock_shutdown = sock_shutdown
 
         # Are we using the chunked-style of transfer encoding?
         self.chunk_left: int | None = None
@@ -1066,6 +1070,9 @@ class HTTPResponse(BaseHTTPResponse):
     def readable(self) -> bool:
         return True
 
+    def shutdown(self) -> None:
+        self._sock_shutdown(socket.SHUT_RD)
+
     def close(self) -> None:
         if not self.closed and self._fp:
             self._fp.close()
```

While we could conceivably call shutdown() by default, that sounds like a breaking change and I don't know what kind of consequences that could have.
Thank you for looking into this more. I really appreciate it.

> Is this something that you could test?

Yes, I did confirm in my earlier comment that calling that line directly fixes the issue. I have also applied your provided patch and then tested those changes within our codebase. It seems like this does in fact resolve the issue for us!

Again, thank you so much for looking into this.
Right, I read your comment when you posted it but then mostly ignored the thread and started again from first principles. It's a good sign that we landed on the same solution!



Patch:
diff --git a/changelog/2868.feature.rst b/changelog/2868.feature.rst
new file mode 100644
index 0000000000..71bbbb1553
--- /dev/null
+++ b/changelog/2868.feature.rst
@@ -0,0 +1,1 @@
+Added ``HTTPResponse.shutdown()`` to stop any ongoing or future reads for a specific response. It calls ``shutdown(SHUT_RD)`` on the underlying socket.
diff --git a/src/urllib3/connection.py b/src/urllib3/connection.py
index 4195ee0306..591ac407bc 100644
--- a/src/urllib3/connection.py
+++ b/src/urllib3/connection.py
@@ -507,6 +507,11 @@ def getresponse(  # type: ignore[override]
         # This is needed here to avoid circular import errors
         from .response import HTTPResponse
 
+        # Save a reference to the shutdown function before ownership is passed
+        # to httplib_response
+        # TODO should we implement it everywhere?
+        _shutdown = getattr(self.sock, "shutdown", None)
+
         # Get the response from http.client.HTTPConnection
         httplib_response = super().getresponse()
 
@@ -535,6 +540,7 @@ def getresponse(  # type: ignore[override]
             enforce_content_length=resp_options.enforce_content_length,
             request_method=resp_options.request_method,
             request_url=resp_options.request_url,
+            sock_shutdown=_shutdown,
         )
         return response
 
diff --git a/src/urllib3/contrib/pyopenssl.py b/src/urllib3/contrib/pyopenssl.py
index c12cb3174d..ed65430661 100644
--- a/src/urllib3/contrib/pyopenssl.py
+++ b/src/urllib3/contrib/pyopenssl.py
@@ -366,9 +366,11 @@ def sendall(self, data: bytes) -> None:
             )
             total_sent += sent
 
-    def shutdown(self) -> None:
-        # FIXME rethrow compatible exceptions should we ever use this
-        self.connection.shutdown()
+    def shutdown(self, how: int) -> None:
+        try:
+            self.connection.shutdown()
+        except OpenSSL.SSL.Error as e:
+            raise ssl.SSLError(f"shutdown error: {e!r}") from e
 
     def close(self) -> None:
         self._closed = True
diff --git a/src/urllib3/response.py b/src/urllib3/response.py
index 51dab24802..66c6a687ec 100644
--- a/src/urllib3/response.py
+++ b/src/urllib3/response.py
@@ -5,6 +5,7 @@
 import json as _json
 import logging
 import re
+import socket
 import sys
 import typing
 import warnings
@@ -440,6 +441,9 @@ def release_conn(self) -> None:
     def drain_conn(self) -> None:
         raise NotImplementedError()
 
+    def shutdown(self) -> None:
+        raise NotImplementedError()
+
     def close(self) -> None:
         raise NotImplementedError()
 
@@ -589,6 +593,7 @@ def __init__(
         request_method: str | None = None,
         request_url: str | None = None,
         auto_close: bool = True,
+        sock_shutdown: typing.Callable[[int], None] | None = None,
     ) -> None:
         super().__init__(
             headers=headers,
@@ -618,6 +623,7 @@ def __init__(
 
         if hasattr(body, "read"):
             self._fp = body  # type: ignore[assignment]
+        self._sock_shutdown = sock_shutdown
 
         # Are we using the chunked-style of transfer encoding?
         self.chunk_left: int | None = None
@@ -1066,7 +1072,14 @@ def stream(
     def readable(self) -> bool:
         return True
 
+    def shutdown(self) -> None:
+        if not self._sock_shutdown:
+            raise ValueError("Cannot shutdown socket as self._sock_shutdown is not set")
+        self._sock_shutdown(socket.SHUT_RD)
+
     def close(self) -> None:
+        self._sock_shutdown = None
+
         if not self.closed and self._fp:
             self._fp.close()
 


Test patch:
diff --git a/test/test_response.py b/test/test_response.py
index 7a3b1a7d1c..3eec0abaab 100644
--- a/test/test_response.py
+++ b/test/test_response.py
@@ -204,6 +204,13 @@ def test_no_preload(self) -> None:
         assert r.data == b"foo"
         assert fp.tell() == len(b"foo")
 
+    def test_no_shutdown(self) -> None:
+        r = HTTPResponse()
+        with pytest.raises(
+            ValueError, match="Cannot shutdown socket as self._sock_shutdown is not set"
+        ):
+            r.shutdown()
+
     def test_decode_bad_data(self) -> None:
         fp = BytesIO(b"\x00" * 10)
         with pytest.raises(DecodeError):
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
index e268869ad5..89db60b422 100644
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -14,6 +14,7 @@
 import ssl
 import tempfile
 import threading
+import time
 import typing
 import zlib
 from collections import OrderedDict
@@ -32,7 +33,13 @@
     get_unreachable_address,
 )
 from dummyserver.testcase import SocketDummyServerTestCase, consume_socket
-from urllib3 import HTTPConnectionPool, HTTPSConnectionPool, ProxyManager, util
+from urllib3 import (
+    BaseHTTPResponse,
+    HTTPConnectionPool,
+    HTTPSConnectionPool,
+    ProxyManager,
+    util,
+)
 from urllib3._collections import HTTPHeaderDict
 from urllib3.connection import HTTPConnection, _get_default_user_agent
 from urllib3.connectionpool import _url_from_pool
@@ -1020,6 +1027,67 @@ def consume_ssl_socket(listener: socket.socket) -> None:
             ssl_sock.sendall(b"hello")
             assert ssl_sock.fileno() > 0
 
+    def test_socket_shutdown_stops_recv(self) -> None:
+        timed_out, starting_read = Event(), Event()
+
+        def socket_handler(listener: socket.socket) -> None:
+            sock = listener.accept()[0]
+
+            ssl_sock = original_ssl_wrap_socket(
+                sock,
+                server_side=True,
+                keyfile=DEFAULT_CERTS["keyfile"],
+                certfile=DEFAULT_CERTS["certfile"],
+                ca_certs=DEFAULT_CA,
+            )
+
+            # Consume request
+            buf = b""
+            while not buf.endswith(b"\r\n\r\n"):
+                buf = ssl_sock.recv(65535)
+
+            # Send incomplete message (note Content-Length)
+            ssl_sock.send(
+                b"HTTP/1.1 200 OK\r\n"
+                b"Content-Type: text/plain\r\n"
+                b"Content-Length: 10\r\n"
+                b"\r\n"
+                b"Hi-"
+            )
+            timed_out.wait(5)
+            ssl_sock.close()
+
+        self._start_server(socket_handler)
+
+        class TestClient(threading.Thread):
+            def __init__(self, host: str, port: int) -> None:
+                super().__init__()
+                self.host, self.port = host, port
+                self.response: BaseHTTPResponse | None = None
+
+            def run(self) -> None:
+                with HTTPSConnectionPool(
+                    self.host, self.port, ca_certs=DEFAULT_CA
+                ) as pool:
+                    self.response = pool.urlopen(
+                        "GET", "/", preload_content=False, retries=0
+                    )
+                    with pytest.raises(ProtocolError, match="Connection broken"):
+                        starting_read.set()
+                        self.response.read()
+
+        test_client = TestClient(self.host, self.port)
+        test_client.start()
+        # First, wait to make sure the client is really stuck reading
+        starting_read.wait(5)
+        time.sleep(LONG_TIMEOUT)
+        # Calling shutdown here calls shutdown() on the underlying socket,
+        # so that the remaining read will fail instead of blocking
+        # indefinitely
+        assert test_client.response is not None
+        test_client.response.shutdown()
+        timed_out.set()
+
 
 class TestProxyManager(SocketDummyServerTestCase):
     def test_simple(self) -> None: