Repository: urllib3/urllib3

Issue:
Bump pytest to 8.0.0
Changelog: https://docs.pytest.org/en/8.0.x/changelog.html#pytest-8-0-0-2024-01-27


Discussion:
Umm, 

In current main  I tested  with pytest 7.4.4  and 
```
nox -rs test-3.12 -- test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3
```
passed.


With pytest 8.0.0 it fails: 

```
FAILED test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3::test_ssl_context_ssl_version_uses_ssl_min_max_versions - Failed: DID NOT WARN. No warnings of type (<class 'DeprecationWarning'>,) were emitted.
```

The changelog mentions some changes regarding `pytest.warns()` but the test should still work ... but it doesn't 




With pytest 8.0.0 
* `test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_3::test_ssl_context_ssl_version_uses_ssl_min_max_versions`  FAILS
* `test/contrib/test_pyopenssl.py::TestHTTPS_TLSv1_2::test_ssl_context_ssl_version_uses_ssl_min_max_versions` WORKS

So I guess it's more related to pytest collect the test, since TestHTTPS_TLSv1_3 was meant to be SKIPPED with ` ssl.PROTOCOL_TLSv1_3 isn't available`. 

I think we can bump the memory limit in `test_get_all_memory_usage_single_chunk` from 10.01 MB to 10.1 MB or even a bit higher if needed to fix the consistent [macOS 3.11](https://github.com/urllib3/urllib3/actions/runs/7868259343/job/21465191410?pr=3335#logs) failures, this shouldn't change the meaning of the test
This sounds good to me! But can we report it to pytest-memray first?
Since it's relevant to this discussion

https://github.com/urllib3/urllib3/pull/3337#issuecomment-1924429650

> This appears to break test_get_all_memory_usage_single_chunk on macOS 3.10/11/12 somehow:
...
> The 528B in response.py at line 246 are allocated by the deque. The 64KiB are allocated in SSLSocket.read() but this makes no sense in this context as the test does not call SSL at all:
...
> This sounds like an incompatibility between pytest 8 and pytest-memray on macOS that we should report to pytest-memray.

Copying my comment from https://github.com/urllib3/urllib3/pull/3337#issuecomment-1926391324

> I wonder , **should we increase the memory limit for this test to "10.064 MB"?** After all we already have "10.01 MB" which I assume the "0.01" was to give some room as indicated in [pytest.mark.limit_memory](https://pytest-memray.readthedocs.io/en/latest/usage.html#pytest.mark.limit_memory)
> 
> `test/test_response.py::TestBytesQueueBuffer::test_get_all_memory_usage_single_chunk` fails not only on macOS
> 
> * macOS 3.10
> * macOS 3.11
> * macOS 3.12
> * Ubuntu 22.04 3.12
> * Ubuntu 3.x test_brotlipy
> 
> I investigated a bit
> 
> * if we run `test/test_response.py::TestBytesQueueBuffer` test alone they will pass
> * if we run `test/contrib/test_pyopenssl.py::TestSocketClosing:test_socket_close_socket_then_file`  or `test/contrib/test_pyopenssl.py::TestSocketClosing::test_socket_close_stays_open_with_makefile_open`  before `test/test_response.py::TestBytesQueueBuffer::test_get_all_memory_usage_single_chunk` then it will fail.
>   
>   * I can reproduce 100% on my mac laptop.
>   * failure seem to be a **consequence of running other tests prior to this one**.
> 
...

> This sounds good to me! But can we report it to pytest-memray first?

I wonder if the warning is relevant to our case
![](https://github.com/urllib3/urllib3/assets/17710133/2f288cbc-ea16-45f7-b32f-2ad8ab79358a)
My understanding is that objects created during one test may not be deleted immediately, so if you allocate two strings of 1MB each, if Python does not release the memory of the first string immediately, your test will use 2MB of memory. But objects created by one should not be leaking into another test, I think?

I created https://github.com/bloomberg/pytest-memray/issues/109

> But objects created by one should not be leaking into another test, I think?

Just to clarify here you refer to the  suspicion  that the allocations of 64.0KiB bllow  on ssl.py:1107 does not really belong to the `test_get_all_memory_usage_single_chunk`. 

When we run this test in isolation the test passes, but if we run `test_socket_close_socket_then_file` before `test_get_all_memory_usage_single_chunk ` it fails. That suggest that the allocation of 64KiB really happens on `test_socket_close_socket_then_file` but  "leaks " into the `test_get_all_memory_usage_single_chunk `. 

```
 _________ TestBytesQueueBuffer.test_get_all_memory_usage_single_chunk __________
Test was limited to 10.0MiB but allocated 10.1MiB
------------------------------ memray-max-memory -------------------------------
List of allocations:
    - 64.0KiB allocated here:
        read:/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py:1107
        ...
    - 528.0B allocated here:
        __init__:/Users/runner/work/urllib3/urllib3/.nox/test-3-12/lib/python3.12/site-packages/urllib3/response.py:246
        ...
    - 10.0MiB allocated here:
        test_get_all_memory_usage_single_chunk:/Users/runner/work/urllib3/urllib3/test/test_response.py:115
        ...
```
See https://github.com/bloomberg/pytest-memray/issues/109#issuecomment-1967627764

Resolving patch:
diff --git a/dummyserver/socketserver.py b/dummyserver/socketserver.py
--- a/dummyserver/socketserver.py
+++ b/dummyserver/socketserver.py
@@ -108,6 +108,7 @@ def __init__(
         socket_handler: typing.Callable[[socket.socket], None],
         host: str = "localhost",
         ready_event: threading.Event | None = None,
+        quit_event: threading.Event | None = None,
     ) -> None:
         super().__init__()
         self.daemon = True
@@ -115,6 +116,7 @@ def __init__(
         self.socket_handler = socket_handler
         self.host = host
         self.ready_event = ready_event
+        self.quit_event = quit_event
 
     def _start_server(self) -> None:
         if self.USE_IPV6:
diff --git a/dummyserver/testcase.py b/dummyserver/testcase.py
--- a/dummyserver/testcase.py
+++ b/dummyserver/testcase.py
@@ -5,6 +5,7 @@
 import ssl
 import threading
 import typing
+from test import LONG_TIMEOUT
 
 import hypercorn
 import pytest
@@ -19,11 +20,19 @@
 
 
 def consume_socket(
-    sock: SSLTransport | socket.socket, chunks: int = 65536
+    sock: SSLTransport | socket.socket,
+    chunks: int = 65536,
+    quit_event: threading.Event | None = None,
 ) -> bytearray:
     consumed = bytearray()
+    sock.settimeout(LONG_TIMEOUT)
     while True:
-        b = sock.recv(chunks)
+        if quit_event and quit_event.is_set():
+            break
+        try:
+            b = sock.recv(chunks)
+        except (TimeoutError, socket.timeout):
+            continue
         assert isinstance(b, bytes)
         consumed += b
         if b.endswith(b"\r\n\r\n"):
@@ -57,11 +66,16 @@ class SocketDummyServerTestCase:
 
     @classmethod
     def _start_server(
-        cls, socket_handler: typing.Callable[[socket.socket], None]
+        cls,
+        socket_handler: typing.Callable[[socket.socket], None],
+        quit_event: threading.Event | None = None,
     ) -> None:
         ready_event = threading.Event()
         cls.server_thread = SocketServerThread(
-            socket_handler=socket_handler, ready_event=ready_event, host=cls.host
+            socket_handler=socket_handler,
+            ready_event=ready_event,
+            host=cls.host,
+            quit_event=quit_event,
         )
         cls.server_thread.start()
         ready_event.wait(5)
@@ -71,23 +85,41 @@ def _start_server(
 
     @classmethod
     def start_response_handler(
-        cls, response: bytes, num: int = 1, block_send: threading.Event | None = None
+        cls,
+        response: bytes,
+        num: int = 1,
+        block_send: threading.Event | None = None,
     ) -> threading.Event:
         ready_event = threading.Event()
+        quit_event = threading.Event()
 
         def socket_handler(listener: socket.socket) -> None:
             for _ in range(num):
                 ready_event.set()
 
-                sock = listener.accept()[0]
-                consume_socket(sock)
+                listener.settimeout(LONG_TIMEOUT)
+                while True:
+                    if quit_event.is_set():
+                        return
+                    try:
+                        sock = listener.accept()[0]
+                        break
+                    except (TimeoutError, socket.timeout):
+                        continue
+                consume_socket(sock, quit_event=quit_event)
+                if quit_event.is_set():
+                    sock.close()
+                    return
                 if block_send:
-                    block_send.wait()
+                    while not block_send.wait(LONG_TIMEOUT):
+                        if quit_event.is_set():
+                            sock.close()
+                            return
                     block_send.clear()
                 sock.send(response)
                 sock.close()
 
-        cls._start_server(socket_handler)
+        cls._start_server(socket_handler, quit_event=quit_event)
         return ready_event
 
     @classmethod
@@ -100,10 +132,25 @@ def start_basic_handler(
             block_send,
         )
 
+    @staticmethod
+    def quit_server_thread(server_thread: SocketServerThread) -> None:
+        if server_thread.quit_event:
+            server_thread.quit_event.set()
+        # in principle the maximum time that the thread can take to notice
+        # the quit_event is LONG_TIMEOUT and the thread should terminate
+        # shortly after that, we give 5 seconds leeway just in case
+        server_thread.join(LONG_TIMEOUT * 2 + 5.0)
+        if server_thread.is_alive():
+            raise Exception("server_thread did not exit")
+
     @classmethod
     def teardown_class(cls) -> None:
         if hasattr(cls, "server_thread"):
-            cls.server_thread.join(0.1)
+            cls.quit_server_thread(cls.server_thread)
+
+    def teardown_method(self) -> None:
+        if hasattr(self, "server_thread"):
+            self.quit_server_thread(self.server_thread)
 
     def assert_header_received(
         self,
@@ -128,11 +175,16 @@ def assert_header_received(
 class IPV4SocketDummyServerTestCase(SocketDummyServerTestCase):
     @classmethod
     def _start_server(
-        cls, socket_handler: typing.Callable[[socket.socket], None]
+        cls,
+        socket_handler: typing.Callable[[socket.socket], None],
+        quit_event: threading.Event | None = None,
     ) -> None:
         ready_event = threading.Event()
         cls.server_thread = SocketServerThread(
-            socket_handler=socket_handler, ready_event=ready_event, host=cls.host
+            socket_handler=socket_handler,
+            ready_event=ready_event,
+            host=cls.host,
+            quit_event=quit_event,
         )
         cls.server_thread.USE_IPV6 = False
         cls.server_thread.start()


Test patch:
diff --git a/test/test_ssltransport.py b/test/test_ssltransport.py
--- a/test/test_ssltransport.py
+++ b/test/test_ssltransport.py
@@ -4,6 +4,7 @@
 import select
 import socket
 import ssl
+import threading
 import typing
 from unittest import mock
 
@@ -111,20 +112,29 @@ def setup_class(cls) -> None:
         cls.server_context, cls.client_context = server_client_ssl_contexts()
 
     def start_dummy_server(
-        self, handler: typing.Callable[[socket.socket], None] | None = None
+        self,
+        handler: typing.Callable[[socket.socket], None] | None = None,
+        validate: bool = True,
     ) -> None:
+        quit_event = threading.Event()
+
         def socket_handler(listener: socket.socket) -> None:
             sock = listener.accept()[0]
             try:
                 with self.server_context.wrap_socket(sock, server_side=True) as ssock:
-                    request = consume_socket(ssock)
+                    request = consume_socket(
+                        ssock,
+                        quit_event=quit_event,
+                    )
+                    if not validate:
+                        return
                     validate_request(request)
                     ssock.send(sample_response())
             except (ConnectionAbortedError, ConnectionResetError):
                 return
 
         chosen_handler = handler if handler else socket_handler
-        self._start_server(chosen_handler)
+        self._start_server(chosen_handler, quit_event=quit_event)
 
     @pytest.mark.timeout(PER_TEST_TIMEOUT)
     def test_start_closed_socket(self) -> None:
@@ -138,7 +148,7 @@ def test_start_closed_socket(self) -> None:
     @pytest.mark.timeout(PER_TEST_TIMEOUT)
     def test_close_after_handshake(self) -> None:
         """Socket errors should be bubbled up"""
-        self.start_dummy_server()
+        self.start_dummy_server(validate=False)
 
         sock = socket.create_connection((self.host, self.port))
         with SSLTransport(
diff --git a/test/with_dummyserver/test_socketlevel.py b/test/with_dummyserver/test_socketlevel.py
--- a/test/with_dummyserver/test_socketlevel.py
+++ b/test/with_dummyserver/test_socketlevel.py
@@ -12,6 +12,7 @@
 import socket
 import ssl
 import tempfile
+import threading
 import typing
 import zlib
 from collections import OrderedDict
@@ -955,7 +956,11 @@ def socket_handler(listener: socket.socket) -> None:
             assert response.connection is None
 
     def test_socket_close_socket_then_file(self) -> None:
-        def consume_ssl_socket(listener: socket.socket) -> None:
+        quit_event = threading.Event()
+
+        def consume_ssl_socket(
+            listener: socket.socket,
+        ) -> None:
             try:
                 with listener.accept()[0] as sock, original_ssl_wrap_socket(
                     sock,
@@ -964,11 +969,11 @@ def consume_ssl_socket(listener: socket.socket) -> None:
                     certfile=DEFAULT_CERTS["certfile"],
                     ca_certs=DEFAULT_CA,
                 ) as ssl_sock:
-                    consume_socket(ssl_sock)
+                    consume_socket(ssl_sock, quit_event=quit_event)
             except (ConnectionResetError, ConnectionAbortedError, OSError):
                 pass
 
-        self._start_server(consume_ssl_socket)
+        self._start_server(consume_ssl_socket, quit_event=quit_event)
         with socket.create_connection(
             (self.host, self.port)
         ) as sock, contextlib.closing(
@@ -983,6 +988,8 @@ def consume_ssl_socket(listener: socket.socket) -> None:
             assert ssl_sock.fileno() == -1
 
     def test_socket_close_stays_open_with_makefile_open(self) -> None:
+        quit_event = threading.Event()
+
         def consume_ssl_socket(listener: socket.socket) -> None:
             try:
                 with listener.accept()[0] as sock, original_ssl_wrap_socket(
@@ -992,11 +999,11 @@ def consume_ssl_socket(listener: socket.socket) -> None:
                     certfile=DEFAULT_CERTS["certfile"],
                     ca_certs=DEFAULT_CA,
                 ) as ssl_sock:
-                    consume_socket(ssl_sock)
+                    consume_socket(ssl_sock, quit_event=quit_event)
             except (ConnectionResetError, ConnectionAbortedError, OSError):
                 pass
 
-        self._start_server(consume_ssl_socket)
+        self._start_server(consume_ssl_socket, quit_event=quit_event)
         with socket.create_connection(
             (self.host, self.port)
         ) as sock, contextlib.closing(
@@ -2232,11 +2239,28 @@ def socket_handler(listener: socket.socket) -> None:
 
 class TestMultipartResponse(SocketDummyServerTestCase):
     def test_multipart_assert_header_parsing_no_defects(self) -> None:
+        quit_event = threading.Event()
+
         def socket_handler(listener: socket.socket) -> None:
             for _ in range(2):
-                sock = listener.accept()[0]
-                while not sock.recv(65536).endswith(b"\r\n\r\n"):
-                    pass
+                listener.settimeout(LONG_TIMEOUT)
+
+                while True:
+                    if quit_event and quit_event.is_set():
+                        return
+                    try:
+                        sock = listener.accept()[0]
+                        break
+                    except (TimeoutError, socket.timeout):
+                        continue
+
+                sock.settimeout(LONG_TIMEOUT)
+                while True:
+                    if quit_event and quit_event.is_set():
+                        sock.close()
+                        return
+                    if sock.recv(65536).endswith(b"\r\n\r\n"):
+                        break
 
                 sock.sendall(
                     b"HTTP/1.1 404 Not Found\r\n"
@@ -2252,7 +2276,7 @@ def socket_handler(listener: socket.socket) -> None:
                 )
                 sock.close()
 
-        self._start_server(socket_handler)
+        self._start_server(socket_handler, quit_event=quit_event)
         from urllib3.connectionpool import log
 
         with mock.patch.object(log, "warning") as log_warning:
@@ -2308,15 +2332,26 @@ def socket_handler(listener: socket.socket) -> None:
     def test_chunked_specified(
         self, method: str, chunked: bool, body_type: str
     ) -> None:
+        quit_event = threading.Event()
         buffer = bytearray()
         expected_bytes = b"\r\n\r\na\r\nxxxxxxxxxx\r\n0\r\n\r\n"
 
         def socket_handler(listener: socket.socket) -> None:
             nonlocal buffer
-            sock = listener.accept()[0]
-            sock.settimeout(0)
+            listener.settimeout(LONG_TIMEOUT)
+            while True:
+                if quit_event.is_set():
+                    return
+                try:
+                    sock = listener.accept()[0]
+                    break
+                except (TimeoutError, socket.timeout):
+                    continue
+            sock.settimeout(LONG_TIMEOUT)
 
             while expected_bytes not in buffer:
+                if quit_event.is_set():
+                    return
                 with contextlib.suppress(BlockingIOError):
                     buffer += sock.recv(65536)
 
@@ -2327,7 +2362,7 @@ def socket_handler(listener: socket.socket) -> None:
             )
             sock.close()
 
-        self._start_server(socket_handler)
+        self._start_server(socket_handler, quit_event=quit_event)
 
         body: typing.Any
         if body_type == "generator":