Repository: python-trio/trio

Issue:
Deadlocks caused by openssl's handling of TLS 1.3 session tickets
We have a test that tries running `SSLStream` over an unbuffered transport, in part to confirm we aren't making assumptions about buffering. It turns out that openssl 1.1.1/TLS 1.3 does make assumptions about buffering, and the test hangs. Now what?

Specifically: we [create an unbuffered transport, then call `do_handshake` on both the client and server sides](https://github.com/python-trio/trio/blob/5a03bee3fbf1a5364a3584ef8150b04143dcecdb/trio/tests/test_ssl.py#L762-L771). The `client.do_handshake()` completes successfully. The `server.do_handshake()` hangs.

I believe what's happening is that when openssl 1.1.1 is doing TLS 1.3, the server automatically sends some session tickets after the handshake completes. Technically these aren't part of the handshake, so the `client.do_handshake()` call doesn't wait for them. But on the server side we don't know this; all we can see is that we called `SSL_do_handshake` and it spit out some data to send, so we assume that the handshake won't complete until we send that data.

I guess there are two scenarios that the designers imagined: (1) the client is trying to read application-level data, so it's reading from the transport, and it gets the session tickets as a bonus. (2) the client isn't trying to read application-level data, so it's not reading from the transport, but the transport has enough intrinsic buffering that the session tickets can just sit there forever without causing any problems. But in our test neither is true, so it breaks.

As far as I know, this shouldn't have too much real-world impact, since real transports almost always have some buffering. (Empirically the session tickets appear to be 510 bytes, which is smaller than pretty much any real transport buffer.) And, there is a workaround available: modify `clogged_stream_maker` so the server does `await server.do_handshake(); await server.send_all(b"x")`, and the client does `await client.do_handshake(); await client.receive_some(1)`. Sending a bit of application level data from server→client like this should flush out the session tickets and get back to the state we want for the rest of the tests.

Nonetheless, it's not ideal that openssl is making subtle assumptions about buffering like this. Shoudl we do something?

Unfortunately, TLS 1.3's decision to move session tickets out of the handshake doesn't give us a lot of options here. We could disable session tickets, but that seems sub-optimal (and currently the `ssl` module doesn't expose a knob for this anyway).

We could potentially delay sending the session tickets until the first time we send application-level data. That way `server.do_handshake()` wouldn't need to make any assumptions about buffering, and it's totally reasonable that `server.send_all(...)` has to wait for the peer to call `client.receive_some(...)`. But, there are two challenges:

* I'm not sure how to implement this. Specifically, it requires figuring out that the data openssl has dumped into its send buffer is the session tickets, as opposed to part of the handshake. Is there any reliable way to do this?

* There are some cases where session tickets are successfully delivered now, but wouldn't be if we implemented this. Specifically, if you have a connection where the server never sends any application-level data, *but* the client does read from network. This sounds a little weird, but it could happen: e.g. any twisted or asyncio client will act like this; or, if you're using trio, you could intentionally implement a client like this exactly because you want to get session tickets.

So.... I'm not having any brilliant ideas here. I think for right now we'll just implement the workaround to get the test suite working, and then wait to see if this becomes an issue or not.

CC: @tiran – I don't think there's anything for you to do here in particular, but it seems like the kind of issue that you like to know about.


Discussion:
Oh, this also causes a mess with `test_closing_nice_case`. (In TLS, there are no nice cases.)

The test that fails is:

https://github.com/python-trio/trio/blob/5a03bee3fbf1a5364a3584ef8150b04143dcecdb/trio/tests/test_ssl.py#L882-L894

So this involves a somewhat delicate thing: if you open a TLS connection and then immediately close it again, how should that be handled? Specifically here we have the client calling `aclose()` as the first thing it does, while the server calls `receive_some()`, and the expectation is that `receive_some()` should return `b""` to signal EOF. It looks pretty straightforward.

And in HTTPS-compatible mode it is straightforward, because the client just closes the transport, and the server interprets that as EOF. But in standards-compliant mode, we're only supposed to report EOF if we got a proper cryptographic notification of the client's intention to close the connection. So in this case, the client's `aclose()` first triggers an implicit handshake, and then after the handshake is finished, the client immediately sends a shutdown notification, and then it closes the transport. And similarly, the server's `receive_some()` triggers an implicit handshake, and then it tries to read from the connection and it gets the clean shutdown notification. Great.

Except... with TLS 1.3, after the handshake, the server tries to send the session tickets. And the standard semantics for transport streams (as determined by e.g. TCP/BSD sockets) are that if your peer has closed the connection, then you can keep *reading* data until you get everything they sent before they closed it, but if you *send* data then you get an immediate error and lose all the data that you might have received. Which is what happens here: the client has closed the transport, so the server's attempt to send the session tickets raises an exception, and the client's clean shutdown notification gets lost, so the server never has a chance to see it.

I... don't think there's anything we can actually do here? I think the server is simply stuck getting `BrokenResourceError` (as if the network had dropped) instead of a clean EOF.

I guess we could swap the roles in the test, so the server's the one who closes it immediately – that would at least let us continue to exercise the aclose-triggering-handshake logic.
Ugh, it's not just that one sequence I analyzed above that triggers this, it's a whole bunch of them that trigger it non-deterministically, probably depending on whether or not the scheduler lets the client close the connection before the server manages to enqueue the session tickets.
Boiling this down further: the major issue with the auto-sending session tickets is that they cannot work reliably in any usage where the server never sends data to the client. As a special case, that includes our toy testcases that just do things like establish a connection and then close it immediately, but it's much more general than that. Also, it doesn't matter whether you use standards-compliant mode or HTTPS-compliant mode.

Here are two openssl issues where people ran into these problems before me:

* https://github.com/openssl/openssl/issues/6342 (hi @tiran!) – not much useful here
* https://github.com/openssl/openssl/issues/6904 – lots of discussion, leading to this PR: https://github.com/openssl/openssl/pull/6944

As noted in the second issue, we could potentially avoid the close-related problems if the client always did a full round-trip shutdown, because waiting for the server's close_notify would automatically consume the session tickets. Currently Trio never does a full round-trip shutdown: if we're in standards-compliant mode then we send a close_notify and then immediately close the connection, and if we're in HTTPS-compatible mode then we skip the close_notify entirely. I suppose we could detect when we're using TLS 1.3 and switch to attempting the full round-trip close_notify dance, and ignore errors as appropriate. This seems suboptimal though; upgrading to TLS 1.3 isn't supposed to *add* round-trips. And anyway, when we're a server we can't control what clients connecting to us do! And clients are specifically allowed by the standard to skip doing a full round-trip shutdown.

The PR attempts to implement a different solution: it ignores EPIPE/ECONNRESET if those occur while attempting to send session tickets. But this has two problems AFAICT. First, we use memory BIOs and then do the I/O ourselves, so any EPIPE/ECONNRESET occurs later, after the call to `SSL_do_handshake`. They [suggest](https://github.com/openssl/openssl/pull/6944#issuecomment-434619051) that in this case it's our job to figure out how to deal with it, which would be fair enough, except that we can't implement the same behavior, because we don't know whether the data we're attempting to send is the session tickets, because openssl doesn't tell us that.

But that doesn't really matter, because there's another problem that's much more serious. Even if you ignore EPIPE/ECONNRESET while sending the session tickets, that doesn't guarantee you can go back to calling `recv` afterwards, because of how TCP works!

...Ugh, I guess I'm filing a bug on openssl.
https://github.com/openssl/openssl/issues/7948
There's a lot more information about the issue in https://github.com/openssl/openssl/issues/7948 and https://github.com/openssl/openssl/issues/7967. At this point I'm pretty sure that the right answer is that the server should wait, and send the session tickets the first time it has some other reason to send data (e.g. because of a call to `SSL_write`). That's also what BoringSSL has switched to doing unconditionally. However, the openssl devs are not yet convinced of this, and are especially unconvinced that they should change this in a stable release, and RHEL 8 has now shipped with a openssl 1.1.1 that contains this bug. So we should keep trying to convince them to fix their stuff, but we should also think about workarounds.

I guess the most elegant workaround would be to detect when openssl tries to send session tickets at the wrong time (= immediately after the handshake), and then instead of sending them we could stash them away to send later, when the right time arrives (= the next time some other operation requires sending data). The stashing part is pretty straightforward. The tricky part is reliably recognizing the session tickets, because openssl doesn't distinguish them from the rest of the handshake – it just gives you a bunch of opaque bytes to send back and forth, and you're not supposed to think too hard about which bytes represent which messages. But here we need to break that abstraction.

My first thought was: maybe we could peek at the raw bytes to find some kind of message type tag. Unfortunately, it turns out that TLS 1.3 starts encrypting everything mid-way through the handshake. In particular, the session tickets, and the messages that come before them, are both encrypted. Unless we somehow extract the secrets from openssl, all we have is a length field. (And we don't have any way to extract the secrets.)

However, I think there actually is a solution. Looking at [RFC 8446](https://tools.ietf.org/html/rfc8446), we see in section 4.6.1:

> At any time after the server has received the client Finished message, it MAY send a NewSessionTicket message

So openssl can't send tickets until after it sees the client's Finished message.

And, the client Finished message always comes after the server Finished message – you can see this by examining Figures 1 through 4 that show all the possible handshake patterns, and also if I'm reading sections 4.4.1 and 4.4.4 correctly, the client Finished contains a hash of the server Finished.

So when we loop calling `SSL_do_handshake` on the server, for TLS 1.3 it always goes through the following states sequentially:

1. until ClientHello arrives: return `WantRead` with no data in the outgoing buffer
2. when ClientHello arrives: return `WantRead`, with the ServerHello ... Finished messages in the outgoing buffer, which we then send. At this point the server's part of the handshake is actually complete.
3. until client Finished arrives: return `WantRead` with no data in the outgoing buffer
4. after client Finished arrives: return successfully, with session tickets in the outgoing buffer, which we then send

But! This means that we can actually reliably identify session tickets! If:

- We're on the server side, and
- `SSL_do_handshake` just completed with no error, and
- we negotiated TLS 1.3 (in Python: `ssl_obj.version() == "TLSv1.3"`)

...then the only thing that can possibly be in the outgoing buffer at this point is session tickets. The server's handshake finished in step 2, so what else can `SSL_do_handshake` be sending in step 4?

AFAICT, this heuristic is currently 100% reliable. Should we worry about things changing in the future in a way that breaks this heuristic?

In principle, TLS 1.3 allows the server to start sending application data before receiving the client Finished message (you can see this in the handshake diagrams), so you might think `SSL_do_handshake` could return before seeing the client Finished. But it turns out that most TLS 1.3 implementations don't make use of this. There are some caveats:

> Servers MAY send data after sending their first flight, but because the handshake is not yet complete, they have no assurance of either the peer's identity or its liveness (i.e., the ClientHello might have been replayed). [section 4.4.4]

And David Benjamin says "I don't think anyone bothers".

OpenSSL *does* have an API for this – in TLS 1.3, "early data" is a concept that only makes sense for clients, so they decided that if a server calls their "early data" functions, then that's how you write application data before seeing the client Finished. But this involves a [completely different API](https://www.openssl.org/docs/man1.1.1/man3/SSL_get_max_early_data.html) that *replaces* `SSL_do_handshake`. So I don't think we need to worry that `SSL_do_handshake` will at some point start returning early before seeing the client Finished message.

Of course at some point we'll want to add support for early data (#223), so we'll need to revisit this then. But for now neither the stdlib `ssl` module nor pyopenssl expose any of the necessary API, so we can kick that can down the road a bit.

Is there anything else that `SSL_do_handshake` might send at the end of the exchange? There are several other "post handshake messages" (see section 4.6), but these don't seem relevant:

- Post-handshake authentication: this allows the server to request a cert from the client. But if the server wanted a cert from the client during the handshake, it can just do that; this is only used if the server decided not to request a cert during the handshake, and then later changes its mind. But if we haven't even left `SSL_do_handshake` yet, it hasn't had a chance to change its mind! Also I think OpenSSL only sends this message if you explicitly call `SSL_verify_client_post_handshake`.

- KeyUpdate: this just tells the other side that hey fyi, all the data I send after this will be using a new set of keys (and it can also suggest to the other side that they do a KeyUpdate as well at their convenience). There's never any reason to send this from `SSL_do_handshake`, and even if openssl did for some reason, it would be fine for us to delay the delivery until we have more data to send.

So...... unless I'm missing something, this is both reliable and safe, so we should probably do it!
Oh, one more thing that's kind of obvious but probably worth saying explicitly: the heuristic above is also robust enough to work correctly if we're using a version of openssl where this bug is already fixed (like boringssl). In this case, at step 4 there simply won't be any session tickets in the outgoing buffer, so our stashing logic will stash 0 bytes, and everything works fine.
David Benjamin was kind enough to read over my post above, and says: "Your state listing is missing HelloRetryRequest, but I think the heuristic works."


Resolving patch:
diff --git a/trio/_ssl.py b/trio/_ssl.py
--- a/trio/_ssl.py
+++ b/trio/_ssl.py
@@ -338,6 +338,7 @@ def __init__(
             )
         self._https_compatible = https_compatible
         self._outgoing = _stdlib_ssl.MemoryBIO()
+        self._delayed_outgoing = None
         self._incoming = _stdlib_ssl.MemoryBIO()
         self._ssl_object = ssl_context.wrap_bio(
             self._incoming,
@@ -428,7 +429,9 @@ def _check_status(self):
     # comments, though, just make sure to think carefully if you ever have to
     # touch it. The big comment at the top of this file will help explain
     # too.
-    async def _retry(self, fn, *args, ignore_want_read=False):
+    async def _retry(
+        self, fn, *args, ignore_want_read=False, is_handshake=False
+    ):
         await trio.hazmat.checkpoint_if_cancelled()
         yielded = False
         finished = False
@@ -473,6 +476,32 @@ async def _retry(self, fn, *args, ignore_want_read=False):
                 finished = True
             to_send = self._outgoing.read()
 
+            # Some versions of SSL_do_handshake have a bug in how they handle
+            # the TLS 1.3 handshake on the server side: after the handshake
+            # finishes, they automatically send session tickets, even though
+            # the client may not be expecting data to arrive at this point and
+            # sending it could cause a deadlock or lost data. This applies at
+            # least to OpenSSL 1.1.1c and earlier, and the OpenSSL devs
+            # currently have no plans to fix it:
+            #
+            #   https://github.com/openssl/openssl/issues/7948
+            #   https://github.com/openssl/openssl/issues/7967
+            #
+            # The correct behavior is to wait to send session tickets on the
+            # first call to SSL_write. (This is what BoringSSL does.) So, we
+            # use a heuristic to detect when OpenSSL has tried to send session
+            # tickets, and we manually delay sending them until the
+            # appropriate moment. For more discussion see:
+            #
+            #   https://github.com/python-trio/trio/issues/819#issuecomment-517529763
+            if (
+                is_handshake and not want_read and self._ssl_object.server_side
+                and self._ssl_object.version() == "TLSv1.3"
+            ):
+                assert self._delayed_outgoing is None
+                self._delayed_outgoing = to_send
+                to_send = b""
+
             # Outputs from the above code block are:
             #
             # - to_send: bytestring; if non-empty then we need to send
@@ -535,6 +564,9 @@ async def _retry(self, fn, *args, ignore_want_read=False):
                 async with self._inner_send_lock:
                     yielded = True
                     try:
+                        if self._delayed_outgoing is not None:
+                            to_send = self._delayed_outgoing + to_send
+                            self._delayed_outgoing = None
                         await self.transport_stream.send_all(to_send)
                     except:
                         # Some unknown amount of our data got sent, and we
@@ -571,7 +603,7 @@ async def _retry(self, fn, *args, ignore_want_read=False):
 
     async def _do_handshake(self):
         try:
-            await self._retry(self._ssl_object.do_handshake)
+            await self._retry(self._ssl_object.do_handshake, is_handshake=True)
         except:
             self._state = _State.BROKEN
             raise


Test patch:
diff --git a/trio/tests/test_highlevel_ssl_helpers.py b/trio/tests/test_highlevel_ssl_helpers.py
--- a/trio/tests/test_highlevel_ssl_helpers.py
+++ b/trio/tests/test_highlevel_ssl_helpers.py
@@ -7,7 +7,7 @@
 import trio
 from trio.socket import AF_INET, SOCK_STREAM, IPPROTO_TCP
 import trio.testing
-from .test_ssl import CLIENT_CTX, SERVER_CTX
+from .test_ssl import client_ctx, SERVER_CTX
 
 from .._highlevel_ssl_helpers import (
     open_ssl_over_tcp_stream, open_ssl_over_tcp_listeners, serve_ssl_over_tcp
@@ -40,7 +40,10 @@ async def getnameinfo(self, *args):  # pragma: no cover
 
 
 # This uses serve_ssl_over_tcp, which uses open_ssl_over_tcp_listeners...
-async def test_open_ssl_over_tcp_stream_and_everything_else():
+# noqa is needed because flake8 doesn't understand how pytest fixtures work.
+async def test_open_ssl_over_tcp_stream_and_everything_else(
+    client_ctx,  # noqa: F811
+):
     async with trio.open_nursery() as nursery:
         (listener,) = await nursery.start(
             partial(
@@ -66,7 +69,7 @@ async def test_open_ssl_over_tcp_stream_and_everything_else():
         stream = await open_ssl_over_tcp_stream(
             "xyzzy.example.org",
             80,
-            ssl_context=CLIENT_CTX,
+            ssl_context=client_ctx,
         )
         with pytest.raises(trio.BrokenResourceError):
             await stream.do_handshake()
@@ -75,7 +78,7 @@ async def test_open_ssl_over_tcp_stream_and_everything_else():
         stream = await open_ssl_over_tcp_stream(
             "trio-test-1.example.org",
             80,
-            ssl_context=CLIENT_CTX,
+            ssl_context=client_ctx,
         )
         assert isinstance(stream, trio.SSLStream)
         assert stream.server_hostname == "trio-test-1.example.org"
@@ -88,7 +91,7 @@ async def test_open_ssl_over_tcp_stream_and_everything_else():
         stream = await open_ssl_over_tcp_stream(
             "trio-test-1.example.org",
             80,
-            ssl_context=CLIENT_CTX,
+            ssl_context=client_ctx,
             https_compatible=True,
             # also, smoke test happy_eyeballs_delay
             happy_eyeballs_delay=1,
diff --git a/trio/tests/test_ssl.py b/trio/tests/test_ssl.py
--- a/trio/tests/test_ssl.py
+++ b/trio/tests/test_ssl.py
@@ -54,16 +54,33 @@
 SERVER_CTX = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
 TRIO_TEST_1_CERT.configure_cert(SERVER_CTX)
 
-CLIENT_CTX = ssl.create_default_context()
-TRIO_TEST_CA.configure_trust(CLIENT_CTX)
-
-# Temporarily disable TLSv1.3, until the issue with openssl's session
-# ticket handling is sorted out one way or another:
-#     https://github.com/python-trio/trio/issues/819
-#     https://github.com/openssl/openssl/issues/7948
-#     https://github.com/openssl/openssl/issues/7967
+# TLS 1.3 has a lot of changes from previous versions. So we want to run tests
+# with both TLS 1.3, and TLS 1.2.
 if hasattr(ssl, "OP_NO_TLSv1_3"):
-    CLIENT_CTX.options |= ssl.OP_NO_TLSv1_3
+    # "tls13" means that we're willing to negotiate TLS 1.3. Usually that's
+    # what will happen, but the renegotiation tests explicitly force a
+    # downgrade on the server side. "tls12" means we refuse to negotiate TLS
+    # 1.3, so we'll almost certainly use TLS 1.2.
+    client_ctx_params = ["tls13", "tls12"]
+else:
+    # We can't control whether we use TLS 1.3, so we just have to accept
+    # whatever openssl wants to use. This might be TLS 1.2 (if openssl is
+    # old), or it might be TLS 1.3 (if openssl is new, but our python version
+    # is too old to expose the configuration knobs).
+    client_ctx_params = ["default"]
+
+
+@pytest.fixture(scope="module", params=client_ctx_params)
+def client_ctx(request):
+    ctx = ssl.create_default_context()
+    TRIO_TEST_CA.configure_trust(ctx)
+    if request.param in ["default", "tls13"]:
+        return ctx
+    elif request.param == "tls12":
+        ctx.options |= ssl.OP_NO_TLSv1_3
+        return ctx
+    else:  # pragma: no cover
+        assert False
 
 
 # The blocking socket server.
@@ -125,11 +142,11 @@ async def ssl_echo_server_raw(**kwargs):
 # echo server (running in a thread)
 @asynccontextmanager
 @async_generator
-async def ssl_echo_server(**kwargs):
+async def ssl_echo_server(client_ctx, **kwargs):
     async with ssl_echo_server_raw(**kwargs) as sock:
         await yield_(
             SSLStream(
-                sock, CLIENT_CTX, server_hostname="trio-test-1.example.org"
+                sock, client_ctx, server_hostname="trio-test-1.example.org"
             )
         )
 
@@ -151,8 +168,21 @@ def __init__(self, sleeper=None):
         # TLSv1_2_METHOD.
         #
         # Discussion: https://github.com/pyca/pyopenssl/issues/624
-        if hasattr(SSL, "OP_NO_TLSv1_3"):
-            ctx.set_options(SSL.OP_NO_TLSv1_3)
+
+        # This is the right way, but we can't use it until this PR is in a
+        # released:
+        #     https://github.com/pyca/pyopenssl/pull/861
+        #
+        # if hasattr(SSL, "OP_NO_TLSv1_3"):
+        #     ctx.set_options(SSL.OP_NO_TLSv1_3)
+        #
+        # Fortunately pyopenssl uses cryptography under the hood, so we can be
+        # confident that they're using the same version of openssl
+        from cryptography.hazmat.bindings.openssl.binding import Binding
+        b = Binding()
+        if hasattr(b.lib, "SSL_OP_NO_TLSv1_3"):
+            ctx.set_options(b.lib.SSL_OP_NO_TLSv1_3)
+
         # Unfortunately there's currently no way to say "use 1.3 or worse", we
         # can only disable specific versions. And if the two sides start
         # negotiating 1.4 at some point in the future, it *might* mean that
@@ -315,19 +345,24 @@ async def test_PyOpenSSLEchoStream_gives_resource_busy_errors():
 
 
 @contextmanager
-def virtual_ssl_echo_server(**kwargs):
+def virtual_ssl_echo_server(client_ctx, **kwargs):
     fakesock = PyOpenSSLEchoStream(**kwargs)
     yield SSLStream(
-        fakesock, CLIENT_CTX, server_hostname="trio-test-1.example.org"
+        fakesock, client_ctx, server_hostname="trio-test-1.example.org"
     )
 
 
 def ssl_wrap_pair(
-    client_transport, server_transport, *, client_kwargs={}, server_kwargs={}
+    client_ctx,
+    client_transport,
+    server_transport,
+    *,
+    client_kwargs={},
+    server_kwargs={}
 ):
     client_ssl = SSLStream(
         client_transport,
-        CLIENT_CTX,
+        client_ctx,
         server_hostname="trio-test-1.example.org",
         **client_kwargs
     )
@@ -337,22 +372,26 @@ def ssl_wrap_pair(
     return client_ssl, server_ssl
 
 
-def ssl_memory_stream_pair(**kwargs):
+def ssl_memory_stream_pair(client_ctx, **kwargs):
     client_transport, server_transport = memory_stream_pair()
-    return ssl_wrap_pair(client_transport, server_transport, **kwargs)
+    return ssl_wrap_pair(
+        client_ctx, client_transport, server_transport, **kwargs
+    )
 
 
-def ssl_lockstep_stream_pair(**kwargs):
+def ssl_lockstep_stream_pair(client_ctx, **kwargs):
     client_transport, server_transport = lockstep_stream_pair()
-    return ssl_wrap_pair(client_transport, server_transport, **kwargs)
+    return ssl_wrap_pair(
+        client_ctx, client_transport, server_transport, **kwargs
+    )
 
 
 # Simple smoke test for handshake/send/receive/shutdown talking to a
 # synchronous server, plus make sure that we do the bare minimum of
 # certificate checking (even though this is really Python's responsibility)
-async def test_ssl_client_basics():
+async def test_ssl_client_basics(client_ctx):
     # Everything OK
-    async with ssl_echo_server() as s:
+    async with ssl_echo_server(client_ctx) as s:
         assert not s.server_side
         await s.send_all(b"x")
         assert await s.receive_some(1) == b"x"
@@ -360,9 +399,9 @@ async def test_ssl_client_basics():
 
     # Didn't configure the CA file, should fail
     async with ssl_echo_server_raw(expect_fail=True) as sock:
-        client_ctx = ssl.create_default_context()
+        bad_client_ctx = ssl.create_default_context()
         s = SSLStream(
-            sock, client_ctx, server_hostname="trio-test-1.example.org"
+            sock, bad_client_ctx, server_hostname="trio-test-1.example.org"
         )
         assert not s.server_side
         with pytest.raises(BrokenResourceError) as excinfo:
@@ -372,7 +411,7 @@ async def test_ssl_client_basics():
     # Trusted CA, but wrong host name
     async with ssl_echo_server_raw(expect_fail=True) as sock:
         s = SSLStream(
-            sock, CLIENT_CTX, server_hostname="trio-test-2.example.org"
+            sock, client_ctx, server_hostname="trio-test-2.example.org"
         )
         assert not s.server_side
         with pytest.raises(BrokenResourceError) as excinfo:
@@ -380,7 +419,7 @@ async def test_ssl_client_basics():
         assert isinstance(excinfo.value.__cause__, ssl.CertificateError)
 
 
-async def test_ssl_server_basics():
+async def test_ssl_server_basics(client_ctx):
     a, b = stdlib_socket.socketpair()
     with a, b:
         server_sock = tsocket.from_stdlib_socket(b)
@@ -390,7 +429,7 @@ async def test_ssl_server_basics():
         assert server_transport.server_side
 
         def client():
-            client_sock = CLIENT_CTX.wrap_socket(
+            client_sock = client_ctx.wrap_socket(
                 a, server_hostname="trio-test-1.example.org"
             )
             client_sock.sendall(b"x")
@@ -410,9 +449,9 @@ def client():
         t.join()
 
 
-async def test_attributes():
+async def test_attributes(client_ctx):
     async with ssl_echo_server_raw(expect_fail=True) as sock:
-        good_ctx = CLIENT_CTX
+        good_ctx = client_ctx
         bad_ctx = ssl.create_default_context()
         s = SSLStream(
             sock, good_ctx, server_hostname="trio-test-1.example.org"
@@ -481,7 +520,7 @@ async def test_attributes():
 # I begin to see why HTTP/2 forbids renegotiation and TLS 1.3 removes it...
 
 
-async def test_full_duplex_basics():
+async def test_full_duplex_basics(client_ctx):
     CHUNKS = 30
     CHUNK_SIZE = 32768
     EXPECTED = CHUNKS * CHUNK_SIZE
@@ -503,7 +542,7 @@ async def receiver(s):
             chunk = await s.receive_some(CHUNK_SIZE // 2)
             received += chunk
 
-    async with ssl_echo_server() as s:
+    async with ssl_echo_server(client_ctx) as s:
         async with _core.open_nursery() as nursery:
             nursery.start_soon(sender, s)
             nursery.start_soon(receiver, s)
@@ -518,8 +557,8 @@ async def receiver(s):
     assert sent == received
 
 
-async def test_renegotiation_simple():
-    with virtual_ssl_echo_server() as s:
+async def test_renegotiation_simple(client_ctx):
+    with virtual_ssl_echo_server(client_ctx) as s:
         await s.do_handshake()
 
         s.transport_stream.renegotiate()
@@ -537,7 +576,7 @@ async def test_renegotiation_simple():
 
 
 @slow
-async def test_renegotiation_randomized(mock_clock):
+async def test_renegotiation_randomized(mock_clock, client_ctx):
     # The only blocking things in this function are our random sleeps, so 0 is
     # a good threshold.
     mock_clock.autojump_threshold = 0
@@ -569,7 +608,7 @@ async def expect(expected):
         with assert_checkpoints():
             assert await s.receive_some(1) == expected
 
-    with virtual_ssl_echo_server(sleeper=sleeper) as s:
+    with virtual_ssl_echo_server(client_ctx, sleeper=sleeper) as s:
         await s.do_handshake()
 
         await send(b"a")
@@ -618,7 +657,9 @@ async def sleep_then_wait_writable():
         await trio.sleep(1000)
         await s.wait_send_all_might_not_block()
 
-    with virtual_ssl_echo_server(sleeper=sleeper_with_slow_send_all) as s:
+    with virtual_ssl_echo_server(
+        client_ctx, sleeper=sleeper_with_slow_send_all
+    ) as s:
         await send(b"x")
         s.transport_stream.renegotiate()
         async with _core.open_nursery() as nursery:
@@ -639,7 +680,7 @@ async def sleeper_with_slow_wait_writable_and_expect(method):
             await trio.sleep(1000)
 
     with virtual_ssl_echo_server(
-        sleeper=sleeper_with_slow_wait_writable_and_expect
+        client_ctx, sleeper=sleeper_with_slow_wait_writable_and_expect
     ) as s:
         await send(b"x")
         s.transport_stream.renegotiate()
@@ -652,7 +693,7 @@ async def sleeper_with_slow_wait_writable_and_expect(method):
         await s.aclose()
 
 
-async def test_resource_busy_errors():
+async def test_resource_busy_errors(client_ctx):
     async def do_send_all():
         with assert_checkpoints():
             await s.send_all(b"x")
@@ -665,28 +706,28 @@ async def do_wait_send_all_might_not_block():
         with assert_checkpoints():
             await s.wait_send_all_might_not_block()
 
-    s, _ = ssl_lockstep_stream_pair()
+    s, _ = ssl_lockstep_stream_pair(client_ctx)
     with pytest.raises(_core.BusyResourceError) as excinfo:
         async with _core.open_nursery() as nursery:
             nursery.start_soon(do_send_all)
             nursery.start_soon(do_send_all)
     assert "another task" in str(excinfo.value)
 
-    s, _ = ssl_lockstep_stream_pair()
+    s, _ = ssl_lockstep_stream_pair(client_ctx)
     with pytest.raises(_core.BusyResourceError) as excinfo:
         async with _core.open_nursery() as nursery:
             nursery.start_soon(do_receive_some)
             nursery.start_soon(do_receive_some)
     assert "another task" in str(excinfo.value)
 
-    s, _ = ssl_lockstep_stream_pair()
+    s, _ = ssl_lockstep_stream_pair(client_ctx)
     with pytest.raises(_core.BusyResourceError) as excinfo:
         async with _core.open_nursery() as nursery:
             nursery.start_soon(do_send_all)
             nursery.start_soon(do_wait_send_all_might_not_block)
     assert "another task" in str(excinfo.value)
 
-    s, _ = ssl_lockstep_stream_pair()
+    s, _ = ssl_lockstep_stream_pair(client_ctx)
     with pytest.raises(_core.BusyResourceError) as excinfo:
         async with _core.open_nursery() as nursery:
             nursery.start_soon(do_wait_send_all_might_not_block)
@@ -707,8 +748,8 @@ async def wait_send_all_might_not_block(self):
     assert record == ["ok"]
 
 
-async def test_checkpoints():
-    async with ssl_echo_server() as s:
+async def test_checkpoints(client_ctx):
+    async with ssl_echo_server(client_ctx) as s:
         with assert_checkpoints():
             await s.do_handshake()
         with assert_checkpoints():
@@ -730,14 +771,14 @@ async def test_checkpoints():
         with assert_checkpoints():
             await s.unwrap()
 
-    async with ssl_echo_server() as s:
+    async with ssl_echo_server(client_ctx) as s:
         await s.do_handshake()
         with assert_checkpoints():
             await s.aclose()
 
 
-async def test_send_all_empty_string():
-    async with ssl_echo_server() as s:
+async def test_send_all_empty_string(client_ctx):
+    async with ssl_echo_server(client_ctx) as s:
         await s.do_handshake()
 
         # underlying SSLObject interprets writing b"" as indicating an EOF,
@@ -753,15 +794,16 @@ async def test_send_all_empty_string():
 
 
 @pytest.mark.parametrize("https_compatible", [False, True])
-async def test_SSLStream_generic(https_compatible):
+async def test_SSLStream_generic(client_ctx, https_compatible):
     async def stream_maker():
         return ssl_memory_stream_pair(
+            client_ctx,
             client_kwargs={"https_compatible": https_compatible},
             server_kwargs={"https_compatible": https_compatible},
         )
 
     async def clogged_stream_maker():
-        client, server = ssl_lockstep_stream_pair()
+        client, server = ssl_lockstep_stream_pair(client_ctx)
         # If we don't do handshakes up front, then we run into a problem in
         # the following situation:
         # - server does wait_send_all_might_not_block
@@ -776,8 +818,8 @@ async def clogged_stream_maker():
     await check_two_way_stream(stream_maker, clogged_stream_maker)
 
 
-async def test_unwrap():
-    client_ssl, server_ssl = ssl_memory_stream_pair()
+async def test_unwrap(client_ctx):
+    client_ssl, server_ssl = ssl_memory_stream_pair(client_ctx)
     client_transport = client_ssl.transport_stream
     server_transport = server_ssl.transport_stream
 
@@ -830,10 +872,10 @@ async def server():
         nursery.start_soon(server)
 
 
-async def test_closing_nice_case():
+async def test_closing_nice_case(client_ctx):
     # the nice case: graceful closes all around
 
-    client_ssl, server_ssl = ssl_memory_stream_pair()
+    client_ssl, server_ssl = ssl_memory_stream_pair(client_ctx)
     client_transport = client_ssl.transport_stream
 
     # Both the handshake and the close require back-and-forth discussion, so
@@ -879,7 +921,7 @@ async def server_closer():
 
     # Check that a graceful close *before* handshaking gives a clean EOF on
     # the other side
-    client_ssl, server_ssl = ssl_memory_stream_pair()
+    client_ssl, server_ssl = ssl_memory_stream_pair(client_ctx)
 
     async def expect_eof_server():
         with assert_checkpoints():
@@ -892,8 +934,8 @@ async def expect_eof_server():
         nursery.start_soon(expect_eof_server)
 
 
-async def test_send_all_fails_in_the_middle():
-    client, server = ssl_memory_stream_pair()
+async def test_send_all_fails_in_the_middle(client_ctx):
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async with _core.open_nursery() as nursery:
         nursery.start_soon(client.do_handshake)
@@ -923,16 +965,16 @@ def close_hook():
     assert closed == 2
 
 
-async def test_ssl_over_ssl():
+async def test_ssl_over_ssl(client_ctx):
     client_0, server_0 = memory_stream_pair()
 
     client_1 = SSLStream(
-        client_0, CLIENT_CTX, server_hostname="trio-test-1.example.org"
+        client_0, client_ctx, server_hostname="trio-test-1.example.org"
     )
     server_1 = SSLStream(server_0, SERVER_CTX, server_side=True)
 
     client_2 = SSLStream(
-        client_1, CLIENT_CTX, server_hostname="trio-test-1.example.org"
+        client_1, client_ctx, server_hostname="trio-test-1.example.org"
     )
     server_2 = SSLStream(server_1, SERVER_CTX, server_side=True)
 
@@ -949,8 +991,8 @@ async def server():
         nursery.start_soon(server)
 
 
-async def test_ssl_bad_shutdown():
-    client, server = ssl_memory_stream_pair()
+async def test_ssl_bad_shutdown(client_ctx):
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async with _core.open_nursery() as nursery:
         nursery.start_soon(client.do_handshake)
@@ -966,8 +1008,9 @@ async def test_ssl_bad_shutdown():
     await server.aclose()
 
 
-async def test_ssl_bad_shutdown_but_its_ok():
+async def test_ssl_bad_shutdown_but_its_ok(client_ctx):
     client, server = ssl_memory_stream_pair(
+        client_ctx,
         server_kwargs={"https_compatible": True},
         client_kwargs={"https_compatible": True}
     )
@@ -1006,10 +1049,10 @@ async def test_ssl_handshake_failure_during_aclose():
             await s.aclose()
 
 
-async def test_ssl_only_closes_stream_once():
+async def test_ssl_only_closes_stream_once(client_ctx):
     # We used to have a bug where if transport_stream.aclose() raised an
     # error, we would call it again. This checks that that's fixed.
-    client, server = ssl_memory_stream_pair()
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async with _core.open_nursery() as nursery:
         nursery.start_soon(client.do_handshake)
@@ -1031,8 +1074,9 @@ def close_hook():
     assert transport_close_count == 1
 
 
-async def test_ssl_https_compatibility_disagreement():
+async def test_ssl_https_compatibility_disagreement(client_ctx):
     client, server = ssl_memory_stream_pair(
+        client_ctx,
         server_kwargs={"https_compatible": False},
         client_kwargs={"https_compatible": True}
     )
@@ -1053,8 +1097,9 @@ async def receive_and_expect_error():
         nursery.start_soon(receive_and_expect_error)
 
 
-async def test_https_mode_eof_before_handshake():
+async def test_https_mode_eof_before_handshake(client_ctx):
     client, server = ssl_memory_stream_pair(
+        client_ctx,
         server_kwargs={"https_compatible": True},
         client_kwargs={"https_compatible": True}
     )
@@ -1067,8 +1112,8 @@ async def server_expect_clean_eof():
         nursery.start_soon(server_expect_clean_eof)
 
 
-async def test_send_error_during_handshake():
-    client, server = ssl_memory_stream_pair()
+async def test_send_error_during_handshake(client_ctx):
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async def bad_hook():
         raise KeyError
@@ -1084,8 +1129,8 @@ async def bad_hook():
             await client.do_handshake()
 
 
-async def test_receive_error_during_handshake():
-    client, server = ssl_memory_stream_pair()
+async def test_receive_error_during_handshake(client_ctx):
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async def bad_hook():
         raise KeyError
@@ -1107,8 +1152,8 @@ async def client_side(cancel_scope):
             await client.do_handshake()
 
 
-async def test_selected_alpn_protocol_before_handshake():
-    client, server = ssl_memory_stream_pair()
+async def test_selected_alpn_protocol_before_handshake(client_ctx):
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     with pytest.raises(NeedHandshakeError):
         client.selected_alpn_protocol()
@@ -1117,10 +1162,10 @@ async def test_selected_alpn_protocol_before_handshake():
         server.selected_alpn_protocol()
 
 
-async def test_selected_alpn_protocol_when_not_set():
+async def test_selected_alpn_protocol_when_not_set(client_ctx):
     # ALPN protocol still returns None when it's not ser,
     # instead of raising an exception
-    client, server = ssl_memory_stream_pair()
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async with _core.open_nursery() as nursery:
         nursery.start_soon(client.do_handshake)
@@ -1133,8 +1178,8 @@ async def test_selected_alpn_protocol_when_not_set():
         server.selected_alpn_protocol()
 
 
-async def test_selected_npn_protocol_before_handshake():
-    client, server = ssl_memory_stream_pair()
+async def test_selected_npn_protocol_before_handshake(client_ctx):
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     with pytest.raises(NeedHandshakeError):
         client.selected_npn_protocol()
@@ -1143,10 +1188,10 @@ async def test_selected_npn_protocol_before_handshake():
         server.selected_npn_protocol()
 
 
-async def test_selected_npn_protocol_when_not_set():
+async def test_selected_npn_protocol_when_not_set(client_ctx):
     # NPN protocol still returns None when it's not ser,
     # instead of raising an exception
-    client, server = ssl_memory_stream_pair()
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async with _core.open_nursery() as nursery:
         nursery.start_soon(client.do_handshake)
@@ -1159,8 +1204,8 @@ async def test_selected_npn_protocol_when_not_set():
         server.selected_npn_protocol()
 
 
-async def test_get_channel_binding_before_handshake():
-    client, server = ssl_memory_stream_pair()
+async def test_get_channel_binding_before_handshake(client_ctx):
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     with pytest.raises(NeedHandshakeError):
         client.get_channel_binding()
@@ -1169,8 +1214,8 @@ async def test_get_channel_binding_before_handshake():
         server.get_channel_binding()
 
 
-async def test_get_channel_binding_after_handshake():
-    client, server = ssl_memory_stream_pair()
+async def test_get_channel_binding_after_handshake(client_ctx):
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async with _core.open_nursery() as nursery:
         nursery.start_soon(client.do_handshake)
@@ -1183,9 +1228,9 @@ async def test_get_channel_binding_after_handshake():
         server.get_channel_binding()
 
 
-async def test_getpeercert():
+async def test_getpeercert(client_ctx):
     # Make sure we're not affected by https://bugs.python.org/issue29334
-    client, server = ssl_memory_stream_pair()
+    client, server = ssl_memory_stream_pair(client_ctx)
 
     async with _core.open_nursery() as nursery:
         nursery.start_soon(client.do_handshake)
@@ -1199,7 +1244,7 @@ async def test_getpeercert():
     )
 
 
-async def test_SSLListener():
+async def test_SSLListener(client_ctx):
     async def setup(**kwargs):
         listen_sock = tsocket.socket()
         await listen_sock.bind(("127.0.0.1", 0))
@@ -1210,7 +1255,7 @@ async def setup(**kwargs):
         transport_client = await open_tcp_stream(*listen_sock.getsockname())
         ssl_client = SSLStream(
             transport_client,
-            CLIENT_CTX,
+            client_ctx,
             server_hostname="trio-test-1.example.org"
         )
         return listen_sock, ssl_listener, ssl_client
@@ -1246,13 +1291,13 @@ async def setup(**kwargs):
     await aclose_forcefully(ssl_server)
 
 
-async def test_deprecated_max_refill_bytes():
+async def test_deprecated_max_refill_bytes(client_ctx):
     stream1, stream2 = memory_stream_pair()
     with pytest.warns(trio.TrioDeprecationWarning):
-        SSLStream(stream1, CLIENT_CTX, max_refill_bytes=100)
+        SSLStream(stream1, client_ctx, max_refill_bytes=100)
     with pytest.warns(trio.TrioDeprecationWarning):
         # passing None is wrong here, but I'm too lazy to make a fake Listener
         # and we get away with it for now. And this test will be deleted in a
         # release or two anyway, so hopefully we'll keep getting away with it
         # for long enough.
-        SSLListener(None, CLIENT_CTX, max_refill_bytes=100)
+        SSLListener(None, client_ctx, max_refill_bytes=100)