Repository: python-trio/trio

Issue:
Work around Windows kernel bad asymptotic behavior when there are too many operations outstanding on the same handle
So [as discussed over here](https://github.com/python-trio/trio/issues/52#issuecomment-548215128), apparently the `CancelIoEx` operation on Windows becomes super-slow when you have a lot of operations outstanding on the same handle. And #1269 makes it so that every `wait_readable`/`wait_writable` call issues an operation on the same handle. So cancelling a bunch of these calls becomes like O(n^3) or something.

And we also issue `CancelIoEx` whenever we transition from 1->2 or 2->1 waiters on the same socket, so this can also be hit during normal usage, without Trio-level cancellation. (I think a typical case would be when you have a socket where one task is blocked for a long time in `wait_readable`, while another task is sending as fast as it can, so it keeps going in and out of `wait_writable`.)

The overhead isn't really noticeable up to like 1000 sockets or so, but eventually it dominates: at 10k sockets, cancelling a `wait_readable` is like 10x slower, and at 30k sockets, it's like 100x slower, getting into multiple ms per call.

@piscisaureus is the one who pointed this out, and he says that he's found that the cost is based on the number of AFD_POLL operations issued *per AFD helper handle*. So as a workaround, we could open multiple AFD helper handles, and spread our AFD_POLL operations around over them. This will involve some annoying bookkeeping, but it shouldn't be *too* bad – basically we'll want to keep a pool of AFD handles, and for each one set a limit on how many total simultaneous operations we'll allow. We'll keep a pool of handles that are below their limit, and pull from there if we can; and if we run out, we'll allocate a new handle.

There's a script in `notes-to-self/socket-scaling.py` to measure the problem, and that can be used to validate our fix.


Discussion:
I just realized also: it's possible that poll operations on different handles don't interact with each other, in which case we might be able to simplify the code a lot by using separate handles for read polls and write polls. Should check this.
...so I just modified my little test script in `notes-to-self/afd-lab.py` to use separate AFD helper handles for the `POLL_SEND` and `POLL_RECEIVE` operations... and the weird interference bugs still happened. Darn.

According to @piscisaureus in https://github.com/python-trio/trio/issues/52#issuecomment-547722895, this doesn't happen if you *also* use different completion ports for the two handles (what the *heck* Windows why does that matter none of this makes any sense). But that's no use to us, because a single event loop has to funnel everything through a single completion port (unless we want to use threads but that's worse than what we're doing now).
For reference, here's how I modified the script to use separate helper handles (and also double-check I wasn't missing any errors):

```diff
modified   notes-to-self/afd-lab.py
@@ -132,7 +132,7 @@ class AFDLab:
             print(f"Poll for {flags!r}: {sys.exc_info()[1]!r}")
             raise
         out_flags = AFDPollFlags(poll_info.Handles[0].Events)
-        print(f"Poll for {flags!r}: got {out_flags!r}")
+        print(f"Poll for {flags!r}: got {out_flags!r} ({lpOverlapped.Internal})")
         return out_flags
 
 
@@ -145,7 +145,8 @@ def fill_socket(sock):
 
 
 async def main():
-    afdlab = AFDLab()
+    afdlab1 = AFDLab()
+    afdlab2 = AFDLab()
 
     a, b = socket.socketpair()
     a.setblocking(False)
@@ -157,13 +158,13 @@ async def main():
         print("-- Iteration start --")
         async with trio.open_nursery() as nursery:
             nursery.start_soon(
-                afdlab.afd_poll,
+                afdlab1.afd_poll,
                 a,
                 AFDPollFlags.AFD_POLL_SEND,
             )
             await trio.sleep(2)
             nursery.start_soon(
-                afdlab.afd_poll,
+                afdlab2.afd_poll,
                 a,
                 AFDPollFlags.AFD_POLL_RECEIVE,
             )
```
I also just double-checked, and confirmed that if you use separate IOCP completion ports, then everything works fine, which matches Bert's experience. Man, I do not understand Windows at all.

<details>

```
    async def t1():
        afdlab = AFDLab()
        with trio.move_on_after(10):
            await afdlab.afd_poll(a, AFDPollFlags.AFD_POLL_SEND)

    async def t2():
        afdlab = AFDLab()
        with trio.move_on_after(10):
            await afdlab.afd_poll(a, AFDPollFlags.AFD_POLL_RECEIVE)

    while True:
        print("-- Iteration start --")
        async with trio.open_nursery() as nursery:
            nursery.start_soon(
                trio.to_thread.run_sync, trio.run, t1
            )
            await trio.sleep(2)
            nursery.start_soon(
                trio.to_thread.run_sync, trio.run, t2
            )
            await trio.sleep(2)
            print("Sending another byte")
            b.send(b"x")
```

</details>
> I just realized also: it's possible that poll operations on different handles don't interact with each other, in which case we might be able to simplify the code a lot by using separate handles for read polls and write polls. Should check this.

Yeah I have tried that too at some point. Just like yourself, I found that it didn't work.

Resolving patch:
diff --git a/notes-to-self/socket-scaling.py b/notes-to-self/socket-scaling.py
--- a/notes-to-self/socket-scaling.py
+++ b/notes-to-self/socket-scaling.py
@@ -11,14 +11,6 @@
 # On Windows: with the old 'select'-based loop, the cost of scheduling grew
 # with the number of outstanding sockets, which was bad.
 #
-# With the new IOCP-based loop, the cost of scheduling is constant, which is
-# good. But, we find that the cost of cancelling a single wait_readable
-# appears to grow like O(n**2) or so in the number of outstanding
-# wait_readables. This is bad -- it means that cancelling all of the
-# outstanding operations here is something like O(n**3)! To avoid this, we
-# should consider creating multiple AFD helper handles and distributing the
-# AFD_POLL operations across them.
-#
 # To run this on Unix systems, you'll probably first have to run:
 #
 #   ulimit -n 31000
diff --git a/trio/_core/_io_windows.py b/trio/_core/_io_windows.py
--- a/trio/_core/_io_windows.py
+++ b/trio/_core/_io_windows.py
@@ -343,6 +343,24 @@ class AFDPollOp:
     lpOverlapped = attr.ib()
     poll_info = attr.ib()
     waiters = attr.ib()
+    afd_group = attr.ib()
+
+
+# The Windows kernel has a weird issue when using AFD handles. If you have N
+# instances of wait_readable/wait_writable registered with a single AFD handle,
+# then cancelling any one of them takes something like O(N**2) time. So if we
+# used just a single AFD handle, then cancellation would quickly become very
+# expensive, e.g. a program with N active sockets would take something like
+# O(N**3) time to unwind after control-C. The solution is to spread our sockets
+# out over multiple AFD handles, so that N doesn't grow too large for any
+# individual handle.
+MAX_AFD_GROUP_SIZE = 500  # at 1000, the cubic scaling is just starting to bite
+
+
+@attr.s(slots=True, eq=False)
+class AFDGroup:
+    size = attr.ib()
+    handle = attr.ib()
 
 
 @attr.s(slots=True, eq=False, frozen=True)
@@ -374,15 +392,14 @@ def __init__(self):
         # touches to safe values up front, before we do anything that can
         # fail.
         self._iocp = None
-        self._afd = None
+        self._all_afd_handles = []
 
         self._iocp = _check(
             kernel32.CreateIoCompletionPort(INVALID_HANDLE_VALUE, ffi.NULL, 0, 0)
         )
         self._events = ffi.new("OVERLAPPED_ENTRY[]", MAX_EVENTS)
 
-        self._afd = _afd_helper_handle()
-        self._register_with_iocp(self._afd, CKeys.AFD_POLL)
+        self._vacant_afd_groups = set()
         # {lpOverlapped: AFDPollOp}
         self._afd_ops = {}
         # {socket handle: AFDWaiters}
@@ -439,10 +456,9 @@ def close(self):
                 self._iocp = None
                 _check(kernel32.CloseHandle(iocp))
         finally:
-            if self._afd is not None:
-                afd = self._afd
-                self._afd = None
-                _check(kernel32.CloseHandle(afd))
+            while self._all_afd_handles:
+                afd_handle = self._all_afd_handles.pop()
+                _check(kernel32.CloseHandle(afd_handle))
 
     def __del__(self):
         self.close()
@@ -581,14 +597,21 @@ def _register_with_iocp(self, handle, completion_key):
     def _refresh_afd(self, base_handle):
         waiters = self._afd_waiters[base_handle]
         if waiters.current_op is not None:
+            afd_group = waiters.current_op.afd_group
             try:
-                _check(kernel32.CancelIoEx(self._afd, waiters.current_op.lpOverlapped))
+                _check(
+                    kernel32.CancelIoEx(
+                        afd_group.handle, waiters.current_op.lpOverlapped
+                    )
+                )
             except OSError as exc:
                 if exc.winerror != ErrorCodes.ERROR_NOT_FOUND:
                     # I don't think this is possible, so if it happens let's
                     # crash noisily.
                     raise  # pragma: no cover
             waiters.current_op = None
+            afd_group.size -= 1
+            self._vacant_afd_groups.add(afd_group)
 
         flags = 0
         if waiters.read_task is not None:
@@ -599,6 +622,14 @@ def _refresh_afd(self, base_handle):
         if not flags:
             del self._afd_waiters[base_handle]
         else:
+            try:
+                afd_group = self._vacant_afd_groups.pop()
+            except KeyError:
+                afd_group = AFDGroup(0, _afd_helper_handle())
+                self._register_with_iocp(afd_group.handle, CKeys.AFD_POLL)
+                self._all_afd_handles.append(afd_group.handle)
+            self._vacant_afd_groups.add(afd_group)
+
             lpOverlapped = ffi.new("LPOVERLAPPED")
 
             poll_info = ffi.new("AFD_POLL_INFO *")
@@ -612,7 +643,7 @@ def _refresh_afd(self, base_handle):
             try:
                 _check(
                     kernel32.DeviceIoControl(
-                        self._afd,
+                        afd_group.handle,
                         IoControlCodes.IOCTL_AFD_POLL,
                         poll_info,
                         ffi.sizeof("AFD_POLL_INFO"),
@@ -632,9 +663,12 @@ def _refresh_afd(self, base_handle):
                     # Do this last, because it could raise.
                     wake_all(waiters, exc)
                     return
-            op = AFDPollOp(lpOverlapped, poll_info, waiters)
+            op = AFDPollOp(lpOverlapped, poll_info, waiters, afd_group)
             waiters.current_op = op
             self._afd_ops[lpOverlapped] = op
+            afd_group.size += 1
+            if afd_group.size >= MAX_AFD_GROUP_SIZE:
+                self._vacant_afd_groups.remove(afd_group)
 
     async def _afd_poll(self, sock, mode):
         base_handle = _get_base_socket(sock)


Test patch:
diff --git a/trio/tests/test_socket.py b/trio/tests/test_socket.py
--- a/trio/tests/test_socket.py
+++ b/trio/tests/test_socket.py
@@ -1,3 +1,5 @@
+import errno
+
 import pytest
 import attr
 
@@ -990,3 +992,24 @@ async def receiver():
             nursery.start_soon(receiver)
             await wait_all_tasks_blocked()
             a.close()
+
+
+async def test_many_sockets():
+    total = 5000  # Must be more than MAX_AFD_GROUP_SIZE
+    sockets = []
+    for x in range(total // 2):
+        try:
+            a, b = stdlib_socket.socketpair()
+        except OSError as e:
+            assert e.errno in (errno.EMFILE, errno.ENFILE)
+            break
+        sockets += [a, b]
+    async with _core.open_nursery() as nursery:
+        for s in sockets:
+            nursery.start_soon(_core.wait_readable, s)
+        await _core.wait_all_tasks_blocked()
+        nursery.cancel_scope.cancel()
+    for sock in sockets:
+        sock.close()
+    if x != total // 2 - 1:
+        print(f"Unable to open more than {(x-1)*2} sockets.")