Repository: python/cpython (Python)

Issue:
Decompose asyncio.run_forever() into parts to improve integration with other event loops
# Feature or enhancement

### Proposal:

This is an admittedly niche use case, but it's one that we've hit in the development of [Toga](https://github.com/beeware/toga).

GUI toolkits all have event loops. If you want to use Python `asyncio` calls in a GUI app, it is necessary to make the GUI's event loop and Python's asyncio event loop co-exist. 

This usually means writing a subclass of EventLoop that overrides `run_forever()` in some way. Either the inner `while True: _run_once()` loop of `run_forever()` needs to be modified to integrate the equivalent "run once" of the GUI's event loop; or the `while True` loop needs to be subverted entirely and `_run_once()` needs to be invoked regularly on the GUI's event loop. 

However, the implementation of `run_forever()` makes this difficult to do without duplicating significant portions of the implementation of `run_forever()`. For example, [Toga's Winforms asyncio integration](https://github.com/beeware/toga/blob/main/winforms/src/toga_winforms/libs/proactor.py#L12) involves duplicating the [startup](https://github.com/beeware/toga/blob/main/winforms/src/toga_winforms/libs/proactor.py#L38) and [shutdown](https://github.com/beeware/toga/blob/main/winforms/src/toga_winforms/libs/proactor.py#L95) of `run_forever()`

The fundamental structure of `run_forever()` is:

```python
def run_forever():
    try:
        run_forever_setup()
        while True:
            run_once()
    finally:
        run_forever_cleanup()    
```

I propose that the implemenation of `run_forever()` be decomposed into exactly these parts - an initial setup, the actual loop, and a post-loop cleanup. This does not impact on the operation of ascynio itself, but it would allow Toga (and any other integrated asyncio loop) to automatically inherit any improvements that are made to the core event loop.

It is worth noting that CPython's own Winforms integration has a similar need. The implementation of [`run_forever()` in `ProactorEventLoop()`](https://github.com/python/cpython/blob/main/Lib/asyncio/windows_events.py#L317) needs to include additional setup and additional cleanup calls. Decomposing the base event loop would allow this additional setup and cleanup to be explcit.



### Has this already been discussed elsewhere?

This is a minor feature, which does not need previous discussion elsewhere

### Links to previous discussion of this feature:

Informal in-person discussion at CPython core team sprint.

<!-- gh-linked-prs -->
### Linked PRs
* gh-110773
<!-- /gh-linked-prs -->



Discussion:
Thanks! I think this (GUI event loop integration) was an originally intended use case, and there was talk of doing a Tkinter integration as an example, but it never panned out, and I stopped caring about Tkinter personally.

It was my understanding that most GUI integrations just run the asyncio event loop in a separate thread. I think it's exciting that Toga managed to do this integration right and that a feature request came out of it.

Could you propose a specific API? It shouldn't be hard to get rough consensus on that, and then you can start implementing. Or you could start with an implementation (so you're perhaps more sure that it actually works, etc.). Up to you.



Patch:
diff --git a/Lib/asyncio/base_events.py b/Lib/asyncio/base_events.py
index 956864e4242799..0476de631a6a52 100644
--- a/Lib/asyncio/base_events.py
+++ b/Lib/asyncio/base_events.py
@@ -400,6 +400,8 @@ def __init__(self):
         self._clock_resolution = time.get_clock_info('monotonic').resolution
         self._exception_handler = None
         self.set_debug(coroutines._is_debug_mode())
+        # The preserved state of async generator hooks.
+        self._old_agen_hooks = None
         # In debug mode, if the execution of a callback or a step of a task
         # exceed this duration in seconds, the slow callback/task is logged.
         self.slow_callback_duration = 0.1
@@ -601,29 +603,52 @@ def _check_running(self):
             raise RuntimeError(
                 'Cannot run the event loop while another loop is running')
 
-    def run_forever(self):
-        """Run until stop() is called."""
+    def _run_forever_setup(self):
+        """Prepare the run loop to process events.
+
+        This method exists so that custom custom event loop subclasses (e.g., event loops
+        that integrate a GUI event loop with Python's event loop) have access to all the
+        loop setup logic.
+        """
         self._check_closed()
         self._check_running()
         self._set_coroutine_origin_tracking(self._debug)
 
-        old_agen_hooks = sys.get_asyncgen_hooks()
-        try:
-            self._thread_id = threading.get_ident()
-            sys.set_asyncgen_hooks(firstiter=self._asyncgen_firstiter_hook,
-                                   finalizer=self._asyncgen_finalizer_hook)
+        self._old_agen_hooks = sys.get_asyncgen_hooks()
+        self._thread_id = threading.get_ident()
+        sys.set_asyncgen_hooks(
+            firstiter=self._asyncgen_firstiter_hook,
+            finalizer=self._asyncgen_finalizer_hook
+        )
+
+        events._set_running_loop(self)
+
+    def _run_forever_cleanup(self):
+        """Clean up after an event loop finishes the looping over events.
 
-            events._set_running_loop(self)
+        This method exists so that custom custom event loop subclasses (e.g., event loops
+        that integrate a GUI event loop with Python's event loop) have access to all the
+        loop cleanup logic.
+        """
+        self._stopping = False
+        self._thread_id = None
+        events._set_running_loop(None)
+        self._set_coroutine_origin_tracking(False)
+        # Restore any pre-existing async generator hooks.
+        if self._old_agen_hooks is not None:
+            sys.set_asyncgen_hooks(*self._old_agen_hooks)
+            self._old_agen_hooks = None
+
+    def run_forever(self):
+        """Run until stop() is called."""
+        try:
+            self._run_forever_setup()
             while True:
                 self._run_once()
                 if self._stopping:
                     break
         finally:
-            self._stopping = False
-            self._thread_id = None
-            events._set_running_loop(None)
-            self._set_coroutine_origin_tracking(False)
-            sys.set_asyncgen_hooks(*old_agen_hooks)
+            self._run_forever_cleanup()
 
     def run_until_complete(self, future):
         """Run until the Future is done.
diff --git a/Lib/asyncio/windows_events.py b/Lib/asyncio/windows_events.py
index 4a4c4bea8948a0..b62ea75fee3858 100644
--- a/Lib/asyncio/windows_events.py
+++ b/Lib/asyncio/windows_events.py
@@ -314,24 +314,25 @@ def __init__(self, proactor=None):
             proactor = IocpProactor()
         super().__init__(proactor)
 
-    def run_forever(self):
-        try:
-            assert self._self_reading_future is None
-            self.call_soon(self._loop_self_reading)
-            super().run_forever()
-        finally:
-            if self._self_reading_future is not None:
-                ov = self._self_reading_future._ov
-                self._self_reading_future.cancel()
-                # self_reading_future was just cancelled so if it hasn't been
-                # finished yet, it never will be (it's possible that it has
-                # already finished and its callback is waiting in the queue,
-                # where it could still happen if the event loop is restarted).
-                # Unregister it otherwise IocpProactor.close will wait for it
-                # forever
-                if ov is not None:
-                    self._proactor._unregister(ov)
-                self._self_reading_future = None
+    def _run_forever_setup(self):
+        assert self._self_reading_future is None
+        self.call_soon(self._loop_self_reading)
+        super()._run_forever_setup()
+
+    def _run_forever_cleanup(self):
+        super()._run_forever_cleanup()
+        if self._self_reading_future is not None:
+            ov = self._self_reading_future._ov
+            self._self_reading_future.cancel()
+            # self_reading_future was just cancelled so if it hasn't been
+            # finished yet, it never will be (it's possible that it has
+            # already finished and its callback is waiting in the queue,
+            # where it could still happen if the event loop is restarted).
+            # Unregister it otherwise IocpProactor.close will wait for it
+            # forever
+            if ov is not None:
+                self._proactor._unregister(ov)
+            self._self_reading_future = None
 
     async def create_pipe_connection(self, protocol_factory, address):
         f = self._proactor.connect_pipe(address)
diff --git a/Misc/NEWS.d/next/Library/2023-10-13-06-47-20.gh-issue-110771.opwdlc.rst b/Misc/NEWS.d/next/Library/2023-10-13-06-47-20.gh-issue-110771.opwdlc.rst
new file mode 100644
index 00000000000000..a22f8a0d5e56c3
--- /dev/null
+++ b/Misc/NEWS.d/next/Library/2023-10-13-06-47-20.gh-issue-110771.opwdlc.rst
@@ -0,0 +1,1 @@
+Expose the setup and cleanup portions of ``asyncio.run_forever()`` as the standalone methods ``asyncio.run_forever_setup()`` and ``asyncio.run_forever_cleanup()``. This allows for tighter integration with GUI event loops.


Test patch:
diff --git a/Lib/test/test_asyncio/test_base_events.py b/Lib/test/test_asyncio/test_base_events.py
index abcb6f55c4b04e..c2080977e9d587 100644
--- a/Lib/test/test_asyncio/test_base_events.py
+++ b/Lib/test/test_asyncio/test_base_events.py
@@ -922,6 +922,43 @@ def test_run_forever_pre_stopped(self):
         self.loop.run_forever()
         self.loop._selector.select.assert_called_once_with(0)
 
+    def test_custom_run_forever_integration(self):
+        # Test that the run_forever_setup() and run_forever_cleanup() primitives
+        # can be used to implement a custom run_forever loop.
+        self.loop._process_events = mock.Mock()
+
+        count = 0
+
+        def callback():
+            nonlocal count
+            count += 1
+
+        self.loop.call_soon(callback)
+
+        # Set up the custom event loop
+        self.loop._run_forever_setup()
+
+        # Confirm the loop has been started
+        self.assertEqual(asyncio.get_running_loop(), self.loop)
+        self.assertTrue(self.loop.is_running())
+
+        # Our custom "event loop" just iterates 10 times before exiting.
+        for i in range(10):
+            self.loop._run_once()
+
+        # Clean up the event loop
+        self.loop._run_forever_cleanup()
+
+        # Confirm the loop has been cleaned up
+        with self.assertRaises(RuntimeError):
+            asyncio.get_running_loop()
+        self.assertFalse(self.loop.is_running())
+
+        # Confirm the loop actually did run, processing events 10 times,
+        # and invoking the callback once.
+        self.assertEqual(self.loop._process_events.call_count, 10)
+        self.assertEqual(count, 1)
+
     async def leave_unfinalized_asyncgen(self):
         # Create an async generator, iterate it partially, and leave it
         # to be garbage collected.