Repository: scrapy/scrapy
Pull request: test_install_asyncio_reactor: restore the original reactor when done
Fixes https://github.com/scrapy/scrapy/issues/5765

Problem statement:
AsyncioTest failures
There are two new failing tests (`AsyncioTest.test_install_asyncio_reactor` and `AsyncioTest.test_is_asyncio_reactor_installed`) on 3.10 and 3.11, and it looks like they started failing after updating to 3.10.9 and 3.11.1 (it may be a coincidence though). I see an asyncio-related change in those versions: "asyncio.get_event_loop() now only emits a deprecation warning when a new event loop was created implicitly. It no longer emits a deprecation warning if the current event loop was set." and it looks like it's the cause for the `test_install_asyncio_reactor` failure as that checks for a warning and maybe there is some new warning. But the second test is about installing the reactor and checking its type and the failure looks like the asyncio reactor is installed when it shouldn't.

Hints from issue discussion:
Disabling `test_install_asyncio_reactor` makes `test_is_asyncio_reactor_installed` pass; I suspect that the former now installs the reactor when previously it didn't (initially this test had a comment "does nothing" but I don't remember why did this call do nothing), this needs future investigation. It may actually just depend on the test run order, maybe it actually broke some time ago. 

As for `test_install_asyncio_reactor`, the new warning is "DeprecationWarning('There is no current event loop')". Which is the same as we got in #5685 and we had a fix for it. Maybe we need a different fix for the current Python version (I wonder if we need to test older point releases, likely we don't).

Commit message: Sturdier fix for asyncio warning

Patch to review:
diff --git a/scrapy/utils/reactor.py b/scrapy/utils/reactor.py
index e6b8de292ef..853c85b2749 100644
--- a/scrapy/utils/reactor.py
+++ b/scrapy/utils/reactor.py
@@ -90,11 +90,16 @@ def set_asyncio_event_loop(event_loop_path):
         asyncio.set_event_loop(event_loop)
     else:
         try:
+            asyncio.get_running_loop()
             event_loop = policy.get_event_loop()
         except RuntimeError:
-            # `get_event_loop` is expected to fail when called from a new thread
+            # Raised by policy.get_running_loop() when called from a new thread
             # with no asyncio event loop yet installed. Such is the case when
-            # called from `scrapy shell`
+            # called from `scrapy shell`.
+            # In Python 3.10, but not in 3.11, policy.get_event_loop() seems to
+            # log a warning when called from the main thread, so we use
+            # asyncio.get_running_loop() to get an earlier RuntimeError as a
+            # workaround.
             event_loop = policy.new_event_loop()
             asyncio.set_event_loop(event_loop)
     return event_loop
diff --git a/tests/test_utils_asyncio.py b/tests/test_utils_asyncio.py
index 741c6a5051b..42780ace7b0 100644
--- a/tests/test_utils_asyncio.py
+++ b/tests/test_utils_asyncio.py
@@ -14,6 +14,9 @@ def test_is_asyncio_reactor_installed(self):
         self.assertEqual(is_asyncio_reactor_installed(), self.reactor_pytest == 'asyncio')
 
     def test_install_asyncio_reactor(self):
+        from twisted.internet import reactor as original_reactor
         with warnings.catch_warnings(record=True) as w:
             install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
             self.assertEqual(len(w), 0)
+        from twisted.internet import reactor
+        assert original_reactor == reactor


Code review comments:
diff_hunk: @@ -90,11 +90,16 @@ def set_asyncio_event_loop(event_loop_path):
         asyncio.set_event_loop(event_loop)
     else:
         try:
+            asyncio.get_running_loop()
             event_loop = policy.get_event_loop()
         except RuntimeError:
-            # `get_event_loop` is expected to fail when called from a new thread
+            # Raised by policy.get_running_loop() when called from a new thread
             # with no asyncio event loop yet installed. Such is the case when
-            # called from `scrapy shell`
+            # called from `scrapy shell`.
+            # In Python 3.10, but not in 3.11, policy.get_event_loop() seems to
+            # log a warning when called from the main thread, so we use
+            # asyncio.get_running_loop() to get an earlier RuntimeError as a
+            # workaround.
path: scrapy/utils/reactor.py
text: @author:
I think this is happening, it feels like a Python 3.10 bug, but I have found no information about it.

If you comment out `asyncio.get_running_loop()` above, `test_install_asyncio_reactor` fails in Python 3.10 but _not_ in Python 3.11. You can try for yourself with:

```
tox -e py311,py310 -- tests/test_utils_asyncio.py::AsyncioTest tests/test_command_shell.py::ShellTest::test_shell_fetch_async
```

@author:
My Python 3.11 is 3.11.0, that explains the 3.10-3.11 difference I get locally.