Repository: scrapy/scrapy

Issue:
Fixes issue with calling fetch in scrapy shell.
Reference Issue Fixes #5740 ,  #5742

You can recreate the issue with the following script.

```
import asyncio
import threading
from twisted.internet import asyncioreactor
from scrapy.utils.defer import deferred_from_coro
from scrapy.utils.reactor import get_asyncio_event_loop_policy

async def test_coro():
    pass

def test_deferred_from_coro():
    return deferred_from_coro(test_coro())

def trigger_warning_message():
    event_loop = get_asyncio_event_loop_policy().new_event_loop()
    asyncio.set_event_loop(event_loop)
    asyncioreactor.install()
    thread = threading.Thread(target=test_deferred_from_coro)
    thread.start()
    thread.join()

trigger_warning_message()
```

I was able to recreate this issue using `scrapy shell` and `fetch` on both windows and linux.  However it only occurs inside of a project with the `TWISTED_REACTOR` setting set to AsyncioSelectorReactor. 

What causes the issue is when `get_asyncio_event_loop_policy().get_event_loop()` is called, there is no event loop in the thread that the function is called from. Which raises an exception.  
asyncio exception in scrapy shell
We got a bug report that using `fetch(<url>)` in `scrapy shell` on Windows in a new project (so with `TWISTED_REACTOR` set to the asyncio one) raises "There is no current event loop" in `deferred_from_coro()`. The Python version is 3.9 and 3.10 and the Scrapy version is 2.7.1. It is possible that the recenty changed loop code indeed doesn't work correctly on Windows but I cannot test or fix it at this time.


Discussion:
# [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/5748?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#5748](https://codecov.io/gh/scrapy/scrapy/pull/5748?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (b67efd2) into [master](https://codecov.io/gh/scrapy/scrapy/commit/fe60c1224e39aa3d85b20afd54566f135d9de085?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (fe60c12) will **decrease** coverage by `4.13%`.
> The diff coverage is `100.00%`.

```diff
@@            Coverage Diff             @@
##           master    #5748      +/-   ##
==========================================
- Coverage   88.91%   84.78%   -4.14%     
==========================================
  Files         162      162              
  Lines       10963    10968       +5     
  Branches     1794     1794              
==========================================
- Hits         9748     9299     -449     
- Misses        932     1398     +466     
+ Partials      283      271      -12     
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/5748?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/utils/defer.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL2RlZmVyLnB5) | `97.41% <100.00%> (+0.08%)` | :arrow_up: |
| [scrapy/pipelines/images.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3BpcGVsaW5lcy9pbWFnZXMucHk=) | `26.47% <0.00%> (-70.59%)` | :arrow_down: |
| [scrapy/core/http2/stream.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvc3RyZWFtLnB5) | `26.58% <0.00%> (-64.74%)` | :arrow_down: |
| [scrapy/core/http2/agent.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvYWdlbnQucHk=) | `36.14% <0.00%> (-60.25%)` | :arrow_down: |
| [scrapy/core/downloader/handlers/http2.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvZG93bmxvYWRlci9oYW5kbGVycy9odHRwMi5weQ==) | `45.07% <0.00%> (-54.93%)` | :arrow_down: |
| [scrapy/core/http2/protocol.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvcHJvdG9jb2wucHk=) | `34.17% <0.00%> (-49.25%)` | :arrow_down: |
| [scrapy/robotstxt.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3JvYm90c3R4dC5weQ==) | `75.30% <0.00%> (-22.23%)` | :arrow_down: |
| [scrapy/utils/test.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL3Rlc3QucHk=) | `51.35% <0.00%> (-14.87%)` | :arrow_down: |
| [scrapy/core/downloader/contextfactory.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvZG93bmxvYWRlci9jb250ZXh0ZmFjdG9yeS5weQ==) | `75.92% <0.00%> (-11.12%)` | :arrow_down: |
| [scrapy/pipelines/media.py](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3BpcGVsaW5lcy9tZWRpYS5weQ==) | `92.90% <0.00%> (-5.68%)` | :arrow_down: |
| ... and [3 more](https://codecov.io/gh/scrapy/scrapy/pull/5748/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | |

It passes all tests locally on both Linux and Windows, so I am not certain why it's failing here.
Can you please also add a test to test_command_shell.py that checks that there are no exceptions printed? I assume it's possible to write one that fails with the current Scrapy on Windows.
> Can you please also add a test to test_command_shell.py that checks that there are no exceptions printed? I assume it's possible to write one that fails with the current Scrapy on Windows.

Sure thing
I wonder if we could go a different way here: rather than create the loop if missing when needed, create it as we create the thread that causes the issue in the first place.

https://github.com/scrapy/scrapy/blob/26ebdbf4efd53bcb48da749576420c9f2fa6e28c/scrapy/commands/shell.py#L71-L80

For example, what if we modify CrawlerProcess.start, so that it takes care of initializing the loop if needed? From there we should have access to settings, i.e. be able to respect ASYNCIO_EVENT_LOOP.
> I wonder if we could go a different way here: rather than create the loop if missing when needed, create it as we create the thread that causes the issue in the first place.
> 
> https://github.com/scrapy/scrapy/blob/26ebdbf4efd53bcb48da749576420c9f2fa6e28c/scrapy/commands/shell.py#L71-L80
> 
> For example, what if we modify CrawlerProcess.start, so that it takes care of initializing the loop if needed? From there we should have access to settings, i.e. be able to respect ASYNCIO_EVENT_LOOP.

I had that same thought when you first made the observation.  But the thread call that causes the issue is the one called from the fetch command I thought, in the `shell` module.

https://github.com/scrapy/scrapy/blob/26ebdbf4efd53bcb48da749576420c9f2fa6e28c/scrapy/shell.py#L110
According to the info in #5742, this seems to only affect `scrapy shell` and also doesn't prevent it from normal usage.
This may be related to the fact that `scrapy shell` runs scraping in a separate thread.

Resolving patch:
diff --git a/scrapy/shell.py b/scrapy/shell.py
--- a/scrapy/shell.py
+++ b/scrapy/shell.py
@@ -21,6 +21,7 @@
 from scrapy.utils.datatypes import SequenceExclude
 from scrapy.utils.misc import load_object
 from scrapy.utils.response import open_in_browser
+from scrapy.utils.reactor import is_asyncio_reactor_installed, set_asyncio_event_loop
 
 
 class Shell:
@@ -76,6 +77,10 @@ def start(self, url=None, request=None, response=None, spider=None, redirect=Tru
                                  banner=self.vars.pop('banner', ''))
 
     def _schedule(self, request, spider):
+        if is_asyncio_reactor_installed():
+            # set the asyncio event loop for the current thread
+            event_loop_path = self.crawler.settings['ASYNCIO_EVENT_LOOP']
+            set_asyncio_event_loop(event_loop_path)
         spider = self._open_spider(request, spider)
         d = _request_deferred(request)
         d.addCallback(lambda x: (x, spider))
diff --git a/scrapy/utils/reactor.py b/scrapy/utils/reactor.py
--- a/scrapy/utils/reactor.py
+++ b/scrapy/utils/reactor.py
@@ -71,14 +71,7 @@ def install_reactor(reactor_path, event_loop_path=None):
     reactor_class = load_object(reactor_path)
     if reactor_class is asyncioreactor.AsyncioSelectorReactor:
         with suppress(error.ReactorAlreadyInstalledError):
-            policy = get_asyncio_event_loop_policy()
-            if event_loop_path is not None:
-                event_loop_class = load_object(event_loop_path)
-                event_loop = event_loop_class()
-                asyncio.set_event_loop(event_loop)
-            else:
-                event_loop = policy.get_event_loop()
-
+            event_loop = set_asyncio_event_loop(event_loop_path)
             asyncioreactor.install(eventloop=event_loop)
     else:
         *module, _ = reactor_path.split(".")
@@ -88,6 +81,25 @@ def install_reactor(reactor_path, event_loop_path=None):
             installer()
 
 
+def set_asyncio_event_loop(event_loop_path):
+    """Sets and returns the event loop with specified import path."""
+    policy = get_asyncio_event_loop_policy()
+    if event_loop_path is not None:
+        event_loop_class = load_object(event_loop_path)
+        event_loop = event_loop_class()
+        asyncio.set_event_loop(event_loop)
+    else:
+        try:
+            event_loop = policy.get_event_loop()
+        except RuntimeError:
+            # `get_event_loop` is expected to fail when called from a new thread
+            # with no asyncio event loop yet installed. Such is the case when
+            # called from `scrapy shell`
+            event_loop = policy.new_event_loop()
+            asyncio.set_event_loop(event_loop)
+    return event_loop
+
+
 def verify_installed_reactor(reactor_path):
     """Raises :exc:`Exception` if the installed
     :mod:`~twisted.internet.reactor` does not match the specified import


Test patch:
diff --git a/tests/test_command_shell.py b/tests/test_command_shell.py
--- a/tests/test_command_shell.py
+++ b/tests/test_command_shell.py
@@ -115,3 +115,14 @@ def test_dns_failures(self):
         errcode, out, err = yield self.execute([url, '-c', 'item'], check_code=False)
         self.assertEqual(errcode, 1, out or err)
         self.assertIn(b'DNS lookup failed', err)
+
+    @defer.inlineCallbacks
+    def test_shell_fetch_async(self):
+        reactor_path = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"
+        url = self.url('/html')
+        code = f"fetch('{url}')"
+        args = ["-c", code, "--set", f"TWISTED_REACTOR={reactor_path}"]
+        _, _, err = yield self.execute(args, check_code=True)
+        self.assertNotIn(
+            b"RuntimeError: There is no current event loop in thread", err
+        )