Repository: scrapy-plugins/scrapy-playwright (language: python)

Issue:
Windows: crawl freezes when multiple schemas are set in DOWNLOAD_HANDLERS
If I have said anything impolite, please forgive me, as I have used a translator. My basic situation is the same as this issue [Spiders hangs](https://github.com/scrapy-plugins/scrapy-playwright/issues/266). I locate the problem to this setting item.
```python
"DOWNLOAD_HANDLERS": {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
},
```
# Code
When I comment out `DOWNLOAD_HANDLERS`, the spider runs normally. Here is my code, you can directly run it and try.
```python
from pathlib import Path
from scrapy.crawler import CrawlerProcess
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"

    def start_requests(self):
        urls = [
            "https://quotes.toscrape.com/page/1/",
            "https://quotes.toscrape.com/page/2/",
        ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse,
                                  meta={"playwright": True},)

    def parse(self, response):
        page = response.url.split("/")[-2]
        filename = f"quotes-{page}.html"
        Path(filename).write_bytes(response.body)
        self.log(f"Saved file {filename}")

if __name__=="__main__":
    process = CrawlerProcess(
        settings={
            "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
            "DOWNLOAD_HANDLERS": {
                "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
                "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            },
            "CONCURRENT_REQUESTS": 32,
        }
    )
    process.crawl(QuotesSpider)
    process.start()
```
# Result
```plaintext
(env4gpt) D:\Repo\crawl>D:/anaconda3/envs/env4gpt/python.exe d:/Repo/crawl/main.py
2024-07-05 19:36:00 [scrapy.utils.log] INFO: Scrapy 2.11.2 started (bot: scrapybot)
2024-07-05 19:36:00 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.11.7, cssselect 1.2.0, parsel 1.9.1, w3lib 2.2.0, Twisted 24.3.0, Python 3.9.18 (main, Sep 11 2023, 14:09:26) [MSC v.1916 64 bit (AMD64)], pyOpenSSL 24.1.0 (OpenSSL 3.2.2 4 Jun 2024), cryptography 42.0.8, Platform Windows-10-10.0.19045-SP0
2024-07-05 19:36:00 [scrapy.addons] INFO: Enabled addons:
[]
2024-07-05 19:36:00 [py.warnings] WARNING: D:\anaconda3\envs\env4gpt\lib\site-packages\scrapy\utils\request.py:254: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.

It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in a future version of Scrapy.

See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
  return cls(crawler)

2024-07-05 19:36:00 [asyncio] DEBUG: Using selector: SelectSelector
2024-07-05 19:36:00 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor
2024-07-05 19:36:00 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop
2024-07-05 19:36:00 [scrapy.extensions.telnet] INFO: Telnet Password: 6b77837564f0d1cc
2024-07-05 19:36:00 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2024-07-05 19:36:00 [scrapy.crawler] INFO: Overridden settings:
{'CONCURRENT_REQUESTS': 32,
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2024-07-05 19:36:01 [asyncio] DEBUG: Using proactor: IocpProactor
2024-07-05 19:36:01 [scrapy-playwright] INFO: Started loop on separate thread: <ProactorEventLoop 
running=True closed=False debug=False>
2024-07-05 19:36:01 [asyncio] DEBUG: Using proactor: IocpProactor
2024-07-05 19:36:01 [scrapy-playwright] INFO: Started loop on separate thread: <ProactorEventLoop 
running=True closed=False debug=False>
2024-07-05 19:36:01 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2024-07-05 19:36:01 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-07-05 19:36:01 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2024-07-05 19:36:01 [scrapy.core.engine] INFO: Spider opened
2024-07-05 19:36:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:36:01 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023   
2024-07-05 19:37:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:38:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:39:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:40:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:41:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:42:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:43:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:44:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:45:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:46:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:47:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:48:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:49:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:50:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:51:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-05 19:52:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
```
The log continues to repeat "[scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)".
# Conda Environment
```plaintext
openpyxl=3.1.4
scrapy==2.11.2
lxml==5.2.2
python-dotenv==1.0.1
scrapy-playwright==0.0.37
```
# System
Windows 10
# Guess
Is it possible that it is related to the computer system? I currently do not have a computer with a Linux system and cannot verify this. Besides, I am in China, could it be related to the IP address? However, I checked the `ScrapyPlaywrightDownloadHandler` class and did not see any bugs. Thank you for your help.



Discussion / hints:
Same issue for me, on windows 10, same components versions, seems that browser is not launched at all, if I add `PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": False}` I see no browser
> Same issue for me, on windows 10, same components versions, seems that browser is not launched at all, if I add `PLAYWRIGHT_LAUNCH_OPTIONS = {"headless": False}` I see no browser

The bug is not fixed yet, you can try `python-playwright` as a alternative solution to develop  your spider program.
I have been developing the package and testing crawls on Windows 11, installing packages with `pip` in a [virtual environment](https://docs.python.org/3/library/venv.html) (without Conda). I do not currently have access to a Windows 10 system, however the CI Windows build is running on [`windows-latest`](https://github.com/scrapy-plugins/scrapy-playwright/blob/f3b1b25e7ded23ebcc4d5dec73ba0367d0a51d41/.github/workflows/tests.yml#L17-L18): according to the [most recent docs on the images](https://github.com/actions/runner-images/tree/567d4a0ea4a16d7a6e3e03b0f5068b99a7af8111?tab=readme-ov-file#available-images) this is the same as `windows-2022`, which includes [Windows 10.0.20348 Build 2529](https://github.com/actions/runner-images/blob/567d4a0ea4a16d7a6e3e03b0f5068b99a7af8111/images/windows/Windows2022-Readme.md).
There was an earlier issue that involved Conda as well: #244. I could not reproduce it without Conda, given that and the above I'm inclined to believe that Conda might be the issue here.
Please try again without Conda to see if the issue is resolved.
> I have been developing the package and testing crawls on Windows 11, installing packages with `pip` in a [virtual environment](https://docs.python.org/3/library/venv.html) (without Conda). I do not currently have access to a Windows 10 system, however the CI Windows build is running on [`windows-latest`](https://github.com/scrapy-plugins/scrapy-playwright/blob/f3b1b25e7ded23ebcc4d5dec73ba0367d0a51d41/.github/workflows/tests.yml#L17-L18): according to the [most recent docs on the images](https://github.com/actions/runner-images/tree/567d4a0ea4a16d7a6e3e03b0f5068b99a7af8111?tab=readme-ov-file#available-images) this is the same as `windows-2022`, which includes [Windows 10.0.20348 Build 2529](https://github.com/actions/runner-images/blob/567d4a0ea4a16d7a6e3e03b0f5068b99a7af8111/images/windows/Windows2022-Readme.md). There was an earlier issue that involved Conda as well: #244. I could not reproduce it without Conda, given that and the above I'm inclined to believe that Conda might be the issue here. Please try again without Conda to see if the issue is resolved.

Thank you for your help. It looks like not the problem of conda in my situation. I closed the conda environment and used the default python.exe on my computer, then downloaded the scrapy-playwright library. Even after ensuring that I'm running the program with the default python.exe rather than the python.exe from the conda environment, the issue persists.

```
D:\Repo\crawl>C:/Python312/python.exe d:/Repo/crawl/main.py
2024-07-09 01:59:19 [scrapy.utils.log] INFO: Scrapy 2.11.2 started (bot: scrapybot)
2024-07-09 01:59:19 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.11.7, cssselect 1.2.0, parsel 1.9.1, w3lib 2.2.1, Twisted 24.3.0, Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct  2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)], pyOpenSSL 24.1.0 (OpenSSL 3.2.2 4 Jun 2024), cryptography 42.0.8, Platform Windows-10-10.0.19045-SP0
2024-07-09 01:59:19 [scrapy.addons] INFO: Enabled addons:
[]
2024-07-09 01:59:19 [py.warnings] WARNING: C:\Python312\Lib\site-packages\scrapy\utils\request.py:254: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting.

It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in 
a future version of Scrapy.

See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
  return cls(crawler)

2024-07-09 01:59:19 [asyncio] DEBUG: Using selector: SelectSelector
2024-07-09 01:59:19 [scrapy.utils.log] DEBUG: Using reactor: twisted.internet.asyncioreactor.AsyncioSelectorReactor   
2024-07-09 01:59:19 [scrapy.utils.log] DEBUG: Using asyncio event loop: asyncio.windows_events._WindowsSelectorEventLoop
2024-07-09 01:59:19 [scrapy.extensions.telnet] INFO: Telnet Password: e6ffaca82a959077
2024-07-09 01:59:19 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.logstats.LogStats']
2024-07-09 01:59:19 [scrapy.crawler] INFO: Overridden settings:
{'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'}
2024-07-09 01:59:19 [asyncio] DEBUG: Using proactor: IocpProactor
2024-07-09 01:59:19 [scrapy-playwright] INFO: Started loop on separate thread: <ProactorEventLoop running=True closed=False debug=False>
2024-07-09 01:59:19 [asyncio] DEBUG: Using proactor: IocpProactor
2024-07-09 01:59:19 [scrapy-playwright] INFO: Started loop on separate thread: <ProactorEventLoop running=True closed=False debug=False>
2024-07-09 01:59:20 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2024-07-09 01:59:20 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-07-09 01:59:20 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2024-07-09 01:59:20 [scrapy.core.engine] INFO: Spider opened
2024-07-09 01:59:20 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-09 01:59:20 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2024-07-09 02:00:20 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-09 02:01:20 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-09 02:02:20 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-09 02:03:20 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
```

I create a venv enviroment and run the code. The problem still persists
I was wrong about Conda, I was able to reproduce the issue in Windows 11 as well. What I found is that the spider works correctly if only one of the schemas is set in the `DOWNLOAD_HANDLERS` settings, e.g.:
```python
"DOWNLOAD_HANDLERS": {
    "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
    # "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
},
```
Fortunately plaintext http is rare nowadays, so only enabling the handler for the `https` schema should be enough for most sites.
I do not know why this is happening, I will need to check in more depth.  
> I was wrong about Conda, I was able to reproduce the issue in Windows 11 as well. What I found is that the spider works correctly if only one of the schemas is set in the `DOWNLOAD_HANDLERS` settings, e.g.:
> 
> ```python
> "DOWNLOAD_HANDLERS": {
>     "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
>     # "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
> },
> ```
> 
> Fortunately plaintext http is rare nowadays, so only enabling the handler for the `https` schema should be enough for most sites. I do not know why this is happening, I will need to check in more depth.

Use conda and set single https schema ❌ 
Dont use conda and set single https schema ✅
Yes, you are right. I use my default `python.exe`(venv also works) and delete the http schema and the spider runs successfully.
The bug has been circumvented, but the cause remains unknown.

Fix patch:
diff --git a/scrapy_playwright/_utils.py b/scrapy_playwright/_utils.py
index a511dff..2659828 100644
--- a/scrapy_playwright/_utils.py
+++ b/scrapy_playwright/_utils.py
@@ -2,7 +2,7 @@
 import logging
 import platform
 import threading
-from typing import Awaitable, Iterator, Optional, Tuple, Union
+from typing import Awaitable, Dict, Iterator, Optional, Tuple, Union
 
 import scrapy
 from playwright.async_api import Error, Page, Request, Response
@@ -103,68 +103,58 @@ async def _get_header_value(
         return None
 
 
-if platform.system() == "Windows":
-
-    class _ThreadedLoopAdapter:
-        """Utility class to start an asyncio event loop in a new thread and redirect coroutines.
-        This allows to run Playwright in a different loop than the Scrapy crawler, allowing to
-        use ProactorEventLoop which is supported by Playwright on Windows.
-        """
-
-        _loop: asyncio.AbstractEventLoop
-        _thread: threading.Thread
-        _coro_queue: asyncio.Queue = asyncio.Queue()
-        _stop_event: asyncio.Event = asyncio.Event()
-
-        @classmethod
-        async def _handle_coro(cls, coro, future) -> None:
-            try:
-                future.set_result(await coro)
-            except Exception as exc:
-                future.set_exception(exc)
-
-        @classmethod
-        async def _process_queue(cls) -> None:
-            while not cls._stop_event.is_set():
-                coro, future = await cls._coro_queue.get()
-                asyncio.create_task(cls._handle_coro(coro, future))
-                cls._coro_queue.task_done()
-
-        @classmethod
-        def _deferred_from_coro(cls, coro) -> Deferred:
-            future: asyncio.Future = asyncio.Future()
-            asyncio.run_coroutine_threadsafe(cls._coro_queue.put((coro, future)), cls._loop)
-            return scrapy.utils.defer.deferred_from_coro(future)
-
-        @classmethod
-        def start(cls) -> None:
-            policy = asyncio.WindowsProactorEventLoopPolicy()  # type: ignore[attr-defined]
+class _ThreadedLoopAdapter:
+    """Utility class to start an asyncio event loop in a new thread and redirect coroutines.
+    This allows to run Playwright in a different loop than the Scrapy crawler, allowing to
+    use ProactorEventLoop which is supported by Playwright on Windows.
+    """
+
+    _loop: asyncio.AbstractEventLoop
+    _thread: threading.Thread
+    _coro_queue: asyncio.Queue = asyncio.Queue()
+    _stop_events: Dict[int, asyncio.Event] = {}
+
+    @classmethod
+    async def _handle_coro(cls, coro, future) -> None:
+        try:
+            future.set_result(await coro)
+        except Exception as exc:
+            future.set_exception(exc)
+
+    @classmethod
+    async def _process_queue(cls) -> None:
+        while any(not ev.is_set() for ev in cls._stop_events.values()):
+            coro, future = await cls._coro_queue.get()
+            asyncio.create_task(cls._handle_coro(coro, future))
+            cls._coro_queue.task_done()
+
+    @classmethod
+    def _deferred_from_coro(cls, coro) -> Deferred:
+        future: asyncio.Future = asyncio.Future()
+        asyncio.run_coroutine_threadsafe(cls._coro_queue.put((coro, future)), cls._loop)
+        return scrapy.utils.defer.deferred_from_coro(future)
+
+    @classmethod
+    def start(cls, caller_id: int) -> None:
+        cls._stop_events[caller_id] = asyncio.Event()
+        if not getattr(cls, "_loop", None):
+            policy = asyncio.DefaultEventLoopPolicy()
+            if platform.system() == "Windows":
+                policy = asyncio.WindowsProactorEventLoopPolicy()  # type: ignore[attr-defined]
             cls._loop = policy.new_event_loop()
             asyncio.set_event_loop(cls._loop)
 
+        if not getattr(cls, "_thread", None):
             cls._thread = threading.Thread(target=cls._loop.run_forever, daemon=True)
             cls._thread.start()
             logger.info("Started loop on separate thread: %s", cls._loop)
-
             asyncio.run_coroutine_threadsafe(cls._process_queue(), cls._loop)
 
-        @classmethod
-        def stop(cls) -> None:
-            cls._stop_event.set()
+    @classmethod
+    def stop(cls, caller_id: int) -> None:
+        """Wait until all handlers are closed to stop the event loop and join the thread."""
+        cls._stop_events[caller_id].set()
+        if all(ev.is_set() for ev in cls._stop_events.values()):
             asyncio.run_coroutine_threadsafe(cls._coro_queue.join(), cls._loop)
             cls._loop.call_soon_threadsafe(cls._loop.stop)
             cls._thread.join()
-
-    _deferred_from_coro = _ThreadedLoopAdapter._deferred_from_coro
-else:
-
-    class _ThreadedLoopAdapter:  # type: ignore[no-redef]
-        @classmethod
-        def start(cls) -> None:
-            pass
-
-        @classmethod
-        def stop(cls) -> None:
-            pass
-
-    _deferred_from_coro = scrapy.utils.defer.deferred_from_coro
diff --git a/scrapy_playwright/handler.py b/scrapy_playwright/handler.py
index e9a2f22..9435016 100644
--- a/scrapy_playwright/handler.py
+++ b/scrapy_playwright/handler.py
@@ -1,5 +1,6 @@
 import asyncio
 import logging
+import platform
 from contextlib import suppress
 from dataclasses import dataclass, field as dataclass_field
 from ipaddress import ip_address
@@ -26,6 +27,7 @@
 from scrapy.http.headers import Headers
 from scrapy.responsetypes import responsetypes
 from scrapy.settings import Settings
+from scrapy.utils.defer import deferred_from_coro
 from scrapy.utils.misc import load_object
 from scrapy.utils.reactor import verify_installed_reactor
 from twisted.internet.defer import Deferred, inlineCallbacks
@@ -34,7 +36,6 @@
 from scrapy_playwright.page import PageMethod
 from scrapy_playwright._utils import (
     _ThreadedLoopAdapter,
-    _deferred_from_coro,
     _encode_body,
     _get_float_setting,
     _get_header_value,
@@ -91,6 +92,7 @@ class Config:
     startup_context_kwargs: dict
     navigation_timeout: Optional[float]
     restart_disconnected_browser: bool
+    use_threaded_loop: bool
 
     @classmethod
     def from_settings(cls, settings: Settings) -> "Config":
@@ -114,6 +116,8 @@ def from_settings(cls, settings: Settings) -> "Config":
             restart_disconnected_browser=settings.getbool(
                 "PLAYWRIGHT_RESTART_DISCONNECTED_BROWSER", default=True
             ),
+            use_threaded_loop=platform.system() == "Windows"
+            or settings.getbool("_PLAYWRIGHT_THREADED_LOOP", False),
         )
         cfg.cdp_kwargs.pop("endpoint_url", None)
         cfg.connect_kwargs.pop("ws_endpoint", None)
@@ -130,13 +134,14 @@ class ScrapyPlaywrightDownloadHandler(HTTPDownloadHandler):
 
     def __init__(self, crawler: Crawler) -> None:
         super().__init__(settings=crawler.settings, crawler=crawler)
-        _ThreadedLoopAdapter.start()
         verify_installed_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor")
         crawler.signals.connect(self._engine_started, signals.engine_started)
         self.stats = crawler.stats
-
         self.config = Config.from_settings(crawler.settings)
 
+        if self.config.use_threaded_loop:
+            _ThreadedLoopAdapter.start(id(self))
+
         self.browser_launch_lock = asyncio.Lock()
         self.context_launch_lock = asyncio.Lock()
         self.context_wrappers: Dict[str, BrowserContextWrapper] = {}
@@ -162,9 +167,14 @@ def __init__(self, crawler: Crawler) -> None:
     def from_crawler(cls: Type[PlaywrightHandler], crawler: Crawler) -> PlaywrightHandler:
         return cls(crawler)
 
+    def _deferred_from_coro(self, coro: Awaitable) -> Deferred:
+        if self.config.use_threaded_loop:
+            return _ThreadedLoopAdapter._deferred_from_coro(coro)
+        return deferred_from_coro(coro)
+
     def _engine_started(self) -> Deferred:
         """Launch the browser. Use the engine_started signal as it supports returning deferreds."""
-        return _deferred_from_coro(self._launch())
+        return self._deferred_from_coro(self._launch())
 
     async def _launch(self) -> None:
         """Launch Playwright manager and configured startup context(s)."""
@@ -333,8 +343,9 @@ def _set_max_concurrent_context_count(self):
     def close(self) -> Deferred:
         logger.info("Closing download handler")
         yield super().close()
-        yield _deferred_from_coro(self._close())
-        _ThreadedLoopAdapter.stop()
+        yield self._deferred_from_coro(self._close())
+        if self.config.use_threaded_loop:
+            _ThreadedLoopAdapter.stop(id(self))
 
     async def _close(self) -> None:
         await asyncio.gather(*[ctx.context.close() for ctx in self.context_wrappers.values()])
@@ -349,7 +360,7 @@ async def _close(self) -> None:
 
     def download_request(self, request: Request, spider: Spider) -> Deferred:
         if request.meta.get("playwright"):
-            return _deferred_from_coro(self._download_request(request, spider))
+            return self._deferred_from_coro(self._download_request(request, spider))
         return super().download_request(request, spider)
 
     async def _download_request(self, request: Request, spider: Spider) -> Response:


Test patch:
diff --git a/tests/__init__.py b/tests/__init__.py
index 7f6a9fb..95e464b 100644
--- a/tests/__init__.py
+++ b/tests/__init__.py
@@ -24,10 +24,11 @@ def allow_windows(test_method):
 
         @wraps(test_method)
         async def wrapped(self, *args, **kwargs):
-            _ThreadedLoopAdapter.start()
+            caller_id = 1234
+            _ThreadedLoopAdapter.start(caller_id)
             coro = test_method(self, *args, **kwargs)
             asyncio.run_coroutine_threadsafe(coro=coro, loop=_ThreadedLoopAdapter._loop).result()
-            _ThreadedLoopAdapter.stop()
+            _ThreadedLoopAdapter.stop(caller_id)
 
         return wrapped