Repository: scrapy-plugins/scrapy-playwright (language: python)

Issue:
Browser breaks with new tab on empty content response
Hello world!

I am building a common Spider that crawls sites and the contained request.
I use scrapy-playwright to load websites first and get the requests that are sent.

I noticed that when I parse urls that have no content on body the execution freezes and playwright's browser shows empty tab.
To be clear reproduction of the problem is when parse a url that has the following condition as true:

```
response_body_text = await response.text()
response_body_text  == ''
```

For the urls that this condition is false spider works perfectly!

For the reproduction, I have a quite common configuration with:

```
CrawlerProcess({
        ...
        # Playwright settings
        'DOWNLOAD_HANDLERS': {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        'TWISTED_REACTOR': "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
        'PLAYWRIGHT_BROWSER_TYPE': 'chromium',
        'PLAYWRIGHT_MAX_PAGES_PER_CONTEXT': 10,
        'PLAYWRIGHT_LAUNCH_OPTIONS': {
            'headless': True,
        }        
})
```

and on each scrapy.Request() I pass the following meta:

```
{
            "playwright": True
}
```

Has anybody else come up with this issue?

Thank you all!


Discussion / hints:
I don't know how to attempt to reproduce. I assume `response` is a Playwright response given [Response.text](https://github.com/microsoft/playwright-python/blob/v1.45.0/playwright/_impl/_network.py#L610) (Scrapy's [TextResponse.text](https://docs.scrapy.org/en/2.11/topics/request-response.html#scrapy.http.TextResponse.text) is an attribute, not a coroutine), however Playwright responses are not directly exposed by scrapy-playwright. Are you maybe connecting to the `response` event with [`playwright_page_event_handlers`](https://github.com/scrapy-plugins/scrapy-playwright?tab=readme-ov-file#playwright_page_event_handlers)?

Please share code that causes the issue, not only configuration settings. Also include logs that show the error. 
The 'await response.text()' was in a response handler that used to try catch the error but  removed the whole handler.
Helped me identify that if avoid parsing this type of urls script would finish normally.

A sample reproduction is the following. It is required to fill the start_urls with a request which will have empty response and status 200.

```
import scrapy

from scrapy.spiders import SitemapSpider
from scrapy.crawler import CrawlerProcess
from scrapy.http import Response
from typing import Any


class MySpider(SitemapSpider):
    name = "MySpider"
    forbiden_calls = set()

    def __init__(self, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.start_urls = []
        self.scrapy_meta = {
            "playwright": True
        }

    def start_requests(self):
        for url in self.start_urls:
            print("Start URL", url)
            yield scrapy.Request(url, self.parse, errback=self.onError, meta=self.scrapy_meta)

    def parse(self, response: Response) -> Any:
        print("Parsing", response.url, response.body, response.status)
                
    def onError(self, failure):
        print(f"Failed to response parsing {failure.request.url}: {failure.value}")

def main():
    process = CrawlerProcess({
        'DOWNLOAD_HANDLERS': {
            "http": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
            "https": "scrapy_playwright.handler.ScrapyPlaywrightDownloadHandler",
        },
        'PLAYWRIGHT_LAUNCH_OPTIONS': {
            'headless': False,
        }        
    })

    process.crawl(MySpider)
    process.start()

if __name__ == '__main__':
    main()
```
I cannot reproduce.

```
$ scrapy version -v
Scrapy       : 2.11.1
lxml         : 5.1.0.0
libxml2      : 2.12.3
cssselect    : 1.2.0
parsel       : 1.8.1
w3lib        : 2.1.2
Twisted      : 23.10.0
Python       : 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0]
pyOpenSSL    : 24.0.0 (OpenSSL 3.2.1 30 Jan 2024)
cryptography : 42.0.5
Platform     : Linux-6.5.0-35-generic-x86_64-with-glibc2.35

$ python -c "import scrapy_playwright; print(scrapy_playwright.__version__)"
0.0.37

$ playwright --version               
Version 1.44.0
```

Running a [minimal Flask application](https://flask.palletsprojects.com/en/3.0.x/quickstart/#a-minimal-application) that returns an empty 200 response:
```python
from flask import Flask

app = Flask(__name__)

@app.route("/")
def hello_world():
    return ""
```

```
$ curl -v http://127.0.0.1:5000
*   Trying 127.0.0.1:5000...
* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
> GET / HTTP/1.1
> Host: 127.0.0.1:5000
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: Werkzeug/3.0.3 Python/3.10.12
< Date: Thu, 04 Jul 2024 18:08:21 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 0
< Connection: close
< 
* Closing connection 0
```

```
$ python
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> r = requests.get("http://127.0.0.1:5000")
>>> r.text, r.status_code
('', 200)
>>> 
```

I modified your code in two ways:
* added "http://127.0.0.1:5000" to `self.start_urls`
* added `"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor"` to the crawler settings

This is the result (notice there's some additional wrapping HTML, this is expected because the Scrapy response is the result of [Page.content](https://playwright.dev/python/docs/api/class-page#page-content) instead of [Response.text](https://playwright.dev/python/docs/api/class-response#response-text)):
```
...
2024-07-04 14:28:55 [scrapy.core.engine] INFO: Spider opened
2024-07-04 14:28:55 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-07-04 14:28:55 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2024-07-04 14:28:55 [scrapy-playwright] INFO: Starting download handler
2024-07-04 14:28:55 [scrapy-playwright] INFO: Starting download handler
Start URL http://127.0.0.1:5000/
2024-07-04 14:29:00 [scrapy-playwright] INFO: Launching browser chromium
2024-07-04 14:29:00 [scrapy-playwright] INFO: Browser chromium launched
2024-07-04 14:29:00 [scrapy-playwright] DEBUG: Browser context started: 'default' (persistent=False, remote=False)
2024-07-04 14:29:00 [scrapy-playwright] DEBUG: [Context=default] New page created, page count is 1 (1 for all contexts)
2024-07-04 14:29:00 [scrapy-playwright] DEBUG: [Context=default] Request: <GET http://127.0.0.1:5000/> (resource type: document)
2024-07-04 14:29:00 [scrapy-playwright] DEBUG: [Context=default] Response: <200 http://127.0.0.1:5000/>
2024-07-04 14:29:01 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://127.0.0.1:5000/> (referer: None) ['playwright']
Parsing http://127.0.0.1:5000/ b'<html><head></head><body></body></html>' 200
2024-07-04 14:29:01 [scrapy.core.engine] INFO: Closing spider (finished)
...
```

Given that the `TWISTED_REACTOR` was not set I suspect you're running on Windows, is this correct? Please share your software versions as explained [here](https://github.com/scrapy-plugins/scrapy-playwright?tab=readme-ov-file#software-versions).
Thank you for your cooperation and the detailed answer. I appreciate it.

Your reproduction works on my example too, probably the endpoint dont return the same format. I have already tested the implementation with TWISTED_REACTOR and without but always get the same result. Freezing browser.

By the way the output of curl on the url that trying to crawl is:
```
StatusCode        : 204
StatusDescription : No Content
Content           : {}
RawContent        : HTTP/1.1 204 No Content
                              Connection: keep-alive
                              ...
RawContentLength  : 0
```

Scrapy logs don't show any information but stop on 

` DEBUG: [Context=default] Response: <204 ...>`

Fix patch:
diff --git a/scrapy_playwright/handler.py b/scrapy_playwright/handler.py
index 84d4ac3..4172a59 100644
--- a/scrapy_playwright/handler.py
+++ b/scrapy_playwright/handler.py
@@ -372,12 +372,12 @@ async def _download_request_with_page(
             request.meta["playwright_page"] = page
 
         start_time = time()
-        response, download = await self._get_response_and_download(request=request, page=page)
+        response, download = await self._get_response_and_download(request, page, spider)
         if isinstance(response, PlaywrightResponse):
             await _set_redirect_meta(request=request, response=response)
             headers = Headers(await response.all_headers())
             headers.pop("Content-Encoding", None)
-        else:
+        elif not download.get("bytes"):
             logger.warning(
                 "Navigating to %s returned None, the response"
                 " will have empty headers and status 200",
@@ -415,7 +415,7 @@ async def _download_request_with_page(
             await page.close()
             self.stats.inc_value("playwright/page_count/closed")
 
-        if download:
+        if download.get("bytes"):
             request.meta["playwright_suggested_filename"] = download.get("suggested_filename")
             respcls = responsetypes.from_args(url=download["url"], body=download["bytes"])
             return respcls(
@@ -440,13 +440,15 @@ async def _download_request_with_page(
         )
 
     async def _get_response_and_download(
-        self, request: Request, page: Page
+        self, request: Request, page: Page, spider: Spider
     ) -> Tuple[Optional[PlaywrightResponse], dict]:
         response: Optional[PlaywrightResponse] = None
         download: dict = {}  # updated in-place in _handle_download
+        download_started = asyncio.Event()
         download_ready = asyncio.Event()
 
         async def _handle_download(dwnld: Download) -> None:
+            download_started.set()
             self.stats.inc_value("playwright/download_count")
             try:
                 if failure := await dwnld.failure():
@@ -460,9 +462,14 @@ async def _handle_download(dwnld: Download) -> None:
             finally:
                 download_ready.set()
 
+        async def _handle_response(response: PlaywrightResponse) -> None:
+            download["response_status"] = response.status
+            download_started.set()
+
         page_goto_kwargs = request.meta.get("playwright_page_goto_kwargs") or {}
         page_goto_kwargs.pop("url", None)
         page.on("download", _handle_download)
+        page.on("response", _handle_response)
         try:
             response = await page.goto(url=request.url, **page_goto_kwargs)
         except PlaywrightError as err:
@@ -473,9 +480,36 @@ async def _handle_download(dwnld: Download) -> None:
                 and "net::ERR_ABORTED" in err.message
             ):
                 raise
+
+            logger.debug(
+                "Navigating to %s failed",
+                request.url,
+                extra={
+                    "spider": spider,
+                    "context_name": request.meta.get("playwright_context"),
+                    "scrapy_request_url": request.url,
+                    "scrapy_request_method": request.method,
+                },
+            )
+            await download_started.wait()
+
+            if download.get("response_status") == 204:
+                raise err
+
+            logger.debug(
+                "Waiting on dowload to finish for %s",
+                request.url,
+                extra={
+                    "spider": spider,
+                    "context_name": request.meta.get("playwright_context"),
+                    "scrapy_request_url": request.url,
+                    "scrapy_request_method": request.method,
+                },
+            )
             await download_ready.wait()
         finally:
             page.remove_listener("download", _handle_download)
+            page.remove_listener("response", _handle_response)
 
         return response, download
 


Test patch:
diff --git a/tests/mockserver.py b/tests/mockserver.py
index 35ff02a..6487a80 100644
--- a/tests/mockserver.py
+++ b/tests/mockserver.py
@@ -59,6 +59,9 @@ def do_GET(self) -> None:
 
         if parsed_path.path == "/headers":
             self._send_json(dict(self.headers))
+        elif parsed_path.path == "/status/204":
+            self.send_response(204)
+            self.end_headers()
         elif parsed_path.path == "/redirect2":
             self.send_response(302)
             self.send_header("Content-Length", "0")
diff --git a/tests/tests_asyncio/test_playwright_requests.py b/tests/tests_asyncio/test_playwright_requests.py
index fc97294..32dd660 100644
--- a/tests/tests_asyncio/test_playwright_requests.py
+++ b/tests/tests_asyncio/test_playwright_requests.py
@@ -465,6 +465,23 @@ async def cancel_download(download):
                     f" exc_type={type(excinfo.value)} exc_msg={str(excinfo.value)}",
                 ) in self._caplog.record_tuples
 
+    @allow_windows
+    async def test_fail_status_204(self):
+        async with make_handler({"PLAYWRIGHT_BROWSER_TYPE": self.browser_type}) as handler:
+            with MockServer() as server:
+                request = Request(
+                    url=server.urljoin("/status/204"),
+                    meta={"playwright": True},
+                )
+                with pytest.raises(PlaywrightError) as excinfo:
+                    await handler._download_request(request, Spider("foo"))
+                assert (
+                    "scrapy-playwright",
+                    logging.WARNING,
+                    f"Closing page due to failed request: {request}"
+                    f" exc_type={type(excinfo.value)} exc_msg={str(excinfo.value)}",
+                ) in self._caplog.record_tuples
+
 
 class TestCaseChromium(IsolatedAsyncioTestCase, MixinTestCase):
     browser_type = "chromium"