Repository: scrapy/scrapy

Issue:
Open in Browser `<base>` replacement will fail if `<head>` has attributes
### Description

When using `open_in_browser()` feature, Scrapy will try to add a `<base>` tag to ensure remote resources are loaded, and to make external links to work in our local browser. This feature rely on the following code:

https://github.com/scrapy/scrapy/blob/06f3d12c1208c380f9f1a16cb36ba2dfa3c244c5/scrapy/utils/response.py#L81-L84

Some website are using attributes on the `<head>` tag, which will prevent the `<base>` tag to be injected, and therefore external resources to be loaded.

### How to reproduce the issue 

Simply create a basic spider [following Scrapy tutorial](https://docs.scrapy.org/en/latest/intro/tutorial.html) and use the following code:

```py
import scrapy
from scrapy.utils.response import open_in_browser

class ExampleSpider(scrapy.Spider):
    name = 'example'
    allowed_domains = ['example.com']
    start_urls = [
        'https://example.com/head-without-argument.html', 
        'https://example.com/head-with-argument.html']

    def parse(self, response):
        open_in_browser(response)
        pass
```

For the scrapped pages itselves, use the simplest code possible (I've not been able to quickly find a public page using arguments on `<head>`, sorry):

```html
<!DOCTYPE html>
<html>
  <!-- head-without-argument.html -->
  <head>
    <title>Title</title>
  </head>
  <body>
    <p>Foo</p>
    <img src="./assets/image.jpg">
  </body>
</html>
```

<!-- -->

```html
<!DOCTYPE html>
<html>
  <!-- head-with-argument.html -->
  <head id="example">
    <title>Title</title>
  </head>
  <body>
    <p>Foo</p>
    <img src="./assets/image.jpg">
  </body>
</html>
```

Then run the spider with `scrapy crawl example` and you'll see that:
1. `head-without-argument.html` output renders resource correctly
2. `head-with-argument.html` output doesn't render resource

### How to fix the issue

At the very least, the literal `replace()` function should be replace by a regex replacement:
```py
 if isinstance(response, HtmlResponse): 
     if b'<base' not in body: 
         repl = f'\\1<base href="{response.url}">' 
         body = re.sub(b"(<head.*?>)", to_bytes(repl), body)
```

### Environment 
```
Scrapy       : 2.5.1
lxml         : 4.6.3.0
libxml2      : 2.9.4
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 21.7.0
Python       : 3.9.7 (default, Sep  3 2021, 04:31:11) - [Clang 12.0.5 (clang-1205.0.22.9)]
pyOpenSSL    : 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021)
cryptography : 35.0.0
Platform     : macOS-11.6-arm64-arm-64bit
```

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/5320?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#5320](https://codecov.io/gh/scrapy/scrapy/pull/5320?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (9d0b5d9) into [master](https://codecov.io/gh/scrapy/scrapy/commit/28eba610e22c0d2a42e830b4e64746edf44598f9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (28eba61) will **decrease** coverage by `5.79%`.
> The diff coverage is `100.00%`.


```diff
@@            Coverage Diff             @@
##           master    #5320      +/-   ##
==========================================
- Coverage   88.59%   82.79%   -5.80%     
==========================================
  Files         163      163              
  Lines       10616    10610       -6     
  Branches     1558     1528      -30     
==========================================
- Hits         9405     8785     -620     
- Misses        936     1564     +628     
+ Partials      275      261      -14     
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/5320?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/utils/response.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL3Jlc3BvbnNlLnB5) | `89.58% <100.00%> (+0.22%)` | :arrow_up: |
| [scrapy/core/http2/stream.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvc3RyZWFtLnB5) | `0.00% <0.00%> (-91.38%)` | :arrow_down: |
| [scrapy/core/downloader/handlers/http2.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvZG93bmxvYWRlci9oYW5kbGVycy9odHRwMi5weQ==) | `14.47% <0.00%> (-85.53%)` | :arrow_down: |
| [scrapy/core/http2/agent.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvYWdlbnQucHk=) | `13.25% <0.00%> (-83.14%)` | :arrow_down: |
| [scrapy/core/http2/protocol.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvcmUvaHR0cDIvcHJvdG9jb2wucHk=) | `3.51% <0.00%> (-79.90%)` | :arrow_down: |
| [scrapy/utils/asyncgen.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL2FzeW5jZ2VuLnB5) | `33.33% <0.00%> (-66.67%)` | :arrow_down: |
| [scrapy/pipelines/images.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3BpcGVsaW5lcy9pbWFnZXMucHk=) | `27.82% <0.00%> (-62.61%)` | :arrow_down: |
| [scrapy/robotstxt.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3JvYm90c3R4dC5weQ==) | `75.30% <0.00%> (-22.23%)` | :arrow_down: |
| [scrapy/utils/test.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL3Rlc3QucHk=) | `43.75% <0.00%> (-17.19%)` | :arrow_down: |
| [scrapy/utils/ssl.py](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL3NzbC5weQ==) | `53.65% <0.00%> (-17.08%)` | :arrow_down: |
| ... and [18 more](https://codecov.io/gh/scrapy/scrapy/pull/5320/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | |

zessx: @Gallaecio I've added your suggestions (fix and format) to the original commit, and a additional commit for test coverage.

Patch:
diff --git a/scrapy/utils/response.py b/scrapy/utils/response.py
index b3ef7b463..8b109dced 100644
--- a/scrapy/utils/response.py
+++ b/scrapy/utils/response.py
@@ -3,8 +3,9 @@ This module provides some useful functions for working with
 scrapy.http.Response objects
 """
 import os
-import webbrowser
+import re
 import tempfile
+import webbrowser
 from typing import Any, Callable, Iterable, Optional, Tuple, Union
 from weakref import WeakKeyDictionary
 
@@ -80,8 +81,9 @@ def open_in_browser(
     body = response.body
     if isinstance(response, HtmlResponse):
         if b'<base' not in body:
-            repl = f'<head><base href="{response.url}">'
-            body = body.replace(b'<head>', to_bytes(repl))
+            repl = fr'\1<base href="{response.url}">'
+            body = re.sub(b"<!--.*?-->", b"", body, flags=re.DOTALL)
+            body = re.sub(rb"(<head(?:>|\s.*?>))", to_bytes(repl), body)
         ext = '.html'
     elif isinstance(response, TextResponse):
         ext = '.txt'


Test patch:
diff --git a/tests/test_utils_response.py b/tests/test_utils_response.py
index d6f4c0bb5..0a09f6109 100644
--- a/tests/test_utils_response.py
+++ b/tests/test_utils_response.py
@@ -83,3 +83,56 @@ class ResponseUtilsTest(unittest.TestCase):
         self.assertEqual(response_status_message(200), '200 OK')
         self.assertEqual(response_status_message(404), '404 Not Found')
         self.assertEqual(response_status_message(573), "573 Unknown Status")
+
+    def test_inject_base_url(self):
+        url = "http://www.example.com"
+
+        def check_base_url(burl):
+            path = urlparse(burl).path
+            if not os.path.exists(path):
+                path = burl.replace('file://', '')
+            with open(path, "rb") as f:
+                bbody = f.read()
+            self.assertEqual(bbody.count(b'<base href="' + to_bytes(url) + b'">'), 1)
+            return True
+
+        r1 = HtmlResponse(url, body=b"""
+        <html>
+            <head><title>Dummy</title></head>
+            <body><p>Hello world.</p></body>
+        </html>""")
+        r2 = HtmlResponse(url, body=b"""
+        <html>
+            <head id="foo"><title>Dummy</title></head>
+            <body>Hello world.</body>
+        </html>""")
+        r3 = HtmlResponse(url, body=b"""
+        <html>
+            <head><title>Dummy</title></head>
+            <body>
+                <header>Hello header</header>
+                <p>Hello world.</p>
+            </body>
+        </html>""")
+        r4 = HtmlResponse(url, body=b"""
+        <html>
+            <!-- <head>Dummy comment</head> -->
+            <head><title>Dummy</title></head>
+            <body><p>Hello world.</p></body>
+        </html>""")
+        r5 = HtmlResponse(url, body=b"""
+        <html>
+            <!--[if IE]>
+            <head><title>IE head</title></head>
+            <![endif]-->
+            <!--[if !IE]>-->
+            <head><title>Standard head</title></head>
+            <!--<![endif]-->
+            <body><p>Hello world.</p></body>
+        </html>""")
+
+        assert open_in_browser(r1, _openfunc=check_base_url), "Inject base url"
+        assert open_in_browser(r2, _openfunc=check_base_url), "Inject base url with argumented head"
+        assert open_in_browser(r3, _openfunc=check_base_url), "Inject unique base url with misleading tag"
+        assert open_in_browser(r4, _openfunc=check_base_url), "Inject unique base url with misleading comment"
+        assert open_in_browser(r5, _openfunc=check_base_url), "Inject unique base url with conditional comment"