Repository: scrapy/scrapy

Issue:
Open in Browser `<base>` replacement will fail if `<head>` has attributes
### Description

When using `open_in_browser()` feature, Scrapy will try to add a `<base>` tag to ensure remote resources are loaded, and to make external links to work in our local browser. This feature rely on the following code:

https://github.com/scrapy/scrapy/blob/06f3d12c1208c380f9f1a16cb36ba2dfa3c244c5/scrapy/utils/response.py#L81-L84

Some website are using attributes on the `<head>` tag, which will prevent the `<base>` tag to be injected, and therefore external resources to be loaded.

### How to reproduce the issue 

Simply create a basic spider [following Scrapy tutorial](https://docs.scrapy.org/en/latest/intro/tutorial.html) and use the following code:

```py
import scrapy
from scrapy.utils.response import open_in_browser

class ExampleSpider(scrapy.Spider):
    name = 'example'
    allowed_domains = ['example.com']
    start_urls = [
        'https://example.com/head-without-argument.html', 
        'https://example.com/head-with-argument.html']

    def parse(self, response):
        open_in_browser(response)
        pass
```

For the scrapped pages itselves, use the simplest code possible (I've not been able to quickly find a public page using arguments on `<head>`, sorry):

```html
<!DOCTYPE html>
<html>
  <!-- head-without-argument.html -->
  <head>
    <title>Title</title>
  </head>
  <body>
    <p>Foo</p>
    <img src="./assets/image.jpg">
  </body>
</html>
```

<!-- -->

```html
<!DOCTYPE html>
<html>
  <!-- head-with-argument.html -->
  <head id="example">
    <title>Title</title>
  </head>
  <body>
    <p>Foo</p>
    <img src="./assets/image.jpg">
  </body>
</html>
```

Then run the spider with `scrapy crawl example` and you'll see that:
1. `head-without-argument.html` output renders resource correctly
2. `head-with-argument.html` output doesn't render resource

### How to fix the issue

At the very least, the literal `replace()` function should be replace by a regex replacement:
```py
 if isinstance(response, HtmlResponse): 
     if b'<base' not in body: 
         repl = f'\\1<base href="{response.url}">' 
         body = re.sub(b"(<head.*?>)", to_bytes(repl), body)
```

### Environment 
```
Scrapy       : 2.5.1
lxml         : 4.6.3.0
libxml2      : 2.9.4
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 21.7.0
Python       : 3.9.7 (default, Sep  3 2021, 04:31:11) - [Clang 12.0.5 (clang-1205.0.22.9)]
pyOpenSSL    : 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021)
cryptography : 35.0.0
Platform     : macOS-11.6-arm64-arm-64bit
```


Discussion:

Resolving patch:
diff --git a/scrapy/utils/response.py b/scrapy/utils/response.py
--- a/scrapy/utils/response.py
+++ b/scrapy/utils/response.py
@@ -3,8 +3,9 @@
 scrapy.http.Response objects
 """
 import os
-import webbrowser
+import re
 import tempfile
+import webbrowser
 from typing import Any, Callable, Iterable, Optional, Tuple, Union
 from weakref import WeakKeyDictionary
 
@@ -80,8 +81,9 @@ def open_in_browser(
     body = response.body
     if isinstance(response, HtmlResponse):
         if b'<base' not in body:
-            repl = f'<head><base href="{response.url}">'
-            body = body.replace(b'<head>', to_bytes(repl))
+            repl = fr'\1<base href="{response.url}">'
+            body = re.sub(b"<!--.*?-->", b"", body, flags=re.DOTALL)
+            body = re.sub(rb"(<head(?:>|\s.*?>))", to_bytes(repl), body)
         ext = '.html'
     elif isinstance(response, TextResponse):
         ext = '.txt'


Test patch:
diff --git a/tests/test_utils_response.py b/tests/test_utils_response.py
--- a/tests/test_utils_response.py
+++ b/tests/test_utils_response.py
@@ -83,3 +83,56 @@ def test_response_status_message(self):
         self.assertEqual(response_status_message(200), '200 OK')
         self.assertEqual(response_status_message(404), '404 Not Found')
         self.assertEqual(response_status_message(573), "573 Unknown Status")
+
+    def test_inject_base_url(self):
+        url = "http://www.example.com"
+
+        def check_base_url(burl):
+            path = urlparse(burl).path
+            if not os.path.exists(path):
+                path = burl.replace('file://', '')
+            with open(path, "rb") as f:
+                bbody = f.read()
+            self.assertEqual(bbody.count(b'<base href="' + to_bytes(url) + b'">'), 1)
+            return True
+
+        r1 = HtmlResponse(url, body=b"""
+        <html>
+            <head><title>Dummy</title></head>
+            <body><p>Hello world.</p></body>
+        </html>""")
+        r2 = HtmlResponse(url, body=b"""
+        <html>
+            <head id="foo"><title>Dummy</title></head>
+            <body>Hello world.</body>
+        </html>""")
+        r3 = HtmlResponse(url, body=b"""
+        <html>
+            <head><title>Dummy</title></head>
+            <body>
+                <header>Hello header</header>
+                <p>Hello world.</p>
+            </body>
+        </html>""")
+        r4 = HtmlResponse(url, body=b"""
+        <html>
+            <!-- <head>Dummy comment</head> -->
+            <head><title>Dummy</title></head>
+            <body><p>Hello world.</p></body>
+        </html>""")
+        r5 = HtmlResponse(url, body=b"""
+        <html>
+            <!--[if IE]>
+            <head><title>IE head</title></head>
+            <![endif]-->
+            <!--[if !IE]>-->
+            <head><title>Standard head</title></head>
+            <!--<![endif]-->
+            <body><p>Hello world.</p></body>
+        </html>""")
+
+        assert open_in_browser(r1, _openfunc=check_base_url), "Inject base url"
+        assert open_in_browser(r2, _openfunc=check_base_url), "Inject base url with argumented head"
+        assert open_in_browser(r3, _openfunc=check_base_url), "Inject unique base url with misleading tag"
+        assert open_in_browser(r4, _openfunc=check_base_url), "Inject unique base url with misleading comment"
+        assert open_in_browser(r5, _openfunc=check_base_url), "Inject unique base url with conditional comment"