Repository: webrecorder/pywb (language: python)

Issue:
No rewritig of <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.3.min.js"></script>
## Describe the bug
A web page in our collection (Seed: http://www.shvr.ch/ ; dating from 2014) contains this snippet:
<script type="text/javascript" src="http://code.jquery.com/jquery-1.6.3.min.js"></script>

Unfortunately the URL is not getting rewritten, which is resulting in a blocked live leak (Refused to load the script 'http://code.jquery.com/jquery-1.6.3.min.js' because it violates the following Content Security Policy directive: "default-src 'unsafe-eval' 'unsafe-inline' 'self' data: blob: mediastream: ws: wss:". Note that 'script-src-elem' was not explicitly set, so 'default-src' is used as a fallback.)

## Expected behavior
The URL should be rewritten to something like (in my setup):
http://localhost:8080/orig/20140725113002/http://code.jquery.com/jquery-1.6.3.min.js

## Environment
- OS: Windows 10
- Browser :chrome
- Version: 100.0.4896.60

## Additional context
OpenWayback perfectly rewrites the URL.

No rewritig of <script type="text/javascript" src="http://code.jquery.com/jquery-1.6.3.min.js"></script>
## Describe the bug
A web page in our collection (Seed: http://www.shvr.ch/ ; dating from 2014) contains this snippet:
<script type="text/javascript" src="http://code.jquery.com/jquery-1.6.3.min.js"></script>

Unfortunately the URL is not getting rewritten, which is resulting in a blocked live leak (Refused to load the script 'http://code.jquery.com/jquery-1.6.3.min.js' because it violates the following Content Security Policy directive: "default-src 'unsafe-eval' 'unsafe-inline' 'self' data: blob: mediastream: ws: wss:". Note that 'script-src-elem' was not explicitly set, so 'default-src' is used as a fallback.)

## Expected behavior
The URL should be rewritten to something like (in my setup):
http://localhost:8080/orig/20140725113002/http://code.jquery.com/jquery-1.6.3.min.js

## Environment
- OS: Windows 10
- Browser :chrome
- Version: 100.0.4896.60

## Additional context
OpenWayback perfectly rewrites the URL.



Discussion / hints:
Are you able to share access to the replay or the WARC files that this is loaded from? Otherwise, not much we can look at, as this doesn't happen in current version of the site. Generally this type of rewriting works very well, so curious about this particular edge case..
Have you tried it with the latest version of pywb? I haven't been able to repro the issue, it could have been fixed in an older version of our tools. The Conifer service is being run by Rhizome. The rewriting is working correctly on the latest version, and the issue is likely due to missing `<html>` tag at the beginning:
```
<!DOCTYPE html>
<head>. 
...
</head>
</html>
```
Hmm, I used version 2.6.7.
this is the last release, no?

Or is my python too old?
I'm using python 3.6.1
I could replay it successfully in https://replayweb.page/ and with the ArchiveWeb.page extension (after having it converted from arc.gz to warc.gz).
But no success in any way in pywb. Not even in pywb 2.6.8
Could you upload the original file somewhere and send a link? Can send via e-mail if you'd like
> Could you upload the original file somewhere and send a link? Can send via e-mail if you'd like

OK, I sent you a link by e-mail.
Are you able to share access to the replay or the WARC files that this is loaded from? Otherwise, not much we can look at, as this doesn't happen in current version of the site. Generally this type of rewriting works very well, so curious about this particular edge case..
Have you tried it with the latest version of pywb? I haven't been able to repro the issue, it could have been fixed in an older version of our tools. The Conifer service is being run by Rhizome. The rewriting is working correctly on the latest version, and the issue is likely due to missing `<html>` tag at the beginning:
```
<!DOCTYPE html>
<head>. 
...
</head>
</html>
```
Hmm, I used version 2.6.7.
this is the last release, no?

Or is my python too old?
I'm using python 3.6.1
I could replay it successfully in https://replayweb.page/ and with the ArchiveWeb.page extension (after having it converted from arc.gz to warc.gz).
But no success in any way in pywb. Not even in pywb 2.6.8
Could you upload the original file somewhere and send a link? Can send via e-mail if you'd like
> Could you upload the original file somewhere and send a link? Can send via e-mail if you'd like

OK, I sent you a link by e-mail.

Fix patch:
diff --git a/pywb/rewrite/content_rewriter.py b/pywb/rewrite/content_rewriter.py
index ce7d7c749..ca0275bfc 100644
--- a/pywb/rewrite/content_rewriter.py
+++ b/pywb/rewrite/content_rewriter.py
@@ -391,7 +391,7 @@ def rewrite_text_stream_to_gen(self, stream, rwinfo):
 # ============================================================================
 class RewriteInfo(object):
     TAG_REGEX = re.compile(b'^(\xef\xbb\xbf)?\s*\<')
-    TAG_REGEX2 = re.compile(b'^.*<\w+[\s>]')
+    TAG_REGEX2 = re.compile(b'^.*<[!]?\w+[\s>]')
     JSON_REGEX = re.compile(b'^\s*[{[][{"]')  # if it starts with this then highly likely not HTML
 
     JSONP_CONTAINS = ['callback=jQuery',


Test patch:
diff --git a/pywb/rewrite/test/test_content_rewriter.py b/pywb/rewrite/test/test_content_rewriter.py
index 7f4298c10..4555291ae 100644
--- a/pywb/rewrite/test/test_content_rewriter.py
+++ b/pywb/rewrite/test/test_content_rewriter.py
@@ -141,6 +141,17 @@ def test_rewrite_html_utf_8(self):
         assert ('Content-Type', 'text/html; charset=utf-8') in headers.headers
         assert b''.join(gen).decode('utf-8') == exp
 
+    def test_rewrite_html_ignore_bom(self):
+        headers = {'Content-Type': 'text/html'}
+        content = u'\ufeff\ufeff\ufeff<!DOCTYPE html>\n<head>\n<a href="http://example.com"></a></body></html>'
+
+        headers, gen, is_rw = self.rewrite_record(headers, content, ts='201701mp_')
+
+        exp = '\ufeff\ufeff\ufeff<!DOCTYPE html>\n<head>\n<a href="http://localhost:8080/prefix/201701/http://example.com"></a></body></html>'
+        assert is_rw
+        assert ('Content-Type', 'text/html') in headers.headers
+        assert b''.join(gen).decode('utf-8') == exp
+
     def test_rewrite_html_utf_8_anchor(self):
         headers = {'Content-Type': 'text/html; charset=utf-8'}
         content = u'<html><body><a href="#éxample-tésté"></a></body></html>'