Repository: mozilla/bleach

Issue:
LinkifyFilter with clean breaks links at &amp;
(Split out to a new issue from [this comment](https://github.com/mozilla/bleach/issues/419#issuecomment-445056894))

When using `LinkifyFilter` instead of the stand-alone `linkify`, links are broken at any `&amp;`, but not bare `&` characters.

Here's a pretty minimal demonstration:

```
import bleach

bleach.linkify("https://www.youtube.com/watch?v=SkcucKDrbOI&amp;feature=youtu.be")
```

Result (expected): `<a href="https://www.youtube.com/watch?v=SkcucKDrbOI&amp;feature=youtu.be" rel="nofollow">https://www.youtube.com/watch?v=SkcucKDrbOI&amp;feature=youtu.be</a>`

```
import bleach
from functools import partial

bleach_linkifier = partial(bleach.linkifier.LinkifyFilter)
cleaner = bleach.Cleaner(filters=[bleach_linkifier])
cleaner.clean("https://www.youtube.com/watch?v=SkcucKDrbOI&amp;feature=youtu.be")
```

Result (not expected, link stops at the `&amp;`): `<a href="https://www.youtube.com/watch?v=SkcucKDrbOI">https://www.youtube.com/watch?v=SkcucKDrbOI</a>&amp;feature=<a href="http://youtu.be">youtu.be</a>`

It works with just a bare `&` instead of the `&amp;`, like `https://www.youtube.com/watch?v=SkcucKDrbOI&feature=youtu.be`. In that case, the output of both methods is:

```
<a href="https://www.youtube.com/watch?v=SkcucKDrbOI&amp;feature=youtu.be">https://www.youtube.com/watch?v=SkcucKDrbOI&amp;feature=youtu.be</a>
```

(with an additional `rel="nofollow"` attr added by `linkify()` by default)

Discussion:

Patch:
diff --git a/bleach/sanitizer.py b/bleach/sanitizer.py
index c60c26b..79b80f5 100644
--- a/bleach/sanitizer.py
+++ b/bleach/sanitizer.py
@@ -395,7 +395,18 @@ class BleachSanitizerFilter(html5lib_shim.SanitizerFilter):
             if part.startswith('&'):
                 entity = html5lib_shim.match_entity(part)
                 if entity is not None:
-                    new_tokens.append({'type': 'Entity', 'name': entity})
+                    if entity == 'amp':
+                        # LinkifyFilter can't match urls across token boundaries
+                        # which is problematic with &amp; since that shows up in
+                        # querystrings all the time. This special-cases &amp;
+                        # and converts it to a & and sticks it in as a
+                        # Characters token. It'll get merged with surrounding
+                        # tokens in the BleachSanitizerfilter.__iter__ and
+                        # escaped in the serializer.
+                        new_tokens.append({'type': 'Characters', 'data': '&'})
+                    else:
+                        new_tokens.append({'type': 'Entity', 'name': entity})
+
                     # Length of the entity plus 2--one for & at the beginning
                     # and and one for ; at the end
                     remainder = part[len(entity) + 2:]


Test patch:
diff --git a/tests/test_linkify.py b/tests/test_linkify.py
index 584a5b0..ab1c513 100644
--- a/tests/test_linkify.py
+++ b/tests/test_linkify.py
@@ -694,6 +694,10 @@ class TestLinkify:
         'http://example.com?b=1&c=2',
         '<a href="http://example.com?b=1&amp;c=2">http://example.com?b=1&amp;c=2</a>'
     ),
+    (
+        'http://example.com?b=1&amp;c=2',
+        '<a href="http://example.com?b=1&amp;c=2">http://example.com?b=1&amp;c=2</a>'
+    ),
     (
         'link: https://example.com/watch#anchor',
         'link: <a href="https://example.com/watch#anchor">https://example.com/watch#anchor</a>'