Repository: mozilla/bleach

Issue:
bug: linkify with entities inside anchor strings are incorrectly escaped
**Describe the bug**

linkify on a string with entities inside anchor element text results in the `&` character of the entity being incorrect escaped to `&amp;`
e.g. `&nbsp;` -> `&amp;nbsp;`

** python and bleach versions (please complete the following information):**

 - Python Version: 3.9.5
 - Bleach Version: 6.0.0

**To Reproduce**

A simple test to reproduce the behavior:
```python
>>> from bleach import linkify
text = r'<p><a href="/">Some&nbsp;entity&rsquo;s</a>More&nbsp;entity&rsquo;s</p>'
expected = r'<p><a href="/" rel="nofollow">Some&nbsp;entity&rsquo;s</a>More&nbsp;entity&rsquo;s</p>'
assert linkify(text) == expected 
```

**Expected behavior**
```python
linkify(r'<a href="/">Some&nbsp;entity&rsquo;s</a>')
'<a href="/" rel="nofollow">Some&nbsp;entity&rsquo;s</a>'
```

**Actual behavior**
```python
linkify(r'<a href="/">Some&nbsp;entity&rsquo;s</a>')
'<a href="/" rel="nofollow">Some&amp;nbsp;entity&amp;rsquo;s</a>'
```

**Additional context**

This bug was introduced in 6.0.0 with the fix for #501 and #692: https://github.com/mozilla/bleach/pull/692


Discussion:

Patch:
diff --git a/bleach/linkifier.py b/bleach/linkifier.py
index 679d7ea..54a432e 100644
--- a/bleach/linkifier.py
+++ b/bleach/linkifier.py
@@ -591,7 +591,7 @@ class LinkifyFilter(html5lib_shim.Filter):
                     in_a = False
                     token_buffer = []
                 else:
-                    token_buffer.append(token)
+                    token_buffer.extend(list(self.extract_entities(token)))
                 continue
 
             if token["type"] in ["StartTag", "EmptyTag"]:


Test patch:
diff --git a/tests/test_linkify.py b/tests/test_linkify.py
index bd83813..4cc01a6 100644
--- a/tests/test_linkify.py
+++ b/tests/test_linkify.py
@@ -323,6 +323,7 @@ def test_link_fragment():
 
 
 def test_link_entities_in_qs():
+    """Entities in the querystring get escaped"""
     assert (
         linkify("http://xx.com/?a=1&b=2")
         == '<a href="http://xx.com/?a=1&amp;b=2" rel="nofollow">http://xx.com/?a=1&amp;b=2</a>'
@@ -330,9 +331,18 @@ def test_link_entities_in_qs():
 
 
 def test_link_entities_in_characters_token():
+    """Entitites in a Characters token don't get escaped"""
     assert linkify("foo &nbsp; bar") == "foo &nbsp; bar"
 
 
+def test_link_entities_in_a_tag():
+    """Entitites between an a start tag and an a end tag don't get escaped"""
+    assert (
+        linkify('<a href="/">Some&nbsp;entity&rsquo;s</a>', callbacks=[])
+        == '<a href="/">Some&nbsp;entity&rsquo;s</a>'
+    )
+
+
 def test_escaped_html():
     """If I pass in escaped HTML, it should probably come out escaped."""
     s = "&lt;em&gt;strong&lt;/em&gt;"