Repository: mozilla/bleach

Issue:
Open bracket '<' still cleaned up without closing bracket
**Describe the bug**

Thanks for the fix provided https://github.com/mozilla/bleach/issues/705!
I think I found regression after that fix

 - Python Version: [e.g. 3.9.6]
 - Bleach Version: [e.g. 6.1.0]

**To Reproduce**

Steps to reproduce the behavior:

```python
# Working!
In [5]: bleach.clean("<test abc")
Out[5]: '&lt;test abc'
```

```python
# Doesn't work (because of duplicated words?)
In [6]: bleach.clean("<test abc abc")
Out[6]: ''
```

```python
# However this work
In [12]: bleach.clean("<test abc abd")
Out[12]: '&lt;test abc abd'
```

```python
# Doesn't work (with space in the end)
In [7]: bleach.clean("<test abc ")
Out[7]: ''
```

```python
# Doesn't work (with space in the end)
In [8]: bleach.clean("asd<test abc ")
Out[8]: 'asd'
```

```python
# However this work
In [9]: bleach.clean("asd<test abc asd")
Out[9]: 'asd&lt;test abc asd'
```

**Expected behavior**

```python
# Doesn't work (because of duplicated words?)
In [6]: bleach.clean("<test abc abc")
Out[6]: '&lt;test abc abc'
```

```python
# Doesn't work (with space in the end)
In [7]: bleach.clean("<test abc ")
Out[7]: '&lt;test abc '
```

```python
# Doesn't work (with space in the end)
In [8]: bleach.clean("asd<test abc ")
Out[8]: 'asd&lt;test abc '
```

**Additional context**

Add any other context about the problem here.

Hints from issue comments:
@willkg: Thank you for putting so much effort into this bug report--I really appreciate it!

I think there are a couple of issues here:

1. It looks like the duplicate token _does_ affect things. It kicks up two parse errors and then everything goes sideways:
    `{'type': 7, 'data': 'eof-in-attribute-name'}`
    `{'type': 7, 'data': 'duplicate-attribute'}`
2. It looks like we need to handle another parse error case:
    `{'type': 7, 'data': 'expected-end-of-tag-but-got-eof'}`

We'll need to fix each issue separately. I'll see what I can do.

Solution patch:
diff --git a/bleach/html5lib_shim.py b/bleach/html5lib_shim.py
index ca1cc8c8..f083db75 100644
--- a/bleach/html5lib_shim.py
+++ b/bleach/html5lib_shim.py
@@ -396,16 +396,25 @@ def __iter__(self):
                 # name that abruptly ends, but we should treat that like
                 # character data
                 yield {"type": TAG_TOKEN_TYPE_CHARACTERS, "data": self.stream.get_tag()}
+
             elif last_error_token["data"] in (
+                "duplicate-attribute",
                 "eof-in-attribute-name",
                 "eof-in-attribute-value-no-quotes",
+                "expected-end-of-tag-but-got-eof",
             ):
                 # Handle the case where the text being parsed ends with <
-                # followed by a series of characters and then space and then
-                # more characters. It's treated as a tag name followed by an
+                # followed by characters and then space and then:
+                #
+                # * more characters
+                # * more characters repeated with a space between (e.g. "abc abc")
+                # * more characters and then a space and then an EOF (e.g. "abc def ")
+                #
+                # These cases are treated as a tag name followed by an
                 # attribute that abruptly ends, but we should treat that like
-                # character data.
+                # character data instead.
                 yield {"type": TAG_TOKEN_TYPE_CHARACTERS, "data": self.stream.get_tag()}
+
             else:
                 yield last_error_token
 


Test patch:
diff --git a/tests/test_clean.py b/tests/test_clean.py
index dc129d0e..da4082a8 100644
--- a/tests/test_clean.py
+++ b/tests/test_clean.py
@@ -167,6 +167,10 @@ def test_bare_entities_get_escaped_correctly(text, expected):
         ("<some thing", "&lt;some thing"),
         # this is an eof-in-attribute-value-no-quotes parser error
         ("<some thing=foo", "&lt;some thing=foo"),
+        # this is a duplicate-attribute parser error
+        ("<some thing thing", "&lt;some thing thing"),
+        # this is an expected-end-of-tag-but-got-eof parser error
+        ("<some thing thing2 ", "&lt;some thing thing2 "),
     ],
 )
 def test_lessthan_escaping(text, expected):