nebius/SWE-bench-extra huggingface dataset:
Repository: mopidy/mopidy-local

Issue:
Not extracting images from m4a files when scanning
Whenever I execute `mopidy local scan --force` in my terminal, I get an error extracting the image from any m4a file.

Example: `WARNING  2021-02-20 22:42:52,743 [111960:MainThread] mopidy_local.storage
  Error extracting images for 'local:track:MDMA/Genie%20%28Deluxe%29/01%20Xccelerate.m4a': ValueError('Unknown image type')`

I can play these files completely fine with ncmpcpp's file browser and they are tagged properly too but I can't scan them into my library.

Comments:

Patch:
diff --git a/mopidy_local/storage.py b/mopidy_local/storage.py
index a8679c3..9f9947c 100644
--- a/mopidy_local/storage.py
+++ b/mopidy_local/storage.py
@@ -58,6 +58,15 @@ def get_image_size_jpeg(data):
     return width, height
 
 
+def test_jpeg(data, file_handle):
+    # Additional JPEG detection looking for JPEG SOI marker
+    if data[:2] == b"\xff\xd8":
+        return "jpeg"
+
+
+imghdr.tests.append(test_jpeg)
+
+
 class LocalStorageProvider:
     def __init__(self, config):
         self._config = ext_config = config[Extension.ext_name]


Test patch:
diff --git a/tests/test_storage.py b/tests/test_storage.py
new file mode 100644
index 0000000..6caed2c
--- /dev/null
+++ b/tests/test_storage.py
@@ -0,0 +1,16 @@
+import pytest
+
+from mopidy_local import storage
+
+
+@pytest.mark.parametrize(
+    "data",
+    [
+        pytest.param("ffd8ffe000104a46494600", id="JFIF"),
+        pytest.param("ffd8ffe100184578696600", id="Exif"),
+        pytest.param("ffd8ffe1095068747470", id="XMP"),
+    ],
+)
+def test_jpeg_detection(data):
+    data_bytes = bytes.fromhex(data)
+    assert storage.imghdr.what(None, data_bytes) is not None
Repository: mozilla/bleach

Issue:
Style filter does not work correctly when supplied non-ASCII style attribute
# Environment

Windows 10 Pro
Python 3.7.1
bleach         3.0.2

I've setup so that `color` style will survive after clean() method like below:
```
>>> bleach.clean('<span style="font-family:consolas;color:red">text</span>', tags = ['span'], attributes = {'span': ['style']}, styles = ['color'])
'<span style="color: red;">text</span>'
```

# Problem

When it contains non-ASCII style description such as Japanese font names, the `color` style is omitted.
```
>>> bleach.clean('<span style="font-family:メイリオ;color:red">text</span>', tags = ['span'], attributes = {'span': ['style']}, styles = ['color'])
'<span style="">text</span>'
```
It should retain the `color` style like below:
```
'<span style="color: red;">text</span>'
```

Comments:

Patch:
diff --git a/bleach/sanitizer.py b/bleach/sanitizer.py
index 79b80f5..6ccd78c 100644
--- a/bleach/sanitizer.py
+++ b/bleach/sanitizer.py
@@ -593,7 +593,8 @@ class BleachSanitizerFilter(html5lib_shim.SanitizerFilter):
         # the whole thing.
         parts = style.split(';')
         gauntlet = re.compile(
-            r"""^([-/:,#%.'"\sa-zA-Z0-9!]|\w-\w|'[\s\w]+'\s*|"[\s\w]+"|\([\d,%\.\s]+\))*$"""
+            r"""^([-/:,#%.'"\s!\w]|\w-\w|'[\s\w]+'\s*|"[\s\w]+"|\([\d,%\.\s]+\))*$""",
+            flags=re.U
         )
 
         for part in parts:


Test patch:
diff --git a/tests/test_css.py b/tests/test_css.py
index 12f27f3..7e34027 100644
--- a/tests/test_css.py
+++ b/tests/test_css.py
@@ -1,6 +1,8 @@
+# -*- coding: utf-8 -*-
 from functools import partial
 
 import pytest
+import six
 
 from bleach import clean
 
@@ -14,6 +16,11 @@ clean = partial(clean, tags=['p'], attributes=['style'])
         ['color'],
         'color: red;'
     ),
+    (
+        u'font-family: メイリオ; color: red; float: left; background-color: red;',
+        [u'color'],
+        u'color: red;'
+    ),
     (
         'border: 1px solid blue; color: red; float: left;',
         ['color'],
@@ -70,11 +77,19 @@ def test_allowed_css(data, styles, expected):
     p_double = "<p style='{0!s}'>bar</p>"
 
     if '"' in data:
+        if is_python2_unicode(data):
+            p_double = unicode(p_double)
         assert clean(p_double.format(data), styles=styles) == p_double.format(expected)
     else:
+        if is_python2_unicode(data):
+            p_single = unicode(p_single)
         assert clean(p_single.format(data), styles=styles) == p_single.format(expected)
 
 
+def is_python2_unicode(data):
+    return six.PY2 and isinstance(data, unicode)
+
+
 def test_valid_css():
     """The sanitizer should fix missing CSS values."""
     styles = ['color', 'float']
Repository: mozilla/bleach

Issue:
Left angle bracket '<'  with any characters (tag or not) and no closing right angle bracket prevents remaining text from being returned
Given some text that contains a left angle bracket and no closing right angle bracket, the `bleach.clean `method returns nothing after the occurrence of the left angle bracket.

Consider the following example strings
`example_str_1 = 'random prefix text <anything any amount of suffix text'`
`example_str_2 = '<e any amount of text here is gone'`
`example_str_3 = '<it works when there is a closing>'`

Expected output
example_str_1  `random prefix text &lt;anything any amount of suffix text`
example_str_2  `&lt;e any amount of text here is gone`
example_str_3  `&lt;it works when there is a closing right bracket&gt;`

Actual output
example_str_1 `random prefix text `                             
example_str_2  _empty string_                                  
example_str_3  `&lt;it works when there is a closing right bracket&gt;` 

Python version 3.6.8
bleach 3.1.5

Comments:

Patch:
diff --git a/bleach/html5lib_shim.py b/bleach/html5lib_shim.py
index 6fc9048..d121953 100644
--- a/bleach/html5lib_shim.py
+++ b/bleach/html5lib_shim.py
@@ -385,7 +385,17 @@ class BleachHTMLTokenizer(HTMLTokenizer):
             yield token
 
         if last_error_token:
-            yield last_error_token
+            if last_error_token["data"] == "eof-in-tag-name":
+                # Handle the case where the text being parsed ends with <
+                # followed by a series of characters. It's treated as a tag
+                # name that abruptly ends, but we should treat that like
+                # character data
+                yield {
+                    "type": TAG_TOKEN_TYPE_CHARACTERS,
+                    "data": "<" + self.currentToken["name"],
+                }
+            else:
+                yield last_error_token
 
     def consumeEntity(self, allowedChar=None, fromAttribute=False):
         # If this tokenizer is set to consume entities, then we can let the


Test patch:
diff --git a/tests/test_clean.py b/tests/test_clean.py
index b9c262a..ab11253 100644
--- a/tests/test_clean.py
+++ b/tests/test_clean.py
@@ -156,6 +156,22 @@ def test_bare_entities_get_escaped_correctly(text, expected):
     assert clean(text) == expected
 
 
+@pytest.mark.parametrize(
+    "text, expected",
+    [
+        ("x<y", "x&lt;y"),
+        ("<y", "&lt;y"),
+        ("x < y", "x &lt; y"),
+        ("<y>", "&lt;y&gt;"),
+    ],
+)
+def test_lessthan_escaping(text, expected):
+    # Tests whether < gets escaped correctly in a series of edge cases where
+    # the html5lib tokenizer hits an error because it's not the beginning of a
+    # tag.
+    assert clean(text) == expected
+
+
 @pytest.mark.parametrize(
     "text, expected",
     [
Repository: mozilla/bleach

Issue:
bug: linkify with entities inside anchor strings are incorrectly escaped
**Describe the bug**

linkify on a string with entities inside anchor element text results in the `&` character of the entity being incorrect escaped to `&amp;`
e.g. `&nbsp;` -> `&amp;nbsp;`

** python and bleach versions (please complete the following information):**

 - Python Version: 3.9.5
 - Bleach Version: 6.0.0

**To Reproduce**

A simple test to reproduce the behavior:
```python
>>> from bleach import linkify
text = r'<p><a href="/">Some&nbsp;entity&rsquo;s</a>More&nbsp;entity&rsquo;s</p>'
expected = r'<p><a href="/" rel="nofollow">Some&nbsp;entity&rsquo;s</a>More&nbsp;entity&rsquo;s</p>'
assert linkify(text) == expected 
```

**Expected behavior**
```python
linkify(r'<a href="/">Some&nbsp;entity&rsquo;s</a>')
'<a href="/" rel="nofollow">Some&nbsp;entity&rsquo;s</a>'
```

**Actual behavior**
```python
linkify(r'<a href="/">Some&nbsp;entity&rsquo;s</a>')
'<a href="/" rel="nofollow">Some&amp;nbsp;entity&amp;rsquo;s</a>'
```

**Additional context**

This bug was introduced in 6.0.0 with the fix for #501 and #692: https://github.com/mozilla/bleach/pull/692


Comments:

Patch:
diff --git a/bleach/linkifier.py b/bleach/linkifier.py
index 679d7ea..54a432e 100644
--- a/bleach/linkifier.py
+++ b/bleach/linkifier.py
@@ -591,7 +591,7 @@ class LinkifyFilter(html5lib_shim.Filter):
                     in_a = False
                     token_buffer = []
                 else:
-                    token_buffer.append(token)
+                    token_buffer.extend(list(self.extract_entities(token)))
                 continue
 
             if token["type"] in ["StartTag", "EmptyTag"]:


Test patch:
diff --git a/tests/test_linkify.py b/tests/test_linkify.py
index bd83813..4cc01a6 100644
--- a/tests/test_linkify.py
+++ b/tests/test_linkify.py
@@ -323,6 +323,7 @@ def test_link_fragment():
 
 
 def test_link_entities_in_qs():
+    """Entities in the querystring get escaped"""
     assert (
         linkify("http://xx.com/?a=1&b=2")
         == '<a href="http://xx.com/?a=1&amp;b=2" rel="nofollow">http://xx.com/?a=1&amp;b=2</a>'
@@ -330,9 +331,18 @@ def test_link_entities_in_qs():
 
 
 def test_link_entities_in_characters_token():
+    """Entitites in a Characters token don't get escaped"""
     assert linkify("foo &nbsp; bar") == "foo &nbsp; bar"
 
 
+def test_link_entities_in_a_tag():
+    """Entitites between an a start tag and an a end tag don't get escaped"""
+    assert (
+        linkify('<a href="/">Some&nbsp;entity&rsquo;s</a>', callbacks=[])
+        == '<a href="/">Some&nbsp;entity&rsquo;s</a>'
+    )
+
+
 def test_escaped_html():
     """If I pass in escaped HTML, it should probably come out escaped."""
     s = "&lt;em&gt;strong&lt;/em&gt;"
Repository: mozilla/bleach

Issue:
Linkify incorrectly parses array arguments
Hi.
Library version up to 3.1.0 incorrectly parses array and object url parameters:
```python
from bleach import DEFAULT_CALLBACKS, Linker
text= 'http://test.com?array[]=1&params_in[]=2'
linker = Linker(url_re=linkifier.URL_RE, callbacks=DEFAULT_CALLBACKS, skip_tags=None, parse_email=False)
print(linker.linkify(text))
# prints: <a href="http://test.com?array" rel="nofollow">http://test.com?array</a>[]=1¶ms_in[]=2
```
As you see, url is split by [], loosing part of the link.

Comments:

Patch:
diff --git a/bleach/linkifier.py b/bleach/linkifier.py
index 54a432e..8fcefb2 100644
--- a/bleach/linkifier.py
+++ b/bleach/linkifier.py
@@ -45,8 +45,8 @@ def build_url_re(tlds=TLDS, protocols=html5lib_shim.allowed_protocols):
         r"""\(*  # Match any opening parentheses.
         \b(?<![@.])(?:(?:{0}):/{{0,3}}(?:(?:\w+:)?\w+@)?)?  # http://
         ([\w-]+\.)+(?:{1})(?:\:[0-9]+)?(?!\.\w)\b   # xx.yy.tld(:##)?
-        (?:[/?][^\s\{{\}}\|\\\^\[\]`<>"]*)?
-            # /path/zz (excluding "unsafe" chars from RFC 1738,
+        (?:[/?][^\s\{{\}}\|\\\^`<>"]*)?
+            # /path/zz (excluding "unsafe" chars from RFC 3986,
             # except for # and ~, which happen in practice)
         """.format(
             "|".join(sorted(protocols)), "|".join(sorted(tlds))


Test patch:
diff --git a/tests/test_linkify.py b/tests/test_linkify.py
index 4cc01a6..0920c47 100644
--- a/tests/test_linkify.py
+++ b/tests/test_linkify.py
@@ -300,6 +300,14 @@ def test_link_ftp():
     )
 
 
+def test_link_with_qs_with_array():
+    """Test that urls pick up [] in querystring"""
+    assert linkify("http://test.com?array[]=1&params_in[]=2") == (
+        '<a href="http://test.com?array[]=1&amp;params_in[]=2" '
+        + 'rel="nofollow">http://test.com?array[]=1&amp;params_in[]=2</a>'
+    )
+
+
 def test_link_query():
     assert (
         linkify("http://xx.com/?test=win")