Repository: pypi/warehouse

Issue:
Invalid Unicode character in package name causing parsing error when trying to parse the RSS feeds
**Describe the bug**
When we tried to parse the [newest packages feed](https://warehouse.pypa.io/api-reference/feeds.html#newest-packages-feed) this morning Python raised an XML parsing exception because there was an invalid Unicode character in the project description of <https://pypi.org/project/dj-test-queries/>.

**Expected behavior**
The RSS feed to not include invalid Unicode characters.

**To Reproduce**
Publish a package with invalid Unicode characters in the description and then attempt to parse the newest packages feed.

**My Platform**
We're using https://pypi.org/project/aiohttp/ to read the feed and https://pypi.org/project/xmltodict/ to dump it to a `dict` for eaiser processing.

**Additional context**
<!-- Add any other context, links, etc. about the feature here. -->



Discussion:

Resolving patch:
diff --git a/warehouse/config.py b/warehouse/config.py
--- a/warehouse/config.py
+++ b/warehouse/config.py
@@ -458,6 +458,9 @@ def configure(settings=None):
     filters.setdefault("is_recent", "warehouse.filters:is_recent")
     filters.setdefault("canonicalize_name", "packaging.utils:canonicalize_name")
     filters.setdefault("format_author_email", "warehouse.filters:format_author_email")
+    filters.setdefault(
+        "remove_invalid_xml_unicode", "warehouse.filters:remove_invalid_xml_unicode"
+    )
 
     # We also want to register some global functions for Jinja
     jglobals = config.get_settings().setdefault("jinja2.globals", {})
diff --git a/warehouse/filters.py b/warehouse/filters.py
--- a/warehouse/filters.py
+++ b/warehouse/filters.py
@@ -188,5 +188,15 @@ def format_author_email(metadata_email: str) -> tuple[str, str]:
     return author_emails[0][0], author_emails[0][1]
 
 
+def remove_invalid_xml_unicode(value: str) -> str:
+    """
+    Remove invalid unicode characters from a string.
+    Useful for XML Templates.
+
+    Ref: https://www.w3.org/TR/REC-xml/#NT-Char
+    """
+    return "".join(c for c in value if ord(c) >= 32)
+
+
 def includeme(config):
     config.add_request_method(_camo_url, name="camo_url")


Test patch:
diff --git a/tests/unit/test_filters.py b/tests/unit/test_filters.py
--- a/tests/unit/test_filters.py
+++ b/tests/unit/test_filters.py
@@ -267,3 +267,20 @@ def test_format_author_email(meta_email, expected_name, expected_email):
     author_name, author_email = filters.format_author_email(meta_email)
     assert author_name == expected_name
     assert author_email == expected_email
+
+
+@pytest.mark.parametrize(
+    ("inp", "expected"),
+    [
+        ("foo", "foo"),  # no change
+        (" foo  bar ", " foo  bar "),  # U+001B : <control> ESCAPE [ESC]
+        ("foo \x1b bar", "foo  bar"),  # U+001B : <control> ESCAPE [ESC]
+        ("foo \x00 bar", "foo  bar"),  # U+0000 : <control> NULL
+        ("foo 🐍 bar", "foo 🐍 bar"),  # U+1F40D : SNAKE [snake] (emoji) [Python]
+    ],
+)
+def test_remove_invalid_xml_unicode(inp, expected):
+    """
+    Test that invalid XML unicode characters are removed.
+    """
+    assert filters.remove_invalid_xml_unicode(inp) == expected