Repository: pypi/warehouse

Issue:
Blocking of stdlib names has holes?
Hi!

Pull request #2409 introduced a blacklist of stdlib names to counter to protect against [pytosquatting](https://www.pytosquatting.org/), aiming to plug #2151.

I found a hole with it. While names like `xmlrpc.client` and `encodings.idna` are blocked because they appear in some `stdlib-list` list, some parent packages like `encodings` are not in their list and do not get blocked by warehouse as a consequence. I was able to register these four packages:
* [encodings](https://pypi.org/project/encodings/)
* [xml.etree](https://pypi.org/project/xml.etree/)
* [xml.parsers](https://pypi.org/project/xml.parsers/)
* [xmlrpc](https://pypi.org/project/xmlrpc/)

According to [pytosquatting.org](https://www.pytosquatting.org/), package `encodings` had about 20 hits per day in the past, not sure how much hits the others would get. Personally, I would have considered at least `encodings` and `xmlrpc` worth blocking.

Should `stdlib-list` be completed or rather parent package names auto-included at warehouse level for a generic fix? What do you think?

PS: I'm happy to delete these four packages from PyPI once this hole is plugged. I would ask to keep them up until then, please feel free to verify their whitehat nature. Thank you!

Best, Sebastian

CC: @benjaoming @hannob


Discussion:
@hartwork thanks for the report. interesting work around :)

i'm going to get a fix in at the warehouse level, then pull down the registered packages.

in the future, PyPI does have a security policy published at https://pypi.org/security/ and https://pypi.python.org/security for issues relating to the security of our service or our users.

Resolving patch:
diff --git a/warehouse/forklift/legacy.py b/warehouse/forklift/legacy.py
--- a/warehouse/forklift/legacy.py
+++ b/warehouse/forklift/legacy.py
@@ -52,10 +52,19 @@
 
 PATH_HASHER = "blake2_256"
 
+
+def namespace_stdlib_list(module_list):
+    for module_name in module_list:
+        parts = module_name.split('.')
+        for i, part in enumerate(parts):
+            yield '.'.join(parts[:i + 1])
+
+
 STDLIB_PROHIBITTED = {
     packaging.utils.canonicalize_name(s.rstrip('-_.').lstrip('-_.'))
-    for s in chain.from_iterable(stdlib_list.stdlib_list(version)
-                                 for version in stdlib_list.short_versions)
+    for s in chain.from_iterable(
+        namespace_stdlib_list(stdlib_list.stdlib_list(version))
+        for version in stdlib_list.short_versions)
 }
 
 # Wheel platform checking


Test patch:
diff --git a/tests/unit/forklift/test_legacy.py b/tests/unit/forklift/test_legacy.py
--- a/tests/unit/forklift/test_legacy.py
+++ b/tests/unit/forklift/test_legacy.py
@@ -882,6 +882,11 @@ def test_fails_with_invalid_names(self, pyramid_config, db_request, name):
                                       "main", "future", "al", "uU", "test",
                                       "encodings.utf_8_sig",
                                       "distutils.command.build_clib",
+                                      "xmlrpc", "xmlrpc.server",
+                                      "xml.etree", "xml.etree.ElementTree",
+                                      "xml.parsers", "xml.parsers.expat",
+                                      "xml.parsers.expat.errors",
+                                      "encodings.idna", "encodings",
                                       "CGIHTTPServer", "cgihttpserver"])
     def test_fails_with_stdlib_names(self, pyramid_config, db_request, name):
         pyramid_config.testing_securitypolicy(userid=1)