Repository: pex-tool/pex

Issue:
pex fails with InvalidSpecifier if any dist has a bad version specifier
After upgrading from 2.1.100 to  2.1.151 ~and so pulling https://github.com/pantsbuild/pex/pull/2264~ `pex` started failing to execute locally with the following error:

```
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "~/.pex/unzipped_pexes/003cdfcf7d5635cb324a06f8b49939c255f2add1/__main__.py", line 108, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "pex/pex_bootstrapper.py", line 627, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "pex/pex.py", line 542, in execute
    self.activate()
  File "pex/pex.py", line 242, in activate
    self.patch_sys()
  File "pex/pex.py", line 445, in patch_sys
    new_sys_path, new_sys_path_importer_cache, new_sys_modules = self.minimum_sys(inherit_path)
  File "pex/pex.py", line 415, in minimum_sys
    sys_path, sys_path_importer_cache = self.minimum_sys_path(isolated_sys_path, inherit_path)
  File "pex/pex.py", line 351, in minimum_sys_path
    site_distributions.update(all_distribution_paths(path_element))
  File "pex/pex.py", line 345, in all_distribution_paths
    locations = set(dist.location for dist in find_distributions([path]))
  File "pex/pex.py", line 345, in <genexpr>
    locations = set(dist.location for dist in find_distributions([path]))
  File "pex/dist_metadata.py", line 1004, in find_distributions
    location=location, metadata=DistMetadata.from_metadata_files(metadata_files)
  File "pex/dist_metadata.py", line 758, in from_metadata_files
    requires_python=requires_python(metadata_files),
  File "pex/dist_metadata.py", line 516, in requires_python
    return SpecifierSet(python_requirement)
  File "pex/vendor/_vendored/packaging_23_1/packaging/specifiers.py", line 714, in __init__
    parsed.add(Specifier(specifier))
  File "pex/vendor/_vendored/packaging_23_1/packaging/specifiers.py", line 245, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
pex.vendor._vendored.packaging_23_1.packaging.specifiers.InvalidSpecifier: Invalid specifier: '>=3.5.*'
```

We managed to trace this back to `apache_libcloud-3.2.0.egg-info/PKG-INFO` which has the following line:

```
Requires-Python: >=3.5.*, <4
```

Although this seems to be a [bad specifier](https://discuss.python.org/t/pep-440-wildcard-usage-in-comparison-clause/10045/7) , some components presumably accept it as the package was working in that state for years, but not the `packaging` stuff used by `pex`. 

Now this [was a bug](https://github.com/apache/libcloud/blob/72de440a508fdb4ea546d4e586a452472f1cbd5d/setup.py#L274) in libcloud [1], but I think `pex` should not fail entirely if some package has this in its version info. ~This was exposed by #2264 because before that this `PKG-INFO` file was not parsed at all.~

---

In any case libcloud has been fixed years ago apparently but of course not every transitive dependency has necessarily been updated to use a fixed version, etc.


Discussion:
@travisdowns I think #2264 may be a bad diagnosis. What prior version of Pex was this same scenario working with?
Yeah, the bisect on Pex versions leads to some change in 2.1.140:
```
$ pex --python python3.8 pex==2.1.139 -cpex -- apache_libcloud==3.2.0 -- -c 'import libcloud; print(libcloud.__file__)'
/home/jsirois/.pex/installed_wheels/e41b8670b5b2cb9b8bf1557f099a0e75dcb0ed9e9b2d667a3a0c1fbec0fa11bb/apache_libcloud-3.2.0-py2.py3-none-any.whl/libcloud/__init__.py
$ pex --python python3.8 pex==2.1.140 -cpex -- apache_libcloud==3.2.0 -- -c 'import libcloud; print(libcloud.__file__)'
Invalid specifier: '>=3.5.*'
```
Yeah, this bisects to #2188 which upgraded vendored `packaging` from the PyPA to 23.1 for Python>=3.7 (packaging 21.3 is used for Python 3.6 and packaging 20.9 is used for any older Python). IOW, the official way to parse any metadata (whether from a dist-info dist (wheel) or PKG-INFO), got stricter in enforcing the PyPAs own spec in [PEP-440](https://peps.python.org/pep-0440).
So, @travisdowns Pex could contemplate ignoring bad `Requires-Python` metadata, except that it actually uses the metadata as part of the tests used at boot-time to decide which embedded dists to activate. This is part of providing multi-platform PEXes. So to drop the check when it fails would mean Pex yolos on your behalf which is definitely against the grain of Pex philosophy of hermetic, exact, verified.

Instead of continuing down that line of thought, perhaps this can be pre-empted by just fixing libcloud? VCS specifiers are very handy for this sort of thing.
Closing the loop on the exact error cause. It looks to be right here (which was included in the packaging 23.1 release): https://github.com/pypa/packaging/commit/068a0b5975a95a5cc30262599ca29bb6ceabe4af#diff-91bbf04171e694df3ef07a8fffc168bfa43e2f00cb531ee37035987210ae0371R230

IOW, PyPA adding better enforcement of their own spec.
> @travisdowns I think #2264 may be a bad diagnosis.

You could be right: I thought before this change the `.egg-info` wasn't examined at all, which would avoid this, but I could be mistaken there.

> What prior version of Pex was this same scenario working with?

2.1.100
> Instead of continuing down that line of thought, perhaps this can be pre-empted by just fixing libcloud? VCS specifiers are very handy for this sort of thing.

Libcloud was fixed a couple of years ago but I can't go and fix all the transitive dependees to update their version. Among other places this version of libcloud is the default shipped with the most recent Ubuntu LTS 22.04, a very popular distro.

> except that it actually uses the metadata as part of the tests used at boot-time to decide which embedded dists to activate. This is part of providing multi-platform PEXes. So to drop the check when it fails would mean Pex yolos on your behalf which is definitely against the grain of Pex philosophy of hermetic, exact, verified.

I don't know much about pex to make a very intelligent comment, but wouldn't treating a failed check in the same way as an unsatisfied python version be better? I.e., don't activate the dist.

> which is definitely against the grain of Pex philosophy of hermetic, exact, verified.

Fair enough, but as it is it is totally not hermetric: installing any random package (even non-python one) on the host may bring in this file and its mere existence on the filesystem means that any existing pex just stops working. So in this sense it seems like it is not at all hermetic.


> You could be right: I thought before this change the .egg-info wasn't examined at all, which would avoid this, but I could be mistaken there.

It would absolutely not avoid this. The same metadata is present in the whl METADATA file.

> I don't know much about pex to make a very intelligent comment, but wouldn't treating a failed check in the same way as an unsatisfied python version be better? I.e., don't activate the dist.

Ok. But then you'd fail in a different way when the code that needs libcloud failed to import it!

> Fair enough, but as it is it is totally not hermetric: installing any random package (even non-python one) on the host may bring in this file and its mere existence on the filesystem means that any existing pex just stops working.

That is just straight up false. That is not what is happening here. See my note above about .whl METADATA.

> Libcloud was fixed a couple of years ago but I can't go and fix all the transitive dependees to update their version. Among other places this version of libcloud is the default shipped with the most recent Ubuntu LTS 22.04, a very popular distro.

It's as easy as this...
So, for example just specify `apache_libcloud @ git+https://github.com/jsirois/libcloud@2cde5937fdfae56113f80f18c7005a4c691a06e5` as a direct requirement or via a constraints file:
```
$ pex -V
2.1.151
$ pex --python python3.11 "apache_libcloud @ git+https://github.com/jsirois/libcloud@2cde5937fdfae56113f80f18c7005a4c691a06e5" -- -c 'import libc
loud; print(libcloud.__file__)'
/home/jsirois/dev/jsirois/libcloud/libcloud/__init__.py
```

That is my fork of the project with a patch applied on top of the v3.2.0 tagged commit.
@travisdowns let me know if my explanation and workaround are sticking for you. If so, I'd like to close this as an answered question. If not, we can discuss in more detail why the workaround is not sufficient or other means of amelioration.
> It would absolutely not avoid this. The same metadata is present in the whl METADATA file.

We did not find any METADATA file with this string, only the PKG-INFO. It comes from the `python3-libcloud` apt package in 22.04 ubuntu repos.

> Ok. But then you'd fail in a different way when the code that needs libcloud failed to import it!

That seems like a good approach: after all libcloud is broken, so it would be up to the user to update it. "Breaking any pex that tries to use libcloud" seems much better than "breaking every pex on the system as soon as the libcloud package is even installed". Note that we did not need libcloud in any way, it was just incidentally installed on the system for some other reason (probably not even explicitly but just as a transitive dependency of some other package). 

> That is just straight up false. That is not what is happening here. See my note above about .whl METADATA.

I am reporting what I saw. `pex` fails on Ubuntu 22.04 if `python3-libcloud` is installed. After removing this package, `pex` immediately works. If installing an unrelated package which isn't used by any pex and is not used by pex itself causes pex bootstrap to break that is "not hermetic", I think? I didn't understand how the .whl file relates to hermeticity, sorry.

> So, for example just specify apache_libcloud @ git+https://github.com/jsirois/libcloud@2cde5937fdfae56113f80f18c7005a4c691a06e5 as a direct requirement or via a constraints file:

As above, and perhaps I did not make this clear enough initially, we are _not using libcloud_. It was just brought in at some unknown point by the host package manager, probably as a transitive dependency.

We don't need a fix: we have already "fixed" this issue by removing the unused package, before this bug was reported. I thought I'd report it here for the benefit of pex maintainers and other users but I'm sure starting to regret my choice.
> As above, and perhaps I did not make this clear enough initially, we are not using libcloud. It was just brought in at some unknown point by the host package manager, probably as a transitive dependency.
> 
> We don't need a fix: we have already "fixed" this issue by removing the unused package, before this bug was reported. I thought I'd report it here for the benefit of pex maintainers and other users but I'm sure starting to regret my choice.

Yeah, that was totally unclear. You did alot of work figuring out what was going on it seems, which is great, but it was not detailed here. If the bug report included the Pex build time command line and runtime command line (this is a runtime error), for example, it would have made it clear libcloud was not a direct or transitive requirement. You have provided more details now though, which is great.

> I am reporting what I saw. pex fails on Ubuntu 22.04 if python3-libcloud is installed. After removing this package, pex immediately works. If installing an unrelated package which isn't used by any pex and is not used by pex itself causes pex bootstrap to break that is "not hermetic", I think?

Ok, that is new information - for me. Thank you.

I repro as such:
```dockerfile
$ cat Dockerfile
FROM ubuntu:22.04

RUN apt update && \
    apt install -y \
        python3-dev \
        python3-distutils \
        python3-venv \
        python3-libcloud

RUN python3 -m venv pex.venv && \
    pex.venv/bin/pip install pex

ENV PATH=pex.venv/bin:$PATH
```

```console
$ docker build -tpex-issues-2289 .
$ docker run --rm -it pex-issues-2289
root@83e1b2ff9d09:/# pex -V
2.1.152
root@83e1b2ff9d09:/# pex cowsay -c cowsay -o cowsay.pex
root@83e1b2ff9d09:/# ./cowsay.pex
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/__main__.py", line 108, in <module>
    bootstrap_pex(__entry_point__, execute=__execute__, venv_dir=__venv_dir__)
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/pex_bootstrapper.py", line 627, in bootstrap_pex
    pex.PEX(entry_point).execute()
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/pex.py", line 542, in execute
    self.activate()
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/pex.py", line 242, in activate
    self.patch_sys()
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/pex.py", line 445, in patch_sys
    new_sys_path, new_sys_path_importer_cache, new_sys_modules = self.minimum_sys(inherit_path)
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/pex.py", line 415, in minimum_sys
    sys_path, sys_path_importer_cache = self.minimum_sys_path(isolated_sys_path, inherit_path)
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/pex.py", line 351, in minimum_sys_path
    site_distributions.update(all_distribution_paths(path_element))
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/pex.py", line 345, in all_distribution_paths
    locations = set(dist.location for dist in find_distributions([path]))
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/pex.py", line 345, in <genexpr>
    locations = set(dist.location for dist in find_distributions([path]))
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/dist_metadata.py", line 1004, in find_distributions
    location=location, metadata=DistMetadata.from_metadata_files(metadata_files)
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/dist_metadata.py", line 758, in from_metadata_files
    requires_python=requires_python(metadata_files),
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/dist_metadata.py", line 516, in requires_python
    return SpecifierSet(python_requirement)
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/specifiers.py", line 714, in __init__
    parsed.add(Specifier(specifier))
  File "/root/.pex/unzipped_pexes/994f8ec4b043ab1eb64b76a7f53d81e99fce2604/.bootstrap/pex/vendor/_vendored/packaging_23_1/packaging/specifiers.py", line 245, in __init__
    raise InvalidSpecifier(f"Invalid specifier: '{spec}'")
pex.vendor._vendored.packaging_23_1.packaging.specifiers.InvalidSpecifier: Invalid specifier: '>=3.5.*'
root@83e1b2ff9d09:/# apt remove -y python3-libcloud
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  python3-certifi python3-cffi-backend python3-chardet python3-cryptography python3-idna python3-lockfile python3-pkg-resources python3-requests python3-simplejson python3-six python3-urllib3
Use 'apt autoremove' to remove them.
The following packages will be REMOVED:
  python3-libcloud
0 upgraded, 0 newly installed, 1 to remove and 12 not upgraded.
After this operation, 19.6 MB disk space will be freed.
(Reading database ... 14211 files and directories currently installed.)
Removing python3-libcloud (3.2.0-2) ...
root@83e1b2ff9d09:/# ./cowsay.pex -t 'Moo!'
  ____
| Moo! |
  ====
    \
     \
       ^__^
       (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||
root@83e1b2ff9d09:/#
```
With this reproducer where which pex commit does the failure bisect to?
I'm sure its the one you identified. That seems clear without bisecting. The fuller picture is what I needed here. I apologize for not being able to read the tea leaves of the initial bug report.
Alrighty, this should clean up easily. The Pex runtime was doing alot of un-necessary work since ~this date 9 years ago it appears in its sys.path scrubbing implementation. Before [this change](https://github.com/pantsbuild/pex/commit/eeb80a5073f64ee4a91b3daf0a3017e0cfeb8abd#diff-07a90ebaf117fe2a7d1010d8dbe6ed42fab40761f07622b3dba999b2e2dbce78L140), each scrubbed distribution in each scrubbed path element was _logged_, but no more. Since then even that bit of use has been absent, and collecting dists was un-needed work.
Aha, thinking more, in 2014 there were egg zips as possible `sys.path` sub-elements; so `find_distributions` was necessary at the time to find such un-exploded eggs. Since none of the Pex machinery, 1st or (vendored) 3rdparty, handles eggs any longer, this is safe to remove.
Thanks for the RCA, and apology accepted.
I am becoming pretty frustrated with OSS expectations. What part of my behavior do you consider unsatisfactory? I apologized for nothing here. I was just trying to run down a bug, and sometimes that takes 20 questions.
Ah, I guess I did apologize - but that was a bit tongue in cheek. 
FWIW this bug will be fixed and released in under 8 hours start to finish. I would guess you don't get that good a response time in many OSS projects; especially those maintained gratis. I am not payed for this or any other software work.

Resolving patch:
diff --git a/pex/pex.py b/pex/pex.py
--- a/pex/pex.py
+++ b/pex/pex.py
@@ -13,7 +13,7 @@
 from pex import bootstrap
 from pex.bootstrap import Bootstrap
 from pex.common import die
-from pex.dist_metadata import CallableEntryPoint, Distribution, EntryPoint, find_distributions
+from pex.dist_metadata import CallableEntryPoint, Distribution, EntryPoint
 from pex.environment import PEXEnvironment
 from pex.executor import Executor
 from pex.finders import get_entry_point_from_console_script, get_script_from_distributions
@@ -335,31 +335,30 @@ def minimum_sys_path(
     ):
         # type: (...) -> Tuple[List[str], Mapping[str, Any]]
         scrub_paths = OrderedSet()  # type: OrderedSet[str]
-        site_distributions = OrderedSet()  # type: OrderedSet[str]
-        user_site_distributions = OrderedSet()  # type: OrderedSet[str]
+        site_paths = OrderedSet()  # type: OrderedSet[str]
+        user_site_paths = OrderedSet()  # type: OrderedSet[str]
 
-        def all_distribution_paths(path):
+        def all_paths(path):
             # type: (Optional[str]) -> Iterable[str]
             if path is None:
                 return ()
-            locations = set(dist.location for dist in find_distributions([path]))
-            return {path} | locations | set(os.path.realpath(path) for path in locations)
+            return path, os.path.realpath(path)
 
         for path_element in sys.path:
             if path_element not in isolated_sys_path:
                 TRACER.log("Tainted path element: %s" % path_element)
-                site_distributions.update(all_distribution_paths(path_element))
+                site_paths.update(all_paths(path_element))
             else:
                 TRACER.log("Not a tainted path element: %s" % path_element, V=2)
 
-        user_site_distributions.update(all_distribution_paths(USER_SITE))
+        user_site_paths.update(all_paths(USER_SITE))
 
         if inherit_path == InheritPath.FALSE:
-            scrub_paths = OrderedSet(site_distributions)
-            scrub_paths.update(user_site_distributions)
-            for path in user_site_distributions:
+            scrub_paths = OrderedSet(site_paths)
+            scrub_paths.update(user_site_paths)
+            for path in user_site_paths:
                 TRACER.log("Scrubbing from user site: %s" % path)
-            for path in site_distributions:
+            for path in site_paths:
                 TRACER.log("Scrubbing from site-packages: %s" % path)
 
         scrubbed_sys_path = list(OrderedSet(sys.path) - scrub_paths)


Test patch: