Repository: pex-tool/pex

Issue:
`_load_type_error` does not handle that `dist` may be `None`
In Pants, I noticed that this command started failing after the Pex 2.1.48 command.

```
❯ ./pants --no-process-execution-local-cache --no-pantsd --no-remote-cache-{read,write} dependencies build-support/bin:changelog
15:55:27.36 [INFO] Starting: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.13.1
15:55:27.72 [INFO] Completed: Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.13.1
1 Exception encountered:

Engine traceback:
  in select
  in pants.init.plugin_resolver.resolve_plugins
  in pants.backend.python.util_rules.pex.create_venv_pex (pants_plugins.pex)
  in pants.backend.python.util_rules.pex.build_pex (pants_plugins.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/ericarellano/code/pants/src/python/pants/engine/process.py", line 258, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Resolving plugins: hdrhistogram, toolchain.pants.plugin==0.13.1' failed with exit code 1.
stdout:

stderr:
Traceback (most recent call last):
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/pex.py", line 475, in execute
    self.activate()
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/pex.py", line 139, in activate
    self._activated_dists = self._activate()
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/pex.py", line 126, in _activate
    activated_dists.extend(env.activate())
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/environment.py", line 289, in activate
    self._activated_dists = self._activate()
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/environment.py", line 633, in _activate
    resolved = self.resolve()
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/environment.py", line 469, in resolve
    self._resolved_dists = self.resolve_dists(all_reqs)
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/environment.py", line 475, in resolve_dists
    self._update_candidate_distributions(self._load_internal_cache())
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/environment.py", line 245, in _update_candidate_distributions
    ranked_dist = self._can_add(dist)
  File "/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/environment.py", line 256, in _can_add
    filename, ext = os.path.splitext(os.path.basename(dist.location))
AttributeError: 'NoneType' object has no attribute 'location'
```

It's fixed by moving my `named_caches` folder so it's not used. Possibly it's from using old cache entries?

Regardless, the type hints are off for this code path. (MyPy should have caught this...I can play with its settings to make it stricter). This helper returns `Optional`:

https://github.com/pantsbuild/pex/blob/cbe7b53188fc79af2b2418f439ce76674bd6f235/pex/util.py#L92-L93

But this is not properly handling the `None` case:

https://github.com/pantsbuild/pex/blob/34700d5b5a9e35a15ae7497d2d523e3d4f4a4153/pex/environment.py#L234-L240


Discussion:
Eric - do you still have that named_caches folder? If so can you move it back and modify the code in /Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.bootstrap/pex/environment.py to print out some actionable information? Handling None is one thing, but secondary - there should never be any - and that is the more pertinent thing to investigate.
Yep, I still have the whole folder. Would it help for me to upload it?

I made this change:

```
234     def _load_internal_cache(self):
235         # type: () -> Iterator[Distribution]
236         internal_cache = os.path.join(self._pex, self._pex_info.internal_cache)
237         with TRACER.timed("Searching dependency cache: %s" % internal_cache, V=2):
238             for distribution_name in self._pex_info.distributions:
239                 dist_path = os.path.join(internal_cache, distribution_name)
240                 dist = DistributionHelper.distribution_from_path(dist_path)
241                 if dist is None:
242                     print(dist_path)
243                 yield dist
244
```

Resulting in:

```
/Users/ericarellano/.cache/pants/named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/.deps/PyYAML-5.4.1-cp39-cp39-macosx_11_0_arm64.whl
```

So then I added a `print()` to `util.py`:

```
 91     @classmethod
 92     def distribution_from_path(cls, path, name=None):
 93         # type: (Text, Optional[str]) -> Optional[Distribution]
 94         """Return a distribution from a path.
 95
 96         If name is provided, find the distribution.  If none is found matching the name, return
 97         None. If name is not provided and there is unambiguously a single distribution, return that
 98         distribution. Otherwise, None.
 99         """
100         if name is None:
101             distributions = set(find_distributions(path))
102             if path.endswith("PyYAML-5.4.1-cp39-cp39-macosx_11_0_arm64.whl"):
103                 print(distributions)
104             if len(distributions) == 1:
105                 return distributions.pop()
106         else:
107             for dist in find_distributions(path):
108                 if dist.project_name == name:
109                     return dist
110         return None
```

Resulting in: `set()`
Yes, if you can upload the whole thing that'd be great.
Ok, Eric provided that offline and you see:
```console
$ tree -a -L 2 named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c
named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c
├── .bootstrap -> ../../bootstraps/05dc9963564f8e8b6c67803a2bcd88a73244617a
├── .deps
│   └── pex-2.1.48-py2.py3-none-any.whl -> ../../../installed_wheels/332e69dd24a824a9104b3b7ec78e7e5445e54e2e/pex-2.1.48-py2.py3-none-any.whl
├── __main__.py -> ../../code/da39a3ee5e6b4b0d3255bfef95601890afd80709/__main__.py
├── PEX-INFO -> ../../code/da39a3ee5e6b4b0d3255bfef95601890afd80709/PEX-INFO
└── __pycache__
    └── __main__.cpython-39.pyc

4 directories, 3 files
```

But PEX-INFO shows many more distributions should be linked `jq . named_caches/pex_root/unzipped_pexes/d49e0d878ea870dc5d0aa3593e33aeea08b9067c/PEX-INFO`:
```json
{
  "bootstrap_hash": "55f4763fb6fadc7d2d58d45949616d1857afd705",
  "build_properties": {
    "pex_version": "2.1.48"
  },
  "code_hash": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "distributions": {
    "PyYAML-5.4.1-cp39-cp39-macosx_11_0_arm64.whl": "abca78579d12532538d9f39c04b6e031ff2269cd",
    "ansicolors-1.1.8-py2.py3-none-any.whl": "c7b5d77e89855f9b02c99e84cd863c5bdc2be329",
    "attrs-21.2.0-py2.py3-none-any.whl": "2d30ac163afe61a501f56152d991607463db80f9",
    "certifi-2021.5.30-py2.py3-none-any.whl": "c1b63440cdd5c303c7268c98b3454996a456e0cd",
    "charset_normalizer-2.0.4-py3-none-any.whl": "5a7cf2d7976cc6d6c538ef27a8a8ed31fa9632f5",
    "fasteners-0.16-py2.py3-none-any.whl": "ef97c8bf1c7ece677c203ea6c1a3580e4dbe0a34",
    "freezegun-1.1.0-py2.py3-none-any.whl": "38db3e74e55ce14a7c42a4b744a82757447717ca",
    "humbug-0.2.6-py3-none-any.whl": "e772468167e997e051e8c20d71551a11bba19f26",
    "idna-3.2-py3-none-any.whl": "f6f18646cd1fcc3b8d439bd113b7cc9ad4ebc3c0",
    "ijson-3.1.4-py3-none-any.whl": "a8e3921dbaed448d425d1e99efcf96168bda858c",
    "iniconfig-1.1.1-py2.py3-none-any.whl": "e772a086ea0855a0dacb5ed123972d1953a243f2",
    "packaging-21.0-py3-none-any.whl": "dd3f7d02ab52ec2c5abdb6bcdd7b0528eeda918b",
    "pex-2.1.48-py2.py3-none-any.whl": "1dd98144a00d5480efc76746c0e4ec0208fb15cc",
    "pluggy-1.0.0-py2.py3-none-any.whl": "e0af88ae2899a7b00b3586b4313183dfa4445d96",
    "psutil-5.8.0-cp39-cp39-macosx_11_0_arm64.whl": "f6a2f9ea6629594cef2b2cf7e8867889cb76476a",
    "py-1.10.0-py2.py3-none-any.whl": "2742038b9554bda558f0835448bd5bc6115968ef",
    "pyparsing-2.4.7-py2.py3-none-any.whl": "401ee8e4d4a08b2e87749a109db29991c85a67c6",
    "pystache-0.5.4-py3-none-any.whl": "2e6e733300384a91d8fde424172e9aeb9d12f91e",
    "pytest-6.2.5-py3-none-any.whl": "c81a0f09ee54ea9496a98d7b1b12d41286248226",
    "python_dateutil-2.8.2-py2.py3-none-any.whl": "ba32f6e5b7ffc0249ce8f14878247e290a4768ea",
    "requests-2.26.0-py2.py3-none-any.whl": "fb70e1b8449a6408d983fb8c56f60f1e827890d8",
    "setproctitle-1.2.2-cp39-cp39-macosx_11_0_arm64.whl": "2634e0f09bb86361723b91ba47fd7e5870a9e520",
    "setuptools-57.5.0-py3-none-any.whl": "59619d37c264739911dbe1f3e71ae4c20bac1bd5",
    "six-1.16.0-py2.py3-none-any.whl": "035d7c208925c1832def39b592f3477ca36397bf",
    "toml-0.10.2-py2.py3-none-any.whl": "941913d720ad4816a848c11a218c9110f1978120",
    "types_PyYAML-5.4.3-py2.py3-none-any.whl": "ff59bd9da4c781186fecb96e9717c458a98db032",
    "types_freezegun-0.1.4-py2.py3-none-any.whl": "17c7703056250722686ad19980195e945fb3069e",
    "types_requests-2.25.0-py3-none-any.whl": "2d1a27d069663b0529ae82cb3bc47c8d815ad6d5",
    "types_setuptools-57.0.0-py3-none-any.whl": "376ce3ec246abbf148b0edc6861c92f368daf4a8",
    "types_toml-0.1.3-py2.py3-none-any.whl": "8599aba9d6d109a743e579345fb4b23b4957c9d7",
    "typing_extensions-3.7.4.3-py3-none-any.whl": "eedab11cf76ead911adc7311c50c4f75212a0fa4",
    "urllib3-1.26.6-py2.py3-none-any.whl": "d58fa3fcc249a15487bc44b2a9afd9b065720684"
  },
  "emit_warnings": false,
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "interpreter_constraints": [
    "CPython==3.9.*"
  ],
  "pex_hash": "d9f2a122a3aa5bd2db73e43b91a754410ecc4263",
  "pex_path": null,
  "requirements": [
    "ansicolors==1.1.8",
    "attrs==21.2.0; python_version >= \"3.6\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.5.0\" and python_version >= \"3.6\"",
    "certifi==2021.5.30; python_version >= \"2.7\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.6.0\"",
    "charset-normalizer==2.0.4; python_full_version >= \"3.6.0\" and python_version >= \"3\"",
    "fasteners==0.16",
    "freezegun==1.1.0; python_version >= \"3.5\"",
    "humbug==0.2.6",
    "idna==3.2; python_version >= \"3.5\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.6.0\" and python_version >= \"3.5\"",
    "ijson==3.1.4",
    "iniconfig==1.1.1; python_version >= \"3.6\"",
    "packaging==21.0; python_version >= \"3.6\"",
    "pex==2.1.48; (python_version >= \"2.7\" and python_full_version < \"3.0.0\") or (python_full_version >= \"3.5.0\" and python_version < \"3.10\")",
    "pluggy==1.0.0; python_version >= \"3.6\"",
    "psutil==5.8.0; (python_version >= \"2.6\" and python_full_version < \"3.0.0\") or python_full_version >= \"3.4.0\"",
    "py==1.10.0; python_version >= \"3.6\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.4.0\" and python_version >= \"3.6\"",
    "pyparsing==2.4.7; python_version >= \"3.6\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.3.0\" and python_version >= \"3.6\"",
    "pystache==0.5.4",
    "pytest==6.2.5; python_version >= \"3.6\"",
    "python-dateutil==2.8.2; python_version >= \"3.5\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.3.0\" and python_version >= \"3.5\"",
    "pyyaml==5.4.1; (python_version >= \"2.7\" and python_full_version < \"3.0.0\") or python_full_version >= \"3.6.0\"",
    "requests==2.26.0; (python_version >= \"2.7\" and python_full_version < \"3.0.0\") or python_full_version >= \"3.6.0\"",
    "setproctitle==1.2.2; python_version >= \"3.6\"",
    "setuptools==57.5.0; python_version >= \"3.6\"",
    "six==1.16.0; python_version >= \"3.5\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.3.0\" and python_version >= \"3.5\"",
    "toml==0.10.2; (python_version >= \"2.6\" and python_full_version < \"3.0.0\") or python_full_version >= \"3.3.0\"",
    "types-freezegun==0.1.4",
    "types-pyyaml==5.4.3",
    "types-requests==2.25.0",
    "types-setuptools==57.0.0",
    "types-toml==0.1.3",
    "typing-extensions==3.7.4.3",
    "urllib3==1.26.6; python_version >= \"2.7\" and python_full_version < \"3.0.0\" or python_full_version >= \"3.6.0\" and python_version < \"4\""
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false
}
```

Its not clear how you could possibly get into that state with atomic_directory, but it does totally explain the `None`(s). I'll look at the layout code afresh tomorrow with this fact in mind.

Interesting in the above is that `__main__.py` and `PEX-INFO` are symlinks into the `code/` cache. This should not be the case. The `--spread` code worked this way - pre-release of Pex 2.1.48, but the `--layout packed` code does not, it copies those files.

zipapp:
https://github.com/pantsbuild/pex/blob/e819336a4a0098e80f3a57c200a7cf571e7a496f/pex/layout.py#L222-L228
packed:
https://github.com/pantsbuild/pex/blob/e819336a4a0098e80f3a57c200a7cf571e7a496f/pex/layout.py#L265-L271
Ok, so the interesting part about the `PEX-INFO` is the `code_hash` - that's the sha1 nothing hash so the PEX in question is a dependencies-only PEX file.
Aha. Ok, old `~/.pex/code/<code_hash>/` dirs (this for --not-zip-safe PEXes), included `__main__.py` and `PEX-INFO` files. This was always a bug since the `<code_hash>` calculation in `PEXBuilder` excludes those 2 files. As such, every dependency-only `--not-zip-safe` PEX shared the same `~/.pex/code/da39a3ee5e6b4b0d3255bfef95601890afd80709/` which had just` __main__.py` and `PEX-INFO`. This used to be harmless since the `PEX-INFO` stored there - even though specific to the 1st dependency only `--not-zip-safe` PEX run in a given `$PEX_ROOT`, was never read / used. This is now harmful, however since PEX zipapp and spread mode pexes accidentally inherit those `PEX-INFO` and `__main__.py` in their `~/.pex/unzipped_pexes/.../` chroot.

Resolving patch:
diff --git a/pex/pex_info.py b/pex/pex_info.py
--- a/pex/pex_info.py
+++ b/pex/pex_info.py
@@ -418,7 +418,7 @@ def install_cache(self):
     @property
     def zip_unsafe_cache(self):
         #: type: () -> str
-        return os.path.join(self.pex_root, "code")
+        return os.path.join(self.pex_root, "user_code")
 
     def update(self, other):
         # type: (PexInfo) -> None


Test patch: