Repository: pex-tool/pex

Issue:
Running opvault 0.4.9 pex leads to infinite recursion in setup tools
Operating system macOS 11.2.3
Python 3.8.2 (installed via pyenv)
Pex 2.1.38.

To reproduce in a clean environment:
```
pex opvault=="0.4.9" -o opvault.pex
<snip>
Saving PEX file to opvault.pex
pex: Zipping PEX file.: 1471.6ms
pex --disable-cache opvault=="0.4.9" -e opvault-cli -vvv -o opvault.pex  14.98s user 5.21s system 158% cpu 12.734 total
```

Running it results in:
```
./opvault.pex
Traceback (most recent call last):
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/pex.py", line 482, in execute
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/pex.py", line 138, in activate
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/pex.py", line 125, in _activate
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/environment.py", line 428, in activate
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/environment.py", line 784, in _activate
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/environment.py", line 608, in resolve
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/environment.py", line 635, in resolve_dists
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/environment.py", line 505, in _resolve_requirement
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/environment.py", line 505, in _resolve_requirement
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/environment.py", line 505, in _resolve_requirement
  [Previous line repeated 955 more times]
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/environment.py", line 484, in _resolve_requirement
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/dist_metadata.py", line 258, in requires_dists
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3180, in parse
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3126, in parse_requirements
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 3133, in __init__
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/packaging/requirements.py", line 106, in __init__
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3552, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 4005, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3552, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 3471, in parseImpl
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1443, in tryParse
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
  File "/Users/zmanji/bin/opvault.pex/.bootstrap/pex/vendor/_vendored/setuptools/pkg_resources/_vendor/pyparsing.py", line 2816, in parseImpl
RecursionError: maximum recursion depth exceeded
```


If I install the same package via pip:
```
 pip install opvault==0.4.9
Collecting opvault==0.4.9
  Using cached opvault-0.4.9-py2.py3-none-any.whl (21 kB)
Requirement already satisfied: pycryptodome==3.9.7 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (3.9.7)
Requirement already satisfied: requests==2.24.0 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (2.24.0)
Requirement already satisfied: shellescape==3.8.1 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (3.8.1)
Requirement already satisfied: certifi==2020.6.20 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (2020.6.20)
Requirement already satisfied: PyYAML==5.3.1 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (5.3.1)
Requirement already satisfied: Naked==0.1.31 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (0.1.31)
Requirement already satisfied: chardet==3.0.4 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (3.0.4)
Requirement already satisfied: urllib3==1.25.9 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (1.25.9)
Requirement already satisfied: idna==2.9 in /Users/zmanji/.pyenv/versions/3.8.2/lib/python3.8/site-packages (from opvault==0.4.9) (2.9)
Installing collected packages: opvault
Successfully installed opvault-0.4.9
```

I can run it successfully:
```
 opvault-cli
Usage: /Users/zmanji/.pyenv/versions/3.8.2/bin/opvault-cli <path_to_opvault> <item_title>
```


Discussion:
Update:

If I run the same original command with an older pex (2.1.21) when I run the built pex I see:

```
./opvault.pex
Python 3.8.2 (default, Apr 19 2020, 14:12:44)
[Clang 11.0.3 (clang-1103.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
(ins)>>>
```

I would say this is a regression when bootstrapping the pex.
Update:

This is a regression between 2.1.24 and 2.1.25.

If I build the pex with 2.1.24 it works fine, but with 2.1.25 I get the observed error.
My reading of changed commits between the two releases seems like it is related to the code introduced in 5a8c50f5ad5a057a43f4e879de078ee1f5c01ffa. 
Beautiful. Naked 0.1.31 depends on ... Naked: https://github.com/chrissimpkins/naked/issues/19

Of course Pex should be robust against this.

Resolving patch:
diff --git a/pex/environment.py b/pex/environment.py
--- a/pex/environment.py
+++ b/pex/environment.py
@@ -481,6 +481,10 @@ def _resolve_requirement(
                 V=9,
             )
 
+        resolved_dists_by_key.update(
+            (key, resolved_distribution) for key in requirement_key.satisfied_keys()
+        )
+
         for dep_requirement in dist_metadata.requires_dists(resolved_distribution):
             # A note regarding extras and why they're passed down one level (we don't pass / use
             # dep_requirement.extras for example):
@@ -510,10 +514,6 @@ def _resolve_requirement(
             ):
                 yield not_found
 
-        resolved_dists_by_key.update(
-            (key, resolved_distribution) for key in requirement_key.satisfied_keys()
-        )
-
     def _root_requirements_iter(self, reqs):
         # type: (Iterable[Requirement]) -> Iterator[QualifiedRequirementOrNotFound]
 


Test patch:
diff --git a/tests/test_integration.py b/tests/test_integration.py
--- a/tests/test_integration.py
+++ b/tests/test_integration.py
@@ -3514,3 +3514,10 @@ def test_require_hashes(tmpdir):
         index + 1
         == error_lines["Got 2f4da4594db7e1e110a944bb1b551fdf4e6c136ad42e4234131391e21eb5b0df"]
     )
+
+
+def test_issues_1316_resolve_cyclic_dependency_graph(tmpdir):
+    # type: (Any) -> None
+    naked_pex = os.path.join(str(tmpdir), "naked.pex")
+    run_pex_command(args=["Naked==0.1.31", "-o", naked_pex]).assert_success()
+    subprocess.check_call(args=[naked_pex, "-c", "import Naked"])