Repository: pex-tool/pex

Issue:
Fails to resolve when same requirement specific multiple times
Error in pants/pex when trying to package multiple modules that depend on each other, with shared external dependencies.

```
AssertionError: More than one direct requirement is satisfied by PyJWT 1.7.1:
0. PyJWT
1. PyJWT==1.7.1
This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment.
```
constraints.txt
```
PyJWT>=1.7.1,<2.0
```
requirements.txt #1
```
PyJWT
```

requirements.txt #2
```
PyJWT==1.7.1
```

To clarify, none of these requirements constraints are mutually exclusive and `pip` **does not** error when multiple of the same requirement are specified. This could potentially be an issue with how pants is passing in the requirements.

### Full traceback

```
stderr:
Traceback (most recent call last):
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 476, in execute
    exit_value = self._wrap_coverage(self._wrap_profiling, self._execute)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 401, in _wrap_coverage
    return runner(*args)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 432, in _wrap_profiling
    return runner(*args)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 532, in _execute
    return self.execute_entry(self._pex_info.entry_point)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 668, in execute_entry
    return self.execute_pkg_resources(entry_point)
  File "/home/noah/.cache/pants/named_caches/pex_root/unzipped_pexes/01a8d22400b26b1d132a9849fb6e28962f1ecf9e/.bootstrap/pex/pex.py", line 700, in execute_pkg_resources
    return runner()
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 666, in main
    do_main(
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 687, in do_main
    pex_builder = build_pex(
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/bin/pex.py", line 578, in build_pex
    result = resolve(
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/resolver.py", line 943, in resolve
    build_and_install_request.install_distributions(
  File "/home/noah/.cache/pants/named_caches/pex_root/installed_wheels/7fea57708be4117c2d962e01abd1585a8846dc06/pex-2.1.54-py2.py3-none-any.whl/pex/resolver.py", line 707, in install_distributions
    raise AssertionError(
AssertionError: More than one direct requirement is satisfied by PyJWT 1.7.1:
0. PyJWT
1. PyJWT==1.7.1
This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment.
```




Discussion:
@chiefnoah what version of Pants are you using?
```
[GLOBAL]
pants_version = "2.9.0.dev2"
```

We're currently using `2.2.0`, but I'm attempting to bump the version again (because that's very old and deprecated), but any version newer than that appears to have this problem.
Ok, thanks @chiefnoah. Could you share the PEX-INFO? If you're seeing this via `./pants run ...` you'd want to `./pants package ...` and then `unzip -qc dist/my.pex PEX-INFO`. If you get this error from some command other than `./pants run` can you provide a bit more detail? Perhaps the full logs from `./pants -ldebug --no-local-cache ...`.
I'm seeing this when trying to run `./pants package ::` to package all of our modules, and also when I attempt to package the specific module that's failing here. Adding `-ldebug --no-local-cache` doesn't add any additional output. 

The actual command appears to be:

```
  ProcessExecutionFailure: Process 'Building redacted.pex with 10 requirements: Flask, PyJWT, PyJWT==1.7.1, cachetools, cryptography, fire>=0.1.3, protobuf, requests==2.25.1, tabulate==0.8.7, typing_extensions>=3.7.4.3' failed with exit code 1.

...
<above stacktrace>
```

I can attempt to package another module and pull the PEX-INFO out from there, but I'm not sure how much that will help.

Thanks, that was enough. This is definitely a Pants bug passing the same requirement 2x.   I'll take a look over there and see if I can't see what conditions lead to that.
Alright, Pants makes no attempt to dedup project names, just whole requirements; so `PyJWT` and `PyJWT==1.7.1` are treated as distinct:
https://github.com/pantsbuild/pants/blob/0b1805739af59f76b1eee86d263dc4c25fe4dcef/src/python/pants/backend/python/util_rules/pex.py#L130-L141

A person sitting down and using the Pex CLI would ~never do this. That said, it appears you're correct re `This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment.` and that's definitely False.

To be clear, what's happening here is at PEX file boot time, the Pex runtime is trying to determine which distributions contained in the PEX file to activate and add to the sys.path. In so doing, its complaining about the same requirement being recorded 2x in its PEX-INFO. Let me dig in the Pex logic a bit here and see what can be done. This is not simple since a common case of multiple requirements is when you use environment markers to express "Choose this version for this environment and that version for that one.". In those cases you cannot collapse dup requirements but there is also no problem at runtime since only one of the two environments is active / present.
Aha, the `This should never happen ...` bit is, at least in part, true (The ERROR line is straight from Pip):
```
$ pex -V
2.1.59
$ pex --resolver-version pip-legacy-resolver PyJWT PyJWT==1.7.1 -o PyJWT.pex --include-tools
ERROR: Double requirement given: PyJWT==1.7.1 (already in PyJWT, name='PyJWT')
pid 25018 -> /home/jsirois/.pex/venvs/52ef3fa0fec32d58685d2164c25fd2d092f23324/9148e449944f6676b304f8f06462e05782fdb928/pex --disable-pip-version-check --no-python-version-warning --exists-action a --use-deprecated legacy-resolver --isolated -q --cache-dir /home/jsirois/.pex download --dest /tmp/tmp5mn2ez7g/home.jsirois..venv.pex.bin.python3 PyJWT PyJWT==1.7.1 --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
None
```

It's only using the new resolver, which Pants now does unconditionally, that it's false:
```
$ pex --resolver-version pip-2020-resolver PyJWT PyJWT==1.7.1 -o PyJWT.pex --include-tools
Traceback (most recent call last):
  File "/home/jsirois/.venv/pex/bin/pex", line 8, in <module>
    sys.exit(main())
  File "/home/jsirois/.venv/pex/lib/python3.10/site-packages/pex/bin/pex.py", line 683, in main
    do_main(
  File "/home/jsirois/.venv/pex/lib/python3.10/site-packages/pex/bin/pex.py", line 704, in do_main
    pex_builder = build_pex(
  File "/home/jsirois/.venv/pex/lib/python3.10/site-packages/pex/bin/pex.py", line 593, in build_pex
    result = resolve(
  File "/home/jsirois/.venv/pex/lib/python3.10/site-packages/pex/resolver.py", line 958, in resolve
    build_and_install_request.install_distributions(
  File "/home/jsirois/.venv/pex/lib/python3.10/site-packages/pex/resolver.py", line 722, in install_distributions
    raise AssertionError(
AssertionError: More than one direct requirement is satisfied by PyJWT 1.7.1:
0. PyJWT
1. PyJWT==1.7.1
This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment.
```
Interesting. For some additional context, what's effectively happening here is we have module A and module B. Module A depends on module B, and both depend on 3rd-party dependency C. Both module A and B exist in the same repo as independent source_roots and packaged separately (one as a `.pex`, another as `.whl`). It seems Pants tries to union all requirements from both modules at package-time and resolve the environment for the `.pex` (which makes sense), but it's not de-duplicating the common dependency. The existence of a `constraints.txt` should disambiguate the version it ultimately resolves and passes to `pex`. 

Should I create an issue on the pants repo?
> Should I create an issue on the pants repo?

I don't think so. You really can't dedup unless you are a resolver. Neither Pants nor Pex are resolvers. Pants delegates to Pex and Pex to Pip. Now Pip has enough context to dedup; i.e.: Its supposed to have enough logic to understand that `PyJWT` and `PyJWT==1.7.1` always dedups to `PyJWT==1.7.1` but that `PyJWT==1.7.1; python_version <= "3.7"` and `PyJWT>1.7.1; python_version > "3.7"` de-dups to one or the other depending on the interpreter being used to perform the resolve, etc.
I'll get this fixed up in Pex 2.1.60 and upgrade Pants to that for the 2.9.x series (It's on 2.9.0rc0 presently).

Resolving patch:
diff --git a/pex/bin/pex.py b/pex/bin/pex.py
--- a/pex/bin/pex.py
+++ b/pex/bin/pex.py
@@ -615,8 +615,8 @@ def build_pex(
                 pex_builder.add_distribution(
                     installed_dist.distribution, fingerprint=installed_dist.fingerprint
                 )
-                if installed_dist.direct_requirement:
-                    pex_builder.add_requirement(installed_dist.direct_requirement)
+                for direct_req in installed_dist.direct_requirements:
+                    pex_builder.add_requirement(direct_req)
         except Unsatisfiable as e:
             die(str(e))
 
diff --git a/pex/resolver.py b/pex/resolver.py
--- a/pex/resolver.py
+++ b/pex/resolver.py
@@ -5,6 +5,7 @@
 from __future__ import absolute_import
 
 import functools
+import itertools
 import os
 import zipfile
 from collections import OrderedDict, defaultdict
@@ -26,6 +27,7 @@
 from pex.resolve.requirement_configuration import RequirementConfiguration
 from pex.resolve.resolver_configuration import ResolverVersion
 from pex.resolve.target_configuration import TargetConfiguration
+from pex.sorted_tuple import SortedTuple
 from pex.third_party.pkg_resources import Distribution, Requirement
 from pex.tracer import TRACER
 from pex.typing import TYPE_CHECKING
@@ -52,6 +54,11 @@ class Unsatisfiable(ResolveError):
     pass
 
 
+def _sorted_requirements(requirements):
+    # type: (Optional[Iterable[Requirement]]) -> SortedTuple[Requirement]
+    return SortedTuple(requirements, key=lambda req: str(req)) if requirements else SortedTuple()
+
+
 @attr.s(frozen=True)
 class InstalledDistribution(object):
     """A distribution target, and the installed distribution that satisfies it.
@@ -62,7 +69,9 @@ class InstalledDistribution(object):
 
     target = attr.ib()  # type: DistributionTarget
     fingerprinted_distribution = attr.ib()  # type: FingerprintedDistribution
-    direct_requirement = attr.ib(default=None)  # type: Optional[Requirement]
+    direct_requirements = attr.ib(
+        converter=_sorted_requirements, factory=SortedTuple
+    )  # type: SortedTuple[Requirement]
 
     @property
     def distribution(self):
@@ -74,12 +83,15 @@ def fingerprint(self):
         # type: () -> str
         return self.fingerprinted_distribution.fingerprint
 
-    def with_direct_requirement(self, direct_requirement=None):
-        # type: (Optional[Requirement]) -> InstalledDistribution
-        if direct_requirement == self.direct_requirement:
+    def with_direct_requirements(self, direct_requirements=None):
+        # type: (Optional[Iterable[Requirement]]) -> InstalledDistribution
+        direct_requirements = _sorted_requirements(direct_requirements)
+        if direct_requirements == self.direct_requirements:
             return self
         return InstalledDistribution(
-            self.target, self.fingerprinted_distribution, direct_requirement=direct_requirement
+            self.target,
+            self.fingerprinted_distribution,
+            direct_requirements=direct_requirements,
         )
 
 
@@ -718,24 +730,8 @@ def add_installation(install_result):
                 if distribution_satisfies_requirement(distribution, req)
                 and installed_distribution.target.requirement_applies(req)
             ]
-            if len(direct_reqs) > 1:
-                raise AssertionError(
-                    "More than one direct requirement is satisfied by {distribution}:\n"
-                    "{requirements}\n"
-                    "This should never happen since Pip fails when more than one requirement for "
-                    "a given project name key is supplied and applies for a given target "
-                    "interpreter environment.".format(
-                        distribution=distribution,
-                        requirements="\n".join(
-                            "{index}. {direct_req}".format(index=index, direct_req=direct_req)
-                            for index, direct_req in enumerate(direct_reqs)
-                        ),
-                    )
-                )
             installed_distributions.add(
-                installed_distribution.with_direct_requirement(
-                    direct_requirement=direct_reqs[0] if direct_reqs else None
-                )
+                installed_distribution.with_direct_requirements(direct_requirements=direct_reqs)
             )
         return installed_distributions
 
@@ -1240,7 +1236,7 @@ def resolve_from_pex(
     )
     direct_requirements_by_project_name = (
         OrderedDict()
-    )  # type: OrderedDict[ProjectName, Requirement]
+    )  # type: OrderedDict[ProjectName, List[Requirement]]
     for direct_requirement in requirement_configuration.parse_requirements(
         network_configuration=network_configuration
     ):
@@ -1249,9 +1245,9 @@ def resolve_from_pex(
                 "Cannot resolve local projects from PEX repositories. Asked to resolve {path} "
                 "from {pex}.".format(path=direct_requirement.path, pex=pex)
             )
-        direct_requirements_by_project_name[
-            ProjectName(direct_requirement.requirement)
-        ] = direct_requirement.requirement
+        direct_requirements_by_project_name.setdefault(
+            ProjectName(direct_requirement.requirement), []
+        ).append(direct_requirement.requirement)
 
     constraints_by_project_name = defaultdict(
         list
@@ -1262,7 +1258,9 @@ def resolve_from_pex(
         ):
             constraints_by_project_name[ProjectName(contraint.requirement)].append(contraint)
 
-    all_reqs = direct_requirements_by_project_name.values()
+    all_reqs = OrderedSet(
+        itertools.chain.from_iterable(direct_requirements_by_project_name.values())
+    )
     unique_targets = TargetConfiguration(
         interpreters=interpreters, platforms=platforms, assume_manylinux=assume_manylinux
     ).unique_targets()
@@ -1276,8 +1274,8 @@ def resolve_from_pex(
 
         for fingerprinted_distribution in fingerprinted_distributions:
             project_name = fingerprinted_distribution.project_name
-            direct_requirement = direct_requirements_by_project_name.get(project_name, None)
-            if not transitive and not direct_requirement:
+            direct_requirements = direct_requirements_by_project_name.get(project_name, [])
+            if not transitive and not direct_requirements:
                 continue
 
             unmet_constraints = [
@@ -1299,7 +1297,7 @@ def resolve_from_pex(
                 InstalledDistribution(
                     target=target,
                     fingerprinted_distribution=fingerprinted_distribution,
-                    direct_requirement=direct_requirement,
+                    direct_requirements=direct_requirements,
                 )
             )
     return Resolved(installed_distributions=tuple(installed_distributions))


Test patch:
diff --git a/tests/integration/test_issue_1550.py b/tests/integration/test_issue_1550.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/test_issue_1550.py
@@ -0,0 +1,39 @@
+# Copyright 2021 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+import json
+import os.path
+import subprocess
+
+from pex import dist_metadata
+from pex.dist_metadata import ProjectNameAndVersion
+from pex.orderedset import OrderedSet
+from pex.pex_info import PexInfo
+from pex.testing import make_env, run_pex_command
+from pex.typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from typing import Any
+
+
+def test_duplicate_requirements_issues_1550(tmpdir):
+    # type: (Any) -> None
+
+    pex_file = os.path.join(str(tmpdir), "pex")
+    run_pex_command(
+        args=[
+            "PyJWT",
+            "PyJWT==1.7.1",
+            "--resolver-version",
+            "pip-2020-resolver",
+            "-o",
+            pex_file,
+        ]
+    ).assert_success()
+
+    subprocess.check_call(args=[pex_file, "-c", "import jwt"])
+    pex_info = PexInfo.from_pex(pex_file)
+    assert 1 == len(pex_info.distributions)
+    assert ProjectNameAndVersion("PyJWT", "1.7.1") == dist_metadata.project_name_and_version(
+        next(iter(pex_info.distributions.keys()))
+    )
+    assert OrderedSet(("PyJWT", "PyJWT==1.7.1")) == pex_info.requirements
diff --git a/tests/test_environment.py b/tests/test_environment.py
--- a/tests/test_environment.py
+++ b/tests/test_environment.py
@@ -140,8 +140,8 @@ def add_requirements(builder, cache):
             requirements, cache=cache, interpreters=[builder.interpreter]
         ).installed_distributions:
             builder.add_distribution(installed_dist.distribution)
-            if installed_dist.direct_requirement:
-                builder.add_requirement(installed_dist.direct_requirement)
+            for direct_req in installed_dist.direct_requirements:
+                builder.add_requirement(direct_req)
 
     def add_wheel(builder, content):
         # type: (PEXBuilder, Dict[str, str]) -> None
@@ -314,8 +314,8 @@ def test_activate_extras_issue_615():
         for installed_dist in resolver.resolve(
             ["pex[requests]==1.6.3"], interpreters=[pb.interpreter]
         ).installed_distributions:
-            if installed_dist.direct_requirement:
-                pb.add_requirement(installed_dist.direct_requirement)
+            for direct_req in installed_dist.direct_requirements:
+                pb.add_requirement(direct_req)
             pb.add_dist_location(installed_dist.distribution.location)
         pb.set_script("pex")
         pb.freeze()
diff --git a/tests/test_resolver.py b/tests/test_resolver.py
--- a/tests/test_resolver.py
+++ b/tests/test_resolver.py
@@ -17,6 +17,7 @@
 from pex.pex_builder import PEXBuilder
 from pex.pex_info import PexInfo
 from pex.platforms import Platform
+from pex.resolve.resolver_configuration import ResolverVersion
 from pex.resolver import (
     InstalledDistribution,
     IntegrityError,
@@ -27,6 +28,7 @@
     resolve,
     resolve_from_pex,
 )
+from pex.sorted_tuple import SortedTuple
 from pex.testing import (
     IS_LINUX,
     IS_PYPY,
@@ -111,7 +113,7 @@ def test_resolve_cache():
         assert installed_dists1 != installed_dists2
         assert len(installed_dists1) == 1
         assert len(installed_dists2) == 1
-        assert installed_dists1[0].direct_requirement == installed_dists2[0].direct_requirement
+        assert installed_dists1[0].direct_requirements == installed_dists2[0].direct_requirements
         assert (
             installed_dists1[0].distribution.location != installed_dists2[0].distribution.location
         )
@@ -530,13 +532,13 @@ def test_resolve_arbitrary_equality_issues_940():
     )
 
     assert len(installed_distributions) == 1
-    requirement = installed_distributions[0].direct_requirement
-    assert requirement is not None, (
+    requirements = installed_distributions[0].direct_requirements
+    assert 1 == len(requirements), (
         "The foo requirement was direct; so the resulting resolved distribution should carry the "
         "associated requirement."
     )
-    assert [("===", "1.0.2-fba4511")] == requirement.specs
-    assert requirement.marker is None
+    assert [("===", "1.0.2-fba4511")] == requirements[0].specs
+    assert requirements[0].marker is None
 
 
 def test_resolve_overlapping_requirements_discriminated_by_markers_issues_1196(py27):
@@ -550,9 +552,10 @@ def test_resolve_overlapping_requirements_discriminated_by_markers_issues_1196(p
     ).installed_distributions
     assert 1 == len(installed_distributions)
     installed_distribution = installed_distributions[0]
+    assert 1 == len(installed_distribution.direct_requirements)
     assert (
         Requirement.parse("setuptools<45; python_full_version == '2.7.*'")
-        == installed_distribution.direct_requirement
+        == installed_distribution.direct_requirements[0]
     )
     assert (
         Requirement.parse("setuptools==44.1.1")
@@ -579,8 +582,8 @@ def create_pex_repository(
         manylinux=manylinux,
     ).installed_distributions:
         pex_builder.add_distribution(installed_dist.distribution)
-        if installed_dist.direct_requirement:
-            pex_builder.add_requirement(installed_dist.direct_requirement)
+        for direct_req in installed_dist.direct_requirements:
+            pex_builder.add_requirement(direct_req)
     pex_builder.freeze()
     return os.path.realpath(cast(str, pex_builder.path()))
 
@@ -788,7 +791,8 @@ def test_resolve_from_pex_intransitive(
             Requirement.parse("requests==2.25.1")
             == installed_distribution.distribution.as_requirement()
         )
-        assert Requirement.parse("requests") == installed_distribution.direct_requirement
+        assert 1 == len(installed_distribution.direct_requirements)
+        assert Requirement.parse("requests") == installed_distribution.direct_requirements[0]
 
 
 def test_resolve_from_pex_constraints(
@@ -836,6 +840,7 @@ def test_resolve_from_pex_ignore_errors(
 
 
 def test_pip_proprietary_url_with_markers_issues_1415():
+    # type: () -> None
     installed_dists = resolve(
         requirements=[
             (
@@ -850,7 +855,27 @@ def test_pip_proprietary_url_with_markers_issues_1415():
 
     installed_dist = installed_dists[0]
     assert Requirement.parse("ansicolors==1.1.8") == installed_dist.distribution.as_requirement()
+    assert 1 == len(installed_dist.direct_requirements)
     assert (
         Requirement.parse("ansicolors==1.1.8; sys_platform == '{}'".format(sys.platform))
-        == installed_dist.direct_requirement
+        == installed_dist.direct_requirements[0]
+    )
+
+
+def test_duplicate_requirements_issues_1550():
+    # type: () -> None
+
+    with pytest.raises(Unsatisfiable):
+        resolve(requirements=["PyJWT", "PyJWT==1.7.1"], resolver_version=ResolverVersion.PIP_LEGACY)
+
+    installed_dists = resolve(
+        requirements=["PyJWT", "PyJWT==1.7.1"], resolver_version=ResolverVersion.PIP_2020
+    )
+    assert len(installed_dists.installed_distributions) == 1
+    installed_distribution = installed_dists.installed_distributions[0]
+    assert {Requirement.parse("PyJWT"), Requirement.parse("PyJWT==1.7.1")} == set(
+        installed_distribution.direct_requirements
     )
+    distribution = installed_distribution.distribution
+    assert "PyJWT" == distribution.project_name
+    assert "1.7.1" == distribution.version