Repository: pex-tool/pex

Issue:
pex pip resolver fails if a dependency is specified differently for different platforms
Hello there,

We recently bumped pex from 2.1.24 to 2.1.45 and now we observe problem with the following requirements file:

```
$ cat requirements.txt
https://<internal hostname>/pip/ray-0.8.4-cp37-cp37m-linux_x86_64.whl; sys_platform == 'linux'
ray==0.8.4; sys_platform == 'darwin'
```

when we call pex like this:

```
$ pex -r requirements.txt
Traceback (most recent call last):
  File "/Users/vitali/.pyenv/versions/pex-bug/bin/pex", line 8, in <module>
    sys.exit(main())
  File "/Users/vitali/.pyenv/versions/3.7.9/envs/pex-bug/lib/python3.7/site-packages/pex/bin/pex.py", line 1149, in main
    pex_builder = build_pex(options.requirements, options, cache=ENV.PEX_ROOT)
  File "/Users/vitali/.pyenv/versions/3.7.9/envs/pex-bug/lib/python3.7/site-packages/pex/bin/pex.py", line 1045, in build_pex
    ignore_errors=options.ignore_errors,
  File "/Users/vitali/.pyenv/versions/3.7.9/envs/pex-bug/lib/python3.7/site-packages/pex/resolver.py", line 1055, in resolve_multi
    ignore_errors=ignore_errors, workspace=workspace, max_parallel_jobs=max_parallel_jobs
  File "/Users/vitali/.pyenv/versions/3.7.9/envs/pex-bug/lib/python3.7/site-packages/pex/resolver.py", line 727, in install_distributions
    for index, direct_req in enumerate(direct_reqs)
AssertionError: More than one direct requirement is satisfied by ray 0.8.4:
0. ray
1. ray==0.8.4; sys_platform == "darwin"
This should never happen since Pip fails when more than one requirement for a given project name key is supplied and applies for a given target interpreter environment.
```

Let me provide details around our specifiers for `ray` package. For the linux environments (production installations run on linux) we use an internally built wheel, because we fixed a bug in ray which is still not fixed in their official version.
For local development on MacOS we're fine to work with the stock ray package, because this bug has only performance implications.

The workaround which I applied is to change our requirements file to download the ray package for both linux and darwin from a https url, something like this:

```
$ cat requirements.txt
https://<internal hostname>/pip/ray-0.8.4-cp37-cp37m-linux_x86_64.whl; sys_platform == 'linux'
https://<internal hostname>/pip/ray-0.8.4-cp37-cp37m-macosx_10_13_intel.whl; sys_platform == 'darwin'
```

Thank you,
Vitali


Discussion:
Thanks for reporting this @vitalihs - the issue is in Pex's requirement parser. It does not retain the environment marker for requirements of the form `https?://...; <marker>` like you're using here. This is easy to repro so I'll get a fix up today.

Resolving patch:
diff --git a/pex/requirements.py b/pex/requirements.py
--- a/pex/requirements.py
+++ b/pex/requirements.py
@@ -434,6 +434,11 @@ def _parse_requirement_line(
             if project_name_and_specifier is not None:
                 project_name = project_name_and_specifier.project_name
                 specifier = project_name_and_specifier.specifier
+        # Pip allows an environment marker after the url which matches the urlparse structure:
+        #   scheme://netloc/path;parameters?query#fragment
+        # See: https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urlparse
+        if not marker and parsed_url.params:
+            marker = Marker(parsed_url.params)
         if project_name is None:
             raise ParseError(
                 line,
@@ -442,7 +447,7 @@ def _parse_requirement_line(
                     "#egg=<project name>."
                 ),
             )
-        url = parsed_url._replace(fragment="").geturl()
+        url = parsed_url._replace(params="", fragment="").geturl()
         requirement = parse_requirement_from_project_name_and_specifier(
             project_name,
             extras=extras,


Test patch:
diff --git a/tests/test_requirements.py b/tests/test_requirements.py
--- a/tests/test_requirements.py
+++ b/tests/test_requirements.py
@@ -245,15 +245,19 @@ def test_parse_requirements_stress(chroot):
                 tmp/tmpW8tdb_[foo]
                 tmp/tmpW8tdb_[foo];python_version == "3.9"
 
-                hg+http://hg.example.com/MyProject@da39a3ee5e6b#egg=AnotherProject[extra,more];python_version=="3.9.*"&subdirectory=foo/bar
+                hg+http://hg.example.com/MyProject@da39a3ee5e6b\\
+                    #egg=AnotherProject[extra,more];python_version=="3.9.*"&subdirectory=foo/bar
 
                 ftp://a/${PROJECT_NAME}-1.0.tar.gz
                 http://a/${PROJECT_NAME}-1.0.zip
                 https://a/numpy-1.9.2-cp34-none-win32.whl
+                https://a/numpy-1.9.2-cp34-none-win32.whl;\\
+                    python_version=="3.4.*" and sys_platform=='win32'
 
                 Django@ git+https://github.com/django/django.git
                 Django@git+https://github.com/django/django.git@stable/2.1.x
-                Django@ git+https://github.com/django/django.git@fd209f62f1d83233cc634443cfac5ee4328d98b8
+                Django@ git+https://github.com/django/django.git\\
+                    @fd209f62f1d83233cc634443cfac5ee4328d98b8
                 Django @ file:projects/django-2.3.zip; python_version >= "3.10"
                 """
             )
@@ -377,6 +381,12 @@ def test_parse_requirements_stress(chroot):
             url="https://a/numpy-1.9.2-cp34-none-win32.whl",
             specifier="==1.9.2",
         ),
+        url_req(
+            project_name="numpy",
+            url="https://a/numpy-1.9.2-cp34-none-win32.whl",
+            specifier="==1.9.2",
+            marker="python_version == '3.4.*' and sys_platform == 'win32'",
+        ),
         url_req(project_name="Django", url="git+https://github.com/django/django.git"),
         url_req(project_name="Django", url="git+https://github.com/django/django.git@stable/2.1.x"),
         url_req(
diff --git a/tests/test_resolver.py b/tests/test_resolver.py
--- a/tests/test_resolver.py
+++ b/tests/test_resolver.py
@@ -4,6 +4,7 @@
 import functools
 import os
 import subprocess
+import sys
 from collections import defaultdict
 from textwrap import dedent
 
@@ -827,3 +828,26 @@ def test_resolve_from_pex_ignore_errors(
     assert len(resolved_distributions_by_key) > 1, "We should resolve at least requests and urllib3"
     assert "requests" in resolved_distributions_by_key
     assert Requirement.parse("urllib3==1.26.1") == resolved_distributions_by_key["urllib3"]
+
+
+def test_pip_proprietary_url_with_markers_issues_1415():
+    # https://<internal hostname>/pip/ray-0.8.4-cp37-cp37m-linux_x86_64.whl; sys_platform == 'linux'
+    # ray==0.8.4; sys_platform == 'darwin'
+    resolved_dists = resolve_multi(
+        requirements=[
+            (
+                "https://files.pythonhosted.org/packages/53/18/"
+                "a56e2fe47b259bb52201093a3a9d4a32014f9d85071ad07e9d60600890ca/"
+                "ansicolors-1.1.8-py2.py3-none-any.whl; sys_platform != '{}'".format(sys.platform)
+            ),
+            "ansicolors==1.1.8; sys_platform == '{}'".format(sys.platform),
+        ]
+    )
+    assert len(resolved_dists) == 1
+
+    resolved_dist = resolved_dists[0]
+    assert Requirement.parse("ansicolors==1.1.8") == resolved_dist.distribution.as_requirement()
+    assert (
+        Requirement.parse("ansicolors==1.1.8; sys_platform == '{}'".format(sys.platform))
+        == resolved_dist.direct_requirement
+    )