Repository: pex-tool/pex

Issue:
Building Pex with requirements.txt that includes local directory + Python version specifier fails
Assume I have a file, `reqs.txt` with the contents below:
```
mylibrary @ file:///Users/rajaiealkorani/code/mylibrary ; python_version >= "3.7" and python_version < "4.0"
```

Let's say I generated this file running `poetry export`.

If I run `pex --resolver-version pip-2020-resolver -r reqs.txt -o /tmp/testing.pex`, then run `/tmp/testing.pex`, and then `import mylibrary` in the Python interpreter, I get an error `No module named 'mylibrary'`

If I try building the Pex file using a reqs.txt as below, then I am able to properly import `mylibrary`
```
mylibrary @ file:///Users/rajaiealkorani/code/mylibrary"
```

It looks like adding Python version specifiers for the local package causes it not to get installed. Is this expected?

I am using Pex `2.1.120`


Discussion:
@rajaie-sg Is there any way you can provide a link to your repository? I can't repro using another Poetry-using repo. My steps:

The poetry-using project with a distribution name of `dagger-io` and a root package named `dagger` on that distribution
```
$ git clone https://github.com/dagger/dagger
$ grep -A28 "tool.poetry]" dagger/sdk/python/pyproject.toml
[tool.poetry]
name = "dagger-io"
version = "0.0.0"
description = "A client package for running Dagger pipelines in Python."
license = "Apache-2.0"
authors = ["Dagger <hello@dagger.io>"]
readme = "README.md"
homepage = "https://dagger.io"
documentation = "https://docs.dagger.io/sdk/python"
repository = "https://github.com/dagger/dagger/tree/main/sdk/python"
classifiers = [
    "Development Status :: 3 - Alpha",
    "Intended Audience :: Developers",
    "Framework :: AnyIO",
    "Framework :: Pytest",
    "Programming Language :: Python",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3 :: Only",
    "Programming Language :: Python :: 3.10",
    "Programming Language :: Python :: 3.11",
    "License :: OSI Approved :: Apache Software License",
    # FIXME: just waiting on windows tests for this
    # "Operating System :: OS Independent",
    "Typing :: Typed",
]
packages = [
    { include = "dagger", from = "src" },
]

$ ls -1 dagger/sdk/python/src/dagger/
api
cli.py
codegen.py
config.py
connection.py
context.py
engine
exceptions.py
__init__.py
log.py
__main__.py
py.typed
server
session.py
transport
```

Setup the same Pex scenario as yours:
```
$ pwd
/home/jsirois/support/pex/issue-2037
$ cat reqs.txt
dagger-io @ file:///home/jsirois/support/pex/issue-2037/dagger/sdk/python; python_version >= "3.7" and python_version < "4.0"
$ pex --resolver-version pip-2020-resolver -r reqs.txt -o testing.pex
$ ./testing.pex
Python 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> import dagger
>>> print(dagger.__file__)
/home/jsirois/.pex/installed_wheels/f484d70da8f2229255c496a2c36f193557e8fa503757d4250defbd1f57252cc7/dagger_io-0.0.0-py3-none-any.whl/dagger/__init__.py
>>>
now exiting InteractiveConsole...
```

So that worked fine.
The PEX-INFO:
```
$ pex-tools ./testing.pex info -i2
{
  "bootstrap_hash": "9ed54e64b9faf41fa3d45c68e63cf8ecfded096c",
  "build_properties": {
    "pex_version": "2.1.120"
  },
  "code_hash": "92b2fed89fd0041bfb5028089dddb2836d6b50c1",
  "distributions": {
    "Pygments-2.14.0-py3-none-any.whl": "73066efa74d4eb1e3e08fa2c826cc02e0a6d4747ccb109ba9f680cbb9c4ebe19",
    "anyio-3.6.2-py3-none-any.whl": "53a34c34cbe93a116a834b73d42831762deba88284d839485219d61a630e3fd4",
    "attrs-22.2.0-py3-none-any.whl": "b7d965a018ed5fbe33529f7ffdd208e203bd38ae6313018bfef8b830e329941c",
    "backoff-2.2.1-py3-none-any.whl": "5c174f49512471d2172cc79b1a753a61015dfca7d5ce0c0398d6807cb81960d8",
    "beartype-0.12.0-py3-none-any.whl": "fccc75121b5f869040b6770bb4cfbcb508f466ac2828fe21c08c88581a44a33d",
    "cattrs-22.2.0-py3-none-any.whl": "5fa669f1f898e6eadd0b6269ba35d4123a4f011de0aaeceac0652feb3bbe523f",
    "certifi-2022.12.7-py3-none-any.whl": "fa660808b834d51826f2d58643aad338e42b6d2bae69df8c8e5a4829dbeabdcf",
    "click-8.1.3-py3-none-any.whl": "78086359bc4a576338dbcaacad4a42784cdd0755b6327b984812fe0913265abf",
    "colorama-0.4.6-py2.py3-none-any.whl": "fd6b1d1a0d109220ed687cd5f6fc6d412195ff651bf4c2d489f43ad1692f9381",
    "commonmark-0.9.1-py2.py3-none-any.whl": "8b1e8f03d3f19f4d61005c0c0002b10c7e2e7ef1e384eae1100de1af0f50fd29",
    "dagger_io-0.0.0-py3-none-any.whl": "daf91252552c5629dd0287019edcbbd7a3c2a76d404a8785f3d66c58b09d414c",
    "exceptiongroup-1.1.0-py3-none-any.whl": "e5b0fb9e8292f20a4acde6ecb2561c7bbf6876fcab0f63488920364b089eab7a",
    "gql-3.4.0-py2.py3-none-any.whl": "0865932b918923c7cd5757e3185d4f3a2dee8d1f27d897596ae14e71a7d27f96",
    "graphql_core-3.2.3-py3-none-any.whl": "975acfdad27f28fe6ddd8e33db8bf9bab7e37504f6c945d4c346f2b9a5568ffe",
    "h11-0.14.0-py3-none-any.whl": "37ba404269b3b437819ec176737a4bd872c1b88cd0543d7b8067b86821221442",
    "httpcore-0.16.3-py3-none-any.whl": "1e3f389af00556ce91d5d52a6b0004005121425d6c9b71bd44cefe9f456f39ad",
    "httpx-0.23.3-py3-none-any.whl": "f64ef9925ce2d486a155f42f4dce5697f7a2d0ebe311166aad7cfd794f858f92",
    "idna-3.4-py3-none-any.whl": "3baea0ffbdea9f349783bacbdb82dbd50eadfec75339388aaf1597754786e3ed",
    "multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "ad5568ce92a1052a77bb75596402940b7edb4e80321a77f63dc448a7ddf54401",
    "platformdirs-2.6.2-py3-none-any.whl": "adfe2a4dd4d028a204cdc6a1e387e4bd9023727e0ec9c9f8b1b88affe2ff8f60",
    "rfc3986-1.5.0-py2.py3-none-any.whl": "38619c25a3ffb7cea10505911075fa58f6393e396d7ad81ba997a7ac4fd0e2dd",
    "rich-12.6.0-py3-none-any.whl": "ee715cddff848a868d2ecc40d4385a81b0ae6529e49a4d983b11291f61ecf27c",
    "shellingham-1.5.0.post1-py2.py3-none-any.whl": "93f1adcb68573b949d805d7d4662a2892467180569976f29d7e135edaf842f81",
    "sniffio-1.3.0-py3-none-any.whl": "feffd920899e99d56efbafd1469bc06ccab4337c224960b4ca43f5c937f56a6a",
    "typer-0.7.0-py3-none-any.whl": "2034b0e305ac03042430d375175c0c865b191edec5326ef46fa039ab2ef32d5b",
    "yarl-1.8.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "38499a67e9e8dd1b615e309e423f29e898738ebd234ac680011494e92033e22a"
  },
  "emit_warnings": true,
  "ignore_errors": false,
  "includes_tools": false,
  "inherit_path": "false",
  "inject_args": [],
  "inject_env": {},
  "interpreter_constraints": [],
  "pex_hash": "fd87987df00f543d01807227049405bad6f4f57a",
  "pex_path": "",
  "pex_paths": [],
  "requirements": [
    "dagger-io==0.0.0; python_version >= \"3.7\" and python_version < \"4.0\""
  ],
  "strip_pex_env": true,
  "venv": false,
  "venv_bin_path": "false",
  "venv_copies": false,
  "venv_site_packages_copies": false,
  "pex_root": "/home/jsirois/.pex"
}
```
I'm going to tentatively label this as a question since Pex appears to work just fine with file:// URL requirements and markers.
@jsirois thank you for your reply. It took me a quite a bit of trying to figure it out, and turns out it's the space before the semicolon that's causing the issue.

In your repro attempt, your `reqs.txt` doesn't contain a space before the semicolon. 
```
cat reqs.txt
dagger-io @ file:///home/jsirois/support/pex/issue-2037/dagger/sdk/python; python_version >= "3.7" and python_version < "4.0"
```

however, in my file, there is a space before the semicolon. I was using the output of `poetry export` which seems to have added that space [in this PR](https://github.com/python-poetry/poetry/pull/4733/files) to fix a related issue. So I wonder if this a bug in Pex?

When I run `pip install -r reqs.txt`, it works whether there is or isn't a space before the semicolon.
Ok. Yeah, that's a bug.

Resolving patch:
diff --git a/pex/requirements.py b/pex/requirements.py
--- a/pex/requirements.py
+++ b/pex/requirements.py
@@ -5,6 +5,7 @@
 
 import os
 import re
+import sys
 from contextlib import contextmanager
 
 from pex import attrs, dist_metadata, pex_warnings
@@ -396,6 +397,8 @@ def _try_parse_pip_local_formats(
         # + Environment markers denoted by `;...`
         #   See: https://www.python.org/dev/peps/pep-0508/#environment-markers
         r";",
+        # + A trailing whitespace character separating any of the above
+        r"\s",
     )
     # N.B.: The basename of the current directory (.) is '' and we allow this.
     match = re.match(
@@ -481,7 +484,9 @@ def _parse_requirement_line(
         specifier = None  # type: Optional[SpecifierSet]
         if not project_name:
             project_name_and_specifier = _try_parse_project_name_and_specifier_from_path(
-                unquote(parsed_url.path)
+                # There may be whitespace separated markers; so we strip the trailing whitespace
+                # used to support those.
+                unquote(parsed_url.path).rstrip()
             )
             if project_name_and_specifier is not None:
                 project_name = project_name_and_specifier.project_name
@@ -499,7 +504,9 @@ def _parse_requirement_line(
                     "#egg=<project name>."
                 ),
             )
-        url = parsed_url._replace(params="", fragment="").geturl()
+        # There may be whitespace separated markers; so we strip the trailing whitespace
+        # used to support those.
+        url = parsed_url._replace(params="", fragment="").geturl().rstrip()
         requirement = parse_requirement_from_project_name_and_specifier(
             project_name,
             extras=extras,


Test patch:
diff --git a/tests/test_requirements.py b/tests/test_requirements.py
--- a/tests/test_requirements.py
+++ b/tests/test_requirements.py
@@ -265,6 +265,7 @@ def test_parse_requirements_stress(chroot):
 
                 a/local/project[foo]; python_full_version == "2.7.8"
                 ./another/local/project;python_version == "2.7.*"
+                ./another/local/project ; python_version == "2.7.*"
                 ./another/local/project
                 ./
                 # Local projects with basenames that are invalid Python project names (trailing _):
@@ -274,18 +275,23 @@ def test_parse_requirements_stress(chroot):
 
                 hg+http://hg.example.com/MyProject@da39a3ee5e6b\\
                     #egg=AnotherProject[extra,more];python_version=="3.9.*"&subdirectory=foo/bar
+                hg+http://hg.example.com/MyProject@da39a3ee5e6b\\
+                    #egg=AnotherProject[extra,more] ; python_version=="3.9.*"&subdirectory=foo/bar
 
                 ftp://a/${PROJECT_NAME}-1.0.tar.gz
                 http://a/${PROJECT_NAME}-1.0.zip
                 https://a/numpy-1.9.2-cp34-none-win32.whl
                 https://a/numpy-1.9.2-cp34-none-win32.whl;\\
                     python_version=="3.4.*" and sys_platform=='win32'
+                https://a/numpy-1.9.2-cp34-none-win32.whl ; \\
+                    python_version=="3.4.*" and sys_platform=='win32'
 
                 Django@ git+https://github.com/django/django.git
                 Django@git+https://github.com/django/django.git@stable/2.1.x
                 Django@ git+https://github.com/django/django.git\\
                     @fd209f62f1d83233cc634443cfac5ee4328d98b8
                 Django @ file:projects/django-2.3.zip; python_version >= "3.10"
+                Django @ file:projects/django-2.3.zip ;python_version >= "3.10"
 
                 # Wheel with local version
                 http://download.pytorch.org/whl/cpu/torch-1.12.1%2Bcpu-cp310-cp310-linux_x86_64.whl
@@ -390,6 +396,10 @@ def test_parse_requirements_stress(chroot):
             path=os.path.realpath("extra/another/local/project"),
             marker="python_version == '2.7.*'",
         ),
+        local_req(
+            path=os.path.realpath("extra/another/local/project"),
+            marker="python_version == '2.7.*'",
+        ),
         local_req(path=os.path.realpath("extra/another/local/project")),
         local_req(path=os.path.realpath("extra")),
         local_req(path=os.path.realpath("extra/tmp/tmpW8tdb_")),
@@ -406,6 +416,13 @@ def test_parse_requirements_stress(chroot):
             extras=["more", "extra"],
             marker="python_version == '3.9.*'",
         ),
+        vcs_req(
+            vcs=VCS.Mercurial,
+            project_name="AnotherProject",
+            url="http://hg.example.com/MyProject@da39a3ee5e6b",
+            extras=["more", "extra"],
+            marker="python_version == '3.9.*'",
+        ),
         url_req(project_name="Project", url="ftp://a/Project-1.0.tar.gz", specifier="==1.0"),
         url_req(project_name="Project", url="http://a/Project-1.0.zip", specifier="==1.0"),
         url_req(
@@ -419,6 +436,12 @@ def test_parse_requirements_stress(chroot):
             specifier="==1.9.2",
             marker="python_version == '3.4.*' and sys_platform == 'win32'",
         ),
+        url_req(
+            project_name="numpy",
+            url="https://a/numpy-1.9.2-cp34-none-win32.whl",
+            specifier="==1.9.2",
+            marker="python_version == '3.4.*' and sys_platform == 'win32'",
+        ),
         vcs_req(vcs=VCS.Git, project_name="Django", url="https://github.com/django/django.git"),
         vcs_req(
             vcs=VCS.Git,
@@ -436,6 +459,12 @@ def test_parse_requirements_stress(chroot):
             specifier="==2.3",
             marker="python_version>='3.10'",
         ),
+        url_req(
+            project_name="Django",
+            url=os.path.realpath("extra/projects/django-2.3.zip"),
+            specifier="==2.3",
+            marker="python_version>='3.10'",
+        ),
         url_req(
             project_name="torch",
             url="http://download.pytorch.org/whl/cpu/torch-1.12.1%2Bcpu-cp310-cp310-linux_x86_64.whl",