Repository: pex-tool/pex

Issue:
Pex 2.1.93 `pex --lock` artifact downloading fails for sdists with ext_modules or PEP-517 build systems.
Even though we download lock artifacts using `pip download --no-deps`, Pip still tries to execute the build system. Maybe to gather metadata? I'm really not sure why it does this; no metadata is needed:
```
$ pex -V
2.1.93
$ pex3 lock create --no-wheel psutil==5.9.1 -o lock.json
```

lock.json:
```json
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": false,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "57f1819b5d9e95cdfb0c881a8a5b7d542ed0b7c522d575706a80bedc848c8954",
              "url": "https://files.pythonhosted.org/packages/d6/de/0999ea2562b96d7165812606b18f7169307b60cd378bc29cf3673322c7e9/psutil-5.9.1.tar.gz"
            }
          ],
          "project_name": "psutil",
          "requires_dists": [
            "enum34; python_version <= \"3.4\" and extra == \"test\"",
            "ipaddress; python_version < \"3.0\" and extra == \"test\"",
            "mock; python_version < \"3.0\" and extra == \"test\"",
            "pywin32; sys_platform == \"win32\" and extra == \"test\"",
            "wmi; sys_platform == \"win32\" and extra == \"test\""
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,>=2.7",
          "version": "5.9.1"
        }
      ],
      "platform_tag": [
        "cp311",
        "cp311",
        "manylinux_2_35_x86_64"
      ]
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.93",
  "prefer_older_binary": false,
  "requirements": [
    "psutil==5.9.1"
  ],
  "requires_python": [],
  "resolver_version": "pip-legacy-resolver",
  "style": "strict",
  "transitive": true,
  "use_pep517": null
}
```

Leads to:
```
$ rm -rf ~/.pex && pex --lock lock.json -o psutil.pex
There was 1 error downloading required artifacts:
1. psutil 5.9.1 from https://files.pythonhosted.org/packages/d6/de/0999ea2562b96d7165812606b18f7169307b60cd378bc29cf3673322c7e9/psutil-5.9.1.tar.gz
    ERROR: Command errored out with exit status 1: /home/jsirois/.pex/venvs/s/0490b204/venv/bin/python3 /home/jsirois/.pex/venvs/s/0490b204/venv/lib/python3.11/site-packages/pip install --ignore-installed --no-user --prefix /home/jsirois/.pex/pip_cache/.tmp/pip-build-env-whexefy7/overlay --no-warn-script-location --no-binary :none: --only-binary :none: --no-index -- 'setuptools>=40.8.0' wheel Check the logs for full command output.
```

So #1811, which switched Pex to using `pip download --no-deps` to download lock artifacts, fails for various sdists.


Discussion:
This was spotted over in https://github.com/pantsbuild/pants/issues/15990
Aha, the explanation is here: https://github.com/pypa/pip/issues/1884#issuecomment-941431284
Pip needs to generate metadata to discover `Requires-Python`. IOW even though we say `--no-deps`; so Pip does not need `Requires-Dist` metadata, it does reasonably need `Requires-Python` in the generic case of downloading a requirement. Granted our requirements in this case are explicit URLs.
But then you'd think this would work with modern Pip that supports `--ignore-requires-python`:
```
^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ python -mvenv pip.venv
^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ source pip.venv/bin/activate
(pip.venv) ^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ pip -q install -U pip
(pip.venv) ^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ pip -V
pip 22.1.2 from /home/jsirois/dev/pantsbuild/jsirois-pex/pip.venv/lib/python3.11/site-packages/pip (python 3.11)
(pip.venv) ^jsirois@gill ~/dev/pantsbuild/jsirois-pex (issues/1817 *) $ pip download --no-deps --ignore-requires-python --dest /tmp/pip-download https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz
Collecting https://files.pythonhosted.org/packages/a0/a4/d63f2d7597e1a4b55aa3b4d6c5b029991d3b824b5bd331af8d4ab1ed687d/PyYAML-5.4.1.tar.gz
  File was already downloaded /tmp/pip-download/PyYAML-5.4.1.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Successfully downloaded PyYAML
```
Aha, the more on-point explanation is here: https://github.com/pypa/pip/issues/1884#issuecomment-941244968 - basically, legacy code in Pip mooshes download and prepare together and that's hard to untangle easily.

Resolving patch:
diff --git a/pex/resolve/downloads.py b/pex/resolve/downloads.py
--- a/pex/resolve/downloads.py
+++ b/pex/resolve/downloads.py
@@ -40,21 +40,10 @@ def get_downloads_dir(pex_root=None):
     return downloads_dir
 
 
-def _strip_indexes_and_repos(package_index_configuration):
-    # type: (PackageIndexConfiguration) -> PackageIndexConfiguration
-    return PackageIndexConfiguration.create(
-        resolver_version=package_index_configuration.resolver_version,
-        indexes=[],
-        find_links=None,
-        network_configuration=package_index_configuration.network_configuration,
-        password_entries=package_index_configuration.password_entries,
-    )
-
-
 @attr.s(frozen=True)
 class ArtifactDownloader(object):
     package_index_configuration = attr.ib(
-        default=PackageIndexConfiguration.create(), converter=_strip_indexes_and_repos
+        default=PackageIndexConfiguration.create()
     )  # type: PackageIndexConfiguration
     target = attr.ib(default=LocalInterpreter.create())  # type: Target
     _pip = attr.ib(init=False)  # type: Pip
diff --git a/pex/resolve/lock_resolver.py b/pex/resolve/lock_resolver.py
--- a/pex/resolve/lock_resolver.py
+++ b/pex/resolve/lock_resolver.py
@@ -331,7 +331,8 @@ def resolve_from_lock(
             if package_index_configuration is None:
                 package_index_configuration = PackageIndexConfiguration.create(
                     resolver_version=resolver_version,
-                    indexes=[],
+                    indexes=indexes,
+                    find_links=find_links,
                     network_configuration=network_configuration,
                     password_entries=PasswordDatabase.from_netrc().append(password_entries).entries,
                 )


Test patch:
diff --git a/tests/integration/test_issue_1817.py b/tests/integration/test_issue_1817.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/test_issue_1817.py
@@ -0,0 +1,65 @@
+# Copyright 2022 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+import os
+import shutil
+
+from pex.cli.testing import run_pex3
+from pex.testing import run_pex_command
+from pex.typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+    from typing import Any
+
+
+def assert_create_and_use_sdist_lock(
+    tmpdir,  # type: Any
+    requirement,  # type: str
+    test,  # type: str
+):
+    # type: (...) -> None
+    pex_root = os.path.join(str(tmpdir), "pex_root")
+    lock = os.path.join(str(tmpdir), "lock.json")
+
+    run_pex3(
+        "lock",
+        "create",
+        "--pex-root",
+        pex_root,
+        "--no-wheel",
+        requirement,
+        "-o",
+        lock,
+        "--indent",
+        "2",
+    ).assert_success()
+
+    shutil.rmtree(pex_root)
+    run_pex_command(
+        args=[
+            "--pex-root",
+            pex_root,
+            "--runtime-pex-root",
+            pex_root,
+            "--lock",
+            lock,
+            "--",
+            "-c",
+            test,
+        ]
+    ).assert_success()
+
+
+def test_sdist_for_project_universal(tmpdir):
+    # type: (Any) -> None
+    assert_create_and_use_sdist_lock(tmpdir, "ansicolors==1.1.8", "import colors")
+
+
+def test_sdist_for_project_with_native_extensions(tmpdir):
+    # type: (Any) -> None
+    assert_create_and_use_sdist_lock(tmpdir, "psutil==5.9.1", "import psutil")
+
+
+def test_sdist_for_project_with_pep517_build(tmpdir):
+    # type: (Any) -> None
+    assert_create_and_use_sdist_lock(tmpdir, "PyYAML==5.4.1", "import yaml")