Repository: pex-tool/pex

Issue:
Lock creation not finishing
I tried running this with pants 2.11.0rc2, meaning Pex 2.1.79

```
["/opt/homebrew/bin/python3", "./pex", "--tmpdir", ".tmp", "lock", "create", "--output=lock.json", "--no-emit-warnings", "--style=universal", "--resolver-version", "pip-2020-resolver", "--indent=2", "--no-pypi", "--index=https://pypi.org/simple/", "--manylinux", "manylinux2014", "--interpreter-constraint", "CPython<3.10,>=3.8", "Django==4.0.4; python_version >= \"3.8\"", "Faker", "ansicolors==1.1.8", "beautifulsoup4==4.11.1", "boto3==1.21.39", "botocore==1.24.39", "bugout==0.1.20", "certifi", "chardet==4.0.0", "colorama", "cryptography==36.0.2", "defusedxml==0.7.1", "django-extensions==3.1.5", "django-prometheus==2.2.0", "django_csp==3.7", "djangorestframework==3.13.1", "djproxy==2.3.6", "docker==5.0.3", "drf-nested-routers==0.93.4", "duo-universal==2.0.1", "elasticsearch-dsl==7.4.0", "elasticsearch==7.5.1", "fasteners", "freezegun", "gitpython==3.1.27", "gunicorn==20.1.0", "hdrhistogram==0.9.1", "httpx==0.22.0", "humanize==4.0.0", "importlib-metadata", "influxdb-client==1.27.0", "jinja2==3.0.3", "jwcrypto==1.0.0", "kubernetes==23.3.0", "locust==2.8.6", "more-itertools==8.12.0", "moto[dynamodb2,ec2,s3,ses]==3.1.4", "ndjson==0.3.1", "networkx==2.7.0; python_version >= \"3.8\"", "opensearch-py==1.1.0", "opentelemetry-api", "opentelemetry-exporter-otlp-proto-grpc", "opentelemetry-instrumentation-botocore", "opentelemetry-instrumentation-django", "opentelemetry-instrumentation-elasticsearch", "opentelemetry-instrumentation-httpx", "opentelemetry-instrumentation-jinja2", "opentelemetry-instrumentation-psycopg2", "opentelemetry-sdk", "packaging", "pdpyras==4.5.0", "pex", "pkginfo==1.8.2", "plyvel==1.4.0", "prometheus-client==0.14.1", "psycopg2-binary==2.9.3", "py", "pyOpenSSL==22.0.0", "pygments==2.11.2", "pytest-django==4.5.2", "pytest-httpx==0.20.0", "pytest-responses==0.5.0", "pytest==7.1.1", "python-dateutil==2.8.2", "python-jose[cryptography]==3.3.0", "redis==4.2.2", "requests-aws4auth==1.1.2", "requests>=2.26.0", "rich==12.2.0", "ruamel.yaml==0.17.21", "selenium==4.1.3", "sentry-sdk==1.5.9", "setuptools<57.0,>=53.1.0", "shortuuid==1.0.8", "six", "social-auth-app-django==5.0.0", "social-auth-core==4.2.0", "stripe==2.71.0", "tomlkit==0.10.1", "twine==4.0.0", "typing-extensions", "urllib3[secure]==1.26.9", "voluptuous", "wheel", "yq==2.8.1", "zipp"]
```

It had been running for 15 minutes before I canceled it. I have not tried looking at any debug logs and I did not think to inspect top (my bad!). Generating with poetry took on a cold cache took close to 2 minutes iirc.

(sorry this is light on info, gotta run to doctor)



Discussion:
I added some logging to the runtime patch that tries to avoid working over full python versions here:
https://github.com/pantsbuild/pex/blob/8d0f40bbb9a0b8a355bc8e9e73786d365402e3d7/pex/pip/runtime_patches.py#L196-L203

That revealed:
```
$ tail -F /tmp/lock.log
tail: cannot open '/tmp/lock.log' for reading: No such file or directory
tail: '/tmp/lock.log' has appeared;  following new file
Python >=3.6.0: Candidate Python 3.10.2 caused needs full versions
```

If I make the code a bit more sophisticated and skip full versions if the operator is `<`, `<=`, `>`, or `>=` and the patch version is 0, the lock completes in 2:30 with a wiped pex root.

The culprits here were:
```
$ jq '.locked_resolves[].locked_requirements[] | select(.requires_python == ">=3.6.0")' lock.json
```
```json
{
  "artifacts": [
    {
      "algorithm": "sha256",
      "hash": "58d5c3d29f5a36ffeb94f02f0d786cd53014cf9b3b3951d42e0080d8a9498d30",
      "url": "https://files.pythonhosted.org/packages/9c/d8/909c4089dbe4ade9f9705f143c9f13f065049a9d5e7d34c828aefdd0a97c/beautifulsoup4-4.11.1-py3-none-any.whl"
    },
    {
      "algorithm": "sha256",
      "hash": "ad9aa55b65ef2808eb405f46cf74df7fcb7044d5cbc26487f96eb2ef2e436693",
      "url": "https://files.pythonhosted.org/packages/e8/b0/cd2b968000577ec5ce6c741a54d846dfa402372369b8b6861720aa9ecea7/beautifulsoup4-4.11.1.tar.gz"
    }
  ],
  "project_name": "beautifulsoup4",
  "requires_dists": [
    "html5lib; extra == \"html5lib\"",
    "lxml; extra == \"lxml\"",
    "soupsieve>1.2"
  ],
  "requires_python": ">=3.6.0",
  "version": "4.11.1"
}
{
  "artifacts": [
    {
      "algorithm": "sha256",
      "hash": "922c5f4edb3aa1beaa47bf61d65d5380011ff6adcd527f26377d05cb73ed8ec8",
      "url": "https://files.pythonhosted.org/packages/e2/a9/efeaeca4928a9a56d04d609b5730994d610c82cf4d9dd7aa173e6ef4233e/Rx-3.2.0-py3-none-any.whl"
    },
    {
      "algorithm": "sha256",
      "hash": "b657ca2b45aa485da2f7dcfd09fac2e554f7ac51ff3c2f8f2ff962ecd963d91c",
      "url": "https://files.pythonhosted.org/packages/34/b5/e0f602453b64b0a639d56f3c05ab27202a4eec993eb64d66c077c821b621/Rx-3.2.0.tar.gz"
    }
  ],
  "project_name": "rx",
  "requires_dists": [],
  "requires_python": ">=3.6.0",
  "version": "3.2"
}
```
So this particular slow lock is solved, but it seems ~any time a lock has to resort to using a full python version for `Requires-Python` evaluation, things will be slow. I'm not sure if there is anything I can do about that, but I'll ruminate and maybe file an issue if there is. If there isn't Pex could try and get this "Oh-oh, things may be slow" warning back to the user in real time, but that's complicated since all this data is in the Pip log and, the way things are structured now, the log data is only available in-program after Pip has completed.
Ah, ok, I can be alot smarter here. Doing so brings the time down to 1:20 and I'm pretty sure this ends this whac-a-mole.

Resolving patch:
diff --git a/pex/pip/runtime_patches.py b/pex/pip/runtime_patches.py
--- a/pex/pip/runtime_patches.py
+++ b/pex/pip/runtime_patches.py
@@ -170,8 +170,6 @@ def check_requires_python(requires_python, *_args, **_kw):
             orig_get_candidate_lookup = RequiresPythonRequirement.get_candidate_lookup
             orig_is_satisfied_by = RequiresPythonRequirement.is_satisfied_by
 
-            py_versions = []
-
             # Ensure we do a proper, but minimal, comparison for Python versions. Previously we
             # always tested all `Requires-Python` specifier sets against Python full versions. That
             # can be pathologically slow (see: https://github.com/pantsbuild/pants/issues/14998); so
@@ -185,22 +183,29 @@ def check_requires_python(requires_python, *_args, **_kw):
             # Do not need full versions to evaluate properly:
             # + Requires-Python: >=3.7,<4
             # + Requires-Python: ==3.7.*
+            # + Requires-Python: >=3.6.0
             #
             def needs_full_versions(spec):
                 components = spec.version.split(".", 2)
                 if len(components) < 3:
                     return False
                 major_, minor_, patch = components
+                if spec.operator in ("<", "<=", ">", ">=") and patch == "0":
+                    return False
                 return patch != "*"
 
             def _py_versions(self):
-                if not py_versions:
-                    py_versions.extend(
-                        python_full_versions
-                        if any(needs_full_versions(spec) for spec in self.specifier)
-                        else python_versions
+                if not hasattr(self, "__py_versions"):
+                    self.__py_versions = (
+                        version
+                        for version in (
+                            python_full_versions
+                            if any(needs_full_versions(spec) for spec in self.specifier)
+                            else python_versions
+                        )
+                        if ".".join(map(str, version)) in self.specifier
                     )
-                return py_versions
+                return self.__py_versions
 
             def get_candidate_lookup(self):
                 for py_version in self._py_versions():
diff --git a/pex/pip/tool.py b/pex/pip/tool.py
--- a/pex/pip/tool.py
+++ b/pex/pip/tool.py
@@ -925,7 +925,7 @@ def spawn_download_distributions(
         log = None
         popen_kwargs = {}
         if log_analyzers:
-            log = os.path.join(safe_mkdtemp(), "pip.log")
+            log = os.path.join(safe_mkdtemp(prefix="pex-pip-log"), "pip.log")
             download_cmd = ["--log", log] + download_cmd
             # N.B.: The `pip -q download ...` command is quiet but
             # `pip -q --log log.txt download ...` leaks download progress bars to stdout. We work


Test patch: