Repository: pex-tool/pex

Issue:
KeyError when locking awscli on Python 3.11
```console
josh@cephandrius:~/.local/bin$ /home/josh/.local/bin/pex3 -V
2.1.141
```

```console
josh@cephandrius:~/.local/bin$ PEX_VERBOSE=2 /home/josh/.local/bin/pex3 lock create $'--output=lock.json' --pip-version 20.3.4-patched --resolver-version pip-2020-resolver  --indent=2 --no-pypi $'--index=https://pypi.org/simple/' --manylinux manylinux2014 --interpreter-constraint $'CPython==3.11.*' awscli==1.28.1
pex: Resolving interpreters: 41.1ms
pex: Resolving for:
pex: Hashing pex: 31.5ms                 
pex: Isolating pex: 0.0ms
pex: Resolving for:
  /usr/bin/python3.11: 8957.7ms
pex: Building setuptools.build_meta:__legacy__ build_backend PEX: 65.2ms                      
running egg_info
creating lib3/PyYAML.egg-info
writing lib3/PyYAML.egg-info/PKG-INFO
writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
writing top-level names to lib3/PyYAML.egg-info/top_level.txt
writing manifest file 'lib3/PyYAML.egg-info/SOURCES.txt'
reading manifest file 'lib3/PyYAML.egg-info/SOURCES.txt'
writing manifest file 'lib3/PyYAML.egg-info/SOURCES.txt'
running dist_info
creating /tmp/tmpm6_hjfef/build/PyYAML.egg-info
writing /tmp/tmpm6_hjfef/build/PyYAML.egg-info/PKG-INFO
writing dependency_links to /tmp/tmpm6_hjfef/build/PyYAML.egg-info/dependency_links.txt
writing top-level names to /tmp/tmpm6_hjfef/build/PyYAML.egg-info/top_level.txt
writing manifest file '/tmp/tmpm6_hjfef/build/PyYAML.egg-info/SOURCES.txt'
reading manifest file '/tmp/tmpm6_hjfef/build/PyYAML.egg-info/SOURCES.txt'
writing manifest file '/tmp/tmpm6_hjfef/build/PyYAML.egg-info/SOURCES.txt'
creating '/tmp/tmpm6_hjfef/build/PyYAML.dist-info'
pex: Creating lock from resolve: 516.4ms
pex:   Building 1 source distribution to gather metadata for lock.: 470.8ms
Traceback (most recent call last):
  File "/home/josh/.local/pipx/venvs/pex/lib/python3.8/site-packages/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/home/josh/.local/pipx/venvs/pex/lib/python3.8/site-packages/pex/cli/command.py", line 84, in run
    return subcommand_func(self)
  File "/home/josh/.local/pipx/venvs/pex/lib/python3.8/site-packages/pex/cli/commands/lock.py", line 482, in _create
    create(
  File "/home/josh/.local/pipx/venvs/pex/lib/python3.8/site-packages/pex/resolve/lockfile/create.py", line 432, in create
    create_lock_download_manager = CreateLockDownloadManager.create(
  File "/home/josh/.local/pipx/venvs/pex/lib/python3.8/site-packages/pex/resolve/lockfile/create.py", line 100, in create
    artifact_and_project_name = source_artifacts_by_pin[pin]
KeyError: Pin(project_name=ProjectName(raw='python_dateutil', normalized='python-dateutil'), version=Version(raw='2.8.2', normalized='2.8.2'))
Pin(project_name=ProjectName(raw='python_dateutil', normalized='python-dateutil'), version=Version(raw='2.8.2', normalized='2.8.2'))
```


Discussion:
(The command was mostly ripped from a `./pants generate-lockfiles` invocation.)
The issue here is triggered by https://github.com/yaml/pyyaml/issues/601 - this leads to a failure to build PyYAML 5.* most recents and backtracking to 5.3.1. The backtracks are not properly handled by the Pex locker.

A workaround until I can get out a fix is to either add a fake top-level dep on "PyYAML==5.3.1" or else a `--constraints`.

In my test rig I repro prior to adding `--constraints` and get success after adding them:
```bash
#!/usr/bin/env bash

set -euo pipefail

VENV="$(mktemp -d)"
trap "rm -rf ${VENV}" EXIT

python3.8 -mvenv "${VENV}"
"${VENV}/bin/pip" install -U pip
"${VENV}/bin/pip" install -U pex

"${VENV}/bin/pex" -V

echo "PyYAML==5.3.1" > "${VENV}/constraints.txt"
"${VENV}/bin/pex3" \
    lock create \
    -v \
    -o lock.json \
    --indent 2 \
    --pip-version vendored \
    --resolver-version pip-2020-resolver \
    --interpreter-constraint "CPython==3.11.*" \
    --style universal \
    --target-system linux \
    --target-system mac \
    --manylinux manylinux2014 \
    --preserve-pip-download-log \
    --constraints "${VENV}/constraints.txt" \
    awscli==1.28.1
```

With `--constraints ...` commented out:
```console
$ ./repro.sh
Requirement already satisfied: pip in /tmp/tmp.w3puaX4Nvu/lib/python3.8/site-packages (23.0.1)
Collecting pip
  Using cached pip-23.2.1-py3-none-any.whl (2.1 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 23.0.1
    Uninstalling pip-23.0.1:
      Successfully uninstalled pip-23.0.1
Successfully installed pip-23.2.1
Collecting pex
  Obtaining dependency information for pex from https://files.pythonhosted.org/packages/30/db/f13bdcd7c770a9f4eca7589a86992b55b8977416a3333c169ebf761382a7/pex-2.1.141-py2.py3-none-any.whl.metadata
  Using cached pex-2.1.141-py2.py3-none-any.whl.metadata (7.3 kB)
Using cached pex-2.1.141-py2.py3-none-any.whl (2.9 MB)
Installing collected packages: pex
Successfully installed pex-2.1.141
2.1.141
pex: Resolving for:
pex: Calculating compatible python versions for ('==3.11.*',): 8.0ms
pex: Hashing pex: 18.3ms
pex: Isolating pex: 0.0ms
pex: Preserving `pip download` log at /tmp/pex-pip-log.sljn2h1x/pip.log
pex: Building setuptools.build_meta:__legacy__ build_backend PEX: 43.1ms
running egg_info
creating lib3/PyYAML.egg-info
writing lib3/PyYAML.egg-info/PKG-INFO
writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
writing top-level names to lib3/PyYAML.egg-info/top_level.txt
writing manifest file 'lib3/PyYAML.egg-info/SOURCES.txt'
reading manifest file 'lib3/PyYAML.egg-info/SOURCES.txt'
writing manifest file 'lib3/PyYAML.egg-info/SOURCES.txt'
running dist_info
creating /tmp/tmpz7a9yjvh/build/PyYAML.egg-info
writing /tmp/tmpz7a9yjvh/build/PyYAML.egg-info/PKG-INFO
writing dependency_links to /tmp/tmpz7a9yjvh/build/PyYAML.egg-info/dependency_links.txt
writing top-level names to /tmp/tmpz7a9yjvh/build/PyYAML.egg-info/top_level.txt
writing manifest file '/tmp/tmpz7a9yjvh/build/PyYAML.egg-info/SOURCES.txt'
reading manifest file '/tmp/tmpz7a9yjvh/build/PyYAML.egg-info/SOURCES.txt'
writing manifest file '/tmp/tmpz7a9yjvh/build/PyYAML.egg-info/SOURCES.txt'
creating '/tmp/tmpz7a9yjvh/build/PyYAML.dist-info'
Traceback (most recent call last):
  File "/tmp/tmp.w3puaX4Nvu/lib/python3.8/site-packages/pex/result.py", line 105, in catch
    return func(*args, **kwargs)
  File "/tmp/tmp.w3puaX4Nvu/lib/python3.8/site-packages/pex/cli/command.py", line 84, in run
    return subcommand_func(self)
  File "/tmp/tmp.w3puaX4Nvu/lib/python3.8/site-packages/pex/cli/commands/lock.py", line 482, in _create
    create(
  File "/tmp/tmp.w3puaX4Nvu/lib/python3.8/site-packages/pex/resolve/lockfile/create.py", line 432, in create
    create_lock_download_manager = CreateLockDownloadManager.create(
  File "/tmp/tmp.w3puaX4Nvu/lib/python3.8/site-packages/pex/resolve/lockfile/create.py", line 100, in create
    artifact_and_project_name = source_artifacts_by_pin[pin]
KeyError: Pin(project_name=ProjectName(raw='colorama', normalized='colorama'), version=Version(raw='0.4.4', normalized='0.4.4'))
Pin(project_name=ProjectName(raw='colorama', normalized='colorama'), version=Version(raw='0.4.4', normalized='0.4.4'))
```

With `--constraints ...` present:
```json
{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [
    "PyYAML==5.3.1"
  ],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "2b0c5cdb25a1d3b56921777596473af2b1314f0059ea451eccb0c3ef72b602b9",
              "url": "https://files.pythonhosted.org/packages/fc/58/aa249041062eef0b0558ca6e198ccc622d4107523a4773c3108985f9deea/awscli-1.28.1-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "4d4841058fe87cc70b8deb27b261cf64337be05d36f45c2abb5078ba8a1e07ef",
              "url": "https://files.pythonhosted.org/packages/a7/cc/9973fa0ae4544b34b4bbe77e6656edcbc79943fcd7311303c70adeb4837a/awscli-1.28.1.tar.gz"
            }
          ],
          "project_name": "awscli",
          "requires_dists": [
            "PyYAML<5.5,>=3.10",
            "botocore==1.30.1",
            "colorama<0.4.5,>=0.2.5",
            "docutils<0.17,>=0.10",
            "rsa<4.8,>=3.1.2",
            "s3transfer<0.7.0,>=0.6.0"
          ],
          "requires_python": ">=3.7",
          "version": "1.28.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "18a32a21bfa9b418b9a38ea5ef4464eba003cbb26fca2cd56e4f51098c5d1a0f",
              "url": "https://files.pythonhosted.org/packages/96/f0/db8b01860d2e9b259831f060a63d7d8d8b6c50b2efd656ac66152d3dbe1c/botocore-1.30.1-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "4d1ac5a796c5c5c87946f25f3d98764288a0ed848e772a7a47cd134847e885e7",
              "url": "https://files.pythonhosted.org/packages/24/57/ee9a4b122f697477bf4d0d9f0dc7951a494937cf78f4321a6c5d3d71d99e/botocore-1.30.1.tar.gz"
            }
          ],
          "project_name": "botocore",
          "requires_dists": [
            "awscrt==0.16.9; extra == \"crt\"",
            "jmespath<2.0.0,>=0.7.1",
            "python-dateutil<3.0.0,>=2.1",
            "urllib3<1.27,>=1.25.4"
          ],
          "requires_python": ">=3.7",
          "version": "1.30.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "9f47eda37229f68eee03b24b9748937c7dc3868f906e8ba69fbcbdd3bc5dc3e2",
              "url": "https://files.pythonhosted.org/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "5941b2b48a20143d2267e95b1c2a7603ce057ee39fd88e7329b0c292aa16869b",
              "url": "https://files.pythonhosted.org/packages/1f/bb/5d3246097ab77fa083a61bd8d3d527b7ae063c7d8e8671b1cf8c4ec10cbe/colorama-0.4.4.tar.gz"
            }
          ],
          "project_name": "colorama",
          "requires_dists": [],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7",
          "version": "0.4.4"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "0c5b78adfbf7762415433f5515cd5c9e762339e23369dbe8000d84a4bf4ab3af",
              "url": "https://files.pythonhosted.org/packages/81/44/8a15e45ffa96e6cf82956dd8d7af9e666357e16b0d93b253903475ee947f/docutils-0.16-py2.py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "c2de3a60e9e7d07be26b7f2b00ca0309c207e06c100f9cc2a94931fc75a478fc",
              "url": "https://files.pythonhosted.org/packages/2f/e0/3d435b34abd2d62e8206171892f174b180cd37b09d57b924ca5c2ef2219d/docutils-0.16.tar.gz"
            }
          ],
          "project_name": "docutils",
          "requires_dists": [],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7",
          "version": "0.16"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "02e2e4cc71b5bcab88332eebf907519190dd9e6e82107fa7f83b1003a6252980",
              "url": "https://files.pythonhosted.org/packages/31/b4/b9b800c45527aadd64d5b442f9b932b00648617eb5d63d2c7a6587b7cafc/jmespath-1.0.1-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "90261b206d6defd58fdd5e85f478bf633a2901798906be2ad389150c5c60edbe",
              "url": "https://files.pythonhosted.org/packages/00/2a/e867e8531cf3e36b41201936b7fa7ba7b5702dbef42922193f05c8976cd6/jmespath-1.0.1.tar.gz"
            }
          ],
          "project_name": "jmespath",
          "requires_dists": [],
          "requires_python": ">=3.7",
          "version": "1.0.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "87a2121042a1ac9358cabcaf1d07680ff97ee6404333bacca15f76aa8ad01a57",
              "url": "https://files.pythonhosted.org/packages/14/e5/b56a725cbde139aa960c26a1a3ca4d4af437282e20b5314ee6a3501e7dfc/pyasn1-0.5.0-py2.py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "97b7290ca68e62a832558ec3976f15cbf911bf5d7c7039d8b861c2a0ece69fde",
              "url": "https://files.pythonhosted.org/packages/61/ef/945a8bcda7895717c8ba4688c08a11ef6454f32b8e5cb6e352a9004ee89d/pyasn1-0.5.0.tar.gz"
            }
          ],
          "project_name": "pyasn1",
          "requires_dists": [],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7",
          "version": "0.5.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "961d03dc3453ebbc59dbdea9e4e11c5651520a876d0f4db161e8674aae935da9",
              "url": "https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "0123cacc1627ae19ddf3c27a5de5bd67ee4586fbdd6440d9748f8abb483d3e86",
              "url": "https://files.pythonhosted.org/packages/4c/c4/13b4776ea2d76c115c1d1b84579f3764ee6d57204f6be27119f13a61d0a9/python-dateutil-2.8.2.tar.gz"
            }
          ],
          "project_name": "python-dateutil",
          "requires_dists": [
            "six>=1.5"
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7",
          "version": "2.8.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "b8eac752c5e14d3eca0e6dd9199cd627518cb5ec06add0de9d32baeee6fe645d",
              "url": "https://files.pythonhosted.org/packages/64/c2/b80047c7ac2478f9501676c988a5411ed5572f35d1beff9cae07d321512c/PyYAML-5.3.1.tar.gz"
            }
          ],
          "project_name": "pyyaml",
          "requires_dists": [],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,>=2.7",
          "version": "5.3.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "78f9a9bf4e7be0c5ded4583326e7461e3a3c5aae24073648b4bdfa797d78c9d2",
              "url": "https://files.pythonhosted.org/packages/e9/93/0c0f002031f18b53af7a6166103c02b9c0667be528944137cc954ec921b3/rsa-4.7.2-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "9d689e6ca1b3038bc82bf8d23e944b6b6037bc02301a574935b2dd946e0353b9",
              "url": "https://files.pythonhosted.org/packages/db/b5/475c45a58650b0580421746504b680cd2db4e81bc941e94ca53785250269/rsa-4.7.2.tar.gz"
            }
          ],
          "project_name": "rsa",
          "requires_dists": [
            "pyasn1>=0.1.3"
          ],
          "requires_python": "<4,>=3.5",
          "version": "4.7.2"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "3c0da2d074bf35d6870ef157158641178a4204a6e689e82546083e31e0311346",
              "url": "https://files.pythonhosted.org/packages/ec/fa/9416461ee05efabe477d588288bdd88acb69a51ee4b31a9b73d3b5b716fc/s3transfer-0.6.1-py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "640bb492711f4c0c0905e1f62b6aaeb771881935ad27884852411f8e9cacbca9",
              "url": "https://files.pythonhosted.org/packages/49/bd/def2ab4c04063a5e114963aae90bcd3e3aca821a595124358b3b00244407/s3transfer-0.6.1.tar.gz"
            }
          ],
          "project_name": "s3transfer",
          "requires_dists": [
            "botocore<2.0a.0,>=1.12.36",
            "botocore[crt]<2.0a.0,>=1.20.29; extra == \"crt\""
          ],
          "requires_python": ">=3.7",
          "version": "0.6.1"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254",
              "url": "https://files.pythonhosted.org/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "1e61c37477a1626458e36f7b1d82aa5c9b094fa4802892072e49de9c60c4c926",
              "url": "https://files.pythonhosted.org/packages/71/39/171f1c67cd00715f190ba0b100d606d440a28c93c7714febeca8b79af85e/six-1.16.0.tar.gz"
            }
          ],
          "project_name": "six",
          "requires_dists": [],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,>=2.7",
          "version": "1.16.0"
        },
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "8d36afa7616d8ab714608411b4a3b13e58f463aee519024578e062e141dce20f",
              "url": "https://files.pythonhosted.org/packages/c5/05/c214b32d21c0b465506f95c4f28ccbcba15022e000b043b72b3df7728471/urllib3-1.26.16-py2.py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "8f135f6502756bde6b2a9b28989df5fbe87c9970cecaa69041edcce7f0589b14",
              "url": "https://files.pythonhosted.org/packages/e2/7d/539e6f0cf9f0b95b71dd701a56dae89f768cd39fd8ce0096af3546aeb5a3/urllib3-1.26.16.tar.gz"
            }
          ],
          "project_name": "urllib3",
          "requires_dists": [
            "PySocks!=1.5.7,<2.0,>=1.5.6; extra == \"socks\"",
            "brotli>=1.0.9; ((os_name != \"nt\" or python_version >= \"3\") and platform_python_implementation == \"CPython\") and extra == \"brotli\"",
            "brotlicffi>=0.8.0; ((os_name != \"nt\" or python_version >= \"3\") and platform_python_implementation != \"CPython\") and extra == \"brotli\"",
            "brotlipy>=0.6.0; (os_name == \"nt\" and python_version < \"3\") and extra == \"brotli\"",
            "certifi; extra == \"secure\"",
            "cryptography>=1.3.4; extra == \"secure\"",
            "idna>=2.0.0; extra == \"secure\"",
            "ipaddress; python_version == \"2.7\" and extra == \"secure\"",
            "pyOpenSSL>=0.14; extra == \"secure\"",
            "urllib3-secure-extra; extra == \"secure\""
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*,>=2.7",
          "version": "1.26.16"
        }
      ],
      "platform_tag": null
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.141",
  "pip_version": "20.3.4-patched",
  "prefer_older_binary": false,
  "requirements": [
    "awscli==1.28.1"
  ],
  "requires_python": [
    "==3.11.*"
  ],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "target_systems": [
    "linux",
    "mac"
  ],
  "transitive": true,
  "use_pep517": null
}
```
Although I have a fix for this issue handling backtracking with build failures, I will note this particular issue is problematic generally. With modern Pip (I use `--pip-version latest` here), you get a hard failure:
```console
...
pex: Preserving `pip download` log at /tmp/pex-pip-log.g51qskjv/pip.log
pid 586513 -> /home/jsirois/.pex/venvs/a29577f0f0d47aa1d10d161479623d0bbf1b234d/cac31ab188cffcbf56f7e1032af40222ace829a3/bin/python -sE /home/jsirois/.pex/venvs/a29577f0f0d47aa1d10d161479623d0bbf1b234d/cac31ab188cffcbf56f7e1032af40222ace829a3/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --isolated -q --cache-dir /home/jsirois/.pex/pip/23.2/pip_cache --log /tmp/pex-pip-log.g51qskjv/pip.log download --dest /tmp/tmpexkdwid_/home.jsirois..pyenv.versions.3.11.4.bin.python3.11 awscli==1.28.1 --index-url https://pypi.org/simple --retries 5 --timeout 15 exited with 1 and STDERR:
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
```

I experimented and `pip==20.*` (20.3.4) and `pip==21.*` (21.3.1) both work, but starting with `pip==22.0` you get:
```console
$ repro.venv/bin/python -V
Python 3.11.4
$ repro.venv/bin/pip -V
pip 22.0 from /home/jsirois/dev/pantsbuild/jsirois-pex/repro.venv/lib/python3.11/site-packages/pip (python 3.11)
$ repro.venv/bin/pip download awscli==1.28.1Collecting awscli==1.28.1
  File was already downloaded /home/jsirois/dev/pantsbuild/jsirois-pex/awscli-1.28.1-py3-none-any.whl
Collecting PyYAML<5.5,>=3.10
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [68 lines of output]
      /tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!

              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.

              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.

              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

      !!
        parsed = self.parsers.get(option_name, lambda x: x)(value)
      running egg_info
      writing lib3/PyYAML.egg-info/PKG-INFO
      writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt
      writing top-level names to lib3/PyYAML.egg-info/top_level.txt
      Traceback (most recent call last):
        File "/home/jsirois/dev/pantsbuild/jsirois-pex/repro.venv/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
          main()
        File "/home/jsirois/dev/pantsbuild/jsirois-pex/repro.venv/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/jsirois/dev/pantsbuild/jsirois-pex/repro.venv/lib/python3.11/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 341, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 323, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in run_setup
          exec(code, locals())
        File "<string>", line 271, in <module>
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/__init__.py", line 107, in setup
          return distutils.core.setup(**attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
                 ^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/dist.py", line 1234, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 314, in run
          self.find_sources()
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 322, in find_sources
          mm.run()
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 551, in run
          self.add_defaults()
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 589, in add_defaults
          sdist.add_defaults(self)
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/command/sdist.py", line 104, in add_defaults
          super().add_defaults()
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
          self._add_defaults_ext()
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
          self.filelist.extend(build_ext.get_source_files())
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<string>", line 201, in get_source_files
        File "/tmp/pip-build-env-099bvt7k/overlay/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
          raise AttributeError(attr)
      AttributeError: cython_sources
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
WARNING: You are using pip version 22.0; however, version 23.2.1 is available.
You should consider upgrading via the '/home/jsirois/dev/pantsbuild/jsirois-pex/repro.venv/bin/python3.11 -m pip install --upgrade pip' command.
```

More experimentation reveals this does not work using the legacy resolver either; so if you want `awscli==1.28.1` you can only get it today with the Pips in the range `>=20.3.2,<22`. Not good!


Resolving patch:
diff --git a/pex/resolve/locker.py b/pex/resolve/locker.py
--- a/pex/resolve/locker.py
+++ b/pex/resolve/locker.py
@@ -217,12 +217,12 @@ class ArtifactBuildResult(object):
 
 @attr.s(frozen=True)
 class ArtifactBuildObserver(object):
-    _done_building_pattern = attr.ib()  # type: Pattern
+    _done_building_patterns = attr.ib()  # type: Iterable[Pattern]
     _artifact_url = attr.ib()  # type: ArtifactURL
 
     def is_done_building(self, line):
         # type: (str) -> bool
-        return self._done_building_pattern.search(line) is not None
+        return any(pattern.search(line) is not None for pattern in self._done_building_patterns)
 
     def build_result(self, line):
         # type: (str) -> Optional[ArtifactBuildResult]
@@ -315,16 +315,17 @@ def analyze(self, line):
         # The log sequence for processing a resolved requirement is as follows (log lines irrelevant
         # to our purposes omitted):
         #
-        #   1.)   "... Found link <url1> ..."
+        #   1.)       "... Found link <url1> ..."
         #   ...
-        #   1.)   "... Found link <urlN> ..."
-        #   2.)   "... Added <varying info ...> to build tracker ..."
-        # * 3.)   Lines related to extracting metadata from <requirement> if the selected
-        #         distribution is an sdist in any form (VCS, local directory, source archive).
-        # * 3.5.) "... Source in <tmp> has version <version>, which satisfies requirement "
-        #         "<requirement> from <url> ..."
-        #   4.)   "... Removed <requirement> from <url> ... from build tracker ..."
-        #   5.)   "... Saved <download dir>/<artifact file>
+        #   1.)       "... Found link <urlN> ..."
+        #   2.)       "... Added <varying info ...> to build tracker ..."
+        # * 3.)       Lines related to extracting metadata from <requirement> if the selected
+        #             distribution is an sdist in any form (VCS, local directory, source archive).
+        # * 3.5. ERR) "... WARNING: Discarding <url> <varying info...>. Command errored out with ...
+        # * 3.5. SUC) "... Source in <tmp> has version <version>, which satisfies requirement "
+        #             "<requirement> from <url> ..."
+        #   4.)       "... Removed <requirement> from <url> ... from build tracker ..."
+        #   5.)       "... Saved <download dir>/<artifact file>
 
         # The lines in section 3 can contain this same pattern of lines if the metadata extraction
         # proceeds via PEP-517 which recursively uses Pip to resolve build dependencies. We want to
@@ -451,10 +452,13 @@ def analyze(self, line):
                 self._selected_path_to_pin[os.path.basename(url.path)] = pin
             else:
                 self._artifact_build_observer = ArtifactBuildObserver(
-                    done_building_pattern=re.compile(
-                        r"Removed {requirement} from {url} (?:.* )?from build tracker".format(
-                            requirement=re.escape(raw_requirement), url=re.escape(url.raw_url)
-                        )
+                    done_building_patterns=(
+                        re.compile(
+                            r"Removed {requirement} from {url} (?:.* )?from build tracker".format(
+                                requirement=re.escape(raw_requirement), url=re.escape(url.raw_url)
+                            )
+                        ),
+                        re.compile(r"WARNING: Discarding {url}".format(url=re.escape(url.raw_url))),
                     ),
                     artifact_url=url,
                 )
@@ -464,10 +468,13 @@ def analyze(self, line):
         if match:
             file_url = match.group("file_url")
             self._artifact_build_observer = ArtifactBuildObserver(
-                done_building_pattern=re.compile(
-                    r"Removed .+ from {file_url} from build tracker".format(
-                        file_url=re.escape(file_url)
-                    )
+                done_building_patterns=(
+                    re.compile(
+                        r"Removed .+ from {file_url} from build tracker".format(
+                            file_url=re.escape(file_url)
+                        )
+                    ),
+                    re.compile(r"WARNING: Discarding {url}".format(url=re.escape(file_url))),
                 ),
                 artifact_url=ArtifactURL.parse(file_url),
             )


Test patch:
diff --git a/tests/integration/cli/commands/test_issue_2211.py b/tests/integration/cli/commands/test_issue_2211.py
new file mode 100644
--- /dev/null
+++ b/tests/integration/cli/commands/test_issue_2211.py
@@ -0,0 +1,79 @@
+# Copyright 2023 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+import os.path
+
+import pytest
+
+from pex.interpreter_constraints import InterpreterConstraint
+from pex.pep_440 import Version
+from pex.pip.version import PipVersion, PipVersionValue
+from pex.typing import TYPE_CHECKING
+from testing import run_pex_command
+from testing.cli import run_pex3
+
+if TYPE_CHECKING:
+    from typing import Any
+
+
+# N.B.: awscli==1.28.1 Only resolves with the pip-2020-resolver for Pip versions earlier than 22.
+# In the end this means only Pip >=20.3.2,<22 can run this test successfully. The underlying issue
+# comes from PyYAML and is documented here: https://github.com/yaml/pyyaml/issues/601
+@pytest.mark.parametrize(
+    "pip_version",
+    [
+        pytest.param(pip_version, id=str(pip_version))
+        for pip_version in PipVersion.values()
+        if Version("20.3.2") <= pip_version.version < Version("22")
+    ],
+)
+def test_backtracking(
+    tmpdir,  # type: Any
+    pip_version,  # type: PipVersionValue
+):
+    # type: (...) -> None
+
+    lock = os.path.join(str(tmpdir), "lock.json")
+    run_pex3(
+        "lock",
+        "create",
+        "-v",
+        "-o",
+        lock,
+        "--indent",
+        "2",
+        "--pip-version",
+        str(pip_version),
+        "--resolver-version",
+        "pip-2020-resolver",
+        "--interpreter-constraint",
+        "CPython==3.11.*",
+        "--style",
+        "universal",
+        "--target-system",
+        "linux",
+        "--target-system",
+        "mac",
+        "--manylinux",
+        "manylinux2014",
+        "awscli==1.28.1",
+    ).assert_success()
+
+    try:
+        python311 = next(InterpreterConstraint.parse("CPython==3.11.*").iter_matching())
+    except StopIteration:
+        pytest.skip("Skipping lock verification since no CPython 3.11 interpreter is available.")
+
+    result = run_pex_command(
+        args=[
+            "--lock",
+            lock,
+            "-c",
+            "aws",
+            "--",
+            "--version",
+        ],
+        python=python311.binary,
+    )
+    result.assert_success()
+    assert result.output.startswith("aws-cli/1.28.1 "), result.output