Repository: pantsbuild/pants

Issue:
Pants not finding wheel for package `cryptography`
**Describe the bug**
Whenever using pants to run type checking with `mypy` (or any other operation that requires installing dependencies, such as running a `pex_binary` target), pants tries to build `cryptography` instead of using the available wheel. I suspect this would happen with any other python package using wheels, `cryptography` just happened to be the one causing me problems right now.

For example, running `./pants check ::` from the root of the repository linked at the bottom of this issue will trigger the problem.

**Pants version**
2.9.0

**OS**
I'm using macOS 12.1 (Monterey) with the M1 Pro chip.

**Additional info**
I'm pretty clueless as to the possible causes for this issue, as I'm fairly new to Pants. It's possible I've made a mistake in my `BUILD` files or my `pants.toml`, although everything seemed to work before I installed `cryptography`, so I think my configuration might be fine.

I should note that poetry does manage to download the wheel just fine. The cryptography package uses abi3 wheels though, and it seems that installing a recent pip version is often recommended as a solution to my issue (without pants), so is it possible that pants simply does not support those types of wheels? I've also tried specifying different `interpreter_constraints` in `pants.toml`, but the issue persists even then.

I read the docs a couple times now, and I'm not really finding anything to help me move forward. I also looked through other issues and wasn't able to find one that helped me solve my problem.

I've created a repository which reproduces the issue : https://github.com/marier-nico/pants-wheel-issue-demo



Discussion:
Yeah, looks like this requires universal2 support which landed in Pip 21.0.1 here https://github.com/pypa/pip/commit/00fb5a0b224cde08e3e5ca034247baadfb646468

Pants uses Pex which uses a vendored copy of Pip at 20.3.4 (both projects support Python 2.7 users still). I'll see about merging that fix into Pex's Pip vendor tree and getting a PEX release out / Pants upgraded.

Since Mac silicon is still rare in CI environments, I might call on you to test if that's ok.
Yeah that would be really great! Let me know if you need me to test anything out, I'd be happy to help :slightly_smiling_face: 
@marier-nico you could help test by installing Pex 2.1.64 and seeing if you can run `pex cryptography --no-build -- -c 'import cryptography; print(cryptography.__file__)'`.

If you'd prefer, instead of installing Pex 2.1.64, you can always use it throw-away:
```console
curl -sSL https://github.com/pantsbuild/pex/releases/download/v2.1.64/pex > pex-2.1.64.pex
python310 pex-2.1.64.pex cryptography --no-build -- -c 'import cryptography; print(cryptography.__file__)'
rm pex-2.1.64.pex
```
@jsirois, I'm having the same issue. It's also a problem for installing `grpcio`.

The above command fails for me. Here's my output:
```
pip list | grep pex && pex cryptography --no-build -- -c 'import cryptography; print(cryptography.__file__)'
pex        2.1.64
Traceback (most recent call last):
  File "/private/var/folders/_5/w6q0pbt55jb_l7pxj5_1ry7c0000gn/T/tmp13vu9qf0/.bootstrap/pex/pex.py", line 475, in execute
    self.activate()
  File "/private/var/folders/_5/w6q0pbt55jb_l7pxj5_1ry7c0000gn/T/tmp13vu9qf0/.bootstrap/pex/pex.py", line 139, in activate
    self._activated_dists = self._activate()
  File "/private/var/folders/_5/w6q0pbt55jb_l7pxj5_1ry7c0000gn/T/tmp13vu9qf0/.bootstrap/pex/pex.py", line 126, in _activate
    activated_dists.extend(env.activate())
  File "/private/var/folders/_5/w6q0pbt55jb_l7pxj5_1ry7c0000gn/T/tmp13vu9qf0/.bootstrap/pex/environment.py", line 322, in activate
    self._activated_dists = self._activate()
  File "/private/var/folders/_5/w6q0pbt55jb_l7pxj5_1ry7c0000gn/T/tmp13vu9qf0/.bootstrap/pex/environment.py", line 673, in _activate
    resolved = self.resolve()
  File "/private/var/folders/_5/w6q0pbt55jb_l7pxj5_1ry7c0000gn/T/tmp13vu9qf0/.bootstrap/pex/environment.py", line 506, in resolve
    for fingerprinted_distribution in self.resolve_dists(all_reqs)
  File "/private/var/folders/_5/w6q0pbt55jb_l7pxj5_1ry7c0000gn/T/tmp13vu9qf0/.bootstrap/pex/environment.py", line 530, in resolve_dists
    for qualified_req_or_not_found in self._root_requirements_iter(reqs):
  File "/private/var/folders/_5/w6q0pbt55jb_l7pxj5_1ry7c0000gn/T/tmp13vu9qf0/.bootstrap/pex/environment.py", line 469, in _root_requirements_iter
    raise ResolveError(message)
pex.environment.ResolveError: A distribution for cryptography could not be resolved in this environment.Found 1 distribution for cryptography that do not apply:
1.) The wheel tags for cryptography 36.0.1 are cp36-abi3-macosx_10_10_universal2 which do not match the supported tags of DistributionTarget(interpreter=PythonInterpreter('/Users/nob/.pyenv/versions/3.9.9/bin/python3.9', PythonIdentity('/Users/nob/.pyenv/versions/3.9.9/bin/python3.9', 'cp39', 'cp39', 'macosx_12_0_arm64', (3, 9, 9)))):
cp39-cp39-macosx_12_0_arm64
cp39-cp39-macosx_12_0_universal2
cp39-cp39-macosx_11_0_arm64
cp39-cp39-macosx_11_0_universal2
cp39-abi3-macosx_12_0_arm64
cp39-abi3-macosx_12_0_universal2
cp39-abi3-macosx_11_0_arm64
cp39-abi3-macosx_11_0_universal2
cp39-none-macosx_12_0_arm64
cp39-none-macosx_12_0_universal2
cp39-none-macosx_11_0_arm64
cp39-none-macosx_11_0_universal2
cp38-abi3-macosx_12_0_arm64
cp38-abi3-macosx_12_0_universal2
cp38-abi3-macosx_11_0_arm64
cp38-abi3-macosx_11_0_universal2
cp37-abi3-macosx_12_0_arm64
cp37-abi3-macosx_12_0_universal2
cp37-abi3-macosx_11_0_arm64
cp37-abi3-macosx_11_0_universal2
cp36-abi3-macosx_12_0_arm64
cp36-abi3-macosx_12_0_universal2
cp36-abi3-macosx_11_0_arm64
cp36-abi3-macosx_11_0_universal2
cp35-abi3-macosx_12_0_arm64
cp35-abi3-macosx_12_0_universal2
cp35-abi3-macosx_11_0_arm64
cp35-abi3-macosx_11_0_universal2
cp34-abi3-macosx_12_0_arm64
cp34-abi3-macosx_12_0_universal2
cp34-abi3-macosx_11_0_arm64
cp34-abi3-macosx_11_0_universal2
cp33-abi3-macosx_12_0_arm64
cp33-abi3-macosx_12_0_universal2
cp33-abi3-macosx_11_0_arm64
cp33-abi3-macosx_11_0_universal2
cp32-abi3-macosx_12_0_arm64
cp32-abi3-macosx_12_0_universal2
cp32-abi3-macosx_11_0_arm64
cp32-abi3-macosx_11_0_universal2
py39-none-macosx_12_0_arm64
py39-none-macosx_12_0_universal2
py39-none-macosx_11_0_arm64
py39-none-macosx_11_0_universal2
py3-none-macosx_12_0_arm64
py3-none-macosx_12_0_universal2
py3-none-macosx_11_0_arm64
py3-none-macosx_11_0_universal2
py38-none-macosx_12_0_arm64
py38-none-macosx_12_0_universal2
py38-none-macosx_11_0_arm64
py38-none-macosx_11_0_universal2
py37-none-macosx_12_0_arm64
py37-none-macosx_12_0_universal2
py37-none-macosx_11_0_arm64
py37-none-macosx_11_0_universal2
py36-none-macosx_12_0_arm64
py36-none-macosx_12_0_universal2
py36-none-macosx_11_0_arm64
py36-none-macosx_11_0_universal2
py35-none-macosx_12_0_arm64
py35-none-macosx_12_0_universal2
py35-none-macosx_11_0_arm64
py35-none-macosx_11_0_universal2
py34-none-macosx_12_0_arm64
py34-none-macosx_12_0_universal2
py34-none-macosx_11_0_arm64
py34-none-macosx_11_0_universal2
py33-none-macosx_12_0_arm64
py33-none-macosx_12_0_universal2
py33-none-macosx_11_0_arm64
py33-none-macosx_11_0_universal2
py32-none-macosx_12_0_arm64
py32-none-macosx_12_0_universal2
py32-none-macosx_11_0_arm64
py32-none-macosx_11_0_universal2
py31-none-macosx_12_0_arm64
py31-none-macosx_12_0_universal2
py31-none-macosx_11_0_arm64
py31-none-macosx_11_0_universal2
py30-none-macosx_12_0_arm64
py30-none-macosx_12_0_universal2
py30-none-macosx_11_0_arm64
py30-none-macosx_11_0_universal2
py39-none-any
py3-none-any
py38-none-any
py37-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any
```
Thanks @nathanielobrown. I need to update Pex's vendored version of packaging to 20.9 as well. In your debug Pip resolved the wheel correctly and then Pex rejected it - facepalm.
Ah, that explains it. I compared the bundled wheels between 2.1.63 and 2.1.64 of pex and thought it was fixed (as the latter got `universal2` wheels), but was then surprised. Nonetheless, thanks for the work on the fix @jsirois. 🙌 
 Looking into my [`grpcio`](https://pypi.org/project/grpcio/#files) issue, I notice that grpcio has a universal build for Python `3.10` but not `3.9` so *possibly* there is a different issue here. Running `pex grpcio --no-build -- -c 'import grpcio; print(grpcio.__file__)'` yields:
```
ERROR: Could not find a version that satisfies the requirement grpcio (from versions: none)
ERROR: No matching distribution found for grpcio
```

while `pip install grpcio` (pip version 21.3.1) yields

```
Collecting grpcio
  Using cached grpcio-1.43.0-cp39-cp39-macosx_12_0_arm64.whl
Collecting six>=1.5.2
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: six, grpcio
Successfully installed grpcio-1.43.0 six-1.16.0
```
So I gather pex 2.1.64 is ignoring `grpcio-1.43.0-cp39-cp39-macosx_12_0_arm64.whl`

Please let me know if I should create a separate issue for this, but I'm thinking maybe this can also be solved by the same upgrade of pex's pip?

Also, is there a way to control the version of pip used by pants/pex? I'm gathering there is not currently, but I imagine using an old version of pip might keep causing issues like this?
@nathanielobrown If I am not mistaken, there is no `arm64` wheel for `grpcio`. Note that your output refers to a cached wheel, and I think that will have been (as it has in the project I'm working on) been built manually.

Might indeed, as you mention, have to upgrade to Python 3.10 in order to get a cross-built PEX on `x86_64` that works on Apple Silicon too.
> Also, is there a way to control the version of pip used by pants/pex? I'm gathering there is not currently, but I imagine using an old version of pip might keep causing issues like this?

Pex vendors Pip so that it can call a stable CLI api. This vendor is stuck at 20.3.4 since Pex (and Pants) maintain support for customers that use Python 2.7 still. https://github.com/pantsbuild/pex/blob/124f809d20a3ebe9170c9fd353317988f0b43f16/pex/vendor/__init__.py#L140-L158
@thor you've got that right. As a mac arm64 customer you need to be able to build sdists or else run under rosetta still for many projects.
> @nathanielobrown If I am not mistaken, there is no `arm64` wheel for `grpcio`. Note that your output refers to a cached wheel, and I think that will have been (as it has in the project I'm working on) been built manually.
> 
> Might indeed, as you mention, have to upgrade to Python 3.10 in order to get a cross-built PEX on `x86_64` that works on Apple Silicon too.

Ah, OK that makes sense. I installed with `pip install --no-cache grpcio` and it built `grpcio-1.43.0.tar.gz`. Thank you!
@jsirois I see that @nathanielobrown has already run the test you showed above. I tried it as well just to see and got the same output they did (on python 3.10)
Ok - @nathanielobrown, @thor and @marier-nico if you could try:
```
pex git+https://github.com/jsirois/pex@5ae1ac1b121b353e83ff8182a71092e7dfa75fa4#egg=pex -cpex -- cryptography --no-build -- -c 'import cryptography; print(cryptography.__file__)'
```
That''s a fairly fancy use of Pex to test an unreleased version of itself (https://github.com/pantsbuild/pex/pull/1591).
Yeah, pretty slick!

Looks like it works:
```
pex 'git+https://github.com/jsirois/pex@5ae1ac1b121b353e83ff8182a71092e7dfa75fa4#egg=pex' -cpex -- cryptography --no-build -- -c 'import cryptography; print(cryptography.__file__)'
/Users/nob/.pex/installed_wheels/efdf1f23daf5e3b056a26d12e37de7ae4e0b2f0f/cryptography-36.0.1-cp36-abi3-macosx_10_10_universal2.whl/cryptography/__init__.py
```
Great - thanks @nathanielobrown.
Can confirm it works for me as well 🚀 

Resolving patch:
diff --git a/src/python/pants/backend/python/util_rules/pex_cli.py b/src/python/pants/backend/python/util_rules/pex_cli.py
--- a/src/python/pants/backend/python/util_rules/pex_cli.py
+++ b/src/python/pants/backend/python/util_rules/pex_cli.py
@@ -37,9 +37,9 @@ class PexBinary(TemplatedExternalTool):
     name = "pex"
     help = "The PEX (Python EXecutable) tool (https://github.com/pantsbuild/pex)."
 
-    default_version = "v2.1.61"
+    default_version = "v2.1.65"
     default_url_template = "https://github.com/pantsbuild/pex/releases/download/{version}/pex"
-    version_constraints = ">=2.1.61,<3.0"
+    version_constraints = ">=2.1.65,<3.0"
 
     @classproperty
     def default_known_versions(cls):
@@ -48,8 +48,8 @@ def default_known_versions(cls):
                 (
                     cls.default_version,
                     plat,
-                    "8072340969ad517279f153551f34d6c43ba51f7984223da4fb0913cc734d0c90",
-                    "3693575",
+                    "e629fe508154f8e298310f9bfce54d8931f299f0112a86a283f157be2c0d8a03",
+                    "3713643",
                 )
             )
             for plat in ["macos_arm64", "macos_x86_64", "linux_x86_64"]
diff --git a/testprojects/src/python/native/setup.py b/testprojects/src/python/native/setup.py
--- a/testprojects/src/python/native/setup.py
+++ b/testprojects/src/python/native/setup.py
@@ -2,7 +2,7 @@
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
 from colors import red
-from setuptools import Extension, setup  # type: ignore[import]
+from setuptools import Extension, setup
 
 native_impl = Extension("native.impl", sources=["impl.c"])
 


Test patch:
diff --git a/src/python/pants/goal/run_tracker_test.py b/src/python/pants/goal/run_tracker_test.py
--- a/src/python/pants/goal/run_tracker_test.py
+++ b/src/python/pants/goal/run_tracker_test.py
@@ -17,7 +17,7 @@
 from pants.version import VERSION
 
 
-@freeze_time(datetime.datetime(2020, 1, 1, 12, 0, 0), as_kwarg="frozen_time")  # type: ignore[call-arg]
+@freeze_time(datetime.datetime(2020, 1, 1, 12, 0, 0), as_kwarg="frozen_time")
 def test_run_tracker_timing_output(**kwargs) -> None:
     with temporary_dir() as buildroot:
         with environment_as(PANTS_BUILDROOT_OVERRIDE=buildroot):
@@ -37,7 +37,7 @@ def test_run_tracker_timing_output(**kwargs) -> None:
     "exit_code,expected",
     [(PANTS_SUCCEEDED_EXIT_CODE, "SUCCESS"), (PANTS_FAILED_EXIT_CODE, "FAILURE")],
 )
-@freeze_time(datetime.datetime(2020, 1, 10, 12, 0, 1), as_kwarg="frozen_time")  # type: ignore[call-arg]
+@freeze_time(datetime.datetime(2020, 1, 10, 12, 0, 1), as_kwarg="frozen_time")
 def test_run_information(exit_code, expected, **kwargs) -> None:
     with temporary_dir() as buildroot:
         with environment_as(PANTS_BUILDROOT_OVERRIDE=buildroot):
@@ -72,7 +72,7 @@ def test_run_information(exit_code, expected, **kwargs) -> None:
             assert run_information_after_ended["outcome"] == expected
 
 
-@freeze_time(datetime.datetime(2020, 1, 10, 12, 0, 1), as_kwarg="frozen_time")  # type: ignore[call-arg]
+@freeze_time(datetime.datetime(2020, 1, 10, 12, 0, 1), as_kwarg="frozen_time")
 def test_anonymous_telemetry(monkeypatch, **kwargs) -> None:
     with temporary_dir() as buildroot:
         with environment_as(PANTS_BUILDROOT_OVERRIDE=buildroot):