Repository: pantsbuild/pants

Issue:
Releases are broken by Linux aarch64
The pantsbuild.pants wheels generated by the Linux aarch64 shard are not manylinux and this is a no-go:
```
$ ./build-support/bin/release.sh publish
...
Uploading distributions to https://upload.pypi.org/legacy/
Signing pantsbuild.pants.testutil-2.16.0.dev5-py3-none-any.whl
Signing pantsbuild.pants-2.16.0.dev5-cp37-cp37m-manylinux2014_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_11_0_arm64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp38-cp38-manylinux2014_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp39-cp39-linux_aarch64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp37-cp37m-linux_aarch64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp37-cp37m-macosx_10_15_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp38-cp38-macosx_10_15_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp38-cp38-linux_aarch64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_10_15_x86_64.whl
Signing pantsbuild.pants-2.16.0.dev5-cp39-cp39-manylinux2014_x86_64.whl
Uploading pantsbuild.pants.testutil-2.16.0.dev5-py3-none-any.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 52.2/52.2 kB • 00:00 • 81.0 MB/s
Uploading pantsbuild.pants-2.16.0.dev5-cp37-cp37m-manylinux2014_x86_64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.2/60.2 MB • 02:31 • 391.7 kB/s
Uploading pantsbuild.pants-2.16.0.dev5-cp39-cp39-macosx_11_0_arm64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.4/10.4 MB • 00:21 • 385.6 kB/s
Uploading pantsbuild.pants-2.16.0.dev5-cp38-cp38-manylinux2014_x86_64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.2/60.2 MB • 02:30 • 403.1 kB/s
Uploading pantsbuild.pants-2.16.0.dev5-cp39-cp39-linux_aarch64.whl
100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.8/59.8 MB • 02:28 • 404.2 kB/s
WARNING  Error during upload. Retry with the --verbose option for more details.
ERROR    HTTPError: 400 Bad Request from https://upload.pypi.org/legacy/
         Binary wheel 'pantsbuild.pants-2.16.0.dev5-cp39-cp39-linux_aarch64.whl' has an unsupported platform tag 'linux_aarch64'.
Traceback (most recent call last):
  File "/tmp/pants-sandbox-8EEcEk/./.cache/pex_root/venvs/371f239c6e05f3b21b727016c72f786b4240472e/c118e6ce068494080810f1d842becda796e2eddb/pex", line 270, in <module>
    runpy.run_module(module_name, run_name="__main__", alter_sys=True)
  File "/home/jsirois/.pyenv/versions/3.7.14/lib/python3.7/runpy.py", line 205, in run_module
    return _run_module_code(code, init_globals, run_name, mod_spec)
  File "/home/jsirois/.pyenv/versions/3.7.14/lib/python3.7/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/jsirois/.pyenv/versions/3.7.14/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/tmp/pants-sandbox-8EEcEk/build-support/bin/_release_helper.py", line 1247, in <module>
    main()
  File "/tmp/pants-sandbox-8EEcEk/build-support/bin/_release_helper.py", line 1223, in main
    publish()
  File "/tmp/pants-sandbox-8EEcEk/build-support/bin/_release_helper.py", line 897, in publish
    upload_wheels_via_twine()
  File "/tmp/pants-sandbox-8EEcEk/build-support/bin/_release_helper.py", line 1020, in upload_wheels_via_twine
    check=True,
  File "/home/jsirois/.pyenv/versions/3.7.14/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/home/jsirois/dev/pantsbuild/pants/build-support/twine-deps.venv/bin/twine', 'upload', '--sign', '--sign-with=gpg2', '--identity=93E55CB567B5C626', '--skip-existing', '/home/jsirois/dev/pantsbuild/pants/dist/deploy/wheels/pantsbuild.pants/7766b9b3cedbfd9a90d5bc934b62e03ee7020ccd/2.16.0.dev5/*.whl']' returned non-zero exit status 1.
```


Discussion:
@benjyw you have the context right now for the new Linux aarch64 shards and what you went through to arrive where things are at. I poked around a bit and we do not use the `manylinux repair ...` tool in either x86_64 or aarch64 which is surprising. So it seems like there is some magic in our Linux x86_64 manylinux image that somehow lets us get away without doing this?
Yeah, I just did whatever we did for x86_64. I guess I was under the impression that building in a manylinux image was sufficient. I'll dig further.
Ah, right. I once knew this. We do something _incredibly_ hokey ... in the release script. We re-write the wheels and do this ourselves, but ~incorrectly and do not use auditwheel. I can fix the hokey to finish the release - it will still be hokey.
Any reason not to use auditwheel? I know we do something weird with the stable_version, am hoping we can stop doing that and just build with the stable version up front.
There is no reason not to. I will not be doing that though to solve this, I'm just fixing the broken jank with working jank for the immediate purpose.
Sounds good.

Resolving patch:
diff --git a/build-support/bin/_release_helper.py b/build-support/bin/_release_helper.py
--- a/build-support/bin/_release_helper.py
+++ b/build-support/bin/_release_helper.py
@@ -967,12 +967,12 @@ def check_roles() -> None:
 
 
 def reversion_prebuilt_wheels() -> None:
-    # First, rewrite to manylinux. See https://www.python.org/dev/peps/pep-0599/. We build on
-    # Centos7, so use manylinux2014.
-    source_platform = "linux_x86_64"
-    dest_platform = "manylinux2014_x86_64"
+    # First, rewrite to manylinux. See https://www.python.org/dev/peps/pep-0599/. We use
+    # manylinux2014 images.
+    source_platform = "linux_"
+    dest_platform = "manylinux2014_"
     unstable_wheel_dir = CONSTANTS.deploy_pants_wheel_dir / CONSTANTS.pants_unstable_version
-    for whl in unstable_wheel_dir.glob(f"*{source_platform}.whl"):
+    for whl in unstable_wheel_dir.glob(f"*{source_platform}*.whl"):
         whl.rename(str(whl).replace(source_platform, dest_platform))
 
     # Now, reversion to use the STABLE_VERSION.
diff --git a/build-support/bin/generate_github_workflows.py b/build-support/bin/generate_github_workflows.py
--- a/build-support/bin/generate_github_workflows.py
+++ b/build-support/bin/generate_github_workflows.py
@@ -763,6 +763,9 @@ def build_wheels_job(platform: Platform, python_versions: list[str]) -> Jobs:
 
 
 def build_wheels_jobs() -> Jobs:
+    # N.B.: When altering the number of total wheels built (currently 10), please edit the expected
+    # total in the _release_helper script. Currently here:
+    # https://github.com/pantsbuild/pants/blob/8c83e4db33d5fe577918ce073f6d89957cb6eef1/build-support/bin/_release_helper.py#L1182-L1192
     return {
         **build_wheels_job(Platform.LINUX_X86_64, ALL_PYTHON_VERSIONS),
         **build_wheels_job(Platform.LINUX_ARM64, ALL_PYTHON_VERSIONS),


Test patch: