Repository: pantsbuild/pants

Issue:
Invalid entry in the RECORD file in pants's wheels
**Describe the bug**
```console
$ bsdtar -xOf pantsbuild.pants-2.14.1-cp39-cp39-manylinux2014_x86_64.whl pantsbuild.pants-2.14.1.dist-info/RECORD | grep VERSION
pants/VERSION,sha256=vNrRse8ZHLdFijrFjxUetx6i8tw/9fa0kvdd3+kbGDU=,6
```

This line is incorrect. According to the [specification](https://packaging.python.org/en/latest/specifications/recording-installed-packages/#the-record-file), the hash digest should a) be using the URL-safe alphabet, and b) have no padding at the end. In other words, the line should look like this:

```
pants/VERSION,sha256=vNrRse8ZHLdFijrFjxUetx6i8tw_9fa0kvdd3-kbGDU,6
```

Every other entry in the file is already correct, it's just `pants/VERSION` that's broken.

**Pants version**
2.14.1, but I looked at some other versions too and they're broken in the same way.

**OS**
All.

**Additional info**
N/A



Discussion:
Looks like the issue comes from here:

https://github.com/pantsbuild/pants/blob/8da1ced43525b59653b818ddfd432c9b6030e4d2/build-support/bin/reversion.py#L61

It should be `b64_encoded = base64.urlsafe_b64encode(fingerprint.digest()).rstrip(b'=')`.
> Every other entry in the file is already correct

Spoke too soon; the entries for the `pantsbuild.pants-2.14.1.dist-info/*` files also exhibit the same problem.

Yeah, so I find:
```
$ unzip -qc ~/Downloads/pantsbuild.pants-2.14.1-cp39-cp39-macosx_11_0_arm64.whl pantsbuild.pants-2.14.1.dist-info/RECORD | grep =,
pants/VERSION,sha256=vNrRse8ZHLdFijrFjxUetx6i8tw/9fa0kvdd3+kbGDU=,6
pantsbuild.pants-2.14.1.dist-info/METADATA,sha256=WUDHsXqSHJjX2R1P+n7t+ujfzobJrNViSo7IkWwsgUU=,34898
pantsbuild.pants-2.14.1.dist-info/WHEEL,sha256=oITnL8r4K9lhj+wWSHlE2XUA0J8WLXdegrKtvi/34zA=,108
pantsbuild.pants-2.14.1.dist-info/entry_points.txt,sha256=PPhLegkjKzPy7mhIYYpl2tyAI6XngR4OI9pqvsoCyKo=,54
pantsbuild.pants-2.14.1.dist-info/namespace_packages.txt,sha256=AbpHGcgLb+kRsJGnwFEktk7uzpZOCcBY74+YBdrKVGs=,1
pantsbuild.pants-2.14.1.dist-info/top_level.txt,sha256=SMY0x5WZgtCApbw/EApPfTQ6I3wIcw0CApL+5veAEiI=,20
pantsbuild.pants-2.14.1.dist-info/zip-safe,sha256=AbpHGcgLb+kRsJGnwFEktk7uzpZOCcBY74+YBdrKVGs=,1
```

It looks like we do potentially modify all dist-info files:
https://github.com/pantsbuild/pants/blob/8da1ced43525b59653b818ddfd432c9b6030e4d2/build-support/bin/reversion.py#L102-L105

So I think you've identified the right bug + fix above.

@SpecLad do you want to post a PR for the fix?
> @SpecLad do you want to post a PR for the fix?

Not really, to be honest. Setting up the build environment seems complicated, and the fix is very simple, so I'm hoping that you can do it yourselves.
@SpecLad thanks for the report. 👍🏽 
@benjyw I can knock this out; just let me know by un-assigning yourself if that sounds good. It would be good to get this fixed before Friday / the march of 2.16.x through alpha and beyond.
@jsirois thanks! Unassigned myself
I've just verified `wheel unpack` serves as a handy check that fails against the current broken-ness:
```
$ rm -rf /tmp/check && wheel-check.venv/bin/wheel unpack dist/deploy/wheels/pantsbuild.pants/33a6cf8a53daf5b39fbaca9798bb6f2459314dec/2.14.1/pantsbuild.pants-2.14.1-cp39-cp39-manylinux2014_x86_64.whl --dest /tmp/check
Traceback (most recent call last):
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/wheel-check.venv/bin/wheel", line 8, in <module>
    sys.exit(main())
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/wheel-check.venv/lib/python3.9/site-packages/wheel/cli/__init__.py", line 91, in main
    args.func(args)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/wheel-check.venv/lib/python3.9/site-packages/wheel/cli/__init__.py", line 19, in unpack_f
    unpack(args.wheelfile, args.dest)
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/wheel-check.venv/lib/python3.9/site-packages/wheel/cli/unpack.py", line 17, in unpack
    with WheelFile(path) as wf:
  File "/home/jsirois/dev/pantsbuild/jsirois-pants/wheel-check.venv/lib/python3.9/site-packages/wheel/wheelfile.py", line 75, in __init__
    algorithm, hash_sum = hash_sum.split("=")
ValueError: too many values to unpack (expected 2)
```

And fails against invalid hashes (I hacked this one manually):
```
$ rm -rf /tmp/check && wheel unpack pantsbuild.pants-2.14.1-cp39-cp39-manylinux2014_aarch64.whl -d /tmp/check
Unpacking to: /tmp/check/pantsbuild.pants-2.14.1...Hash mismatch for file 'pantsbuild.pants-2.14.1.dist-info/zip-safe'
```

And succeeds with a valid RECORD only.

Resolving patch:
diff --git a/build-support/bin/reversion.py b/build-support/bin/reversion.py
--- a/build-support/bin/reversion.py
+++ b/build-support/bin/reversion.py
@@ -10,6 +10,8 @@
 import hashlib
 import os
 import re
+import subprocess
+import tempfile
 import zipfile
 
 from pants.util.contextutil import open_zip, temporary_dir
@@ -52,14 +54,16 @@ def locate_dist_info_dir(workspace):
 
 
 def fingerprint_file(workspace, filename):
-    """Given a relative filename located in a workspace, fingerprint the file.
+    """Given a relative filename located in a workspace, fingerprint the file for a RECORD entry.
 
     Returns a tuple of fingerprint string and size string.
     """
+    # See the spec here:
+    # https://packaging.python.org/en/latest/specifications/recording-installed-packages/#the-record-file
     content = read_file(os.path.join(workspace, filename), binary_mode=True)
     fingerprint = hashlib.sha256(content)
-    b64_encoded = base64.b64encode(fingerprint.digest())
-    return f"sha256={b64_encoded.decode()}", str(len(content))
+    record_encoded = base64.urlsafe_b64encode(fingerprint.digest()).rstrip(b"=")
+    return f"sha256={record_encoded.decode()}", str(len(content))
 
 
 def rewrite_record_file(workspace, src_record_file, mutated_file_tuples):
@@ -146,10 +150,15 @@ def reversion(
         # Create a new output whl in the destination.
         dst_whl_filename = os.path.basename(whl_file).replace(input_version, target_version)
         dst_whl_file = os.path.join(dest_dir, dst_whl_filename)
-        with open_zip(dst_whl_file, "w", zipfile.ZIP_DEFLATED) as whl:
-            for dst_filename in dst_filenames:
-                whl.write(os.path.join(workspace, dst_filename), dst_filename)
-
+        with tempfile.TemporaryDirectory() as chroot:
+            tmp_whl_file = os.path.join(chroot, dst_whl_filename)
+            with open_zip(tmp_whl_file, "w", zipfile.ZIP_DEFLATED) as whl:
+                for dst_filename in dst_filenames:
+                    whl.write(os.path.join(workspace, dst_filename), dst_filename)
+            check_dst = os.path.join(chroot, "check-wheel")
+            os.mkdir(check_dst)
+            subprocess.run(args=["wheel", "unpack", "-d", check_dst, tmp_whl_file], check=True)
+            os.rename(tmp_whl_file, dst_whl_file)
         print("Wrote whl with version {} to {}.\n".format(target_version, dst_whl_file))
 
 


Test patch: