Repository: pypa/pip

Issue:
RECORD size and hash do not reflect rewritten shebangs
### Description

When installing a wheel, pip rewrites the shebang in scripts in `{dist}-{ver}.dist-info/scripts/` from `#!python` to the appropriate path to the Python interpreter.  However, the hash and size in `RECORD` for the script correspond to the original version of the script, not the rewritten version.

### Expected behavior

I expected the `RECORD` to reflect the files that actually get installed, so that an install can be integrity-checked.

### pip version

20.3.4

### Python version

3.9.9

### OS

Parabola GNU/Linux-libre (like Arch Linux)

### How to Reproduce

1. Use pip to install a wheel that contains a `{dist}-{ver}.dist-info/scripts/*` script with a `#!python` shebang.  One such wheel is websocket-client 0.57.0.
2. Check whether the hash and size of the script match what got recorded in `RECORD`.


### Output

```sh-session
$ wget https://files.pythonhosted.org/packages/4c/5f/f61b420143ed1c8dc69f9eaec5ff1ac36109d52c80de49d66e0c36c3dfdf/websocket_client-0.57.0-py2.py3-none-any.whl
…
$ pip install --ignore-installed --no-deps --prefix=testdir ./websocket_client-0.57.0-py2.py3-none-any.whl
…

$ # Observe what the resulting RECORD says
$ grep bin/w testdir/lib/python3.9/site-packages/websocket_client-0.57.0.dist-info/RECORD 
../../../bin/wsdump.py,sha256=S54et6zebnxb2VJcgBadSnvXblK1iBF93ap54hlc5O8,6403

$ # Observe whether this matches the resulting script file
$ sha256sum testdir/bin/wsdump.py | xargs python -c 'import sys, base64; print(base64.b64encode(bytes.fromhex(sys.argv[1])).decode("utf-8"))'
6GQkITdeQFmlpL7/T9+O/X0sWsKeddIZCvwtU0ld+hc=
$ wc -c < testdir/bin/wsdump.py 
6412

$ # Observe whether this matches the original script in the wheel
$ bsdtar xfO websocket_client-0.57.0-py2.py3-none-any.whl websocket_client-0.57.0.data/scripts/wsdump.py | sha256sum | xargs python -c 'import sys, base64; print(base64.b64encode(bytes.fromhex(sys.argv[1])).decode("utf-8"))'
S54et6zebnxb2VJcgBadSnvXblK1iBF93ap54hlc5O8=
$ bsdtar xfO websocket_client-0.57.0-py2.py3-none-any.whl websocket_client-0.57.0.data/scripts/wsdump.py |wc -c
6403
```


### Code of Conduct

- [X] I agree to follow the [PSF Code of Conduct](https://www.python.org/psf/conduct/).


Discussion:
Is this reproducible with the latest pip release?
> Is this reproducible with the latest pip release?

Whoop, I get used to thinking that Arch/Parabola have the latest everything, but I guess Arch has a blocker for upgrading to pip 21.

OK, so using a virtualenv to get the latest pip: yes, it is reproducible with the latest pip release.

```console
$ python -m virtualenv venv                                                                                                                                                                                                      
created virtual environment CPython3.9.9.final.0-64 in 468ms
  creator CPython3Posix(dest=…/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/lukeshu/.local/share/virtualenv)
    added seed packages: pip==21.3.1, setuptools==59.2.0, wheel==0.37.0
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

$ . venv/bin/activate

(venv) $ pip --version
pip 21.3.1 from …/venv/lib/python3.9/site-packages/pip (python 3.9)

(venv) $ pip install --ignore-installed --no-deps --prefix=testdir ./websocket_client-0.57.0-py2.py3-none-any.whl
Processing ./websocket_client-0.57.0-py2.py3-none-any.whl
Installing collected packages: websocket-client
Successfully installed websocket-client-0.57.0

(venv) $ grep bin/w testdir/lib/python3.9/site-packages/websocket_client-0.57.0.dist-info/RECORD 
../../../bin/wsdump.py,sha256=S54et6zebnxb2VJcgBadSnvXblK1iBF93ap54hlc5O8,6403
```
I do wonder what changed during the time, there’s nothing I recall around this. Maybe this is not even a pip issue but an Arch issue? Could you try versions between 20.3.4 and 21.3.1 and see if the behaviour changed in a particular version?
@uranusjr am I confused here? I thought the OP stated that the behaviour was the same in 20.3.4 and 21.3.1? Which makes this feel like a straight bug. In fact, I just confirmed it with Ubuntu (on WSL) with Python 3.8 and pip 21.3.1.

I suspect this is "simply" a case of us copying RECORD from the wheel without recalculating the hash for scripts where we rewrite the shebang. It's probably not been reported before because (a) scripts are uncommon (most people use entry points these days) and (b) very few people check hashes, I suspect.
We do compute these things for generated files.

https://github.com/pypa/pip/blob/main/src/pip/_internal/operations/install/wheel.py#L266

I think we might want to start validating these hashes from wheels at some point, similar to how we'd want to validate the hashes by default in some contexts and get TUF-based protection of all the index interactions. 🤷🏻‍♂️
And we check for files that have been changed; to recompute hashes and sizes.

https://github.com/pypa/pip/blob/main/src/pip/_internal/operations/install/wheel.py#L260

Someone should step through this chunk of code with a debugger to figure out what is going wrong here; because we certainly have code to account for changed files. 

Resolving patch:
diff --git a/src/pip/_internal/operations/install/wheel.py b/src/pip/_internal/operations/install/wheel.py
--- a/src/pip/_internal/operations/install/wheel.py
+++ b/src/pip/_internal/operations/install/wheel.py
@@ -224,19 +224,16 @@ def _normalized_outrows(
     )
 
 
-def _record_to_fs_path(record_path: RecordPath) -> str:
-    return record_path
-
-
-def _fs_to_record_path(path: str, relative_to: Optional[str] = None) -> RecordPath:
-    if relative_to is not None:
-        # On Windows, do not handle relative paths if they belong to different
-        # logical disks
-        if (
-            os.path.splitdrive(path)[0].lower()
-            == os.path.splitdrive(relative_to)[0].lower()
-        ):
-            path = os.path.relpath(path, relative_to)
+def _record_to_fs_path(record_path: RecordPath, lib_dir: str) -> str:
+    return os.path.join(lib_dir, record_path)
+
+
+def _fs_to_record_path(path: str, lib_dir: str) -> RecordPath:
+    # On Windows, do not handle relative paths if they belong to different
+    # logical disks
+    if os.path.splitdrive(path)[0].lower() == os.path.splitdrive(lib_dir)[0].lower():
+        path = os.path.relpath(path, lib_dir)
+
     path = path.replace(os.path.sep, "/")
     return cast("RecordPath", path)
 
@@ -259,7 +256,7 @@ def get_csv_rows_for_installed(
         old_record_path = cast("RecordPath", row[0])
         new_record_path = installed.pop(old_record_path, old_record_path)
         if new_record_path in changed:
-            digest, length = rehash(_record_to_fs_path(new_record_path))
+            digest, length = rehash(_record_to_fs_path(new_record_path, lib_dir))
         else:
             digest = row[1] if len(row) > 1 else ""
             length = row[2] if len(row) > 2 else ""
@@ -475,7 +472,7 @@ def record_installed(
         newpath = _fs_to_record_path(destfile, lib_dir)
         installed[srcfile] = newpath
         if modified:
-            changed.add(_fs_to_record_path(destfile))
+            changed.add(newpath)
 
     def is_dir_path(path: RecordPath) -> bool:
         return path.endswith("/")


Test patch:
diff --git a/tests/functional/test_install_wheel.py b/tests/functional/test_install_wheel.py
--- a/tests/functional/test_install_wheel.py
+++ b/tests/functional/test_install_wheel.py
@@ -1,5 +1,7 @@
+import base64
 import csv
 import distutils
+import hashlib
 import os
 import shutil
 from pathlib import Path
@@ -364,6 +366,44 @@ def test_wheel_record_lines_have_hash_for_data_files(
     ]
 
 
+def test_wheel_record_lines_have_updated_hash_for_scripts(
+    script: PipTestEnvironment,
+) -> None:
+    """
+    pip rewrites "#!python" shebang lines in scripts when it installs them;
+    make sure it updates the RECORD file correspondingly.
+    """
+    package = make_wheel(
+        "simple",
+        "0.1.0",
+        extra_data_files={
+            "scripts/dostuff": "#!python\n",
+        },
+    ).save_to_dir(script.scratch_path)
+    script.pip("install", package)
+    record_file = script.site_packages_path / "simple-0.1.0.dist-info" / "RECORD"
+    record_text = record_file.read_text()
+    record_rows = list(csv.reader(record_text.splitlines()))
+    records = {r[0]: r[1:] for r in record_rows}
+
+    script_path = script.bin_path / "dostuff"
+    script_contents = script_path.read_bytes()
+    assert not script_contents.startswith(b"#!python\n")
+
+    script_digest = hashlib.sha256(script_contents).digest()
+    script_digest_b64 = (
+        base64.urlsafe_b64encode(script_digest).decode("US-ASCII").rstrip("=")
+    )
+
+    script_record_path = os.path.relpath(
+        script_path, script.site_packages_path
+    ).replace(os.path.sep, "/")
+    assert records[script_record_path] == [
+        f"sha256={script_digest_b64}",
+        str(len(script_contents)),
+    ]
+
+
 @pytest.mark.incompatible_with_test_venv
 @pytest.mark.usefixtures("with_wheel")
 def test_install_user_wheel(