Repository: pantsbuild/pants

Issue:
`build_runtime_package_dependencies` might not be deterministic
I haven't investigated deeply yet, but it looks like `pants.core.goals.test.build_runtime_package_dependencies` might not be deterministic.

While inspecting traces of Pants' own CI, I noticed that `pep561_integration_test.py` and its inputs tend to (always?) miss the remote cache, and are the long pole in CI:
<img width="1369" alt="pep561-integration" src="https://user-images.githubusercontent.com/46740/158910184-c00c77e2-08cf-4db0-aac1-0abba5653276.png">

It's not clear how the non-determinism gets kicked off, but it's unlikely to be a process (at least initially), because hitting the cache for a non-deterministic process is still deterministic. So most likely either a `@rule` or uncacheable process.


Discussion:
It's possible that this would be a good testbed for #14195.
Noticed this again tonight (without it, `pantsbuild/pants` CI would complete about 1 minute faster on a warm cache).

It reproduces locally if you restart `pantsd` between attempts to run:
```
./pants test tests/python/pants_test/integration/pep561_integration_test.py
```

I think that rather than `build_runtime_package_dependencies`, this is likely to be the code below it: either `src/python/pants/backend/python/util_rules/dists.py` or `src/python/pants/backend/python/goals/setup_py.py`. In the absence of #14195, the best bet is probably to run in `-ldebug` and compare the input/output digests from the relevant processes.
Running with `-ldebug`, and grepping out and diffing the `spawning local process` lines, it looks like the culprit is that:
```
../build_backend.pex_pex_shim.sh backend_shim.py
```
... has a different input digest in each run. That input was probably the output of some other process (or `@rule`) which was non-deterministic (we don't log process results currently: maybe we should).
I can't make sense of inputs to that process being the culprit, but the outputs are almost certainly non-deterministic without extra effort. The sdists and wheels built by that process are being built by known and unknown 3rd parties (setuptools+wheel, poetry, flit, ...). They may respect `SOURCE_DATE_EPOCH`, but we should probably be setting it regardless just in case here: https://github.com/pantsbuild/pants/blob/e7c6095829304bd180e922fffeac3f8f0e4650f8/src/python/pants/backend/python/util_rules/dists.py#L212-L215
Yea, let's definitely do `SOURCE_DATE_EPOCH`: thanks.

The reason inputs are relevant here though is that even if the `backend_shim.py` process is non-deterministic, if we hit the cache for it like we should in this case, the non-determinism would be papered over, and it wouldn't need to re-run. So we have multiple culprits.

Resolving patch:
diff --git a/src/python/pants/backend/python/goals/setup_py.py b/src/python/pants/backend/python/goals/setup_py.py
--- a/src/python/pants/backend/python/goals/setup_py.py
+++ b/src/python/pants/backend/python/goals/setup_py.py
@@ -954,7 +954,7 @@ def maybe_add_resource(fp: str) -> None:
     return (
         tuple(sorted(packages)),
         tuple(sorted(namespace_packages)),
-        tuple((pkg, tuple(sorted(files))) for pkg, files in package_data.items()),
+        tuple((pkg, tuple(sorted(files))) for pkg, files in sorted(package_data.items())),
     )
 
 


Test patch:
diff --git a/src/python/pants/backend/python/goals/setup_py_integration_test.py b/src/python/pants/backend/python/goals/setup_py_integration_test.py
--- a/src/python/pants/backend/python/goals/setup_py_integration_test.py
+++ b/src/python/pants/backend/python/goals/setup_py_integration_test.py
@@ -3,13 +3,17 @@
 
 from __future__ import annotations
 
+import hashlib
 import os
 import subprocess
 import sys
 import venv
+from pathlib import Path
 from tempfile import TemporaryDirectory
+from textwrap import dedent
 
-from pants.testutil.pants_integration_test import run_pants
+from pants.testutil.pants_integration_test import run_pants, setup_tmpdir
+from pants.util.dirutil import safe_rmtree
 
 
 def test_native_code() -> None:
@@ -43,3 +47,64 @@ def test_native_code() -> None:
             capture_output=True,
         )
         assert proc.stdout == b"Professor Native\n"
+
+
+def package_determinism(expected_artifact_count: int, files: dict[str, str]) -> None:
+    """Tests that the given sources can be `package`d reproducibly."""
+
+    def digest(path: str) -> tuple[str, str]:
+        d = hashlib.sha256(Path(path).read_bytes()).hexdigest()
+        return path, d
+
+    def run_and_digest(address: str) -> dict[str, str]:
+        safe_rmtree("dist")
+        pants_run = run_pants(
+            [
+                "--backend-packages=pants.backend.python",
+                "--no-pantsd",
+                "package",
+                address,
+            ],
+        )
+        pants_run.assert_success()
+        return dict(digest(os.path.join("dist", f)) for f in os.listdir("dist"))
+
+    with setup_tmpdir(files) as source_dir:
+        one = run_and_digest(f"{source_dir}:dist")
+        two = run_and_digest(f"{source_dir}:dist")
+
+    assert len(one) == expected_artifact_count
+    assert one == two
+
+
+def test_deterministic_package_data() -> None:
+    package_determinism(
+        2,
+        {
+            "BUILD": dedent(
+                """\
+                python_distribution(
+                    name="dist",
+                    dependencies=["{tmpdir}/a", "{tmpdir}/b"],
+                    provides=python_artifact(name="det", version="2.3.4"),
+                )
+                """
+            ),
+            "a/BUILD": dedent(
+                """\
+                python_sources(dependencies=[":resources"])
+                resources(name="resources", sources=["*.txt"])
+                """
+            ),
+            "a/source.py": "",
+            "a/a.txt": "",
+            "b/BUILD": dedent(
+                """\
+                python_sources(dependencies=[":resources"])
+                resources(name="resources", sources=["*.txt"])
+                """
+            ),
+            "b/source.py": "",
+            "b/b.txt": "",
+        },
+    )