Repository: pantsbuild/pants

Issue:
Include/exclude and heuristic-based `immutable_inputs`
`Process.immutable_inputs` is currently configured as a mapping from a file path to a `Digest` to materialize at that path. It is merged into the `input_digest` of a `Process` to form a "complete" `Digest` of inputs.

But this separation is a bit unfortunate, because it means that using `immutable_inputs` (which should be fairly widely used: see discussion on #14070) means a lot of change to how your `Process` is constructed.

----

From an API-changes perspective, we should:
1. remove the guarantee that inputs are mutable by default, to allow us to automatically choose the strategy that we use for each path, and symlink wherever we choose to.
    * Then, by default, we should automatically use symlinking for files or directories over a certain size, etc.
2. convert the existing `immutable_inputs` list into a flat list of paths which should be _forced_/includelisted to be materialized as immutable (regardless of the heuristic from above), but which must already exist in the `input_digest` (maybe "force" should go in the name? `force_immutable_paths=` ...?).
    * This will mean that unlike today (where you need to change how your `Process` is constructed to stop adding an input to the `input_digests` and instead add it to the `immutable_inputs` as a dict), you would only need to add a path to the `immutable_inputs`, _without_ changing your `input_digest`.
3. add an `mutable_inputs` (or `force_mutable_paths=`...?) flat list of paths to act as an excludelist that _prevents_ a path from being materialized as a symlink.

----

From an implementation perspective, this would look like moving the support for creating the symlinks for `ImmutableInputs` from an explicit step before running a process:
https://github.com/pantsbuild/pants/blob/08410e5b94d3e3aa86b2538ee1991d964e27e72c/src/rust/engine/process_execution/src/local.rs#L641-L662
...to a thing that happens inside `fn materialize_directory` when an includelisted path is encountered:
https://github.com/pantsbuild/pants/blob/08410e5b94d3e3aa86b2538ee1991d964e27e72c/src/rust/engine/fs/store/src/lib.rs#L1188-L1201

`materialize_directory(_helper)` would take either:
1. a reference to a collection of paths which were includelisted as immutable
2. a trait or function which implemented the heuristic for which `directory::Entry`s / paths to includelist

...and a reference to `ImmutableInputs`, and then would create a symlink to the `ImmutableInputs` whenever it encountered one of the includelisted paths.
Use immutable_inputs for `PEX`s
#13848 (designed in #12716) added support for immutable process inputs, which are symlinked into the sandbox, rather than copied there. Once immutable inputs have stabilized (notably, once #13899 is fixed), we should provide the `repository.pex` as an immutable input to significantly reduce IO.

It's also possible that in most cases where a PEX is used as an input to a process, that supplying it as an immutable input would make sense: in particular, cases where a PEX contains only thirdparty code, and firstparty code is materialized as loose files.


Discussion:
In the most common case (fixers) couldn't we deduce the mutable paths by the intersection of the input and output paths?

The output paths for any process will either a) not exist yet or b) do exist (from the inputs) and therefore need to be mutable (or else why are you capturing them)
> In the most common case (fixers) couldn't we deduce the mutable paths by the intersection of the input and output paths?
> 
> The output paths for any process will either a) not exist yet or b) do exist (from the inputs) and therefore need to be mutable (or else why are you capturing them)

Yea, for sure.
This is currently blocked on #13899.
#13899 has been stably fixed for a while, so we should be good to do this now.
Is this too risky to backport to 2.13? It sounds like very low hanging performance fruit
> Is this too risky to backport to 2.13? It sounds like very low hanging performance fruit

Depends how large the patch was probably. But 2.13 is already pretty large (~7 weeks).
@stuhood I'm interested in working on this one 👀

The only reference to `repository.pex` I see is [here](https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/util_rules/pex_from_targets.py#L644). Walking backwards from there, the code jumps through various rules & helpers before landing on [generalize_requirements_pex_request](https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/util_rules/pex_from_targets.py#L683), which I see is invoked (via `Get(RequirementsPexRequest, ...)`) in a handful of modules under `src/python/pants/backend/python/`. Is the idea to update how the outputs of those invocations are used, so they're linked in as immutable inputs?
Without fully answering your question (@Eric-Arellano would probably be better equipped for that), I'll say that: `repository.pex` is no longer used with PEX lockfiles (only with non-PEX/native lockfiles), because #14127 moved to using PEX's lockfile subsetting support directly.
Oh nice - does that mean this is no longer relevant since non-PEX lockfiles are being deprecated?
(Just thinking "out loud") I see that `VenvPexProcess` doesn't have an `immutable_input_digests` field to match the plain `Process` - maybe this ticket evolves to not focus on `repository.pex`, but instead add that plumbing & ensure we're using it appropriately everywhere we can?
What are the downsides of using `immutable_inputs`, beyond code complexity? I'm wondering if we should use this ~every where? Like `pytest_runner.pex` for example.
> maybe this ticket evolves to not focus on `repository.pex`, but instead add that plumbing & ensure we're using it appropriately everywhere we can?

Correct.

> What are the downsides of using `immutable_inputs`, beyond code complexity? I'm wondering if we should use this ~every where? Like `pytest_runner.pex` for example.

The overhead of something being used as an immutable input is very low, but not zero: so if something is always used exactly once as an input, it won't be worth it to use it that way.

But yes: I expect that almost all PEXes will make sense to provide as `immutable_inputs`.

----

One other consideration before starting this would just be whether the API of `immutable_inputs` should be adjusted at all. We had briefly discussed adjusting it from a `dict[path, Digest]` to a `set[path]`... i.e., _paths_ in the sandbox are marked as being materialized immutably, and are assumed to already exist in the input `Digest` for a process. The [`materialize_digest` call in local process execution](https://github.com/pantsbuild/pants/blob/8b03d1340be5ce00cc12bb2efc202556179c20dd/src/rust/engine/process_execution/src/local.rs#L658-L672) would then take the `set[path]`, and when it encountered one of those paths, would materialize it as immutable rather than mutable (replacing [this code](https://github.com/pantsbuild/pants/blob/8b03d1340be5ce00cc12bb2efc202556179c20dd/src/rust/engine/process_execution/src/local.rs#L637-L639)).
> But yes: I expect that almost all PEXes will make sense to provide as immutable_inputs.

Maybe starting with tool PEXes would make sense? For almost every workflow, we materialize the same PEX multiple times: `pytest.pex`, `black.pex`, and so on. That's because Pants often partitions things, like Pytest running one `python_test` at at time.

I suspect we also materialize the same `requirements.pex` multiple times, but it's less of a guarantee because there are theoretically as many `requirements.pex` as unique combinations of your third-party requirements.

Those tool PEXes all get built using this helper:

https://github.com/pantsbuild/pants/blob/8b03d1340be5ce00cc12bb2efc202556179c20dd/src/python/pants/backend/python/subsystems/python_tool_base.py#L212-L228

> One other consideration before starting this would just be whether the API of immutable_inputs should be adjusted at all.

While I agree with the change, I don't think we should block on it. This is an internal API mostly, and it's not very costly for us to change now vs in a month or two. Sounds like neither you or me have time this month to do this, so it'd be better to not block @danxmoran making an awesome performance improvement.

> does that mean this is no longer relevant since non-PEX lockfiles are being deprecated?

`repository.pex` will still exist because we will continue to support manually-managed lockfiles using `requirements.txt` style. There, we will use `repository.pex`. We only are deprecating Pants generating `requirements.txt`-style locks via Poetry with the `generate-lockfiles` goal, given how many issues there are like `[python-repos]` not being hooked up.

So, `repository.pex` will still exist, just most users won't trigger that codepath, so it's lower priority.
> What are the downsides of using `immutable_inputs`

Because it's one of those things that belong in the giant (useful) hack list. Just like using Pex for its packed layout instead of using Pip and venvs directly. We really need to keep clear eyes about where our hacks are and why and when we need them. Packed layout reduces most tool venvs to O(10) files (zips) I think, so the immutable inputs hack is only likely useful for user packed PEXes with O(100) files.

In other words, our hacks here are:
1. Use packed PEXes to reduce O(10k) files to O(10-100) files because materializing sandboxes is slow.
2. Use immutable inputs to further hack that and bring down the materialization of the O(10-100) files to a 1 time event per pantsd restart.
Thanks for explaining that, John. Iiuc, it's not only the # of files that is an issue, but the size of those files? Symlinking a large file is much faster than copying it? So, even though tool PEXes are roughly O(10), we may still get a speed up?
> > One other consideration before starting this would just be whether the API of immutable_inputs should be adjusted at all.
> 
> While I agree with the change, I don't think we should block on it. This is an internal API mostly, and it's not very costly for us to change now vs in a month or two. Sounds like neither you or me have time this month to do this, so it'd be better to not block @danxmoran making an awesome performance improvement.

I would consider blocking on it, just because I'm sure that @danxmoran is capable of doing it, and doing it would _drastically_ reduce the API surface area of the change. See #13862 for an example of converting the JVM to use `immutable_inputs` with the current API.

With the altered API, using `immutable_inputs` would instead amount to includelisting a few paths on the Process constructor.
> So, even though tool PEXes are roughly O(10), we may still get a speed up?

Yes, but tools are small in practice in addition to having small dependency sets. So again we're only likely to hit this in user packed PEXes.
I've opened #17282 to cover the API change that I think would be worth making before working on this. It's also possible that depending on the heuristic that is used there (i.e. if it is size based), then there wouldn't actually be any PEX/Python-specific work to do here.

Resolving patch:
diff --git a/conftest.py b/conftest.py
--- a/conftest.py
+++ b/conftest.py
@@ -1,6 +1,7 @@
 # Copyright 2022 Pants project contributors (see CONTRIBUTORS.md).
 # Licensed under the Apache License, Version 2.0 (see LICENSE).
 
+
 import pytest
 
 


Test patch:
diff --git a/src/rust/engine/fs/store/src/tests.rs b/src/rust/engine/fs/store/src/tests.rs
--- a/src/rust/engine/fs/store/src/tests.rs
+++ b/src/rust/engine/fs/store/src/tests.rs
@@ -1,4 +1,4 @@
-use std::collections::{BTreeMap, HashMap, HashSet};
+use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
 use std::fs::File;
 use std::io::Read;
 use std::os::unix::fs::PermissionsExt;
@@ -9,7 +9,8 @@ use testutil::data::{TestData, TestDirectory};
 
 use bytes::{Bytes, BytesMut};
 use fs::{
-  DigestEntry, DirectoryDigest, FileEntry, Link, PathStat, Permissions, EMPTY_DIRECTORY_DIGEST,
+  DigestEntry, DirectoryDigest, FileEntry, Link, PathStat, Permissions, RelativePath,
+  EMPTY_DIRECTORY_DIGEST,
 };
 use grpc_util::prost::MessageExt;
 use grpc_util::tls;
@@ -19,7 +20,8 @@ use protos::gen::build::bazel::remote::execution::v2 as remexec;
 use workunit_store::WorkunitStore;
 
 use crate::{
-  EntryType, FileContent, Snapshot, Store, StoreError, StoreFileByDigest, UploadSummary, MEGABYTES,
+  EntryType, FileContent, ImmutableInputs, Snapshot, Store, StoreError, StoreFileByDigest,
+  UploadSummary, MEGABYTES,
 };
 
 pub(crate) const STORE_BATCH_API_SIZE_LIMIT: usize = 4 * 1024 * 1024;
@@ -1046,7 +1048,12 @@ async fn materialize_missing_file() {
   let store_dir = TempDir::new().unwrap();
   let store = new_local_store(store_dir.path());
   store
-    .materialize_file(file.clone(), TestData::roland().digest(), 0o644)
+    .materialize_file(
+      file.clone(),
+      TestData::roland().digest(),
+      Permissions::ReadOnly,
+      false,
+    )
     .await
     .expect_err("Want unknown digest error");
 }
@@ -1065,7 +1072,12 @@ async fn materialize_file() {
     .await
     .expect("Error saving bytes");
   store
-    .materialize_file(file.clone(), testdata.digest(), 0o644)
+    .materialize_file(
+      file.clone(),
+      testdata.digest(),
+      Permissions::ReadOnly,
+      false,
+    )
     .await
     .expect("Error materializing file");
   assert_eq!(file_contents(&file), testdata.bytes());
@@ -1082,6 +1094,8 @@ async fn materialize_missing_directory() {
     .materialize_directory(
       materialize_dir.path().to_owned(),
       TestDirectory::recursive().directory_digest(),
+      &BTreeSet::new(),
+      None,
       Permissions::Writable,
     )
     .await
@@ -1114,6 +1128,8 @@ async fn materialize_directory(perms: Permissions, executable_file: bool) {
     .materialize_directory(
       materialize_dir.path().to_owned(),
       recursive_testdir.directory_digest(),
+      &BTreeSet::new(),
+      None,
       perms,
     )
     .await
@@ -1584,6 +1600,8 @@ async fn explicitly_overwrites_already_existing_file() {
     .materialize_directory(
       dir_to_write_to.path().to_owned(),
       contents_dir.directory_digest(),
+      &BTreeSet::new(),
+      None,
       Permissions::Writable,
     )
     .await
@@ -1592,3 +1610,89 @@ async fn explicitly_overwrites_already_existing_file() {
   let file_contents = std::fs::read(&file_path).unwrap();
   assert_eq!(file_contents, b"abc123".to_vec());
 }
+
+#[tokio::test]
+async fn big_file_immutable_link() {
+  let materialize_dir = TempDir::new().unwrap();
+  let input_file = materialize_dir.path().join("input_file");
+  let output_file = materialize_dir.path().join("output_file");
+  let output_dir = materialize_dir.path().join("output_dir");
+  let nested_output_file = output_dir.join("file");
+
+  let file_bytes = extra_big_file_bytes();
+  let file_digest = extra_big_file_digest();
+
+  let nested_directory = remexec::Directory {
+    files: vec![remexec::FileNode {
+      name: "file".to_owned(),
+      digest: Some(file_digest.into()),
+      is_executable: true,
+      ..remexec::FileNode::default()
+    }],
+    ..remexec::Directory::default()
+  };
+  let directory = remexec::Directory {
+    files: vec![
+      remexec::FileNode {
+        name: "input_file".to_owned(),
+        digest: Some(file_digest.into()),
+        is_executable: true,
+        ..remexec::FileNode::default()
+      },
+      remexec::FileNode {
+        name: "output_file".to_owned(),
+        digest: Some(file_digest.into()),
+        is_executable: true,
+        ..remexec::FileNode::default()
+      },
+    ],
+    directories: vec![remexec::DirectoryNode {
+      name: "output_dir".to_string(),
+      digest: Some(hashing::Digest::of_bytes(&nested_directory.to_bytes()).into()),
+    }],
+    ..remexec::Directory::default()
+  };
+  let directory_digest =
+    fs::DirectoryDigest::from_persisted_digest(hashing::Digest::of_bytes(&directory.to_bytes()));
+
+  let store_dir = TempDir::new().unwrap();
+  let store = new_local_store(store_dir.path());
+  let immutable_inputs_dir = TempDir::new().unwrap();
+  let immutable_inputs = ImmutableInputs::new(store.clone(), immutable_inputs_dir.path()).unwrap();
+  store
+    .record_directory(&nested_directory, false)
+    .await
+    .expect("Error saving Directory");
+  store
+    .record_directory(&directory, false)
+    .await
+    .expect("Error saving Directory");
+  store
+    .store_file_bytes(file_bytes.clone(), false)
+    .await
+    .expect("Error saving bytes");
+
+  store
+    .materialize_directory(
+      materialize_dir.path().to_owned(),
+      directory_digest,
+      &BTreeSet::from([
+        RelativePath::new("output_file").unwrap(),
+        RelativePath::new("output_dir").unwrap(),
+      ]),
+      Some(&immutable_inputs),
+      Permissions::Writable,
+    )
+    .await
+    .expect("Error materializing file");
+
+  let assert_is_linked = |path: &PathBuf, is_linked: bool| {
+    assert_eq!(file_contents(&path), file_bytes);
+    assert!(is_executable(&path));
+    assert_eq!(path.metadata().unwrap().permissions().readonly(), is_linked);
+  };
+
+  assert_is_linked(&input_file, true);
+  assert_is_linked(&output_file, false);
+  assert_is_linked(&nested_output_file, false);
+}
diff --git a/src/rust/engine/process_execution/src/cache_tests.rs b/src/rust/engine/process_execution/src/cache_tests.rs
--- a/src/rust/engine/process_execution/src/cache_tests.rs
+++ b/src/rust/engine/process_execution/src/cache_tests.rs
@@ -4,7 +4,7 @@ use std::path::PathBuf;
 
 use cache::PersistentCache;
 use sharded_lmdb::DEFAULT_LEASE_TIME;
-use store::Store;
+use store::{ImmutableInputs, Store};
 use tempfile::TempDir;
 use testutil::data::TestData;
 use testutil::relative_paths;
@@ -12,7 +12,7 @@ use workunit_store::{RunningWorkunit, WorkunitStore};
 
 use crate::{
   local::KeepSandboxes, CacheContentBehavior, CommandRunner as CommandRunnerTrait, Context,
-  FallibleProcessResultWithPlatform, ImmutableInputs, NamedCaches, Process, ProcessError,
+  FallibleProcessResultWithPlatform, NamedCaches, Process, ProcessError,
 };
 
 struct RoundtripResults {
diff --git a/src/rust/engine/process_execution/src/local_tests.rs b/src/rust/engine/process_execution/src/local_tests.rs
--- a/src/rust/engine/process_execution/src/local_tests.rs
+++ b/src/rust/engine/process_execution/src/local_tests.rs
@@ -9,7 +9,7 @@ use spectral::{assert_that, string::StrAssertions};
 use tempfile::TempDir;
 
 use fs::EMPTY_DIRECTORY_DIGEST;
-use store::Store;
+use store::{ImmutableInputs, Store};
 use testutil::data::{TestData, TestDirectory};
 use testutil::path::{find_bash, which};
 use testutil::{owned_string_vec, relative_paths};
@@ -17,8 +17,8 @@ use workunit_store::{RunningWorkunit, WorkunitStore};
 
 use crate::{
   local, local::KeepSandboxes, CacheName, CommandRunner as CommandRunnerTrait, Context,
-  FallibleProcessResultWithPlatform, ImmutableInputs, InputDigests, NamedCaches, Platform, Process,
-  ProcessError, RelativePath,
+  FallibleProcessResultWithPlatform, InputDigests, NamedCaches, Platform, Process, ProcessError,
+  RelativePath,
 };
 
 #[derive(PartialEq, Debug)]
diff --git a/src/rust/engine/process_execution/src/nailgun/tests.rs b/src/rust/engine/process_execution/src/nailgun/tests.rs
--- a/src/rust/engine/process_execution/src/nailgun/tests.rs
+++ b/src/rust/engine/process_execution/src/nailgun/tests.rs
@@ -1,13 +1,13 @@
 use std::path::PathBuf;
 
-use store::Store;
+use store::{ImmutableInputs, Store};
 use task_executor::Executor;
 use tempfile::TempDir;
 use testutil::owned_string_vec;
 use workunit_store::WorkunitStore;
 
 use crate::nailgun::NailgunPool;
-use crate::{ImmutableInputs, NamedCaches, Process};
+use crate::{NamedCaches, Process};
 
 fn pool(size: usize) -> (NailgunPool, NamedCaches, ImmutableInputs) {
   let _ = WorkunitStore::setup_for_tests();