Repository: pantsbuild/pants

Issue:
Dirty the dependents of an uncacheable Node
#6146 deals with adding support for never storing the result of a Node that is not `cacheable`: this is used to avoid memoizing the result of a `@console_rule` execution. `@console_rule`s are "effectively" roots in the graph... but not _technically_ roots in the graph: the actual roots are `Select` nodes in the graph. In order to land #6146, `Select` was also marked `!cacheable` (which is totally fine as a workaround, because `Select` nodes are always cheap to run).

But in order to use uncacheable Nodes for usecases involving non-root Nodes (such as the interactive processes defined in #6002), we will need to dirty the dependents of an uncacheable Node each time it runs. This will have the effect of causing the dirtied dependents to re-evaluate whether they need to run (and consider whether they can be "cleaned"), while always re-triggering the uncacheable Node (either as part of the cleaning process, or as part of re-running one of the dirtied dependents).


Discussion:
A thing that occurred to me while reviewing #7350, is that rather than `@rules` being uncacheable, it might always be particular _parameters_ that are uncacheable. It's already the case that our `@console_rule` un-cacheability is protected by `Console` intentionally not having an `__eq__` impl... but formalizing the hack such that any `@rule` with an uncachable parameter is uncachable would... fix this issue I think? We would leak some memory for each abandoned subgraph, but probably fine until we begin implementing garbage collection for the `Graph`?
> We would leak some memory for each abandoned subgraph

Is this just the inevitable consequence of allowing uncacheable rules (because their values can never be retrieved) / are you saying this is basically what happens now with `@console_rule`s?
> > We would leak some memory for each abandoned subgraph
> 
> Is this just the inevitable consequence of allowing uncacheable rules (because their values can never be retrieved) / are you saying this is basically what happens now with `@console_rule`s?

No... it's just code that we would have to write. Basically: the effect of using a value that is never equal to another value as a key in the graph is that the node is immediately orphaned after use (because no other instance of the type will ever match it). Implementing GC would cause it to be cleaned up eventually. But if you formalized this, you would want to eagerly discard or clear those nodes at the end of a `Session`, or something.
Post #6880, and with a solid solution for this issue (that makes it very obvious what is cacheable and what isn't), I think that we should be able to remove `@console_rule`.

Resolving patch:
diff --git a/src/python/pants/engine/scheduler.py b/src/python/pants/engine/scheduler.py
--- a/src/python/pants/engine/scheduler.py
+++ b/src/python/pants/engine/scheduler.py
@@ -118,9 +118,9 @@ def __init__(
   def _root_type_ids(self):
     return self._to_ids_buf(self._root_subject_types)
 
-  def graph_trace(self, execution_request):
+  def graph_trace(self, session, execution_request):
     with temporary_file_path() as path:
-      self._native.lib.graph_trace(self._scheduler, execution_request, path.encode())
+      self._native.lib.graph_trace(self._scheduler, session, execution_request, path.encode())
       with open(path, 'r') as fd:
         for line in fd.readlines():
           yield line.rstrip()
@@ -302,8 +302,8 @@ def _run_and_return_roots(self, session, execution_request):
                            '`self._native._peek_cffi_extern_method_runtime_exceptions()`.')
     return roots
 
-  def lease_files_in_graph(self):
-    self._native.lib.lease_files_in_graph(self._scheduler)
+  def lease_files_in_graph(self, session):
+    self._native.lib.lease_files_in_graph(self._scheduler, session)
 
   def garbage_collect_store(self):
     self._native.lib.garbage_collect_store(self._scheduler)
@@ -350,7 +350,7 @@ def graph_len(self):
 
   def trace(self, execution_request):
     """Yields a stringified 'stacktrace' starting from the scheduler's roots."""
-    for line in self._scheduler.graph_trace(execution_request.native):
+    for line in self._scheduler.graph_trace(self._session, execution_request.native):
       yield line
 
   def visualize_graph_to_file(self, filename):
@@ -613,7 +613,7 @@ def materialize_directories(
     return result
 
   def lease_files_in_graph(self):
-    self._scheduler.lease_files_in_graph()
+    self._scheduler.lease_files_in_graph(self._session)
 
   def garbage_collect_store(self):
     self._scheduler.garbage_collect_store()


Test patch:
diff --git a/src/rust/engine/graph/src/tests.rs b/src/rust/engine/graph/src/tests.rs
--- a/src/rust/engine/graph/src/tests.rs
+++ b/src/rust/engine/graph/src/tests.rs
@@ -19,7 +19,7 @@ use crate::{EntryId, Graph, InvalidationResult, Node, NodeContext, NodeError};
 #[test]
 fn create() {
   let graph = Arc::new(Graph::new());
-  let context = TContext::new(0, graph.clone());
+  let context = TContext::new(graph.clone());
   assert_eq!(
     graph.create(TNode(2), &context).wait(),
     Ok(vec![T(0, 0), T(1, 0), T(2, 0)])
@@ -29,7 +29,7 @@ fn create() {
 #[test]
 fn invalidate_and_clean() {
   let graph = Arc::new(Graph::new());
-  let context = TContext::new(0, graph.clone());
+  let context = TContext::new(graph.clone());
 
   // Create three nodes.
   assert_eq!(
@@ -58,14 +58,14 @@ fn invalidate_and_clean() {
 #[test]
 fn invalidate_and_rerun() {
   let graph = Arc::new(Graph::new());
-  let context0 = TContext::new(0, graph.clone());
+  let context = TContext::new(graph.clone());
 
   // Create three nodes.
   assert_eq!(
-    graph.create(TNode(2), &context0).wait(),
+    graph.create(TNode(2), &context).wait(),
     Ok(vec![T(0, 0), T(1, 0), T(2, 0)])
   );
-  assert_eq!(context0.runs(), vec![TNode(2), TNode(1), TNode(0)]);
+  assert_eq!(context.runs(), vec![TNode(2), TNode(1), TNode(0)]);
 
   // Clear the middle Node, which dirties the upper node.
   assert_eq!(
@@ -76,20 +76,20 @@ fn invalidate_and_rerun() {
     }
   );
 
-  // Request with a new context, which will cause both the middle and upper nodes to rerun since
+  // Request with a different salt, which will cause both the middle and upper nodes to rerun since
   // their input values have changed.
-  let context1 = TContext::new(1, graph.clone());
+  let context = context.new_session(1).with_salt(1);
   assert_eq!(
-    graph.create(TNode(2), &context1).wait(),
+    graph.create(TNode(2), &context).wait(),
     Ok(vec![T(0, 0), T(1, 1), T(2, 1)])
   );
-  assert_eq!(context1.runs(), vec![TNode(1), TNode(2)]);
+  assert_eq!(context.runs(), vec![TNode(1), TNode(2)]);
 }
 
 #[test]
 fn invalidate_with_changed_dependencies() {
   let graph = Arc::new(Graph::new());
-  let context = TContext::new(0, graph.clone());
+  let context = TContext::new(graph.clone());
 
   // Create three nodes.
   assert_eq!(
@@ -107,11 +107,8 @@ fn invalidate_with_changed_dependencies() {
   );
 
   // Request with a new context that truncates execution at the middle Node.
-  let context = TContext::new_with_dependencies(
-    0,
-    vec![(TNode(1), None)].into_iter().collect(),
-    graph.clone(),
-  );
+  let context =
+    TContext::new(graph.clone()).with_dependencies(vec![(TNode(1), None)].into_iter().collect());
   assert_eq!(
     graph.create(TNode(2), &context).wait(),
     Ok(vec![T(1, 0), T(2, 0)])
@@ -160,7 +157,7 @@ fn invalidate_randomly() {
   let mut iterations = 0;
   let mut max_distinct_context_values = 0;
   loop {
-    let context = TContext::new(iterations, graph.clone());
+    let context = TContext::new(graph.clone()).with_salt(iterations);
 
     // Compute the root, and validate its output.
     let node_output = match graph.create(TNode(range), &context).wait() {
@@ -194,6 +191,42 @@ fn invalidate_randomly() {
   );
 }
 
+#[test]
+fn dirty_dependents_of_uncacheable_node() {
+  let graph = Arc::new(Graph::new());
+
+  // Create a context for which the bottommost Node is not cacheable.
+  let context = {
+    let mut uncacheable = HashSet::new();
+    uncacheable.insert(TNode(0));
+    TContext::new(graph.clone()).with_uncacheable(uncacheable)
+  };
+
+  // Create three nodes.
+  assert_eq!(
+    graph.create(TNode(2), &context).wait(),
+    Ok(vec![T(0, 0), T(1, 0), T(2, 0)])
+  );
+  assert_eq!(context.runs(), vec![TNode(2), TNode(1), TNode(0)]);
+
+  // Re-request the root in a new session and confirm that only the bottom node re-runs.
+  let context = context.new_session(1);
+  assert_eq!(
+    graph.create(TNode(2), &context).wait(),
+    Ok(vec![T(0, 0), T(1, 0), T(2, 0)])
+  );
+  assert_eq!(context.runs(), vec![TNode(0)]);
+
+  // Re-request with a new session and different salt, and confirm that everything re-runs bottom
+  // up (the order of node cleaning).
+  let context = context.new_session(2).with_salt(1);
+  assert_eq!(
+    graph.create(TNode(2), &context).wait(),
+    Ok(vec![T(0, 1), T(1, 1), T(2, 1)])
+  );
+  assert_eq!(context.runs(), vec![TNode(0), TNode(1), TNode(2)]);
+}
+
 #[test]
 fn drain_and_resume() {
   // Confirms that after draining a Graph that has running work, we are able to resume the work
@@ -208,7 +241,7 @@ fn drain_and_resume() {
   let context = {
     let mut delays = HashMap::new();
     delays.insert(TNode(1), delay_in_task);
-    TContext::new_with_delays(0, delays, graph.clone())
+    TContext::new(graph.clone()).with_delays(delays)
   };
 
   // Spawn a background thread that will mark the Graph draining after a short delay.
@@ -243,11 +276,9 @@ fn cyclic_failure() {
   // Confirms that an attempt to create a cycle fails.
   let graph = Arc::new(Graph::new());
   let top = TNode(2);
-  let context = TContext::new_with_dependencies(
-    0,
+  let context = TContext::new(graph.clone()).with_dependencies(
     // Request creation of a cycle by sending the bottom most node to the top.
     vec![(TNode(0), Some(top))].into_iter().collect(),
-    graph.clone(),
   );
 
   assert_eq!(graph.create(TNode(2), &context).wait(), Err(TError::Cyclic));
@@ -262,7 +293,7 @@ fn cyclic_dirtying() {
   let initial_bot = TNode(0);
 
   // Request with a context that creates a path downward.
-  let context_down = TContext::new(0, graph.clone());
+  let context_down = TContext::new(graph.clone());
   assert_eq!(
     graph.create(initial_top.clone(), &context_down).wait(),
     Ok(vec![T(0, 0), T(1, 0), T(2, 0)])
@@ -270,13 +301,11 @@ fn cyclic_dirtying() {
 
   // Clear the bottom node, and then clean it with a context that causes the path to reverse.
   graph.invalidate_from_roots(|n| n == &initial_bot);
-  let context_up = TContext::new_with_dependencies(
-    1,
+  let context_up = context_down.with_salt(1).with_dependencies(
     // Reverse the path from bottom to top.
     vec![(TNode(1), None), (TNode(0), Some(TNode(1)))]
       .into_iter()
       .collect(),
-    graph.clone(),
   );
 
   let res = graph.create(initial_bot, &context_up).wait();
@@ -407,7 +436,7 @@ impl Node for TNode {
 
   fn run(self, context: TContext) -> BoxFuture<Vec<T>, TError> {
     context.ran(self.clone());
-    let token = T(self.0, context.id());
+    let token = T(self.0, context.salt());
     if let Some(dep) = context.dependency_of(&self) {
       context.maybe_delay(&self);
       context
@@ -426,8 +455,8 @@ impl Node for TNode {
     None
   }
 
-  fn cacheable(&self) -> bool {
-    true
+  fn cacheable(&self, context: &Self::Context) -> bool {
+    !context.uncacheable.contains(self)
   }
 }
 
@@ -489,30 +518,43 @@ impl TNode {
 ///
 #[derive(Clone)]
 struct TContext {
-  id: usize,
+  session_id: usize,
+  // A value that is included in every value computed by this context. Stands in for "the state of the
+  // outside world". A test that wants to "change the outside world" and observe its effect on the
+  // graph should change the salt to do so.
+  salt: usize,
   // A mapping from source to optional destination that drives what values each TNode depends on.
   // If there is no entry in this map for a node, then TNode::run will default to requesting
   // the next smallest node. Finally, if a None entry is present, a node will have no
   // dependencies.
   edges: Arc<HashMap<TNode, Option<TNode>>>,
-  delays: HashMap<TNode, Duration>,
+  delays: Arc<HashMap<TNode, Duration>>,
+  uncacheable: Arc<HashSet<TNode>>,
   graph: Arc<Graph<TNode>>,
   runs: Arc<Mutex<Vec<TNode>>>,
   entry_id: Option<EntryId>,
 }
 impl NodeContext for TContext {
   type Node = TNode;
+  type SessionId = usize;
+
   fn clone_for(&self, entry_id: EntryId) -> TContext {
     TContext {
-      id: self.id,
+      session_id: self.session_id,
+      salt: self.salt,
       edges: self.edges.clone(),
       delays: self.delays.clone(),
+      uncacheable: self.uncacheable.clone(),
       graph: self.graph.clone(),
       runs: self.runs.clone(),
       entry_id: Some(entry_id),
     }
   }
 
+  fn session_id(&self) -> &usize {
+    &self.session_id
+  }
+
   fn graph(&self) -> &Graph<TNode> {
     &self.graph
   }
@@ -529,49 +571,50 @@ impl NodeContext for TContext {
 }
 
 impl TContext {
-  fn new(id: usize, graph: Arc<Graph<TNode>>) -> TContext {
+  fn new(graph: Arc<Graph<TNode>>) -> TContext {
     TContext {
-      id,
+      session_id: 0,
+      salt: 0,
       edges: Arc::default(),
-      delays: HashMap::default(),
+      delays: Arc::default(),
+      uncacheable: Arc::default(),
       graph,
       runs: Arc::new(Mutex::new(Vec::new())),
       entry_id: None,
     }
   }
 
-  fn new_with_dependencies(
-    id: usize,
-    edges: HashMap<TNode, Option<TNode>>,
-    graph: Arc<Graph<TNode>>,
-  ) -> TContext {
-    TContext {
-      id,
-      edges: Arc::new(edges),
-      delays: HashMap::default(),
-      graph,
-      runs: Arc::new(Mutex::new(Vec::new())),
-      entry_id: None,
-    }
+  fn with_dependencies(mut self, edges: HashMap<TNode, Option<TNode>>) -> TContext {
+    self.edges = Arc::new(edges);
+    self
   }
 
-  fn new_with_delays(
-    id: usize,
-    delays: HashMap<TNode, Duration>,
-    graph: Arc<Graph<TNode>>,
-  ) -> TContext {
-    TContext {
-      id,
-      edges: Arc::default(),
-      delays,
-      graph,
-      runs: Arc::new(Mutex::new(Vec::new())),
-      entry_id: None,
+  fn with_delays(mut self, delays: HashMap<TNode, Duration>) -> TContext {
+    self.delays = Arc::new(delays);
+    self
+  }
+
+  fn with_uncacheable(mut self, uncacheable: HashSet<TNode>) -> TContext {
+    self.uncacheable = Arc::new(uncacheable);
+    self
+  }
+
+  fn with_salt(mut self, salt: usize) -> TContext {
+    self.salt = salt;
+    self
+  }
+
+  fn new_session(mut self, new_session_id: usize) -> TContext {
+    self.session_id = new_session_id;
+    {
+      let mut runs = self.runs.lock();
+      runs.clear();
     }
+    self
   }
 
-  fn id(&self) -> usize {
-    self.id
+  fn salt(&self) -> usize {
+    self.salt
   }
 
   fn get(&self, dst: TNode) -> BoxFuture<Vec<T>, TError> {