Repository: pantsbuild/pants

Issue:
Mark nodes "dirty" during invalidation, rather than deleting them
After a change event and during invalidation, we transitively delete `Nodes` from the graph. But rather than deleting, we should mark `Nodes` "dirty" instead.

As describe in the `bazel`/`skyframe` design, and in the `adapton` design, marking a `Node` dirty allows it to be "resurrected" without re-running if all of its inputs are subsequently marked clean with the same values they had before.

This is an extremely efficient cache of the previous execution, because it can happen purely in memory.
Inlining of PathGlob expansion causes excess invalidation with the daemon
Inlining of `PathGlobsExpansion` causes `Snapshots` to be more fragile than necessary. In particular, because `VFS::expand` recursively walks from the root down to the relevant directories without memoizing `PathGlobsExpansion`s, any change in the root invalidates _all_ `Snapshots` from a changed path up to the root of the repo (but not the directory listings and stats).

This significantly affects warm performance in the presence of the daemon.

We should experiment with memoizing (portions of?) `PathGlobsExpansion` in the `Graph` (similar to what we do for `ReadLink`/`Scandir`), and see how much it affects cold/warm performance.

NB: As a wildcard: it's possible that fixing #4558 would be a better approach than increasing memoization would be.

----

The relevant codepath centers around https://github.com/pantsbuild/pants/blob/cdba694c38b4d60342c75a44ff97c5642429e585/src/rust/engine/fs/src/lib.rs#L570-L576 . When we call that method from the engine (here: https://github.com/pantsbuild/pants/blob/01bcda6e3c352d99dc6a583ed57295d3a4c5afe8/src/rust/engine/src/nodes.rs#L913-L940), the `VFS` in question is https://github.com/pantsbuild/pants/blob/01bcda6e3c352d99dc6a583ed57295d3a4c5afe8/src/rust/engine/src/nodes.rs#L73 , which has the effect of memoizing/recording `ReadLink` and `Scandir` calls in the `Graph` (which later allows for filesystem invalidation).

The effect of memoizing `ReadLink` and `Scandir` (but none of the rest of the other steps of a `PathGlobsExpansion`) is that a `Snapshot` `Node` in the `Graph` is flat: the `Snapshot` depends directly on the syscalls, and to recompute it the entire `VFS::expand` call reruns.

----

When we fix this, it would be good to get a simple benchmark in place on the rust side. But it's also important to directly measure the performance of:
* cold `./pants list ::`
* warm `./pants list ::` after touching one file at the root of the repo


Discussion:
Relates to #3365.
Also relates to #4394... it's possible that implementing this issue would make fixing that one unnecessary.
For me, this maps onto how `.pants.d` workdirs are handled in v1, where running invalidate causes pants to check the existing workdirs if they still end up with the same hashes, but may result in less work being performed depending on the task.
> but may result in less work being performed depending on the task.

Actually, we never reuse them currently... they're always cleared if we toggle back to an older version. They're mostly stable for the purposes of symlinks into `dist`, etc.
Assigned to @jsirois 
@jsirois : One thing I hadn't thought of (and which you may now have thought of): we currently don't account for equality of `Node::Output` types... a `TaskNode` outputs `core::Value` instances, which we do not currently implement equality (in the form of rust's `Eq`) for. In places where we need both `Eq` and `Hash` we convert to a `core::Key`, which memoizes the `core::Value` to assign it a unique id... at least in the case of putting a python value in a hashtable, I think the Key abstraction is worthwhile, because we avoid a very large number of calls into python.

I think that in the case of un-dirtying a `Node`, you would only need equality (and not hash)... in that case, I think that you should be able to use `externs::call_method(value1, '__equals__', &[value2])`, to avoid converting to a `Key`.

EDIT: An update on this: I added a `externs::equals(Value, Value) -> Boolean` method, so this should be unblocked... although we might have to remove a few cases where we override `__eq__` for performance.
1) I'm slightly worried about pantsd memory utilization with this model (particularly for large repos in e.g. the `::` case) - do we have an eviction strategy in place yet for `Graph` nodes?

2) we'll probably get this as a side-effect of the new model, but it'd be great to think about optimizing the backing impl here for invalidation performance. as a random datapoint, it takes ~30s currently to invalidate from just touching the finagle root dirs in our monorepo:

```
I0202 21:13:32.313214 18899 scheduler_service.py:55] enqueuing 2 changes for subscription all_files
I0202 21:14:02.581753 18899 scheduler.py:402] invalidated 356056 nodes for: set([u'', u'finagle-internal', u'finagle'])
```

(this may also be exacerbated by #4394 tho)
> I'm slightly worried about pantsd memory utilization with this model (particularly for large repos in e.g. the :: case) - do we have an eviction strategy in place yet for Graph nodes?

Since we can't have two copies of the same Node (ie, with the same NodeKey), I think that the worst case is that "all of the nodes that represent the `::` case" stay in memory. But it should not be the case that you ever end up with 2x that many Nodes in memory unless the entire repository has changed in such a way that all of the NodeKeys change.
> Since we can't have two copies of the same Node (ie, with the same NodeKey)

ah, gotcha
#4394 was less of a solution than we thought, so this is now hindering the efficiency of `pantsd` in cases where files are created near the root of the repo.

Going to switch to this to see if I can knock it out today.
This is primarily annoying when changing files in the root of the repo, which is relatively rare. And since it doesn't effect correctness, going to defer to `1.4.0`.
@jsirois : One thing to be cautious of: @illicitonion will be changing this code a bit in order to integrate it with #4585. The memoization that is done in this codepath will continue to be necessary to provide good incremental performance in the context of the engine though, so that part will not change.

@illicitonion : Worth being aware of this ticket, as it explains a bit of why the `fs` module does what it does.
This will be resolved by #5106.
assuming fixed by #5106 (not sure why github didn't auto-close based on the annotation) - thanks @illicitonion !
err, guess that was because #5106 was 'closed' vs 'merged' and superceded by #5496.
So, it looks like this is still an issue, in that touching a file in the root of the repository will invalidate the listing there, which will invalidate all `Snapshots`, because they depend on the filesystem operations. Basically everything else in the graph depends on a `Snapshot`, so the result is that although only one new IO happens, we end up re-constructing the `Snapshot`s (which requires some round trips to the database) and all rule logic re-runs.

Fixing #4558 seems like the least contorted way to resolve this, so will re-open this until we resolve #4558. cc @jsirois 

Resolving patch:
diff --git a/src/python/pants/engine/native.py b/src/python/pants/engine/native.py
--- a/src/python/pants/engine/native.py
+++ b/src/python/pants/engine/native.py
@@ -224,6 +224,7 @@
 
 uint64_t graph_len(Scheduler*);
 uint64_t graph_invalidate(Scheduler*, BufferBuffer);
+uint64_t graph_invalidate_all_paths(Scheduler*);
 PyResult graph_visualize(Scheduler*, Session*, char*);
 void graph_trace(Scheduler*, ExecutionRequest*, char*);
 
diff --git a/src/python/pants/engine/scheduler.py b/src/python/pants/engine/scheduler.py
--- a/src/python/pants/engine/scheduler.py
+++ b/src/python/pants/engine/scheduler.py
@@ -296,6 +296,11 @@ def invalidate_files(self, direct_filenames):
     logger.info('invalidated %d nodes for: %s', invalidated, filenames)
     return invalidated
 
+  def invalidate_all_files(self):
+    invalidated =  self._native.lib.graph_invalidate_all_paths(self._scheduler)
+    logger.info('invalidated all %d nodes', invalidated)
+    return invalidated
+
   def graph_len(self):
     return self._native.lib.graph_len(self._scheduler)
 
@@ -453,11 +458,17 @@ def execution_request(self, products, subjects):
     return self.execution_request_literal(roots)
 
   def invalidate_files(self, direct_filenames):
-    """Calls `Graph.invalidate_files()` against an internal product Graph instance."""
+    """Invalidates the given filenames in an internal product Graph instance."""
     invalidated = self._scheduler.invalidate_files(direct_filenames)
     self._maybe_visualize()
     return invalidated
 
+  def invalidate_all_files(self):
+    """Invalidates all filenames in an internal product Graph instance."""
+    invalidated = self._scheduler.invalidate_all_files()
+    self._maybe_visualize()
+    return invalidated
+
   def node_count(self):
     return self._scheduler.graph_len()
 


Test patch:
diff --git a/tests/python/pants_test/test_base.py b/tests/python/pants_test/test_base.py
--- a/tests/python/pants_test/test_base.py
+++ b/tests/python/pants_test/test_base.py
@@ -331,6 +331,7 @@ def scan():
   def _reset_engine(self):
     if self._scheduler is not None:
       self._build_graph.reset()
+      self._scheduler.invalidate_all_files()
       # Eagerly free file handles, threads, connections, etc, held by the scheduler. In theory,
       # dropping the scheduler is equivalent, but it's easy for references to the scheduler to leak.
       self._scheduler.pre_fork()