Repository: dbt-labs/dbt-core

Issue:
[CT-1927] [Feature] Consider only successors of the deleted nodes when populating the GraphQueue
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

The current implementation [here](https://github.com/dbt-labs/dbt-core/blob/c65ba11ae6dca5203228320a7f6388ce1796f104/core/dbt/graph/queue.py#L159-L166) looks through all four of these nodes, no matter what.

<img width="250" alt="image" src="https://user-images.githubusercontent.com/44704949/214682644-3e27e481-522b-464f-b97c-5540d426bd76.png">

We can do better! In fact, we only need to add nodes with an in-degree of 0. This means we only need to visit nodes that could possibly have an in-degree of zero. This only happens to a node whose predecessor is removed, so we can restrict visits to successor nodes of removed nodes.

#### Example
We can make it so that if Node 2 is removed, then only its successors (Nodes 3 and 4) will be visited, and only Node 3 will change to an in-degree of zero. Only nodes with an in-degree of zero need to be re-added to the queue -- in this case, only Node 3.

### Average case
We can see that before it would visit all X nodes in the average case. But now the average case is just the typical number of successors (i.e., the mean number of out-degrees per node).

I would guess for most dbt projects, the mean number of out-degrees will be WAAAAY closer to 0 than it will be to X, hence "taking almost no time at all" 😉

### Describe alternatives you've considered

The status quo is _an_ alternative, but not a very competitive one 😄

There might be some other optimizations we can pursue, but this is a good place to start.

### Who will this benefit?

This should help somewhat with large dbt projects with 1000+ nodes like described in https://github.com/dbt-labs/dbt-core/issues/6073

### Are you interested in contributing this feature?

See https://github.com/dbt-labs/dbt-core/pull/6720

### Anything else?

_No response_


Discussion:

Resolving patch:
diff --git a/core/dbt/graph/queue.py b/core/dbt/graph/queue.py
--- a/core/dbt/graph/queue.py
+++ b/core/dbt/graph/queue.py
@@ -40,7 +40,7 @@ def __init__(self, graph: nx.DiGraph, manifest: Manifest, selected: Set[UniqueId
         # store the 'score' of each node as a number. Lower is higher priority.
         self._scores = self._get_scores(self.graph)
         # populate the initial queue
-        self._find_new_additions()
+        self._find_new_additions(list(self.graph.nodes()))
         # awaits after task end
         self.some_task_done = threading.Condition(self.lock)
 
@@ -156,12 +156,12 @@ def _already_known(self, node: UniqueId) -> bool:
         """
         return node in self.in_progress or node in self.queued
 
-    def _find_new_additions(self) -> None:
+    def _find_new_additions(self, candidates) -> None:
         """Find any nodes in the graph that need to be added to the internal
         queue and add them.
         """
-        for node, in_degree in self.graph.in_degree():
-            if in_degree == 0 and not self._already_known(node):
+        for node in candidates:
+            if self.graph.in_degree(node) == 0 and not self._already_known(node):
                 self.inner.put((self._scores[node], node))
                 self.queued.add(node)
 
@@ -174,8 +174,9 @@ def mark_done(self, node_id: UniqueId) -> None:
         """
         with self.lock:
             self.in_progress.remove(node_id)
+            successors = list(self.graph.successors(node_id))
             self.graph.remove_node(node_id)
-            self._find_new_additions()
+            self._find_new_additions(successors)
             self.inner.task_done()
             self.some_task_done.notify_all()
 


Test patch: