Repository: dbt-labs/dbt-core
Pull request: Attempt to skip saved query processing when no semantic manifest changes
Resolves #10563

### Problem

We run "process_saved_queries" even when nothing has changed in the semantic manifest.

### Solution

Check for changes to saved queries, semantic models, and metrics before executing process saved queries. This will have false positives because of cascading re-parsing, but will enable us to skip at least some unnecessary saved query processing.

### Checklist

- [x] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me.
- [x] I have run this code in development, and it appears to resolve the stated issue.
- [x] This PR includes tests, or tests are not required or relevant for this PR.
- [x] This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
- [x] This PR includes [type annotations](https://docs.python.org/3/library/typing.html) for new and modified functions.


Problem statement:
Do not execute process_saved_queries if nothing in semantic manifest has changed
### Description

Currently we run process_saved_queries for every saved query in a manifest, even if nothing has changed in the semantic manifest. In the right circumstances this can be a performance problem.

### Acceptance Criteria

Check for actual changes to the semantic manifest before executing process_saved_queries for every saved query.

Hints from issue discussion:
Saved queries, metrics and semantic models are in the semantic manifest. In some cases semantic_models may be "scheduled for parsing" because a model that it refers to has changed, but nothing in the actual definition has changed. If we want to minimize the cases where we run process_saved_queries, it might make sense to create a flag in the partial parser ("semantic_manifest_changed") and set it if we encounter a saved query, metric or semantic model in "handle_schema_file_changes".

Commit message: Attempt to skip saved query processing when no semantic manifest changes

Patch to review:
diff --git a/.changes/unreleased/Fixes-20240926-101220.yaml b/.changes/unreleased/Fixes-20240926-101220.yaml
new file mode 100644
index 00000000000..677ee458048
--- /dev/null
+++ b/.changes/unreleased/Fixes-20240926-101220.yaml
@@ -0,0 +1,6 @@
+kind: Fixes
+body: Attempt to skip saved query processing when no semantic manifest changes
+time: 2024-09-26T10:12:20.193453-04:00
+custom:
+  Author: gshank
+  Issue: "10563"
diff --git a/core/dbt/parser/manifest.py b/core/dbt/parser/manifest.py
index 5536bf39270..ce396b97fef 100644
--- a/core/dbt/parser/manifest.py
+++ b/core/dbt/parser/manifest.py
@@ -1141,6 +1141,24 @@ def process_metrics(self, config: RuntimeConfig):
 
     def process_saved_queries(self, config: RuntimeConfig):
         """Processes SavedQuery nodes to populate their `depends_on`."""
+        # Note: This will also capture various nodes which have been re-parsed
+        # because they refer to some other changed node, so there will be
+        # false positives. Ideally we would compare actual changes.
+        semantic_manifest_changed = False
+        SemanticManifestNode = Union[SavedQuery, SemanticModel, Metric]
+        semantic_manifest_nodes: chain[SemanticManifestNode] = chain(
+            self.manifest.saved_queries.values(),
+            self.manifest.semantic_models.values(),
+            self.manifest.metrics.values(),
+        )
+        for node in semantic_manifest_nodes:
+            # Check if this node has been modified in this parsing run
+            if node.created_at > self.started_at:
+                semantic_manifest_changed = True
+                break  # as soon as we run into one changed node we can stop
+        if semantic_manifest_changed is False:
+            return
+
         current_project = config.project_name
         for saved_query in self.manifest.saved_queries.values():
             # TODO:


Code review comments:
diff_hunk: @@ -1141,6 +1141,24 @@ def process_metrics(self, config: RuntimeConfig):
 
     def process_saved_queries(self, config: RuntimeConfig):
         """Processes SavedQuery nodes to populate their `depends_on`."""
+        # Note: This will also capture various nodes which have been re-parsed
+        # because they refer to some other changed node, so there will be
+        # false positives. Ideally we would compare actual changes.
+        semantic_manifest_changed = False
+        SemanticManifestNode = Union[SavedQuery, SemanticModel, Metric]
path: core/dbt/parser/manifest.py
text: @user1:
Nit: Can we put this definition in a constants file somewhere?

@author:
Moved it to core/dbt/contracts/graph/nodes.py