Repository: dbt-labs/dbt-core

Issue:
[CT-94] Holistic tracking for perceived start-up performance
There are three time-consuming steps that contribute to perceived slowness at the start of `dbt run`:
1. **Parsing**: Reading files &rarr; creating the internal manifest. Scales with size of project, but helped by partial + static parsing, and with [fairly solid telemetry](https://github.com/dbt-labs/dbt-core/blob/3032594b26aaafaf1003db513645660d0e7c89dc/core/dbt/parser/manifest.py#L774) today.
2. **Graph compilation** (called [here](https://github.com/dbt-labs/dbt-core/blob/3032594b26aaafaf1003db513645660d0e7c89dc/core/dbt/task/runnable.py#L91)): Scales with the number of nodes in a project, limited by memory / networkx algorithm
3. **Adapter cache construction** (called [here](https://github.com/dbt-labs/dbt-core/blob/3032594b26aaafaf1003db513645660d0e7c89dc/core/dbt/task/runnable.py#L410): Once we have a database connection, run metadata queries (or call metadata endpoints) to populate the cache, saving repeated queries by subsequent materializations. Scales with number of objects in the database, and number of unique namespaces (databases/schemas) that need caching. Speed also varies _significantly_ across different databases; this is particularly slow on Spark/Databricks today (https://github.com/dbt-labs/dbt-spark/issues/228).

Not all commands complete all of these steps. For instance, `dbt ls` and `dbt run-operation` need only step 1. But all of these steps must run, each time and in sequence, for "graph runnable" tasks (`run`, `test`, `build`, etc)—and they contribute to the perceived delay between submitting the command, and seeing the first node's query hit the database.

While we don't have present appetite for top-to-bottom highly involved performance work, the sooner we can add telemetry, the better served we'll be when we do. Let's start by adding high-level telemetry for steps 2+3.


Discussion:
@jtcohen6  Let's refine this to get definitive exit goals and split this up
- Are we trying to improve perf? Profiling is better
- Are we trying to verify customers are hitting a certain SLA? Then metrics are the way to go
Really appreciate the ask for clarification here!

Last year, we set an SLA of "5 seconds start-up time," but the performance work we targeted was narrowly oriented around parsing. I'd like to capture more telemetry so we can see how that SLA exists in the real world: What's the total number of seconds between the very first step in `main.py`, and the first node that actually starts running?

Today, we lack visibility into:
- Adapter-specific performance across different databases, especially when those databases are operating at scale (thousands of objects). This is the thing I'm most interested in tracking, and I'm not sure we have good ways to profile it.
- Non-standard DAG shapes where our graph compilation algorithms perform poorly (e.g. >100 tests for every model). This is something we could better achieve with profiling and a wider variety of performance testing projects.
Another important input: the number of nodes in a project, versus the number of nodes actually selected. This feels like a prerequisite to really understanding the potential impact of heavier lifts we might make to ameliorate start-up time, such as #4688.
@jtcohen6 Just to clarify in terms of work-- This would boil down to adding telemetry for `Graph compilation` and `Adapter cache construction`.  That's how we are estimating this so please let us know if that doesn't sound right. 
@iknox-fa Correct!

Resolving patch:
diff --git a/core/dbt/task/runnable.py b/core/dbt/task/runnable.py
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -50,6 +50,7 @@
 
 from dbt.graph import GraphQueue, NodeSelector, SelectionSpec, parse_difference, Graph
 from dbt.parser.manifest import ManifestLoader
+import dbt.tracking
 
 import dbt.exceptions
 from dbt import flags
@@ -88,7 +89,12 @@ def compile_manifest(self):
 
     def _runtime_initialize(self):
         self.load_manifest()
+
+        start_compile_manifest = time.perf_counter()
         self.compile_manifest()
+        compile_time = time.perf_counter() - start_compile_manifest
+        if dbt.tracking.active_user is not None:
+            dbt.tracking.track_runnable_timing({"graph_compilation_elapsed": compile_time})
 
 
 class GraphRunnableTask(ManifestTask):
@@ -391,7 +397,13 @@ def _mark_dependent_errors(self, node_id, result, cause):
             self._skipped_children[dep_node_id] = cause
 
     def populate_adapter_cache(self, adapter):
+        start_populate_cache = time.perf_counter()
         adapter.set_relations_cache(self.manifest)
+        cache_populate_time = time.perf_counter() - start_populate_cache
+        if dbt.tracking.active_user is not None:
+            dbt.tracking.track_runnable_timing(
+                {"adapter_cache_construction_elapsed": cache_populate_time}
+            )
 
     def before_hooks(self, adapter):
         pass
diff --git a/core/dbt/tracking.py b/core/dbt/tracking.py
--- a/core/dbt/tracking.py
+++ b/core/dbt/tracking.py
@@ -44,6 +44,7 @@
 RESOURCE_COUNTS = "iglu:com.dbt/resource_counts/jsonschema/1-0-0"
 EXPERIMENTAL_PARSER = "iglu:com.dbt/experimental_parser/jsonschema/1-0-0"
 PARTIAL_PARSER = "iglu:com.dbt/partial_parser/jsonschema/1-0-1"
+RUNNABLE_TIMING = "iglu:com.dbt/runnable/jsonschema/1-0-0"
 DBT_INVOCATION_ENV = "DBT_INVOCATION_ENV"
 
 
@@ -412,6 +413,18 @@ def track_partial_parser(options):
     )
 
 
+def track_runnable_timing(options):
+    context = [SelfDescribingJson(RUNNABLE_TIMING, options)]
+    assert active_user is not None, "Cannot track runnable info when active user is None"
+    track(
+        active_user,
+        category="dbt",
+        action="runnable_timing",
+        label=get_invocation_id(),
+        context=context,
+    )
+
+
 def flush():
     fire_event(FlushEvents())
     try:


Test patch: