Repository: dbt-labs/dbt-core

Issue:
Add detailed tracking for start-of-invocation parse time
### Describe the feature
- We know that the time it takes to parse especially big projects is a major pain point, and that this parse time is increasing linearly with project size
- We plan to dive deep into parsing over the coming months and identify areas for improvement
- As we roll out changes that promise to decrease parse time, we need the ability to measure empirical treatment effects for active projects in the wild

Proposal: Extend our existing opt-out anonymous usage tracking to present a detailed picture of project size, resource distribution, and start-of-invocation steps with associated execution times.

As far as the implementation, I'm taking some indirect inspiration from Snowplow's [performance timing context](https://discourse.snowplowanalytics.com/t/measuring-page-load-times-with-the-performance-timing-context-tutorial/100). I'm envisioning a custom structured event that does two things:
- Splits out start-of-invocation tasks into discrete components—read files from disk, create hologrammed objects, run metadata queries against database, etc—and calculates the time it takes to do each of them
- Includes a dictionary representation of project attributes:
    - `Found 1 model, 1 test, 1 snapshot, 1 analysis, 143 macros, 0 operations, 4 seed files, 0 sources`
    - Total number of files

### Describe alternatives you've considered
- Use current tracking to accomplish a crummier version of this. We fire events on invocation start and for each model (if `dbt run`); we could take the difference from the first model run to approximate total start-of-invocation-stuff time.
- Develop an internal suite of sample projects that represent most possible project permutations (1000s of models, 1000s of schema tests, 1000s of macros, 1000s of docs blocks, etc.). Test potential changes against that suite during CI and hope it translates to real-world projects.
    - I don't think this is an alternative as much as something we should do in addition. We'll still want to know the on-the-ground impact of our changes, no matter how promising they seem in development.

### Who will this benefit?
- Especially large organizations that will have large dbt projects from day 1
- Mature projects that continue to grow, slowly but surely


Discussion:
@jtcohen6 I've been thinking about this as a breakdown across node type parsing. Maybe an example payload would be:
```
[
  {
    "type": "model",
    "parse_time": 10.2,
    "count_parsed": 1000
  },
  {
    "type": "source",
    "parse_time": 2.4,
    "count_parsed": 128
  }
]
```

I think it might be challenging to break down parse times across file loading / jinja interpolation / data encoding/decoding. Instead, we can use this dataset to understand empirical performance across node types, then drill into specific performance improvements in a testing/sample project using something like cProfile and snakeviz. That's just my 2 cents - you buy it?
Totally. I imagined the difficulties here to be reversed, whereby it'd be reasonable to break down parse time by functional area of the codebase and quite challenging to break it down by specific node. I'm not exactly sure why I thought that, but I'm equally happy with the approach you outline.
We settled on tracking higher level timing info that is already being recorded by https://github.com/fishtown-analytics/dbt/pull/2883

Resolving patch:
diff --git a/core/dbt/parser/manifest.py b/core/dbt/parser/manifest.py
--- a/core/dbt/parser/manifest.py
+++ b/core/dbt/parser/manifest.py
@@ -1,3 +1,5 @@
+from dataclasses import dataclass
+from dataclasses import field
 import os
 import pickle
 from typing import (
@@ -6,6 +8,7 @@
 import time
 
 import dbt.exceptions
+import dbt.tracking
 import dbt.flags as flags
 
 from dbt.adapters.factory import (
@@ -24,6 +27,7 @@
 from dbt.contracts.graph.parsed import (
     ParsedSourceDefinition, ParsedNode, ParsedMacro, ColumnInfo, ParsedExposure
 )
+from dbt.contracts.util import Writable
 from dbt.exceptions import (
     ref_target_not_found,
     get_target_not_found_or_disabled_msg,
@@ -47,12 +51,39 @@
 from dbt.ui import warning_tag
 from dbt.version import __version__
 
+from hologram import JsonSchemaMixin
 
 PARTIAL_PARSE_FILE_NAME = 'partial_parse.pickle'
 PARSING_STATE = DbtProcessState('parsing')
 DEFAULT_PARTIAL_PARSE = False
 
 
+@dataclass
+class ParserInfo(JsonSchemaMixin):
+    parser: str
+    elapsed: float
+    path_count: int = 0
+
+
+@dataclass
+class ProjectLoaderInfo(JsonSchemaMixin):
+    project_name: str
+    elapsed: float
+    parsers: List[ParserInfo]
+    path_count: int = 0
+
+
+@dataclass
+class ManifestLoaderInfo(JsonSchemaMixin, Writable):
+    path_count: int = 0
+    is_partial_parse_enabled: Optional[bool] = None
+    parse_project_elapsed: Optional[float] = None
+    patch_sources_elapsed: Optional[float] = None
+    process_manifest_elapsed: Optional[float] = None
+    load_all_elapsed: Optional[float] = None
+    projects: List[ProjectLoaderInfo] = field(default_factory=list)
+
+
 _parser_types: List[Type[Parser]] = [
     ModelParser,
     SnapshotParser,
@@ -120,10 +151,26 @@ def __init__(
             root_project, all_projects,
         )
         self._loaded_file_cache: Dict[str, FileBlock] = {}
-        partial_parse = self._partial_parse_enabled()
-        self._perf_info: Dict[str, Any] = {
-            'path_count': 0, 'projects': [],
-            'is_partial_parse_enabled': partial_parse}
+        self._perf_info = ManifestLoaderInfo(
+            is_partial_parse_enabled=self._partial_parse_enabled()
+        )
+
+    def track_project_load(self):
+        invocation_id = dbt.tracking.active_user.invocation_id
+        dbt.tracking.track_project_load({
+            "invocation_id": invocation_id,
+            "project_id": self.root_project.hashed_name(),
+            "path_count": self._perf_info.path_count,
+            "parse_project_elapsed": self._perf_info.parse_project_elapsed,
+            "patch_sources_elapsed": self._perf_info.patch_sources_elapsed,
+            "process_manifest_elapsed": (
+                self._perf_info.process_manifest_elapsed
+            ),
+            "load_all_elapsed": self._perf_info.load_all_elapsed,
+            "is_partial_parse_enabled": (
+                self._perf_info.is_partial_parse_enabled
+            ),
+        })
 
     def parse_with_cache(
         self,
@@ -175,7 +222,7 @@ def parse_project(
         # per-project cache.
         self._loaded_file_cache.clear()
 
-        project_info: Dict[str, Any] = {'parsers': []}
+        project_parser_info: List[ParserInfo] = []
         start_timer = time.perf_counter()
         total_path_count = 0
         for parser in parsers:
@@ -188,21 +235,26 @@ def parse_project(
                     print("..", end='', flush=True)
 
             if parser_path_count > 0:
-                parser_elapsed = time.perf_counter() - parser_start_timer
-                project_info['parsers'].append({'parser': type(
-                    parser).__name__, 'path_count': parser_path_count,
-                    'elapsed': '{:.2f}'.format(parser_elapsed)})
+                project_parser_info.append(ParserInfo(
+                    parser=parser.resource_type,
+                    path_count=parser_path_count,
+                    elapsed=time.perf_counter() - parser_start_timer
+                ))
                 total_path_count = total_path_count + parser_path_count
         if total_path_count > 100:
             print("..")
 
         elapsed = time.perf_counter() - start_timer
-        project_info['project_name'] = project.project_name
-        project_info['path_count'] = total_path_count
-        project_info['elapsed'] = '{:.2f}'.format(elapsed)
-        self._perf_info['projects'].append(project_info)
-        self._perf_info['path_count'] = self._perf_info['path_count'] + \
-            total_path_count
+        project_info = ProjectLoaderInfo(
+            project_name=project.project_name,
+            path_count=total_path_count,
+            elapsed=elapsed,
+            parsers=project_parser_info
+        )
+        self._perf_info.projects.append(project_info)
+        self._perf_info.path_count = (
+            self._perf_info.path_count + total_path_count
+        )
 
     def load_only_macros(self) -> Manifest:
         old_results = self.read_parse_results()
@@ -228,11 +280,14 @@ def load(self, macro_manifest: Manifest):
         self.results.files.update(macro_manifest.files)
 
         start_timer = time.perf_counter()
+
         for project in self.all_projects.values():
             # parse a single project
             self.parse_project(project, macro_manifest, old_results)
-        self._perf_info['parse_project_elapsed'] = '{:.2f}'.format(
-            time.perf_counter() - start_timer)
+
+        self._perf_info.parse_project_elapsed = (
+            time.perf_counter() - start_timer
+        )
 
     def write_parse_results(self):
         path = os.path.join(self.root_project.target_path,
@@ -335,8 +390,9 @@ def create_manifest(self) -> Manifest:
         # list is created
         start_patch = time.perf_counter()
         sources = patch_sources(self.results, self.root_project)
-        self._perf_info['patch_sources_elapsed'] = '{:.2f}'.format(
-            time.perf_counter() - start_patch)
+        self._perf_info.patch_sources_elapsed = (
+            time.perf_counter() - start_patch
+        )
         disabled = []
         for value in self.results.disabled.values():
             disabled.extend(value)
@@ -360,8 +416,11 @@ def create_manifest(self) -> Manifest:
         manifest.patch_macros(self.results.macro_patches)
         start_process = time.perf_counter()
         self.process_manifest(manifest)
-        self._perf_info['process_manifest_elapsed'] = '{:.2f}'.format(
-            time.perf_counter() - start_process)
+
+        self._perf_info.process_manifest_elapsed = (
+            time.perf_counter() - start_process
+        )
+
         return manifest
 
     @classmethod
@@ -373,6 +432,7 @@ def load_all(
     ) -> Manifest:
         with PARSING_STATE:
             start_load_all = time.perf_counter()
+
             projects = root_config.load_dependencies()
             loader = cls(root_config, projects, macro_hook)
             loader.load(macro_manifest=macro_manifest)
@@ -380,8 +440,13 @@ def load_all(
             manifest = loader.create_manifest()
             _check_manifest(manifest, root_config)
             manifest.build_flat_graph()
-            loader._perf_info['load_all_elapsed'] = '{:.2f}'.format(
-                time.perf_counter() - start_load_all)
+
+            loader._perf_info.load_all_elapsed = (
+                time.perf_counter() - start_load_all
+            )
+
+            loader.track_project_load()
+
             return manifest
 
     @classmethod
diff --git a/core/dbt/task/parse.py b/core/dbt/task/parse.py
--- a/core/dbt/task/parse.py
+++ b/core/dbt/task/parse.py
@@ -10,10 +10,7 @@
 from dbt.adapters.factory import get_adapter
 from dbt.parser.manifest import Manifest, ManifestLoader, _check_manifest
 from dbt.logger import DbtProcessState, print_timestamped_line
-from dbt.clients.system import write_file
 from dbt.graph import Graph
-import dbt.utils
-import json
 import time
 from typing import Optional
 import os
@@ -36,8 +33,7 @@ def write_manifest(self):
 
     def write_perf_info(self):
         path = os.path.join(self.config.target_path, PERF_INFO_FILE_NAME)
-        write_file(path, json.dumps(self.loader._perf_info,
-                                    cls=dbt.utils.JSONEncoder, indent=4))
+        self.loader._perf_info.write(path)
         print_timestamped_line(f"Performance info: {path}")
 
     # This method takes code that normally exists in other files
@@ -71,8 +67,9 @@ def get_full_manifest(self):
             print_timestamped_line("Manifest checked")
             manifest.build_flat_graph()
             print_timestamped_line("Flat graph built")
-            loader._perf_info['load_all_elapsed'] = '{:.2f}'.format(
-                time.perf_counter() - start_load_all)
+            loader._perf_info.load_all_elapsed = (
+                time.perf_counter() - start_load_all
+            )
 
         self.loader = loader
         self.manifest = manifest
diff --git a/core/dbt/tracking.py b/core/dbt/tracking.py
--- a/core/dbt/tracking.py
+++ b/core/dbt/tracking.py
@@ -27,6 +27,7 @@
 PACKAGE_INSTALL_SPEC = 'iglu:com.dbt/package_install/jsonschema/1-0-0'
 RPC_REQUEST_SPEC = 'iglu:com.dbt/rpc_request/jsonschema/1-0-1'
 DEPRECATION_WARN_SPEC = 'iglu:com.dbt/deprecation_warn/jsonschema/1-0-0'
+LOAD_ALL_TIMING_SPEC = 'iglu:com.dbt/load_all_timing/jsonschema/1-0-0'
 
 DBT_INVOCATION_ENV = 'DBT_INVOCATION_ENV'
 
@@ -273,6 +274,20 @@ def track_invocation_start(config=None, args=None):
     )
 
 
+def track_project_load(options):
+    context = [SelfDescribingJson(LOAD_ALL_TIMING_SPEC, options)]
+    assert active_user is not None, \
+        'Cannot track project loading time when active user is None'
+
+    track(
+        active_user,
+        category='dbt',
+        action='load_project',
+        label=active_user.invocation_id,
+        context=context
+    )
+
+
 def track_model_run(options):
     context = [SelfDescribingJson(RUN_MODEL_SPEC, options)]
     assert active_user is not None, \


Test patch:
diff --git a/test/integration/033_event_tracking_test/test_events.py b/test/integration/033_event_tracking_test/test_events.py
--- a/test/integration/033_event_tracking_test/test_events.py
+++ b/test/integration/033_event_tracking_test/test_events.py
@@ -87,6 +87,24 @@ def run_event_test(
             populated_contexts
         )
 
+    def load_context(self):
+
+        def populate(project_id, user_id, invocation_id, version):
+            return [{
+                'schema': 'iglu:com.dbt/load_all_timing/jsonschema/1-0-0',
+                'data': {
+                    'invocation_id': invocation_id,
+                    'project_id': project_id,
+                    'path_count': ANY,
+                    'is_partial_parse_enabled': ANY,
+                    'load_all_elapsed': ANY,
+                    'parse_project_elapsed': ANY,
+                    'patch_sources_elapsed': ANY,
+                    'process_manifest_elapsed': ANY,
+                },
+            }]
+        return populate
+
     def build_context(
         self,
         command,
@@ -205,6 +223,12 @@ def test__postgres_event_tracking_compile(self):
                 label='start',
                 context=ANY
             ),
+            call(
+                category='dbt',
+                action='load_project',
+                label=ANY,
+                context=ANY
+            ),
             call(
                 category='dbt',
                 action='invocation',
@@ -215,6 +239,7 @@ def test__postgres_event_tracking_compile(self):
 
         expected_contexts = [
             self.build_context('compile', 'start'),
+            self.load_context(),
             self.build_context('compile', 'end', result_type='ok')
         ]
 
@@ -312,6 +337,12 @@ def seed_context(project_id, user_id, invocation_id, version):
                 label='start',
                 context=ANY
             ),
+            call(
+                category='dbt',
+                action='load_project',
+                label=ANY,
+                context=ANY
+            ),
             call(
                 category='dbt',
                 action='run_model',
@@ -328,6 +359,7 @@ def seed_context(project_id, user_id, invocation_id, version):
 
         expected_contexts = [
             self.build_context('seed', 'start'),
+            self.load_context(),
             seed_context,
             self.build_context('seed', 'end', result_type='ok')
         ]
@@ -343,6 +375,12 @@ def test__postgres_event_tracking_models(self):
                 label='start',
                 context=ANY
             ),
+            call(
+                category='dbt',
+                action='load_project',
+                label=ANY,
+                context=ANY
+            ),
             call(
                 category='dbt',
                 action='run_model',
@@ -371,6 +409,7 @@ def test__postgres_event_tracking_models(self):
 
         expected_contexts = [
             self.build_context('run', 'start'),
+            self.load_context(),
             self.run_context(
                 hashed_contents='1e5789d34cddfbd5da47d7713aa9191c',
                 model_id='4fbacae0e1b69924b22964b457148fb8',
@@ -408,6 +447,12 @@ def test__postgres_event_tracking_model_error(self):
                 label='start',
                 context=ANY
             ),
+            call(
+                category='dbt',
+                action='load_project',
+                label=ANY,
+                context=ANY
+            ),
             call(
                 category='dbt',
                 action='run_model',
@@ -424,6 +469,7 @@ def test__postgres_event_tracking_model_error(self):
 
         expected_contexts = [
             self.build_context('run', 'start'),
+            self.load_context(),
             self.run_context(
                 hashed_contents='4419e809ce0995d99026299e54266037',
                 model_id='576c3d4489593f00fad42b97c278641e',
@@ -454,6 +500,12 @@ def test__postgres_event_tracking_tests(self):
                 label='start',
                 context=ANY
             ),
+            call(
+                category='dbt',
+                action='load_project',
+                label=ANY,
+                context=ANY
+            ),
             call(
                 category='dbt',
                 action='invocation',
@@ -464,6 +516,7 @@ def test__postgres_event_tracking_tests(self):
 
         expected_contexts = [
             self.build_context('test', 'start'),
+            self.load_context(),
             self.build_context('test', 'end', result_type='ok')
         ]
 
@@ -558,6 +611,12 @@ def test__postgres_event_tracking_unable_to_connect(self):
                 label='start',
                 context=ANY
             ),
+            call(
+                category='dbt',
+                action='load_project',
+                label=ANY,
+                context=ANY
+            ),
             call(
                 category='dbt',
                 action='invocation',
@@ -568,6 +627,7 @@ def test__postgres_event_tracking_unable_to_connect(self):
 
         expected_contexts = [
             self.build_context('run', 'start'),
+            self.load_context(),
             self.build_context('run', 'end', result_type='error')
         ]
 
@@ -598,6 +658,12 @@ def test__postgres_event_tracking_snapshot(self):
                 label='start',
                 context=ANY
             ),
+            call(
+                category='dbt',
+                action='load_project',
+                label=ANY,
+                context=ANY
+            ),
             call(
                 category='dbt',
                 action='run_model',
@@ -615,6 +681,7 @@ def test__postgres_event_tracking_snapshot(self):
         # the model here has a raw_sql that contains the schema, which changes
         expected_contexts = [
             self.build_context('snapshot', 'start'),
+            self.load_context(),
             self.run_context(
                 hashed_contents=ANY,
                 model_id='820793a4def8d8a38d109a9709374849',
@@ -646,6 +713,12 @@ def test__postgres_event_tracking_catalog_generate(self):
                 label='start',
                 context=ANY
             ),
+            call(
+                category='dbt',
+                action='load_project',
+                label=ANY,
+                context=ANY,
+            ),
             call(
                 category='dbt',
                 action='invocation',
@@ -656,6 +729,7 @@ def test__postgres_event_tracking_catalog_generate(self):
 
         expected_contexts = [
             self.build_context('generate', 'start'),
+            self.load_context(),
             self.build_context('generate', 'end', result_type='ok')
         ]