Repository: dbt-labs/dbt-core

Issue:
[Bug] Huge slowdown of dbt initialization for projects with many tests since v0.21
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

This is a continuation of the closed #4012. 
Since the upgrade from v0.20 to v0.21, when the project has a lot of schema tests, dbt startup time is severely increased. In one of the projects (343 models, 3882 tests) the startup time increased from 7 seconds to over 3 minutes! Using the timing recording, I managed to track down the root cause to #4022 (`compilation.py:114(<listcomp>)`). In addition, it seems that the `digraph.py:638(add_edges_from)` also performs significantly worse as well. 
With #4012, the first issue seems to be resolved and the startup gets faster but is still impacted by the `digraph.py:638(add_edges_from)`. 
Disabling all tests makes the performance on par with 0.20.

Here are the details from each version tests. I used pre-parsed simple `dbt compile` to showcase the issue:
### 0.20.2

* **dbt.log**
```
2021-10-26 13:09:56.378448 (MainThread): Running with dbt=0.20.2
2021-10-26 13:09:57.385906 (MainThread): running dbt with arguments Namespace(cls=<class 'dbt.task.compile.CompileTask'>, debug=False, defer=None, exclude=None, full_refresh=False, log_cache_events=False, log_format='default', models=None, parse_only=False, partial_parse=None, profile=None, profiles_dir='C:\\Users\\dvalchev\\.dbt', project_dir=None, record_timing_info='..\\timimng-compile-0.20.txt', rpc_method='compile', selector_name=None, single_threaded=False, state=None, strict=False, target=None, test_new_parser=False, threads=15, use_cache=True, use_colors=None, use_experimental_parser=False, vars='{}', version_check=True, warn_error=False, which='compile', write_json=True)
2021-10-26 13:09:57.387901 (MainThread): Tracking: do not track
2021-10-26 13:09:59.117617 (MainThread): Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
2021-10-26 13:09:59.117617 (MainThread): Partial parsing enabled, no changes found, skipping parsing
2021-10-26 13:10:01.176648 (MainThread): write_gpickle is deprecated and will be removed in 3.0.Use ``pickle.dump(G, path, protocol)``
2021-10-26 13:10:01.239455 (MainThread): Found 343 models, 3882 tests, 0 snapshots, 7 analyses, 387 macros, 0 operations, 1 seed file, 5 sources, 0 exposures
2021-10-26 13:10:04.374412 (MainThread): 
```

* **Timings**
![image](https://user-images.githubusercontent.com/30269223/138926843-f48635bb-b57d-45ac-b02c-0b353e37daec.png)

### 0.21

* **dbt.log**
```
2021-10-26 12:44:30.354647 (MainThread): Running with dbt=0.21.0
2021-10-26 12:44:31.292324 (MainThread): running dbt with arguments Namespace(cls=<class 'dbt.task.compile.CompileTask'>, debug=False, defer=None, exclude=None, full_refresh=False, log_cache_events=False, log_format='default', parse_only=False, partial_parse=None, profile=None, profiles_dir='C:\\Users\\dvalchev\\.dbt', project_dir=None, record_timing_info='..\\timimng-compile-0.21.txt', rpc_method='compile', select=None, selector_name=None, single_threaded=False, state=None, strict=False, target=None, test_new_parser=False, threads=15, use_cache=True, use_colors=None, use_experimental_parser=False, vars='{}', version_check=True, warn_error=False, which='compile', write_json=True)
2021-10-26 12:44:31.292324 (MainThread): Tracking: do not track
2021-10-26 12:44:32.226488 (MainThread): Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
2021-10-26 12:44:32.227487 (MainThread): Partial parsing enabled, no changes found, skipping parsing
2021-10-26 12:46:55.233418 (MainThread): write_gpickle is deprecated and will be removed in 3.0.Use ``pickle.dump(G, path, protocol)``
2021-10-26 12:46:55.329506 (MainThread): Found 343 models, 3882 tests, 0 snapshots, 7 analyses, 408 macros, 0 operations, 1 seed file, 5 sources, 0 exposures
2021-10-26 12:47:59.318122 (MainThread): 
```

* **Timings**
![image](https://user-images.githubusercontent.com/30269223/138927028-12a02779-2acd-4284-8683-2532fded15a2.png)

### 1.0.0-b2

* ***dbt.log***
```
2021-10-26 13:15:11.414604 (MainThread): Running with dbt=1.0.0-b2
2021-10-26 13:15:12.295335 (MainThread): * Deprecation Warning: The `source-paths` config has been deprecated in favor of
`model-paths`. Please update your `dbt_project.yml` configuration to reflect
this change.
2021-10-26 13:15:12.341246 (MainThread): running dbt with arguments Namespace(cls=<class 'dbt.task.compile.CompileTask'>, debug=None, defer=None, exclude=None, fail_fast=None, full_refresh=False, log_cache_events=False, log_format=None, parse_only=False, partial_parse=None, printer_width=None, profile=None, profiles_dir='C:\\Users\\dvalchev\\.dbt', project_dir=None, record_timing_info='..\\timimng-compile-1.00.txt', rpc_method='compile', select=None, selector_name=None, send_anonymous_usage_stats=None, single_threaded=False, state=None, static_parser=None, target=None, threads=15, use_colors=None, use_experimental_parser=None, vars='{}', version_check=False, warn_error=None, which='compile', write_json=None)
2021-10-26 13:15:12.342210 (MainThread): Tracking: do not track
2021-10-26 13:15:13.291790 (MainThread): Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
2021-10-26 13:15:13.291790 (MainThread): Partial parsing enabled, no changes found, skipping parsing
2021-10-26 13:15:16.127705 (MainThread): write_gpickle is deprecated and will be removed in 3.0.Use ``pickle.dump(G, path, protocol)``
2021-10-26 13:15:16.210482 (MainThread): Found 343 models, 3882 tests, 0 snapshots, 7 analyses, 408 macros, 0 operations, 1 seed file, 5 sources, 0 exposures
2021-10-26 13:16:20.580208 (MainThread): 
```

* **Timings**
![image](https://user-images.githubusercontent.com/30269223/138927586-73593eff-3426-47b2-9a87-935b7be60ffd.png)


### Expected Behavior

dbt startup performance is on par with 0.20

### Steps To Reproduce

Execute `dbt compile` or `dbt run` on a project with a large number of tests

### Relevant log output

```shell
see above
```


### Environment

```markdown
- OS: Windows 10
- Python: 3.8.7
- dbt: 0.21+
```


### What database are you using dbt with?

snowflake

### Additional Context

_No response_


Discussion:
@dvalchev Thanks for the detailed report!

I'm glad to see that the improvement in #4022 cut down the graph compilation time significantly, though I'm unpleasantly surprised to see this line is still chewing up 54.1s:

https://github.com/dbt-labs/dbt-core/blob/73af9a56e50c18ff116aabfbd3fca448db113d5e/core/dbt/graph/graph.py#L85

When I run a sample project with 2k models and 6k tests on my machine locally, using `1.0.0-b2`, and output detailed timing information, I see that method taking **5.5s**—still much longer than I'd like, but an order of magnitude faster than the performance you're seeing.

In order to achieve the test-blocking behavior desirable in `dbt build`, we need to create graph edges between upstream tests and downstream models. The performance improvement in #4022 made the process of adding those edges (linking nodes) much, much faster. Yet we now need to operate over a much bigger graph, one with many more edges. In this regard, we're limited by the performance of `networkx`, and the technical capabilities of each machine running dbt.

I think there are a few paths we could pursue:
1. Add the extra edges for the `build` task only. This would add complexity to our task architecture, but it would restore previous behavior for other tasks, such as `dbt compile`.
2. Apply selection criteria _before_ passing the graph into `networkx` methods. This wouldn't help performance for a full `dbt build`, but it would be much better in development, where you're likely running a handful of models and tests.
3. Save the graph as an intermediate artifact (`graph.gpickle`), and reuse it if there are no file changes (known via partial parsing). We wouldn't attempt partial graph reconstruction, just full reuse or full invalidation.

I don't think we'd be able to do both (2) and (3); if selection criteria impacts the graph we build, then we can't reuse it in subsequent invocations with different selection criteria.

From my point of view, all options are on the table, including ones not listed here. We've been working hard on improving performance all year, and I'd like to see us carry that through into v1.0.
Thanks @jtcohen6 for the quick response! 
I think probably option (2) would be potentially a good trade-off. The biggest negative impact on the issue I'm expecting in our case would be really during the day-to-day development. Definitely it would be better at least to scale the processing time with the selection size. 
On that train of thought, (1) can be also beneficial for minimizing the dev impact. We're still not yet onboard with the `build` setup but I expect it to be used much more often in the automated pipeline executions (where the impact will be minimized) as opposed on more of a manual run/test/selection scenarios during the daily development. 

Resolving patch:
diff --git a/core/dbt/graph/graph.py b/core/dbt/graph/graph.py
--- a/core/dbt/graph/graph.py
+++ b/core/dbt/graph/graph.py
@@ -1,6 +1,7 @@
 from typing import (
     Set, Iterable, Iterator, Optional, NewType
 )
+from itertools import product
 import networkx as nx  # type: ignore
 
 from dbt.exceptions import InternalException
@@ -77,17 +78,26 @@ def select_successors(self, selected: Set[UniqueId]) -> Set[UniqueId]:
             successors.update(self.graph.successors(node))
         return successors
 
-    def get_subset_graph(self, selected: Iterable[UniqueId]) -> 'Graph':
+    def get_subset_graph(self, selected: Iterable[UniqueId]) -> "Graph":
         """Create and return a new graph that is a shallow copy of the graph,
         but with only the nodes in include_nodes. Transitive edges across
         removed nodes are preserved as explicit new edges.
         """
-        new_graph = nx.algorithms.transitive_closure(self.graph)
 
+        new_graph = self.graph.copy()
         include_nodes = set(selected)
 
         for node in self:
             if node not in include_nodes:
+                source_nodes = [x for x, _ in new_graph.in_edges(node)]
+                target_nodes = [x for _, x in new_graph.out_edges(node)]
+
+                new_edges = product(source_nodes, target_nodes)
+                non_cyclic_new_edges = [
+                    (source, target) for source, target in new_edges if source != target
+                ]  # removes cyclic refs
+
+                new_graph.add_edges_from(non_cyclic_new_edges)
                 new_graph.remove_node(node)
 
         for node in include_nodes:
@@ -96,6 +106,7 @@ def get_subset_graph(self, selected: Iterable[UniqueId]) -> 'Graph':
                     "Couldn't find model '{}' -- does it exist or is "
                     "it disabled?".format(node)
                 )
+
         return Graph(new_graph)
 
     def subgraph(self, nodes: Iterable[UniqueId]) -> 'Graph':


Test patch: