Repository: dbt-labs/dbt-core

Issue:
[CT-2315] [Feature] Ignore duplicate edges while building the subgraph for the job queue 
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Context: I'm investigating an issue with the command `dbt build` that became very slow in our project (1,300+ models and 20,000+ tests). It impacts user productivity (waiting time) and time-sensitive scheduled jobs. 

I suspect the function [Graph.get_subset_graph](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/graph/graph.py#L68) can become slow when running `dbt build` in a project with wide tables and many tests.

Let's consider a simple example:
```
Source A -----> Model B --------> Model C
     |             |                |
    \/             \/               \/ 
  Test A         Test B           Test C
```

The [function](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/graph/graph.py#L68) loops through all the graph nodes and extract the upstream & downstream nodes. It creates the product of these nodes and adds the edges to the subgraph.
```
for node in self:
         if node not in include_nodes:
                source_nodes = [x for x, _ in new_graph.in_edges(node)]
                target_nodes = [x for _, x in new_graph.out_edges(node)]

                new_edges = product(source_nodes, target_nodes)
                non_cyclic_new_edges = [
                    (source, target) for source, target in new_edges if source != target
                ]  # removes cyclic refs

                new_graph.add_edges_from(non_cyclic_new_edges)
                new_graph.remove_node(node)
```

During `dbt build`, when evaluating the node `Test B` (and looking at the test nodes only), `source_nodes = [Test A]` and `target_nodes = [Test C]`. So, the edges to add to the subgraph are `[(Test A, Test C)]`. 
-> Is `Test A` considered a source nodes due to the [add_test_edges function](https://github.com/dbt-labs/dbt-core/blob/4186f99b742b47e0e95aca4f604cc09e5c67a449/core/dbt/compilation.py#L427) in the [build task](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/task/build.py#L72)?

Now, let's consider we have 30 tests in `Source A` and 500 tests in `Model C` --> We get 15000 new edges to add to the graph for node `Test B`.
And now let's consider we also have 100 tests in `Model B`. For all of them, we have the same source nodes and target nodes, so we have the same 15000 edges to add 100 times to the new graph (1,500,000 edges). The high number of edges to add makes the function `new_graph.add_edges_from` slow.

As the edges are 100 times the same, I think we could prevent duplicate work by ignoring edges that already exist in the new graph.

```
new_edges = [
     (source, target) for source, target in new_edges if source != target and not new_graph.has_edge(source, target)
]  # removes cyclic refs and edges already existing in the new graph
```

What do you think? 

Note that the issue doesn't seem to happen if we run `dbt run` or `dbt test` (because `source_nodes` is empty in these cases). 




### Describe alternatives you've considered

I don't observe the same behavior when running `dbt run` or `dbt test`. So, alternatively, we can use these 2 commands sequentially instead of `dbt build`, but it's not exactly the same


### Who will this benefit?

`dbt build ...` is a standard CLI command many DBT users execute.

### Are you interested in contributing this feature?

Yes

### Anything else?

_No response_


Discussion:

Resolving patch:
diff --git a/core/dbt/graph/graph.py b/core/dbt/graph/graph.py
--- a/core/dbt/graph/graph.py
+++ b/core/dbt/graph/graph.py
@@ -81,8 +81,10 @@ def get_subset_graph(self, selected: Iterable[UniqueId]) -> "Graph":
 
                 new_edges = product(source_nodes, target_nodes)
                 non_cyclic_new_edges = [
-                    (source, target) for source, target in new_edges if source != target
-                ]  # removes cyclic refs
+                    (source, target)
+                    for source, target in new_edges
+                    if source != target and not new_graph.has_edge(source, target)
+                ]  # removes cyclic refs and edges already existing in new graph
 
                 new_graph.add_edges_from(non_cyclic_new_edges)
                 new_graph.remove_node(node)


Test patch: