Repository: dbt-labs/dbt-core

Issue:
[CT-1150] Include `flat_graph` in `manifest.deepcopy()`
Why isn't `flat_graph` included in `manifest.deepcopy()`? Probably not for any reason in particular.

The original implementation of `manifest.deepcopy()` was the early days of the dbt-RPC server (https://github.com/dbt-labs/dbt-core/commit/1a0df174c9f252d0f93e03dbf4ef5005b2429072), and that's still the only place we really call it. The idea is to support dynamically adding new temporary nodes (arbitrary SQL compilation/execution) that might need to [`add_refs`](https://github.com/dbt-labs/dbt-rpc/blob/b45d09e487fedb1b12fd3214081e593d52332bb4/dbt_rpc/task/sql_commands.py#L30-L50) in the manifest — or rather, a full copy of the manifest.

This does mean, however, that any time you want to deepcopy of a manifest, and you want that copied manifest to support introspection of the `graph` variable in the Jinja context, you actually need an extra step:
```python
from dbt.contracts.graph.manifest import Manifest

manifest = Manifest(...)

# should just be this, right?
copy_of_manifest = manifest.deepcopy()

# still missing something
copy_of_manifest.build_flat_graph()
# now you're good to go
```

The fix here might be as simple as taking this:
https://github.com/dbt-labs/dbt-core/blob/b26280d1cf1af1c715753f73eb3b46f6eaa717f5/core/dbt/contracts/graph/manifest.py#L781-L794
And adding in:
```python
import copy
...

    def deepcopy(self):
        return Manifest(
            ...
            flat_graph={k: copy.deepcopy(v) for k, v in self.flat_graph.items()},
            ...
        )
```


Discussion:
@emmyoop maybe this is something small you could pick up with extra bandwidth

Resolving patch:
diff --git a/core/dbt/contracts/graph/manifest.py b/core/dbt/contracts/graph/manifest.py
--- a/core/dbt/contracts/graph/manifest.py
+++ b/core/dbt/contracts/graph/manifest.py
@@ -780,7 +780,7 @@ def get_used_databases(self):
         return frozenset(x.database for x in chain(self.nodes.values(), self.sources.values()))
 
     def deepcopy(self):
-        return Manifest(
+        copy = Manifest(
             nodes={k: _deepcopy(v) for k, v in self.nodes.items()},
             sources={k: _deepcopy(v) for k, v in self.sources.items()},
             macros={k: _deepcopy(v) for k, v in self.macros.items()},
@@ -793,6 +793,8 @@ def deepcopy(self):
             files={k: _deepcopy(v) for k, v in self.files.items()},
             state_check=_deepcopy(self.state_check),
         )
+        copy.build_flat_graph()
+        return copy
 
     def build_parent_and_child_maps(self):
         edge_members = list(


Test patch:
diff --git a/test/unit/test_manifest.py b/test/unit/test_manifest.py
--- a/test/unit/test_manifest.py
+++ b/test/unit/test_manifest.py
@@ -547,6 +547,34 @@ def test_get_resource_fqns(self):
         resource_fqns = manifest.get_resource_fqns()
         self.assertEqual(resource_fqns, expect)
 
+    def test__deepcopy_copies_flat_graph(self):
+        test_node = ParsedModelNode(
+                name='events',
+                database='dbt',
+                schema='analytics',
+                alias='events',
+                resource_type=NodeType.Model,
+                unique_id='model.snowplow.events',
+                fqn=['snowplow', 'events'],
+                package_name='snowplow',
+                refs=[],
+                sources=[],
+                metrics=[],
+                depends_on=DependsOn(),
+                config=self.model_config,
+                tags=[],
+                path='events.sql',
+                original_file_path='events.sql',
+                root_path='',
+                meta={},
+                language='sql',
+                raw_code='does not matter',
+                checksum=FileHash.empty())
+
+        original = make_manifest(nodes=[test_node])
+        original.build_flat_graph()
+        copy = original.deepcopy()
+        self.assertEqual(original.flat_graph, copy.flat_graph)
 
 class MixedManifestTest(unittest.TestCase):
     def setUp(self):