Repository: dbt-labs/dbt-core

Issue:
[CT-899] Update manifest flat_graph after deferral, to support dbt-metrics compilation
Metrics + deferral don't work perfectly together, because of how the `dbt-metrics` package is doing the "lookup" for a metric's model parent.

I can envision two ways to solve this problem:
1. Use a "real" `ref` in the `dbt-metrics` package [here](https://github.com/dbt-labs/dbt_metrics/blob/main/macros/graph_parsing/get_model_relation.sql), instead of re-constructing the model relation via `graph` lookup. I'd certainly feel better about it, if we did it this way :)
2. Rebuild the `flat_graph` after deferral. This is actually very simple, and could be a good way to go regardless. It will also enable users who are extracting metadata from the `graph` variable (e.g. `dbt_artifacts` package rewrite: https://github.com/brooklyn-data/dbt_artifacts/pull/132). There are two places we could do this: https://github.com/dbt-labs/dbt-core/commit/b26c7ca3efb277b9fefe778e92c2c488ef816d7b

Those aren't mutually exclusive. Both could be good ideas in their own right.

### Reproduction case

```sql
-- models/model_a.sql
select 1 as id, current_timestamp as ts

```
```yml
# models/metric.yml
version: 2

metrics:
  - name: some_metric
    label: Some Metric
    model: ref('model_a')

    type: count
    sql: id # superfluous here, but shown as an example

    timestamp: ts
    time_grains: [day]
```
```sql
-- models/model_b.sql

-- {{ metric('some_metric') }}

{% if execute %}
  {% set model_ref_node = graph.nodes.values() | selectattr('name', 'equalto', 'model_a') | first %}
  {% set relation = api.Relation.create(
      database = model_ref_node.database,
      schema = model_ref_node.schema,
      identifier = model_ref_node.alias
  )
  %}
{% else %}
  {% set relation = "" %}
{% endif %}

-- this one is a real ref
select * from {{ ref('model_a') }}
union all
-- this one is synthesized via 'graph' var
select * from {{ relation }}
```

I have two profile targets set up: `prod` (with `schema: analytics`) and `dev` (with `schema: dbt_jcohen`). The default is `dev`.

1. Run into a "prod" schema: `dbt run --target prod`
2. Move the "prod" manifest (`target/manifest.json`) into a new folder, e.g. `state/`
3. Change the metric definition (e.g. `sql: id` &rarr; `sql: user_id`)
4. Switch to using the `dev` target. `dbt ls -s state:modified+ --state state/ --target dev` to confirm that `some_metric` + `model_b` are both selected, and `model_a` is not selected
5. Ensure that dev schema is empty (e.g. `drop schema dbt_jcohen cascade`)
6. `dbt run -s state:modified+ --state state/ --defer --target dev`
7. Check logs / compiled SQL:
```sql
-- this one is a real ref
select * from "jerco"."analytics"."model_a"
union all
-- this one is synthesized via 'graph' var
select * from "jerco"."dbt_jcohen"."model_a"
```

### Resolution

Following proposal 2 above, after either of the additions in https://github.com/dbt-labs/dbt-core/commit/b26c7ca3efb277b9fefe778e92c2c488ef816d7b, this works:
```sql
-- this one is a real ref
select * from "jerco"."analytics"."model_a"
union all
-- this one is synthesized via 'graph' var
select * from "jerco"."analytics"."model_a"
```

### Any concerns?

https://github.com/dbt-labs/dbt-core/blob/1071a4681df91633301fdf23e34de819b66fbeb7/core/dbt/contracts/graph/manifest.py#L690-L701

This does require us to rebuild `flat_graph` after parsing is over, at the beginning of execution, since that's where deferral happens. Deferral is resolved before any individual nodes are run in threads, though, so I don't believe we'd need to worry about thread safety.


Discussion:
Let's make sure to do a test case here before we close this out
### Current State
As @jtcohen6 raised in the above issue, there are problems with how the metrics package is currently handling model references. For the curious, this is because metrics can have nested dependencies that resolve as models 1+n nodes upstream. For example, if you have an expression metric that is built on a chain of 4 other metrics, we need to resolve the model(s) reference at the end of that chain. 

Right now we construct that information from `graph`. We can find the edges of the metric dependency, get the model name from the metric, construct that as a relation and then use it in our sql generation. This produces issues, however, when it comes to state deferral.

### Desired End State

As most of this will happen behind the scenes, there are few constraints when it comes to resolving this issue. The end goal is that the relationship would be correctly picked up and understood . @jtcohen6 raised a few ideas on how to resolve this:

- **Add dependencies at parse time**. Whenever dbt-core detects a call to metric(), it also tries to look up that metric's model dependencies, to record them in depends_on.nodes, too. This is potentially very slow, and I'm not even sure it's possible — the metrics haven't been parsed yet.
- **More complex check in the RuntimeRefResolver.** We adjust the logic in RuntimeRefResolver (code linked above) so it checks to ensure that either:
  - the ref'd model is in self.model.depends_on.nodes, OR
  - the ref'd model is upstream of this model, because it's upstream of a metric that is in self.model.depends_on.nodes / self.model.metrics. This feels sorta gross, and like it requires information that's not in scope here.
- **Resolve the ref for the metric, rather than the downstream model**. If there were a way to "compile" a metric so that it's the one resolving the ref, everything's golden. We'd need to store the resolved form of the metric's model reference ... in the graph somewhere ... or make it accessible as another Python method.

One thing I do want to call out is that we'd need this functionality to be able to support n+1 node depth. IE not just understanding `metric --> model` but also understanding `metric --> ... --> model`. 

Would love to hear thoughts about solutions from the languages team!
@callum-mcdata Thanks for the detailed comment! I just opened it as a new issue, for a spike/investigation: https://github.com/dbt-labs/dbt-core/issues/5637

Let's keep using this ticket for the narrowly scoped fix described above: rebuild `flat_graph` after deferral has been resolved

Resolving patch:
diff --git a/core/dbt/contracts/graph/manifest.py b/core/dbt/contracts/graph/manifest.py
--- a/core/dbt/contracts/graph/manifest.py
+++ b/core/dbt/contracts/graph/manifest.py
@@ -1018,6 +1018,10 @@ def merge_from_artifact(
                 merged.add(unique_id)
                 self.nodes[unique_id] = node.replace(deferred=True)
 
+        # Rebuild the flat_graph, which powers the 'graph' context variable,
+        # now that we've deferred some nodes
+        self.build_flat_graph()
+
         # log up to 5 items
         sample = list(islice(merged, 5))
         fire_event(MergedFromState(nbr_merged=len(merged), sample=sample))


Test patch:
diff --git a/tests/functional/metrics/test_metric_deferral.py b/tests/functional/metrics/test_metric_deferral.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/metrics/test_metric_deferral.py
@@ -0,0 +1,131 @@
+import os
+import pytest
+from dbt.tests.util import run_dbt, copy_file, write_file
+from pathlib import Path
+
+metrics_yml = """
+version: 2
+
+metrics:
+  - name: some_metric
+    label: Some Metric
+    model: ref('model_a')
+
+    type: count
+    sql: id
+
+    timestamp: ts
+    time_grains: [day]
+"""
+
+metrics_2_yml = """
+version: 2
+
+metrics:
+  - name: some_metric
+    label: Some Metric
+    model: ref('model_a')
+
+    type: count
+    sql: user_id
+
+    timestamp: ts
+    time_grains: [day]
+"""
+
+model_a_sql = """
+select 1 as fun
+"""
+
+model_b_sql = """
+-- {{ metric('some_metric') }}
+
+{% if execute %}
+  {% set model_ref_node = graph.nodes.values() | selectattr('name', 'equalto', 'model_a') | first %}
+  {% set relation = api.Relation.create(
+      database = model_ref_node.database,
+      schema = model_ref_node.schema,
+      identifier = model_ref_node.alias
+  )
+  %}
+{% else %}
+  {% set relation = "" %}
+{% endif %}
+
+-- this one is a real ref
+select * from {{ ref('model_a') }}
+union all
+-- this one is synthesized via 'graph' var
+select * from {{ relation }}
+"""
+
+
+class TestMetricDeferral:
+    @pytest.fixture(scope="class", autouse=True)
+    def setup(self, project):
+        # Create "prod" schema
+        prod_schema_name = project.test_schema + "_prod"
+        project.create_test_schema(schema_name=prod_schema_name)
+        # Create "state" directory
+        path = Path(project.project_root) / "state"
+        Path.mkdir(path)
+
+    @pytest.fixture(scope="class")
+    def dbt_profile_data(self, unique_schema):
+        return {
+            "config": {"send_anonymous_usage_stats": False},
+            "test": {
+                "outputs": {
+                    "default": {
+                        "type": "postgres",
+                        "threads": 4,
+                        "host": "localhost",
+                        "port": int(os.getenv("POSTGRES_TEST_PORT", 5432)),
+                        "user": os.getenv("POSTGRES_TEST_USER", "root"),
+                        "pass": os.getenv("POSTGRES_TEST_PASS", "password"),
+                        "dbname": os.getenv("POSTGRES_TEST_DATABASE", "dbt"),
+                        "schema": unique_schema,
+                    },
+                    "prod": {
+                        "type": "postgres",
+                        "threads": 4,
+                        "host": "localhost",
+                        "port": int(os.getenv("POSTGRES_TEST_PORT", 5432)),
+                        "user": os.getenv("POSTGRES_TEST_USER", "root"),
+                        "pass": os.getenv("POSTGRES_TEST_PASS", "password"),
+                        "dbname": os.getenv("POSTGRES_TEST_DATABASE", "dbt"),
+                        "schema": unique_schema + "_prod",
+                    },
+                },
+                "target": "default",
+            },
+        }
+
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "model_a.sql": model_a_sql,
+            "model_b.sql": model_b_sql,
+            "metrics.yml": metrics_yml,
+        }
+
+    def test_metric_deferral(self, project):
+        results = run_dbt(["run", "--target", "prod"])
+        assert len(results) == 2
+
+        # copy manifest.json to "state" directory
+        target_path = os.path.join(project.project_root, "target")
+        copy_file(target_path, "manifest.json", project.project_root, ["state", "manifest.json"])
+
+        # Change metrics file
+        write_file(metrics_2_yml, project.project_root, "models", "metrics.yml")
+
+        # Confirm that some_metric + model_b are both selected, and model_a is not selected
+        results = run_dbt(["ls", "-s", "state:modified+", "--state", "state/", "--target", "prod"])
+        assert results == ["metric:test.some_metric", "test.model_b"]
+
+        # Run in default schema
+        results = run_dbt(
+            ["run", "-s", "state:modified+", "--state", "state/", "--defer", "--target", "default"]
+        )
+        assert len(results) == 1