Repository: dbt-labs/dbt-core

Issue:
[CT-948] [Feature] Adding Metric Tree Parsing Helper Functions to dbt Core
### Is this your first time opening an issue?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)

### Describe the Feature

Right now, the `dbt_metrics` package uses a macro called `get_metric_tree` to traverse the graph in reverse order and get the following information for each metric that is used:
- `full_set`: the full list of metrics that are used in the macro call.
- `parent_set`: the list of metrics upstream of the metrics used in the macro call. NOT inclusive of expression metrics - this list is only comprised of base metrics
- `expression_set`: the list of expression metrics used in the macro call. Inclusive of ones listed or upstrea
- `base_set`: the list of metrics listed in the macro call. 
- `ordered_expression_set`: the list of expression metrics + their depth. Starting at 999 and moving donwards to limit to 1k. Ex:
  - metric a: 999
  - metric b (downstream of a): 998

### Helper Functions
This is a list of helper functions that I believe would satisfy the same needs that we currently use the metric tree for:
1. **`get_full_metric_list`**: 
    - **Function**: This function would traverse through the dag in the opposite direction of 1 or more provided metrics and return a list of all metrics upstream of them. **Inclusive of metrics listed in input**
    - **Input**: 1+ metrics via the format `metric('metric_name')`
2. **`get_base_metric_list`**: 
    - **Function**: This function would traverse through the dag in the opposite direction of 1 or more provided metrics and return a list of all base (non-expression) metrics upstream. **Inclusive of metrics listed in input**
    - **Input**: 1+ metrics via the format `metric('metric_name')`
3. **`get_expression_metric_list`**: 
    - **Function**: This function would traverse through the dag in the opposite direction of 1 or more provided metrics and return a list of all expression metrics upstream of them. **Inclusive of metrics listed in input**
    - **Input**: 1+ metrics via the format `metric('metric_name')`
4. **`get_ordered_expression_metric_list`**: 
    - **Function**: This function would traverse through the dag in the opposite direction of 1 or more provided metrics and return a list of all expression metrics upstream of them. It would also contain a numeric representation of the depth of the expression metric  **Inclusive of metrics listed in input**
    - **Input**: 1+ metrics via the format `metric('metric_name')`
    - **Output**: A list of dictionaries like: `['metric_a':1, 'metric_b':2]`

I believe we can get away from having a `get_base_metric_list` because we can have that just be the input from the macro now that we don't need to iterate recursively over a dictionary like we're doing now. Additionally we might be able to get away with just the second expression metric list .`

### Describe alternatives you've considered

The alternative is the current implementation in the `dbt_metrics` package.

### Who will this benefit?

Anyone using the `dbt_metrics` package (ie anyone using the Semantic Layer) and the developer/maintainers of the package!

### Are you interested in contributing this feature?

Yes! I'd love to try and write some of these helper functions.

### Anything else?

[This is the link to the jinja code that currently accomplishes similar functionality.](https://github.com/dbt-labs/dbt_metrics/blob/main/macros/variables/get_metric_tree.sql)


Discussion:
@callum-mcdata Thanks for the thorough write-up! I believe we must prioritize some or all of this refactoring work, ahead of declaring metrics + the `dbt-metrics` package ready for mature use.

There's one important piece I don't see here, which is constructing the relations for **models upstream of metrics**. Currently, the `dbt-metrics` package does that via `graph` introspection. I think we should investigate using real `ref` resolution instead, provided by a `dbt-core` helper. It yielded the issue in #5525. We can resolve that one by other means, but we can't guarantee that we won't run into more weird edge cases.
> There's one important piece I don't see here, which is constructing the relations for models upstream of metrics. Currently, the dbt-metrics package does that via graph introspection. I think we should investigate using real ref resolution instead, provided by a dbt-core helper. It yielded the issue in https://github.com/dbt-labs/dbt-core/issues/5525. We can resolve that one by other means, but we can't guarantee that we won't run into more weird edge cases.

I'd hoped using a `ref` resolution would be a simple swap in the `dbt_metrics` package but unfortunately it ran into some [issues as seen here](https://github.com/dbt-labs/dbt_metrics/issues/64). Would a `dbt-core` helper allow us to sidestep the conditional block `ref` issue?

Resolving patch:
diff --git a/core/dbt/contracts/graph/metrics.py b/core/dbt/contracts/graph/metrics.py
--- a/core/dbt/contracts/graph/metrics.py
+++ b/core/dbt/contracts/graph/metrics.py
@@ -38,33 +38,52 @@ def parent_metrics(cls, metric_node, manifest):
             if node and node.resource_type == NodeType.Metric:
                 yield from cls.parent_metrics(node, manifest)
 
-    def parent_models(self):
+    @classmethod
+    def parent_metrics_names(cls, metric_node, manifest):
+        yield metric_node.name
+
+        for parent_unique_id in metric_node.depends_on.nodes:
+            node = manifest.metrics.get(parent_unique_id)
+            if node and node.resource_type == NodeType.Metric:
+                yield from cls.parent_metrics_names(node, manifest)
+
+    @classmethod
+    def reverse_dag_parsing(cls, metric_node, manifest, metric_depth_count):
+        if metric_node.type == "expression":
+            yield {metric_node.name: metric_depth_count}
+            metric_depth_count = metric_depth_count + 1
+
+        for parent_unique_id in metric_node.depends_on.nodes:
+            node = manifest.metrics.get(parent_unique_id)
+            if node and node.resource_type == NodeType.Metric and node.type == "expression":
+                yield from cls.reverse_dag_parsing(node, manifest, metric_depth_count)
+
+    def full_metric_dependency(self):
+        to_return = list(set(self.parent_metrics_names(self.node, self.manifest)))
+        return to_return
+
+    def base_metric_dependency(self):
         in_scope_metrics = list(self.parent_metrics(self.node, self.manifest))
 
-        to_return = {
-            "base": [],
-            "derived": [],
-        }
+        to_return = []
         for metric in in_scope_metrics:
-            if metric.type == "expression":
-                to_return["derived"].append(
-                    {"metric_source": None, "metric": metric, "is_derived": True}
-                )
-            else:
-                for node_unique_id in metric.depends_on.nodes:
-                    node = self.manifest.nodes.get(node_unique_id)
-                    if node and node.resource_type in NodeType.refable():
-                        to_return["base"].append(
-                            {
-                                "metric_relation_node": node,
-                                "metric_relation": self.Relation.create(
-                                    database=node.database,
-                                    schema=node.schema,
-                                    identifier=node.alias,
-                                ),
-                                "metric": metric,
-                                "is_derived": False,
-                            }
-                        )
+            if metric.type != "expression" and metric.name not in to_return:
+                to_return.append(metric.name)
+
+        return to_return
+
+    def derived_metric_dependency(self):
+        in_scope_metrics = list(self.parent_metrics(self.node, self.manifest))
+
+        to_return = []
+        for metric in in_scope_metrics:
+            if metric.type == "expression" and metric.name not in to_return:
+                to_return.append(metric.name)
+
+        return to_return
+
+    def derived_metric_dependency_depth(self):
+        metric_depth_count = 1
+        to_return = list(self.reverse_dag_parsing(self.node, self.manifest, metric_depth_count))
 
         return to_return


Test patch:
diff --git a/tests/functional/metrics/test_metric_helper_functions.py b/tests/functional/metrics/test_metric_helper_functions.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/metrics/test_metric_helper_functions.py
@@ -0,0 +1,103 @@
+import pytest
+
+from dbt.tests.util import run_dbt, get_manifest
+from dbt.contracts.graph.metrics import ResolvedMetricReference
+
+metrics__yml = """
+version: 2
+
+metrics:
+
+  - name: number_of_people
+    label: "Number of people"
+    description: Total count of people
+    model: "ref('people')"
+    type: count
+    sql: "*"
+    timestamp: created_at
+    time_grains: [day, week, month]
+    dimensions:
+      - favorite_color
+      - loves_dbt
+    meta:
+        my_meta: 'testing'
+
+  - name: collective_tenure
+    label: "Collective tenure"
+    description: Total number of years of team experience
+    model: "ref('people')"
+    type: sum
+    sql: tenure
+    timestamp: created_at
+    time_grains: [day, week, month]
+    filters:
+      - field: loves_dbt
+        operator: 'is'
+        value: 'true'
+
+  - name: average_tenure
+    label: "Average tenure"
+    description: "The average tenure per person"
+    type: expression
+    sql: "{{metric('collective_tenure')}} / {{metric('number_of_people')}} "
+    timestamp: created_at
+    time_grains: [day, week, month]
+
+  - name: average_tenure_plus_one
+    label: "Average tenure"
+    description: "The average tenure per person"
+    type: expression
+    sql: "{{metric('average_tenure')}} + 1 "
+    timestamp: created_at
+    time_grains: [day, week, month]
+"""
+
+models__people_sql = """
+select 1 as id, 'Drew' as first_name, 'Banin' as last_name, 'yellow' as favorite_color, true as loves_dbt, 5 as tenure, current_timestamp as created_at
+union all
+select 1 as id, 'Jeremy' as first_name, 'Cohen' as last_name, 'indigo' as favorite_color, true as loves_dbt, 4 as tenure, current_timestamp as created_at
+union all
+select 1 as id, 'Callum' as first_name, 'McCann' as last_name, 'emerald' as favorite_color, true as loves_dbt, 0 as tenure, current_timestamp as created_at
+"""
+
+
+class TestMetricHelperFunctions:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "metrics.yml": metrics__yml,
+            "people.sql": models__people_sql,
+        }
+
+    def test_expression_metric(
+        self,
+        project,
+    ):
+
+        # initial parse
+        run_dbt(["compile"])
+
+        # make sure all the metrics are in the manifest
+        manifest = get_manifest(project.project_root)
+        parsed_metric = manifest.metrics["metric.test.average_tenure_plus_one"]
+        testing_metric = ResolvedMetricReference(parsed_metric, manifest, None)
+
+        full_metric_dependency = set(testing_metric.full_metric_dependency())
+        expected_full_metric_dependency = set(
+            ["average_tenure_plus_one", "average_tenure", "collective_tenure", "number_of_people"]
+        )
+        assert full_metric_dependency == expected_full_metric_dependency
+
+        base_metric_dependency = set(testing_metric.base_metric_dependency())
+        expected_base_metric_dependency = set(["collective_tenure", "number_of_people"])
+        assert base_metric_dependency == expected_base_metric_dependency
+
+        derived_metric_dependency = set(testing_metric.derived_metric_dependency())
+        expected_derived_metric_dependency = set(["average_tenure_plus_one", "average_tenure"])
+        assert derived_metric_dependency == expected_derived_metric_dependency
+
+        derived_metric_dependency_depth = list(testing_metric.derived_metric_dependency_depth())
+        expected_derived_metric_dependency_depth = list(
+            [{"average_tenure_plus_one": 1}, {"average_tenure": 2}]
+        )
+        assert derived_metric_dependency_depth == expected_derived_metric_dependency_depth