Repository: dbt-labs/dbt-core

Issue:
[CT-2734] Add metricflow_time_spine to the semantic manifest. 
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

We need to create a time spine model in the users dbt project to support cumulative metrics (full context here: https://www.notion.so/dbtlabs/Time-spine-for-SL-Beta-7c10850fcaf44d47ba0cf23b0a03f4b4). This will be a model that is fully managed by dbt. The user shouldn't have to think about configuring this model or need to write a seperate semantic manifest config. 

We should automatically add this model to the semantic manifest, even if there is no semantic model config,  so metricflow  can use it for constructing cumulative metrics. 

### Describe alternatives you've considered

The user would create a semantic manifest file manually. This feels like unnecessary friction for since this model is an internal MetricFlow model, and not something a user should have to think about configuring. 

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_


Discussion:
Link to spec: https://github.com/dbt-labs/dbt-semantic-interfaces/pull/95/files/db47347be2568f58a015b952fd7cbbae7651f5aa#diff-bcadf38027b48517e9746238936cd9be52c0e28e6c74603d1eb11583a3157a9f
I understand the current plan here is:
- For v1.6: Document how users can define this model themselves
- For v1.7: Find an elegant-but-not-too-magical way to automatically create this model, fully managed by dbt — perhaps using an approach similar to what we'd do for a "latest" pointer to a versioned model (#7442)

> We should automatically add this model to the semantic manifest, even if there is no semantic model config, so metricflow can use it for constructing cumulative metrics.

Question: Should dbt automatically create this model in _every_ project? Even projects that are not using the Semantic Layer (defining semantic models / metrics) at all?
I love the idea of using similar logic for finding the latest versioned model. Thanks for surfacing that discussion! 

> Question: Should dbt automatically create this model in every project? Even projects that are not using the Semantic Layer (defining semantic models / metrics) at all?

I don't think we should create the time spine if the project doesn't have semantic models for two reasons:
1. We're creating an extra table/view in the users warehouse that they don't need. It would be confusing to see this pop up if you're just using dbt for transformations. 
2. Although it's probably negligible, we're adding extra latency and compute to your project when we don't need to be. 

Ideally we're smart enough to know if a project has semantic models, and only create the time spine table if they do. 
Link to DSI changes task: https://linear.app/dbt-labs/issue/SEM-342/project-key-in-semantic-manifest
Also, here's the schema from DSI for the project config: https://github.com/dbt-labs/dbt-semantic-interfaces/pull/95/files#diff-5c4c38db4204c3cf53b401e057f98e3686d3aa2d13ff5c391bd5036097115f21R258
From the standup meeting what I gathered was that the first step was for users to create this model themselves, from documentation. Is that correct? Is there documentation of what that would look like?
@gshank That is correct! Here is the documentation https://github.com/dbt-labs/docs.getdbt.com/pull/3392/files#diff-526134beefd89cf44f090a6108f73da6cc639c7e0e6a5d35fa639a55057f9c7c
Do we want the whole model stored in the SemanticManifest? Where? It's neither a semantic_model nor a metric... Or do we just want the relation name in config? 

Resolving patch:
diff --git a/core/dbt/contracts/graph/manifest.py b/core/dbt/contracts/graph/manifest.py
--- a/core/dbt/contracts/graph/manifest.py
+++ b/core/dbt/contracts/graph/manifest.py
@@ -50,6 +50,7 @@
     DuplicateMacroInPackageError,
     DuplicateMaterializationNameError,
     AmbiguousResourceNameRefError,
+    ParsingError,
 )
 from dbt.helper_types import PathSet
 from dbt.events.functions import fire_event
@@ -62,6 +63,13 @@
 from dbt_semantic_interfaces.implementations.metric import PydanticMetric
 from dbt_semantic_interfaces.implementations.semantic_manifest import PydanticSemanticManifest
 from dbt_semantic_interfaces.implementations.semantic_model import PydanticSemanticModel
+from dbt_semantic_interfaces.implementations.project_configuration import (
+    PydanticProjectConfiguration,
+)
+from dbt_semantic_interfaces.implementations.time_spine_table_configuration import (
+    PydanticTimeSpineTableConfiguration,
+)
+from dbt_semantic_interfaces.type_enums import TimeGranularity
 
 NodeEdgeMap = Dict[str, List[str]]
 PackageName = str
@@ -982,7 +990,12 @@ def analysis_lookup(self) -> AnalysisLookup:
 
     @property
     def pydantic_semantic_manifest(self) -> PydanticSemanticManifest:
-        pydantic_semantic_manifest = PydanticSemanticManifest(metrics=[], semantic_models=[])
+        project_config = PydanticProjectConfiguration(
+            time_spine_table_configurations=[],
+        )
+        pydantic_semantic_manifest = PydanticSemanticManifest(
+            metrics=[], semantic_models=[], project_configuration=project_config
+        )
 
         for semantic_model in self.semantic_models.values():
             pydantic_semantic_manifest.semantic_models.append(
@@ -992,6 +1005,25 @@ def pydantic_semantic_manifest(self) -> PydanticSemanticManifest:
         for metric in self.metrics.values():
             pydantic_semantic_manifest.metrics.append(PydanticMetric.parse_obj(metric.to_dict()))
 
+        # Look for time-spine table model and create time spine table configuration
+        if self.semantic_models:
+            # Get model for time_spine_table
+            time_spine_model_name = "metricflow_time_spine"
+            model = self.ref_lookup.find(time_spine_model_name, None, None, self)
+            if not model:
+                raise ParsingError(
+                    "The semantic layer requires a 'metricflow_time_spine' model in the project, but none was found. Guidance on creating this model can be found on our docs site (https://docs.getdbt.com/docs/build/metricflow-time-spine)"
+                )
+            # Create time_spine_table_config, set it in project_config, and add to semantic manifest
+            time_spine_table_config = PydanticTimeSpineTableConfiguration(
+                location=model.relation_name,
+                column_name="date_day",
+                grain=TimeGranularity.DAY,
+            )
+            pydantic_semantic_manifest.project_configuration.time_spine_table_configurations = [
+                time_spine_table_config
+            ]
+
         return pydantic_semantic_manifest
 
     @property
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -77,7 +77,7 @@
         "minimal-snowplow-tracker~=0.0.2",
         # DSI is under active development, so we're pinning to specific dev versions for now.
         # TODO: Before RC/final release, update to use ~= pinning.
-        "dbt-semantic-interfaces==0.1.0.dev7",
+        "dbt-semantic-interfaces==0.1.0.dev8",
         # ----
         # Expect compatibility with all new versions of these packages, so lower bounds only.
         "packaging>20.9",


Test patch:
diff --git a/tests/functional/semantic_models/test_semantic_model_parsing.py b/tests/functional/semantic_models/test_semantic_model_parsing.py
--- a/tests/functional/semantic_models/test_semantic_model_parsing.py
+++ b/tests/functional/semantic_models/test_semantic_model_parsing.py
@@ -66,6 +66,24 @@
   1000 as revenue,
   current_timestamp as created_at"""
 
+metricflow_time_spine_sql = """
+with days as (
+    {{dbt_utils.date_spine('day'
+    , "to_date('01/01/2000','mm/dd/yyyy')"
+    , "to_date('01/01/2027','mm/dd/yyyy')"
+    )
+    }}
+),
+
+final as (
+    select cast(date_day as date) as date_day
+    from days
+)
+
+select *
+from final
+"""
+
 
 class TestSemanticModelParsing:
     @pytest.fixture(scope="class")
@@ -73,6 +91,7 @@ def models(self):
         return {
             "schema.yml": schema_yml,
             "fct_revenue.sql": fct_revenue_sql,
+            "metricflow_time_spine.sql": metricflow_time_spine_sql,
         }
 
     def test_semantic_model_parsing(self, project):
@@ -111,7 +130,6 @@ def test_semantic_model_changed_partial_parsing(self, project):
         semantic_model = manifest.semantic_models["semantic_model.test.revenue"]
         assert semantic_model.dimensions[0].type_params.time_granularity == TimeGranularity.WEEK
 
-    @pytest.mark.skip(reason="Tracking in https://github.com/dbt-labs/dbt-core/issues/7977")
     def test_semantic_model_deleted_partial_parsing(self, project):
         # First, use the default schema.yml to define our semantic model, and
         # run the dbt parse command
@@ -128,4 +146,4 @@ def test_semantic_model_deleted_partial_parsing(self, project):
         assert result.success
 
         # Finally, verify that the manifest reflects the deletion
-        assert "semanticmodel.test.revenue" not in result.result.semantic_models
+        assert "semantic_model.test.revenue" not in result.result.semantic_models