Repository: dbt-labs/dbt-core

Issue:
[CT-3227] [Bug] KeyError: 'my_project://models/metrics.yml' after deleting YAML files related to semantic layer
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

Failure with long stacktrace after deleting YAML files related to semantic models and/or metrics.

### Workaround

The easiest workaround is do a [`dbt clean`](https://docs.getdbt.com/reference/commands/clean):
```shell
dbt clean
```

<details>
<summary>More detailed workaround</summary>

Here's a workaround that may work in the meantime:

Use `--no-partial-parse` to [disable partial parsing](https://docs.getdbt.com/docs/dbt-versions/release-notes/Feb-2023/no-partial-parse-config):
```shell
dbt --no-partial-parse compile
```

Alternatively, delete the [`partial_parse.msgpack`](https://docs.getdbt.com/reference/parsing#partial-parsing) file from your `target` directory (for example, via [`dbt clean`](https://docs.getdbt.com/reference/commands/clean)).
</details>

### Expected Behavior

Either no failure or at least no stacktrace.

### Steps To Reproduce

1. create a bunch of semantic models / metrics that reference each other
2. parse project: `dbt parse`
3. delete a bunch of them
4. re-parse project: `dbt parse`
5. get a KeyError
```
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 659, in parse_project
    block = FileBlock(self.manifest.files[file_id])
KeyError: 'salesforce_opportunity://models/marts/core/metrics.yml'
```

### Relevant log output

```shell
05:53:09 Running with dbt=1.6.6
05:53:11 Registered adapter: snowflake=1.6.4
05:53:11 Encountered an error:
'salesforce_opportunity://models/marts/core/metrics.yml'
05:53:11 Traceback (most recent call last):
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 87, in wrapper
    result, success = func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 72, in wrapper
    return func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 143, in wrapper
    return func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 172, in wrapper
    return func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 219, in wrapper
    return func(*args, **kwargs)
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/cli/requires.py", line 246, in wrapper
    manifest = ManifestLoader.get_full_manifest(
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 316, in get_full_manifest
    manifest = loader.load()
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 492, in load
    self.parse_project(
  File "/venv/dbt-1.6.0-latest/lib/python3.8/site-packages/dbt/parser/manifest.py", line 659, in parse_project
    block = FileBlock(self.manifest.files[file_id])
KeyError: 'salesforce_opportunity://models/marts/core/metrics.yml'
05:53:09 'salesforce_opportunity://models/marts/core/metrics.yml'
```


### Environment

```markdown
- OS:
- Python:
- dbt: 1.6.6
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

Maybe be related (or a duplicate of):
- https://github.com/dbt-labs/dbt-core/issues/8859
- https://github.com/dbt-labs/dbt-core/issues/8609


Discussion:
### Workaround

The easiest workaround is do a [`dbt clean`](https://docs.getdbt.com/reference/commands/clean):
```shell
dbt clean
```

<details>
<summary>More detailed workaround</summary>

Here's a workaround that may work in the meantime:

Use `--no-partial-parse` to [disable partial parsing](https://docs.getdbt.com/docs/dbt-versions/release-notes/Feb-2023/no-partial-parse-config):
```shell
dbt --no-partial-parse compile
```

Alternatively, delete the [`partial_parse.msgpack`](https://docs.getdbt.com/reference/parsing#partial-parsing) file from your `target` directory (for example, via [`dbt clean`](https://docs.getdbt.com/reference/commands/clean)).
</details>
The key thing to triggering this error was defining a [ratio metric](https://docs.getdbt.com/docs/build/ratio), parsing the project, and then deleting the file it appears within:
```yaml
metrics:
  - name: average_value_per_opportunity
    description: The average value per opportunity
    type: ratio
    label: Average Value per Opportunity
    type_params:
      numerator: total_value
      denominator: total_opportunities
```

<details>
<summary>

### Reprex</summary>

👉 This reproducible example assumes dbt-snowflake>=1.7.0-rc1. You'll need to adjust the `metricflow_time_spine` model in order to use dbt_utils to make it work for dbt ~=1.6.0.

`seeds/orders.csv`
```csv
"order_id","location_id","customer_id","order_total","tax_paid","ordered_at","count_food_items","count_drink_items","count_items","subtotal_drink_items","subtotal_food_items","subtotal","order_cost","location_name","is_food_order","is_drink_order"
"3d12eb16-1bbb-47ad-a732-bf156c322fa6","7f790ed7-0fc4-4de2-a1b0-cce72e657fc4","7cd5e7f3-39c8-48ce-950a-1b70449e7829",6.3600000000000000,0.36000000000000000000,2016-09-01 19:58:00.000,0,1,1,6.0000000000000000,0,6.0000000000000000,0.82000000000000000000,Philadelphia,false,true
"2c54134f-2322-4704-9f82-e2e9b77bcb19","7f790ed7-0fc4-4de2-a1b0-cce72e657fc4","96c8937f-d99c-4515-acec-db24d07768f9",50.8800000000000000,2.8800000000000000,2016-09-01 09:35:00.000,3,2,5,11.0000000000000000,37.0000000000000000,48.0000000000000000,10.32000000000000000000,Philadelphia,true,true
```

Create `models/metricflow_time_spine.sql` as described [here](https://docs.getdbt.com/docs/build/metricflow-time-spine):
```sql
{{ config(materialized='table') }}

with days as (

    {{
        dbt.date_spine(
            'day',
            "to_date('01/01/2000','mm/dd/yyyy')",
            to_date('2027-01-02')
        )
    }}

),

final as (
    select cast(date_day as date) as date_day
    from days
)

select * from final
```

`models/semantic_models.yml`
```yaml

semantic_models:
  - name: orders_2
    model: ref('orders')
    description: |
      Order fact table. This table is at the order grain with one row per order.
    defaults:
      agg_time_dimension: ordered_at
    entities:
      - name: order_id
        type: primary
    dimensions:
      - name: ordered_at
        expr: ordered_at
        type: time
        type_params:
          time_granularity: day
    measures:
      - name: customers_with_orders_2
        description: Distinct count of customers placing orders
        agg: count_distinct
        expr: customer_id
      - name: total_value
        agg: sum
        expr: order_total
      - name: total_opportunities
        agg: count_distinct
        expr: customer_id
```

`models/metrics.yml`
```yaml
metrics:
  - name: average_value_per_opportunity
    description: The average value per opportunities
    type: ratio
    label: Average Value per Opportunity
    type_params:
      numerator: total_value
      denominator: total_opportunities

  - name: total_value
    description: The total opportunity value for the business
    type: simple
    label: Total Value
    type_params:
      measure: total_value

  - name: total_opportunities
    description: The total opportunities for the business
    type: simple
    label: Total Opportunities
    type_params:
      measure: total_opportunities
```

At the end, your project folder will look like this:
```
├── dbt_project.yml
├── models
│   ├── metricflow_time_spine.sql
│   ├── metrics.yml
│   └── semantic_models.yml
└── seeds
    └── orders.csv
```

Run these set of commands to see the error:
```shell
dbt clean
dbt parse
mv models/metrics.yml models/metrics.yml.x
dbt parse
```

Here's the error I got:
```
15:27:46  Encountered an error:
'my_project://models/metrics.yml'
...
  File "/venv/snowflake_1.7/lib/python3.10/site-packages/dbt/parser/manifest.py", line 663, in parse_project
    block = FileBlock(self.manifest.files[file_id])
KeyError: 'my_project://models/metrics.yml'
```

Then run this to make the problem "go away":
```shell
dbt --no-partial-parse parse
```

</details>
@dbeatty10 Is this only affecting 1.6 and 1.7?
> @dbeatty10 Is this only affecting 1.6 and 1.7?

I didn't check 1.5 since the reprex was using `semantic_models` that first appeared in [1.6](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md).

@graciegoheen Do you think we should try out a similar non-MetricFlow `ratio` metric with 1.5 also?
Got it - let's contain this issue to `semantic_models`. I'll go ahead and add the 1.6 and 1.7 backport labels
We had hoped this was resolved by some related work we did. However I just tried the reproduction with 1.7.latest and the error is still occurring.
The error specifically happens when a yaml file which contains definitions for both semantic models and metrics is deleted

Resolving patch:
diff --git a/core/dbt/parser/partial.py b/core/dbt/parser/partial.py
--- a/core/dbt/parser/partial.py
+++ b/core/dbt/parser/partial.py
@@ -211,6 +211,7 @@ def add_to_pp_files(self, source_file):
         if (
             file_id not in self.project_parser_files[project_name][parser_name]
             and file_id not in self.file_diff["deleted"]
+            and file_id not in self.file_diff["deleted_schema_files"]
         ):
             self.project_parser_files[project_name][parser_name].append(file_id)
 
@@ -467,7 +468,11 @@ def schedule_nodes_for_parsing(self, unique_ids):
 
     def _schedule_for_parsing(self, dict_key: str, element, name, delete: Callable) -> None:
         file_id = element.file_id
-        if file_id in self.saved_files and file_id not in self.file_diff["deleted"]:
+        if (
+            file_id in self.saved_files
+            and file_id not in self.file_diff["deleted"]
+            and file_id not in self.file_diff["deleted_schema_files"]
+        ):
             schema_file = self.saved_files[file_id]
             elements = []
             assert isinstance(schema_file, SchemaSourceFile)


Test patch:
diff --git a/tests/functional/partial_parsing/fixtures.py b/tests/functional/partial_parsing/fixtures.py
--- a/tests/functional/partial_parsing/fixtures.py
+++ b/tests/functional/partial_parsing/fixtures.py
@@ -452,6 +452,60 @@
       agg_time_dimension: created_at
 """
 
+people_sl_yml = """
+version: 2
+
+semantic_models:
+  - name: semantic_people
+    model: ref('people')
+    dimensions:
+      - name: favorite_color
+        type: categorical
+      - name: created_at
+        type: TIME
+        type_params:
+          time_granularity: day
+    measures:
+      - name: years_tenure
+        agg: SUM
+        expr: tenure
+      - name: people
+        agg: count
+        expr: id
+    entities:
+      - name: id
+        type: primary
+    defaults:
+      agg_time_dimension: created_at
+
+metrics:
+
+  - name: number_of_people
+    description: Total count of people
+    label: "Number of people"
+    type: simple
+    type_params:
+      measure: people
+    meta:
+        my_meta: 'testing'
+
+  - name: collective_tenure
+    description: Total number of years of team experience
+    label: "Collective tenure"
+    type: simple
+    type_params:
+      measure:
+        name: years_tenure
+        filter: "{{ Dimension('id__loves_dbt') }} is true"
+
+  - name: average_tenure
+    label: Average Tenure
+    type: ratio
+    type_params:
+      numerator: collective_tenure
+      denominator: number_of_people
+"""
+
 env_var_metrics_yml = """
 
 metrics:
diff --git a/tests/functional/partial_parsing/test_pp_metrics.py b/tests/functional/partial_parsing/test_pp_metrics.py
--- a/tests/functional/partial_parsing/test_pp_metrics.py
+++ b/tests/functional/partial_parsing/test_pp_metrics.py
@@ -1,6 +1,8 @@
 import pytest
 
-from dbt.tests.util import run_dbt, write_file, get_manifest
+from dbt.cli.main import dbtRunner
+from dbt.contracts.graph.manifest import Manifest
+from dbt.tests.util import run_dbt, rm_file, write_file, get_manifest
 from tests.functional.partial_parsing.fixtures import (
     people_sql,
     metricflow_time_spine_sql,
@@ -9,6 +11,7 @@
     people_metrics2_yml,
     metric_model_a_sql,
     people_metrics3_yml,
+    people_sl_yml,
 )
 
 from dbt.exceptions import CompilationError
@@ -84,3 +87,29 @@ def test_metrics(self, project):
             # We use "parse" here and not "run" because we're checking that the CompilationError
             # occurs at parse time, not compilation
             results = run_dbt(["parse"])
+
+
+class TestDeleteFileWithMetricsAndSemanticModels:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "people.sql": people_sql,
+            "metricflow_time_spine.sql": metricflow_time_spine_sql,
+            "people_sl.yml": people_sl_yml,
+        }
+
+    def test_metrics(self, project):
+        # Initial parsing
+        runner = dbtRunner()
+        result = runner.invoke(["parse"])
+        assert result.success
+        manifest = result.result
+        assert isinstance(manifest, Manifest)
+        assert len(manifest.metrics) == 3
+
+        # Remove metric file
+        rm_file(project.project_root, "models", "people_sl.yml")
+
+        # Rerun parse, shouldn't fail
+        result = runner.invoke(["parse"])
+        assert result.exception is None, result.exception