Repository: dbt-labs/dbt-core

Issue:
[CT-1208] [Bug] Manifest deserialization error with disabled models (v1.2 -> v1.3)
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

While running v1.3, dbt encounters a thorny error while trying to deserialize a manifest produced by v1.2 — for use with `--state` comparison functionality — if the previous manifest included one or more disabled models

### Expected Behavior

dbt is able to deserialize a compatible manifest (v1.0+) for use in `--state` comparison / Slim CI use cases

### Steps To Reproduce

Create a project with a disabled model:
```sql
{{ config(enabled = False) }}

select 1 as id
```
```shell
$ pip install dbt-postgres==1.2.1
$ dbt ls
$ mkdir state
$ mv target/manifest.json state/
$ pip install dbt-postgres==1.3.0b2
$ dbt ls -s state:modified --state state
```
Should see `mashumaro.exceptions.InvalidFieldValue`

### Relevant log output

```shell
$ dbt list -s state:modified --state state/
20:15:40  Encountered an error:
Field "disabled" of type Optional[Mapping[str, List[Union[CompiledAnalysisNode, CompiledSingularTestNode, CompiledModelNode, CompiledHookNode, CompiledRPCNode, CompiledSqlNode, CompiledGenericTestNode, CompiledSeedNode, CompiledSnapshotNode, ParsedAnalysisNode, ParsedSingularTestNode, ParsedHookNode, ParsedModelNode, ParsedRPCNode, ParsedSqlNode, ParsedGenericTestNode, ParsedSeedNode, ParsedSnapshotNode, ParsedSourceDefinition]]]] in WritableManifest has invalid value {'model.test.some_model': [{'raw_sql': '{{ config(enabled = False) }}\n\nselect 1 as id', 'resource_type': 'model', 'depends_on': {'macros': [], 'nodes': []}, 'config': {'enabled': False, 'alias': None, 'schema': None, 'database': None, 'tags': [], 'meta': {}, 'materialized': 'view', 'persist_docs': {}, 'quoting': {}, 'column_types': {}, 'full_refresh': None, 'unique_key': None, 'on_schema_change': 'ignore', 'grants': {}, 'post-hook': [], 'pre-hook': []}, 'database': 'jerco', 'schema': 'dbt_jcohen', 'fqn': ['test', 'some_model'], 'unique_id': 'model.test.some_model', 'package_name': 'test', 'root_path': '/Users/jerco/dev/scratch/testy', 'path': 'some_model.sql', 'original_file_path': 'models/some_model.sql', 'name': 'some_model', 'alias': 'some_model', 'checksum': {'name': 'sha256', 'checksum': '10791597936d57b87ec620692cd3c5a0d3ee9375eeca13eed056b1794879a5c6'}, 'tags': [], 'refs': [], 'sources': [], 'metrics': [], 'description': '', 'columns': {}, 'meta': {}, 'docs': {'show': True}, 'patch_path': None, 'compiled_path': None, 'build_path': None, 'deferred': False, 'unrendered_config': {'enabled': False}, 'created_at': 1663618342.081204, 'config_call_dict': {'enabled': False}}]}
20:15:40  Traceback (most recent call last):
  File "<string>", line 93, in from_dict
  File "<string>", line 93, in <dictcomp>
  File "<string>", line 93, in <listcomp>
  File "<string>", line 79, in __unpack_union_WritableManifest_disabled__0a00ae6d19904011b1aa529e4813ef81
mashumaro.exceptions.InvalidFieldValue: Field "disabled" of type Union[CompiledAnalysisNode, CompiledSingularTestNode, CompiledModelNode, CompiledHookNode, CompiledRPCNode, CompiledSqlNode, CompiledGenericTestNode, CompiledSeedNode, CompiledSnapshotNode, ParsedAnalysisNode, ParsedSingularTestNode, ParsedHookNode, ParsedModelNode, ParsedRPCNode, ParsedSqlNode, ParsedGenericTestNode, ParsedSeedNode, ParsedSnapshotNode, ParsedSourceDefinition] in WritableManifest has invalid value {'raw_sql': '{{ config(enabled = False) }}\n\nselect 1 as id', 'resource_type': 'model', 'depends_on': {'macros': [], 'nodes': []}, 'config': {'enabled': False, 'alias': None, 'schema': None, 'database': None, 'tags': [], 'meta': {}, 'materialized': 'view', 'persist_docs': {}, 'quoting': {}, 'column_types': {}, 'full_refresh': None, 'unique_key': None, 'on_schema_change': 'ignore', 'grants': {}, 'post-hook': [], 'pre-hook': []}, 'database': 'jerco', 'schema': 'dbt_jcohen', 'fqn': ['test', 'some_model'], 'unique_id': 'model.test.some_model', 'package_name': 'test', 'root_path': '/Users/jerco/dev/scratch/testy', 'path': 'some_model.sql', 'original_file_path': 'models/some_model.sql', 'name': 'some_model', 'alias': 'some_model', 'checksum': {'name': 'sha256', 'checksum': '10791597936d57b87ec620692cd3c5a0d3ee9375eeca13eed056b1794879a5c6'}, 'tags': [], 'refs': [], 'sources': [], 'metrics': [], 'description': '', 'columns': {}, 'meta': {}, 'docs': {'show': True}, 'patch_path': None, 'compiled_path': None, 'build_path': None, 'deferred': False, 'unrendered_config': {'enabled': False}, 'created_at': 1663618342.081204, 'config_call_dict': {'enabled': False}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/main.py", line 129, in main
    results, succeeded = handle_and_check(args)
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/main.py", line 191, in handle_and_check
    task, res = run_from_args(parsed)
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/main.py", line 218, in run_from_args
    task = parsed.cls.from_args(args=parsed)
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/task/base.py", line 185, in from_args
    return super().from_args(args)
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/task/base.py", line 132, in from_args
    return cls(args, config)
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/task/list.py", line 43, in __init__
    super().__init__(args, config)
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/task/runnable.py", line 118, in __init__
    self.set_previous_state()
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/task/runnable.py", line 122, in set_previous_state
    self.previous_state = PreviousState(
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/contracts/state.py", line 21, in __init__
    self.manifest = WritableManifest.read_and_check_versions(str(manifest_path))
  File "/Users/jerco/dev/scratch/testy/env/lib/python3.9/site-packages/dbt/contracts/util.py", line 263, in read_and_check_versions
    return cls.from_dict(data)  # type: ignore
  File "<string>", line 95, in from_dict
mashumaro.exceptions.InvalidFieldValue: Field "disabled" of type Optional[Mapping[str, List[Union[CompiledAnalysisNode, CompiledSingularTestNode, CompiledModelNode, CompiledHookNode, CompiledRPCNode, CompiledSqlNode, CompiledGenericTestNode, CompiledSeedNode, CompiledSnapshotNode, ParsedAnalysisNode, ParsedSingularTestNode, ParsedHookNode, ParsedModelNode, ParsedRPCNode, ParsedSqlNode, ParsedGenericTestNode, ParsedSeedNode, ParsedSnapshotNode, ParsedSourceDefinition]]]] in WritableManifest has invalid value {'model.test.some_model': [{'raw_sql': '{{ config(enabled = False) }}\n\nselect 1 as id', 'resource_type': 'model', 'depends_on': {'macros': [], 'nodes': []}, 'config': {'enabled': False, 'alias': None, 'schema': None, 'database': None, 'tags': [], 'meta': {}, 'materialized': 'view', 'persist_docs': {}, 'quoting': {}, 'column_types': {}, 'full_refresh': None, 'unique_key': None, 'on_schema_change': 'ignore', 'grants': {}, 'post-hook': [], 'pre-hook': []}, 'database': 'jerco', 'schema': 'dbt_jcohen', 'fqn': ['test', 'some_model'], 'unique_id': 'model.test.some_model', 'package_name': 'test', 'root_path': '/Users/jerco/dev/scratch/testy', 'path': 'some_model.sql', 'original_file_path': 'models/some_model.sql', 'name': 'some_model', 'alias': 'some_model', 'checksum': {'name': 'sha256', 'checksum': '10791597936d57b87ec620692cd3c5a0d3ee9375eeca13eed056b1794879a5c6'}, 'tags': [], 'refs': [], 'sources': [], 'metrics': [], 'description': '', 'columns': {}, 'meta': {}, 'docs': {'show': True}, 'patch_path': None, 'compiled_path': None, 'build_path': None, 'deferred': False, 'unrendered_config': {'enabled': False}, 'created_at': 1663618342.081204, 'config_call_dict': {'enabled': False}}]}
```


### Environment

```markdown
- OS: macOS 12.4
- Python: 3.9.13
- dbt: 1.2.1 -> 1.3.0b2
```


### Which database adapter are you using with dbt?

_No response_

### Additional Context

We just need to update the `upgrade_manifest_json` method, added in https://github.com/dbt-labs/dbt-core/pull/5421, to handle the `disabled` section of the manifest as it does the `nodes` section.

This works for me locally (not saying it's best, though):
```python
def rename_sql_attr(node_content: dict) -> dict:
    if "raw_sql" in node_content:
        node_content["raw_code"] = node_content.pop("raw_sql")
    if "compiled_sql" in node_content:
        node_content["compiled_code"] = node_content.pop("compiled_sql")
    node_content["language"] = "sql"
    return node_content


def upgrade_manifest_json(manifest: dict) -> dict:
    for node_content in manifest.get("nodes", {}).values():
        node_content = rename_sql_attr(node_content)
    for node_content in manifest.get("disabled", {}).values():
        # here it's a list, for some reason
        node_content = [rename_sql_attr(node_content[0])]
    for metric_content in manifest.get("metrics", {}).values():
        # handle attr renames + value translation ("expression" -> "derived")
        metric_content = rename_metric_attr(metric_content)
    return manifest
```


Discussion:

Resolving patch:
diff --git a/core/dbt/contracts/util.py b/core/dbt/contracts/util.py
--- a/core/dbt/contracts/util.py
+++ b/core/dbt/contracts/util.py
@@ -243,13 +243,22 @@ def rename_metric_attr(data: dict, raise_deprecation_warning: bool = False) -> d
     return data
 
 
+def rename_sql_attr(node_content: dict) -> dict:
+    if "raw_sql" in node_content:
+        node_content["raw_code"] = node_content.pop("raw_sql")
+    if "compiled_sql" in node_content:
+        node_content["compiled_code"] = node_content.pop("compiled_sql")
+    node_content["language"] = "sql"
+    return node_content
+
+
 def upgrade_manifest_json(manifest: dict) -> dict:
     for node_content in manifest.get("nodes", {}).values():
-        if "raw_sql" in node_content:
-            node_content["raw_code"] = node_content.pop("raw_sql")
-        if "compiled_sql" in node_content:
-            node_content["compiled_code"] = node_content.pop("compiled_sql")
-        node_content["language"] = "sql"
+        node_content = rename_sql_attr(node_content)
+    for disabled in manifest.get("disabled", {}).values():
+        # There can be multiple disabled nodes for the same unique_id
+        # so make sure all the nodes get the attr renamed
+        disabled = [rename_sql_attr(n) for n in disabled]
     for metric_content in manifest.get("metrics", {}).values():
         # handle attr renames + value translation ("expression" -> "derived")
         metric_content = rename_metric_attr(metric_content)


Test patch: