Repository: nicholasyager/dbt-loom

Issue:
Pydantic warnings after upgrade to dbt-core 1.9.1. State selection does not work.
**Describe the bug**
updating a downstream project to dbt 1.9 fails on `seed` and `operation` nodes.


**Expected behavior**
upgrading from dbt 1.8.3 to dbt 1.9.1 should be possible.

```
15:17:05  Running with dbt=1.9.1
15:17:07  Initializing dbt-loom=0.7.3
15:17:07  dbt-loom: Patching ref protection methods to support dbt-loom dependencies.
15:17:07  dbt-loom: Loading manifest for `baseline` from `s3`
/workspace/dap/.venv/lib/python3.11/site-packages/pydantic/main.py:426: UserWarning: Pydantic serializer warnings:
  Expected `enum` but got `str` with value `'seed'` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
15:17:08  dbt-loom: Loading manifest for `dm_felles_dimensjoner` from `s3`
/workspace/dap/.venv/lib/python3.11/site-packages/pydantic/main.py:426: UserWarning: Pydantic serializer warnings:
  Expected `enum` but got `str` with value `'operation'` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
15:17:08  dbt-loom: Loading manifest for `dap_test_upstream` from `s3`
15:17:08  dbt-loom: Loading manifest for `dap_test` from `s3`
15:17:09  Registered adapter: snowflake=1.9.0
15:17:09  Unable to do partial parsing because saved manifest not found. Starting full parse.
15:17:11  dbt-loom: Injecting nodes
15:17:13  Found 1367 models, 12 data tests, 1 source, 660 macros
15:17:13  The selection criterion 'state:modified+' does not match any enabled nodes
15:17:13  The selection criterion 'state:modified+' does not match any enabled nodes
15:17:13  The selection criterion 'state:modified+' does not match any enabled nodes
15:17:13  Nothing to do. Try checking your model configs and model specification args
```
**Additional context**
Add any other context about the problem here.

Discussion:
@nicholasyager: Hi @fredrikhgrelland 👋🏻 We have unit tests running in CI/CD that verifies that dbt-loom functions with dbt-core 1.9.1 as expected. 

Two things stand out to me based on the log provided:
- The Pydantic warnings are not something I've seen before. I'm curious if you're able to trace them down to a specific model or value in your manifests.
- Why are you expecting `sate:modified+`to select anything? Unfortunately, dbt-core's plugin interface does not provide model checksums or raw code, so dbt-core has no way to know if an upstream model has modified state. @nicholasyager: Removing the `bug` label until this can be confirmed to be a defect in dbt-loom.@fredrikhgrelland: Hi, the modified+ part is not specifically related to this case, it is simply a part of my cicd, and I did not remove it.

We are currently running with loom 7.3 and dbt 1.8.9 successfully in this same repository. This error came when we made no other change than updating to dbt 1.9.1

As you see it only happens with one of the manifests.
I can certainly look into the manifest, but it contains more than 1000 models.. 
are you testing with seeds and run-operations? I can try to create a narrow case with that@nicholasyager: Thanks, @fredrikhgrelland ! I'll be on the lookout for your findings.

> are you testing with seeds and run-operations?

- Seeds, yes! 
- Macros (run-operations), no. Macros are not injectable by loom (or dbt Cloud, as far as I know). Current recommendation from dbt Labs is to create a separate macro-centric package and import it into the downstream projects.@fredrikhgrelland: I have had some time to look this over, and two separate issues combined made me create this bug report.

1. The pydantic warning started with dbt 1.9, but it does not seem to affect the operation of dbt loom. I am unsure however if the same nodes are actually injected in the graph...
2. dbt 1.9 had a [breaking change](https://github.com/dbt-labs/dbt-core/issues/8997). Previously changing the working directory to DBT_PROJECT_DIR while running `deps` in dbt.invoke(). This allowed dbt_loom to pick up its config file without needing the explicit `DBT_LOOM_CONFIG` env variable. In dbt 1.9 this variable has to be set if running dbt programatically in python, rather than via cli.

I was not able to find the exact code change that made the pydantic warning happen, but there has to be some significant changes in manifest handling with 1.9.
One of my largest projects where we use loom are compiling fine with dbt 1.8.8 but are failing on 1.9.1 with a compilation error:
```
Compilation Error
  'model.dbt_baseline_datavault.kuf_basis_rutinekode_pskf_custombridge' depends on 'model.dbt_baseline_datavault.kuf_pensjons_sparing_custombridge' which is not in the graph!
```

This model is absolutely there. If you follow the dependencies a few levels down, it has a reference to a separate project, referenced with loom, which again has reference to the current project. So we have a cyclic dependency! We did not catch that until now because it compiles and runs without any issues in dbt 1.8.

Solution patch:
diff --git a/README.md b/README.md
index 6628cd6..5b9af29 100644
--- a/README.md
+++ b/README.md
@@ -67,8 +67,8 @@ By default, `dbt-loom` will look for `dbt_loom.config.yml` in your working direc
 ## How does it work?
 
 As of dbt-core 1.6.0-b8, there now exists a `dbtPlugin` class which defines functions that can
-be called by dbt-core's `PluginManger`. During different parts of the dbt-core lifecycle (such as graph linking and
-manifest writing), the `PluginManger` will be called and all plugins registered with the appropriate hook will be executed.
+be called by dbt-core's `PluginManager`. During different parts of the dbt-core lifecycle (such as graph linking and
+manifest writing), the `PluginManager` will be called and all plugins registered with the appropriate hook will be executed.
 
 dbt-loom implements a `get_nodes` hook, and uses a configuration file to parse manifests, identify public models, and
 inject those public models when called by `dbt-core`.
@@ -226,6 +226,19 @@ manifests:
       - dbt_project_evaluator
 ```
 
+### Optional manifests
+
+If you want to allow a manifest reference to be missing (e.g. using dbt-loom for an upstream project to see dependencies), you can set `optional: true` for that manifest entry. When `optional` is true and the manifest file does not exist, dbt-loom will skip loading it without raising an error. If `optional` is false or omitted (the default), missing manifests will cause an error.
+
+```yaml
+manifests:
+  - name: revenue
+    type: file
+    config:
+      path: ../revenue/target/manifest.json
+    optional: true  # If the manifest file is missing, do not raise an error
+```
+
 ## Known Caveats
 
 Cross-project dependencies are a relatively new development, and dbt-core plugins
diff --git a/dbt_loom/config.py b/dbt_loom/config.py
index 234b027..619aa53 100644
--- a/dbt_loom/config.py
+++ b/dbt_loom/config.py
@@ -59,6 +59,7 @@ class ManifestReference(BaseModel):
         SnowflakeReferenceConfig,
     ]
     excluded_packages: List[str] = Field(default_factory=list)
+    optional: bool = False
 
 
 class dbtLoomConfig(BaseModel):
diff --git a/dbt_loom/manifests.py b/dbt_loom/manifests.py
index 0aa9e0f..0ff8fcd 100644
--- a/dbt_loom/manifests.py
+++ b/dbt_loom/manifests.py
@@ -36,7 +36,7 @@ class DependsOn(BaseModel):
     macros: List[str] = Field(default_factory=list)
 
 
-class ManifestNode(BaseModel):
+class ManifestNode(BaseModel, use_enum_values=True):
     """A basic ManifestNode that can be referenced across projects."""
 
     name: str
@@ -233,8 +233,13 @@ class ManifestLoader:
                 "not have a valid type."
             )
 
-        manifest = self.loading_functions[manifest_reference.type](
-            manifest_reference.config
-        )
+        try:
+            manifest = self.loading_functions[manifest_reference.type](
+                manifest_reference.config
+            )
+        except LoomConfigurationError as e:
+            if getattr(manifest_reference, "optional", False):
+                return None
+            raise
 
         return manifest


Test patch:
diff --git a/test_projects/revenue/dbt_loom.config.yml b/test_projects/revenue/dbt_loom.config.yml
new file mode 100644
index 0000000..8e0a673
--- /dev/null
+++ b/test_projects/revenue/dbt_loom.config.yml
@@ -0,0 +1,8 @@
+manifests:
+  - name: potato
+    type: file
+    optional: true
+    config:
+      path: ../customer_success/target/manifest.json
+    excluded_packages:
+      - dbt_project_evaluator
diff --git a/tests/test_manifest_loaders.py b/tests/test_manifest_loaders.py
index 88e8547..9ace7c1 100644
--- a/tests/test_manifest_loaders.py
+++ b/tests/test_manifest_loaders.py
@@ -9,6 +9,7 @@ from dbt_loom.config import (
     FileReferenceConfig,
     ManifestReference,
     ManifestReferenceType,
+    LoomConfigurationError,
 )
 from dbt_loom.manifests import ManifestLoader, UnknownManifestPathType
 
@@ -97,3 +98,35 @@ def test_manifest_loader_selection(example_file):
     manifest = manifest_loader.load(manifest_reference)
 
     assert manifest == example_content
+
+
+def test_load_from_local_filesystem_optional_missing():
+    """If the manifest file does not exist, it should not raise an error if optional=True."""
+    file_config = FileReferenceConfig(
+        path="not_exist_manifest.json"
+    )
+    manifest_reference = ManifestReference(
+        name="missing",
+        type=ManifestReferenceType.file,
+        config=file_config,
+        optional=True,
+    )
+    manifest_loader = ManifestLoader()
+    manifest = manifest_loader.load(manifest_reference)
+    assert manifest is None
+
+
+def test_load_from_local_filesystem_not_optional_missing():
+    """If the manifest file does not exist, it should raise an error if optional=False."""
+    file_config = FileReferenceConfig(
+        path="not_exist_manifest.json"
+    )
+    manifest_reference = ManifestReference(
+        name="missing",
+        type=ManifestReferenceType.file,
+        config=file_config,
+        optional=False,
+    )
+    manifest_loader = ManifestLoader()
+    with pytest.raises(LoomConfigurationError):
+        manifest_loader.load(manifest_reference)