Repository: dbt-labs/dbt-core

Issue:
Env vars + partial parsing
If an env var value needed at parse time (input to `ref`/`source`/`config` or yaml property) has changed, and partial parsing is enabled, dbt will not re-parse affected nodes and is likely to yield incorrect results.

Reproduction case: [gist](https://gist.github.com/jtcohen6/8e178b5c96234969ef2d7fb299b572be)

- **Non-solution** (better than nothing): Continue to document this in [known partial parsing limitations](https://docs.getdbt.com/reference/parsing#known-limitations); add to [env var docs](https://docs.getdbt.com/reference/dbt-jinja-functions/env_var) as well. Instruct users to manually trigger a full re-parse (`--no-partial-parse`) when they change an env var value needed at parse time.
- **Simpler solution:** Store a dictionary of all available env vars in the internal manifest (`partial_parse.msgpack`). If any env var values change, trigger a full re-parse, and say that a change in env vars is the reason. Note: I'm not sure how this would work in deployment environments that set contextual env vars, such as dbt Cloud scheduled runs.
- **Complex, good solution:** For each node in the graph, statically analyze calls to the `env_var` macro, and then store those env vars in ~`depends_on` for that node~ a separate map object, similar to how we handle `docs` blocks. Then, if we detect a change in env var values, we can trigger re-parse for only those nodes which depend on that env var, plus (if the node is a macro) any nodes that depend on the macro. (It would be great to do this for `--vars` someday as well.)


Discussion:
@gshank Up to you whether we keep this issue open, or close it and open a new one for env vars in other files (`.yml` files, `dbt_project.yml`, `profiles.yml`)

Resolving patch:
diff --git a/core/dbt/parser/schemas.py b/core/dbt/parser/schemas.py
--- a/core/dbt/parser/schemas.py
+++ b/core/dbt/parser/schemas.py
@@ -57,6 +57,7 @@
     TestBuilder, GenericTestBlock, TargetBlock, YamlBlock,
     TestBlock, Testable
 )
+from dbt.ui import warning_tag
 from dbt.utils import (
     get_pseudo_test_path, coerce_dict_str
 )
@@ -886,11 +887,13 @@ def parse_patch(
                     # re-application of the patch in partial parsing.
                     node.patch_path = source_file.file_id
             else:
-                raise ParsingException(
+                msg = (
                     f"Did not find matching node for patch with name '{patch.name}' "
                     f"in the '{patch.yaml_key}' section of "
                     f"file '{source_file.path.original_file_path}'"
                 )
+                warn_or_error(msg, log_fmt=warning_tag('{}'))
+                return
 
         # patches can't be overwritten
         node = self.manifest.nodes.get(unique_id)


Test patch:
diff --git a/test/integration/068_partial_parsing_tests/test_partial_parsing.py b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
--- a/test/integration/068_partial_parsing_tests/test_partial_parsing.py
+++ b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
@@ -143,7 +143,7 @@ def test_postgres_pp_models(self):
         # referred to in schema file
         self.copy_file('test-files/models-schema2.yml', 'models/schema.yml')
         self.rm_file('models/model_three.sql')
-        with self.assertRaises(ParsingException):
+        with self.assertRaises(CompilationException):
             results = self.run_dbt(["--partial-parse", "--warn-error", "run"])
 
         # Put model back again
@@ -312,7 +312,7 @@ def test_postgres_pp_sources(self):
 
         # Change seed name to wrong name
         self.copy_file('test-files/schema-sources5.yml', 'models/sources.yml')
-        with self.assertRaises(ParsingException):
+        with self.assertRaises(CompilationException):
             results = self.run_dbt(["--partial-parse", "--warn-error", "run"])
 
         # Put back seed name to right name