Repository: dbt-labs/dbt-core

Issue:
[PP] delete_schema_macro_patch: local variable 'macro_unique_id' referenced before assignment
Bug crops up in this line, if the condition in the previous for loop is never met:
https://github.com/dbt-labs/dbt/blob/72722635f275666b34efc9baf8485be79dfcaa88/core/dbt/parser/partial.py#L643

```
2021-07-26 13:04:12.526071 (MainThread): Running with dbt=0.20.0
2021-07-26 13:04:12.722541 (MainThread): running dbt with arguments Namespace(cls=<class 'dbt.task.test.TestTask'>, data=False, debug=False, defer=None, exclude=None, fail_fast=False, log_cache_events=False, log_format='default', models=None, partial_parse=None, profile=None, profiles_dir='/Users/jerco/.dbt', project_dir=None, record_timing_info=None, rpc_method='test', schema=False, selector_name=None, single_threaded=False, state=None, store_failures=True, strict=False, target=None, test_new_parser=False, threads=None, use_cache=True, use_colors=None, use_experimental_parser=False, vars='{}', version_check=True, warn_error=False, which='test', write_json=True)
2021-07-26 13:04:12.724824 (MainThread): Tracking: tracking
2021-07-26 13:04:12.753859 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x1088752e0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108b2cf10>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108b4eac0>]}
2021-07-26 13:04:12.841350 (MainThread): Partial parsing enabled: 2 files deleted, 1 files added, 0 files changed.
2021-07-26 13:04:12.841663 (MainThread): Partial parsing: added file: my_project://models/sources.yml
2021-07-26 13:04:12.841937 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108c093a0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108c09430>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x108c093d0>]}
2021-07-26 13:04:12.842190 (MainThread): Flushing usage events
2021-07-26 13:04:13.126158 (MainThread): Encountered an error:
2021-07-26 13:04:13.126434 (MainThread): local variable 'macro_unique_id' referenced before assignment
2021-07-26 13:04:13.131442 (MainThread): Traceback (most recent call last):
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 256, in run_from_args
    results = task.run()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 406, in run
    self._runtime_initialize()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 122, in _runtime_initialize
    super()._runtime_initialize()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 79, in _runtime_initialize
    self.load_manifest()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 66, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/manifest.py", line 170, in get_full_manifest
    manifest = loader.load()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/manifest.py", line 212, in load
    project_parser_files = partial_parsing.get_parsing_files()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 127, in get_parsing_files
    self.delete_schema_file(file_id)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 447, in delete_schema_file
    self.handle_schema_file_changes(saved_schema_file, saved_yaml_dict, new_yaml_dict)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 499, in handle_schema_file_changes
    self.delete_schema_macro_patch(schema_file, macro)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 643, in delete_schema_macro_patch
    if macro_unique_id and macro_unique_id in self.saved_manifest.macros:
UnboundLocalError: local variable 'macro_unique_id' referenced before assignment
```


Discussion:
I came across this bug this morning. I tried retracing my steps, but I haven't been able to reproduce it. I'm happy closing this one for now, and coming back to it if we see the same error crop up again later.

Resolving patch:
diff --git a/core/dbt/contracts/files.py b/core/dbt/contracts/files.py
--- a/core/dbt/contracts/files.py
+++ b/core/dbt/contracts/files.py
@@ -220,7 +220,7 @@ class SchemaSourceFile(BaseSourceFile):
     # node patches contain models, seeds, snapshots, analyses
     ndp: List[str] = field(default_factory=list)
     # any macro patches in this file by macro unique_id.
-    mcp: List[str] = field(default_factory=list)
+    mcp: Dict[str, str] = field(default_factory=dict)
     # any source patches in this file. The entries are package, name pairs
     # Patches are only against external sources. Sources can be
     # created too, but those are in 'sources'
diff --git a/core/dbt/contracts/graph/manifest.py b/core/dbt/contracts/graph/manifest.py
--- a/core/dbt/contracts/graph/manifest.py
+++ b/core/dbt/contracts/graph/manifest.py
@@ -759,7 +759,7 @@ def add_macro_patch(
         if macro.patch_path:
             package_name, existing_file_path = macro.patch_path.split('://')
             raise_duplicate_macro_patch_name(patch, existing_file_path)
-        source_file.macro_patches.append(unique_id)
+        source_file.macro_patches[patch.name] = unique_id
         macro.patch(patch)
 
     def add_source_patch(
diff --git a/core/dbt/parser/partial.py b/core/dbt/parser/partial.py
--- a/core/dbt/parser/partial.py
+++ b/core/dbt/parser/partial.py
@@ -668,19 +668,17 @@ def delete_schema_source(self, schema_file, source_dict):
 
     def delete_schema_macro_patch(self, schema_file, macro):
         # This is just macro patches that need to be reapplied
-        for unique_id in schema_file.macro_patches:
-            parts = unique_id.split('.')
-            macro_name = parts[-1]
-            if macro_name == macro['name']:
-                macro_unique_id = unique_id
-                break
+        macro_unique_id = None
+        if macro['name'] in schema_file.macro_patches:
+            macro_unique_id = schema_file.macro_patches[macro['name']]
+            del schema_file.macro_patches[macro['name']]
         if macro_unique_id and macro_unique_id in self.saved_manifest.macros:
             macro = self.saved_manifest.macros.pop(macro_unique_id)
             self.deleted_manifest.macros[macro_unique_id] = macro
             macro_file_id = macro.file_id
-            self.add_to_pp_files(self.saved_files[macro_file_id])
-        if macro_unique_id in schema_file.macro_patches:
-            schema_file.macro_patches.remove(macro_unique_id)
+            if macro_file_id in self.new_files:
+                self.saved_files[macro_file_id] = self.new_files[macro_file_id]
+                self.add_to_pp_files(self.saved_files[macro_file_id])
 
     # exposures are created only from schema files, so just delete
     # the exposure.


Test patch:
diff --git a/test/integration/068_partial_parsing_tests/extra-files/macros.yml b/test/integration/068_partial_parsing_tests/extra-files/macros.yml
new file mode 100644
--- /dev/null
+++ b/test/integration/068_partial_parsing_tests/extra-files/macros.yml
@@ -0,0 +1,4 @@
+version: 2
+macros:
+    - name: do_something
+      description: "This is a test macro"
diff --git a/test/integration/068_partial_parsing_tests/test_partial_parsing.py b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
--- a/test/integration/068_partial_parsing_tests/test_partial_parsing.py
+++ b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
@@ -163,8 +163,17 @@ def test_postgres_pp_models(self):
         with self.assertRaises(CompilationException):
             results = self.run_dbt(["--partial-parse", "run"])
 
-        # Remove the macro patch
+        # put back macro file, got back to schema file with no macro
+        # add separate macro patch schema file
         shutil.copyfile('extra-files/models-schema2.yml', 'models-a/schema.yml')
+        shutil.copyfile('extra-files/my_macro.sql', 'macros/my_macro.sql')
+        shutil.copyfile('extra-files/macros.yml', 'macros/macros.yml')
+        results = self.run_dbt(["--partial-parse", "run"])
+
+        # delete macro and schema file
+        print(f"\n\n*** remove macro and macro_patch\n\n")
+        os.remove(normalize('macros/my_macro.sql'))
+        os.remove(normalize('macros/macros.yml'))
         results = self.run_dbt(["--partial-parse", "run"])
         self.assertEqual(len(results), 3)
 
@@ -193,6 +202,8 @@ def tearDown(self):
             os.remove(normalize('macros/my_macro.sql'))
         if os.path.exists(normalize('models-a/eschema.yml')):
             os.remove(normalize('models-a/eschema.yml'))
+        if os.path.exists(normalize('macros/macros.yml')):
+            os.remove(normalize('macros/macros.yml'))
 
 
 class TestSources(DBTIntegrationTest):