Repository: dbt-labs/dbt-core

Issue:
[PP] ValueError in handle_macro_file_links
Not sure if this is already fixed in one of the open PRs. It looked similar to, but not exactly the same as, some of the other open issues.

The case where this cropped up for me is a pretty weird one, but I've been able to reliably reproduce it. In my work for #3510, I was adding a macro named `snowflake__truncate_relation` to `plugins/snowflake/dbt/include/snowflake/macros/adapters.sql`. If I remove that macro, do a full re-parse, then add it back and use partial parsing, I see the error again.

If the `unique_id` is not in `source_file.macros` (potentially because it was removed already), it raises the a `ValueError` when it tries to remove it from the list here:

https://github.com/dbt-labs/dbt/blob/9c58f3465bf9907a2b62942de548f80650cd6288/core/dbt/parser/partial.py#L391

```
2021-07-27 17:38:12.543828 (MainThread): Partial parsing enabled: 0 files deleted, 0 files added, 1 files changed.
2021-07-27 17:38:12.574882 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x116d1fdf0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x116d1fd00>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x116d1fd30>]}
2021-07-27 17:38:12.578412 (MainThread): Flushing usage events
2021-07-27 17:38:12.870885 (MainThread): Encountered an error:
2021-07-27 17:38:12.871082 (MainThread): list.remove(x): x not in list
Traceback (most recent call last):
  File "/Users/jerco/dev/product/dbt/core/dbt/main.py", line 127, in main
    results, succeeded = handle_and_check(args)
  File "/Users/jerco/dev/product/dbt/core/dbt/main.py", line 205, in handle_and_check
    task, res = run_from_args(parsed)
  File "/Users/jerco/dev/product/dbt/core/dbt/main.py", line 258, in run_from_args
    results = task.run()
  File "/Users/jerco/dev/product/dbt/core/dbt/task/runnable.py", line 408, in run
    self._runtime_initialize()
  File "/Users/jerco/dev/product/dbt/core/dbt/task/runnable.py", line 124, in _runtime_initialize
    super()._runtime_initialize()
  File "/Users/jerco/dev/product/dbt/core/dbt/task/runnable.py", line 81, in _runtime_initialize
    self.load_manifest()
  File "/Users/jerco/dev/product/dbt/core/dbt/task/runnable.py", line 68, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/manifest.py", line 170, in get_full_manifest
    manifest = loader.load()
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/manifest.py", line 212, in load
    project_parser_files = partial_parsing.get_parsing_files()
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/partial.py", line 131, in get_parsing_files
    self.update_in_saved(file_id)
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/partial.py", line 213, in update_in_saved
    self.update_macro_in_saved(new_source_file, old_source_file)
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/partial.py", line 271, in update_macro_in_saved
    self.handle_macro_file_links(old_source_file, follow_references=True)
  File "/Users/jerco/dev/product/dbt/core/dbt/parser/partial.py", line 391, in handle_macro_file_links
    source_file.macros.remove(unique_id)
ValueError: list.remove(x): x not in list
```


Discussion:
Jeremy: This code has changed in a couple of tickets. I don't see from the log how to reproduce it. Could you try with the latest in develop (or 0.20.latest) and see if it still happens?
@gshank I retried this with the latest version of `develop`, and I was able to reproduce the issue. Again, this is a niche one, since it involves adding/removing macros from internal packages (in this case `dbt_snowflake`), so I don't know how likely it is to come up in practice. Users don't tend to edit internal packages, outside of upgrading to a new adapter plugin version. I'm interested in it insofar as it could represent an error that crops up elsewhere.

Steps to reproduce:
1. Install dbt from source (`pip install -r editable-requirements.txt`)
2. Parse a sample project to create `target/partial_parse.msgpack`
3. Add any macro to an internal project (e.g. `plugins/postgres/dbt/include/postgres/macros/adapters.sql`)
4. Re-parse the project with partial parsing enabled:
```
Partial parsing enabled: 0 files deleted, 0 files added, 1 files changed.
Encountered an error:
list.remove(x): x not in list
```
5. Trigger a full re-parse (`dbt --no-partial-parse parse`)
6. Remove the previously added macro
7. Re-parse the project with partial parsing enabled:
```
Partial parsing enabled: 0 files deleted, 0 files added, 1 files changed.
Encountered an error:
list.remove(x): x not in list
```

Resolving patch:
diff --git a/core/dbt/parser/partial.py b/core/dbt/parser/partial.py
--- a/core/dbt/parser/partial.py
+++ b/core/dbt/parser/partial.py
@@ -362,7 +362,8 @@ def handle_macro_file_links(self, source_file, follow_references=False):
         for unique_id in macros:
             if unique_id not in self.saved_manifest.macros:
                 # This happens when a macro has already been removed
-                source_file.macros.remove(unique_id)
+                if unique_id in source_file.macros:
+                    source_file.macros.remove(unique_id)
                 continue
 
             base_macro = self.saved_manifest.macros.pop(unique_id)
@@ -388,7 +389,9 @@ def handle_macro_file_links(self, source_file, follow_references=False):
                     macro_patch = self.get_schema_element(macro_patches, base_macro.name)
                     self.delete_schema_macro_patch(schema_file, macro_patch)
                     self.merge_patch(schema_file, 'macros', macro_patch)
-            source_file.macros.remove(unique_id)
+            # The macro may have already been removed by handling macro children
+            if unique_id in source_file.macros:
+                source_file.macros.remove(unique_id)
 
     # similar to schedule_nodes_for_parsing but doesn't do sources and exposures
     # and handles schema tests


Test patch: