Repository: dbt-labs/dbt-core

Issue:
[PP] KeyError in remove_tests
This is using dbt v0.20.1-rc1.

I haven't been able to identify root cause, but I get this error when switching from the `main` branch in our internal analytics project to another working branch (`try-partial-parser`) that's 1 commit ahead, 14 commits behind.

Weirdly, the error seems to crop up on a test/model that are completely untouched by the changes in each branch, except that the test is defined elsewhere in a `yml` file that has some minor edits in the `main` branch.

```
2021-08-09 11:52:02.164951 (MainThread): Partial parsing enabled: 1 files deleted, 2 files added, 4 files changed.
2021-08-09 11:52:02.170847 (MainThread): Partial parsing: added file: fishtown_internal_analytics://models/staging/snowplow/raw/current/dbt/current_snowplow_partial_parser.sql
2021-08-09 11:52:02.171018 (MainThread): Partial parsing: added file: fishtown_internal_analytics://models/marts/dbt/dbt_partial_parser.sql
2021-08-09 11:52:02.171302 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10958dfa0>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10958df70>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10958df10>]}
2021-08-09 11:52:02.171526 (MainThread): Flushing usage events
2021-08-09 11:52:06.365964 (MainThread): The 'warn' method is deprecated, use 'warning' instead
2021-08-09 11:52:06.366286 (MainThread): Error sending message, disabling tracking
2021-08-09 11:52:06.366594 (MainThread): Encountered an error:
2021-08-09 11:52:06.366857 (MainThread): 'test.fishtown_internal_analytics.not_null_fct_cloud_accounts_has_successful_session.f841b7dce3'
2021-08-09 11:52:06.427303 (MainThread): Traceback (most recent call last):
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 256, in run_from_args
    results = task.run()
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 406, in run
    self._runtime_initialize()
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 122, in _runtime_initialize
    super()._runtime_initialize()
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 79, in _runtime_initialize
    self.load_manifest()
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 66, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/manifest.py", line 180, in get_full_manifest
    manifest = loader.load()
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/manifest.py", line 222, in load
    project_parser_files = partial_parsing.get_parsing_files()
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 125, in get_parsing_files
    self.change_schema_file(file_id)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 467, in change_schema_file
    self.handle_schema_file_changes(saved_schema_file, saved_yaml_dict, new_yaml_dict)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 513, in handle_schema_file_changes
    self.delete_schema_source(schema_file, source)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 669, in delete_schema_source
    self.schedule_referencing_nodes_for_parsing(unique_id)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 296, in schedule_referencing_nodes_for_parsing
    self.schedule_nodes_for_parsing(self.saved_manifest.child_map[unique_id])
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 308, in schedule_nodes_for_parsing
    self.remove_mssat_file(source_file)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 291, in remove_mssat_file
    self.schedule_referencing_nodes_for_parsing(unique_id)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 296, in schedule_referencing_nodes_for_parsing
    self.schedule_nodes_for_parsing(self.saved_manifest.child_map[unique_id])
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 308, in schedule_nodes_for_parsing
    self.remove_mssat_file(source_file)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 290, in remove_mssat_file
    self.remove_node_in_saved(source_file, unique_id)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 263, in remove_node_in_saved
    self.delete_schema_mssa_links(schema_file, dict_key, elem_patch)
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 646, in delete_schema_mssa_links
    self.remove_tests(schema_file, dict_key, elem['name'])
  File "/usr/local/Cellar/dbt@0.20.1-rc1/0.20.1rc1_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 651, in remove_tests
    node = self.saved_manifest.nodes.pop(test_unique_id)
KeyError: 'test.fishtown_internal_analytics.not_null_fct_cloud_accounts_has_successful_session.f841b7dce3'
```


Discussion:
@gshank Could you take a look at this one when you get a chance? This has only come up once for me, but I was able to reliably reproduce it by switching between the two branches mentioned above.

Resolving patch:
diff --git a/core/dbt/parser/manifest.py b/core/dbt/parser/manifest.py
--- a/core/dbt/parser/manifest.py
+++ b/core/dbt/parser/manifest.py
@@ -231,7 +231,7 @@ def load(self):
                                 "Switching to a full re-parse.")
 
                     # Get traceback info
-                    tb_info = traceback.format_exc(10)
+                    tb_info = traceback.format_exc()
                     formatted_lines = tb_info.splitlines()
                     (_, line, method) = formatted_lines[-3].split(', ')
                     exc_info = {
diff --git a/core/dbt/parser/partial.py b/core/dbt/parser/partial.py
--- a/core/dbt/parser/partial.py
+++ b/core/dbt/parser/partial.py
@@ -654,8 +654,9 @@ def delete_schema_mssa_links(self, schema_file, dict_key, elem):
     def remove_tests(self, schema_file, dict_key, name):
         tests = schema_file.get_tests(dict_key, name)
         for test_unique_id in tests:
-            node = self.saved_manifest.nodes.pop(test_unique_id)
-            self.deleted_manifest.nodes[test_unique_id] = node
+            if test_unique_id in self.saved_manifest.nodes:
+                node = self.saved_manifest.nodes.pop(test_unique_id)
+                self.deleted_manifest.nodes[test_unique_id] = node
         schema_file.remove_tests(dict_key, name)
 
     def delete_schema_source(self, schema_file, source_dict):


Test patch: