Repository: dbt-labs/dbt-core

Issue:
[PP] IndexError in update_mssat_in_saved
I experienced this error after switching branches. The branch I switched to had three additional models and one additional analysis.

```
2021-07-16 18:34:23.889246 (MainThread): Partial parsing enabled: 0 files deleted, 4 files added, 6 files changed.
2021-07-16 18:34:23.893682 (MainThread): Partial parsing: added file: fishtown_internal_analytics://models/marts/dbt/dbt_experimental_parser_samples.sql
2021-07-16 18:34:23.893852 (MainThread): Partial parsing: added file: fishtown_internal_analytics://models/marts/dbt/dbt_experimental_parser_misses.sql
2021-07-16 18:34:23.893953 (MainThread): Partial parsing: added file: fishtown_internal_analytics://analysis/experimental_parser_missed_files.sql
2021-07-16 18:34:23.894057 (MainThread): Partial parsing: added file: fishtown_internal_analytics://models/staging/snowplow/raw/current/dbt/current_snowplow_experimental_parser.sql
2021-07-16 18:34:23.894222 (MainThread): Partial parsing: update schema file: fishtown_internal_analytics://models/staging/snowplow/schema.yml
2021-07-16 18:34:23.894419 (MainThread): Partial parsing: update schema file: fishtown_internal_analytics://models/marts/dbt/schema.yml
2021-07-16 18:34:23.894523 (MainThread): Partial parsing: updated file: fishtown_internal_analytics://models/staging/snowplow/raw/current/dbt/current_snowplow_load_all_timing.sql
2021-07-16 18:34:23.894736 (MainThread): Partial parsing: deleted exposure exposure.fishtown_internal_analytics.dbt_minor_version_adoption
2021-07-16 18:34:23.894890 (MainThread): Partial parsing: deleted exposure exposure.fishtown_internal_analytics.metrics_dashboard
2021-07-16 18:34:23.895036 (MainThread): Partial parsing: deleted exposure exposure.fishtown_internal_analytics.weekly_dbt_metrics
2021-07-16 18:34:23.895196 (MainThread): Partial parsing: updated file: fishtown_internal_analytics://models/marts/dbt/docs.md
2021-07-16 18:34:23.895365 (MainThread): Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10e4a4340>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10e4a4310>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0x10e4a42b0>]}
2021-07-16 18:34:23.895583 (MainThread): Flushing usage events
2021-07-16 18:34:24.018905 (MainThread): Encountered an error:
2021-07-16 18:34:24.019179 (MainThread): list index out of range
2021-07-16 18:34:24.025747 (MainThread): Traceback (most recent call last):
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 125, in main
    results, succeeded = handle_and_check(args)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 203, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/main.py", line 256, in run_from_args
    results = task.run()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 406, in run
    self._runtime_initialize()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 122, in _runtime_initialize
    super()._runtime_initialize()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 79, in _runtime_initialize
    self.load_manifest()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/task/runnable.py", line 66, in load_manifest
    self.manifest = ManifestLoader.get_full_manifest(self.config)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/manifest.py", line 170, in get_full_manifest
    manifest = loader.load()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/manifest.py", line 212, in load
    project_parser_files = partial_parsing.get_parsing_files()
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 131, in get_parsing_files
    self.update_in_saved(file_id)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 199, in update_in_saved
    self.update_mssat_in_saved(new_source_file, old_source_file)
  File "/usr/local/Cellar/dbt/0.20.0_1/libexec/lib/python3.8/site-packages/dbt/parser/partial.py", line 215, in update_mssat_in_saved
    unique_id = old_source_file.nodes[0]
IndexError: list index out of range
```


Discussion:
Another report from a dbt Cloud IDE user. We don't have the exact stacktrace, so I'm not positive it's the same method, but we do have debug-level partial parsing logging. I've scrubbed the names of the projects and models being added/updated.
```
2021-07-19T17:48:15.549005Z: Partial parsing enabled: 19 files deleted, 0 files added, 29 files changed.
2021-07-19T17:48:15.556639Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.556989Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.557126Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.557334Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.557512Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.557664Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.557902Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.558058Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.558231Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.558335Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.558658Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.558786Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.558961Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.559061Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.559159Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.559633Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.559760Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.559875Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.559985Z: Partial parsing: deleted file: *****://models/*****.sql
2021-07-19T17:48:15.560139Z: Partial parsing: updated file: *****://models/*****.sql
2021-07-19T17:48:15.560281Z: Partial parsing: updated file: *****://models/*****.sql
2021-07-19T17:48:15.560404Z: Partial parsing: updated file: *****://models/*****.sql
list index out of range
```
I am experiencing this also, immediately after switching branches.

For anyone who gets this, a `dbt --no-partial-parse compile` fixes it for me (after that you can go back to partial parsing without issue.
Right on. You can also delete `target/partial_parse.msgpack` to trigger a full re-parse. That's especially useful when submitting commands to an RPC server that already has partial parsing turned on (e.g. in the dbt Cloud IDE).
This issue seems to recur after it's been deleted once (as it gets recreated). A user reached out with their experience:

- I had the compilation error I couldn't shake
- Deleted partial_parse, error cleared
- Deleted one of the snapshot .sql files I had created while trying to clear the error
- partial_parse came back and so did the error
- Deleted partial_parse again, error cleared
- Ran dbt snapshot, partial parse came back again, no error this time (probably because I didn't delete or rename anything)
Also having this error on new branch, removing the target directory (which contains the `partial_parse.msgpack` as mentioned above) fixes it.

Resolving patch:
diff --git a/core/dbt/parser/partial.py b/core/dbt/parser/partial.py
--- a/core/dbt/parser/partial.py
+++ b/core/dbt/parser/partial.py
@@ -147,6 +147,18 @@ def add_to_pp_files(self, source_file):
                 file_id not in self.file_diff['deleted']):
             self.project_parser_files[project_name][parser_name].append(file_id)
 
+    def already_scheduled_for_parsing(self, source_file):
+        file_id = source_file.file_id
+        project_name = source_file.project_name
+        if project_name not in self.project_parser_files:
+            return False
+        parser_name = parse_file_type_to_parser[source_file.parse_file_type]
+        if parser_name not in self.project_parser_files[project_name]:
+            return False
+        if file_id not in self.project_parser_files[project_name][parser_name]:
+            return False
+        return True
+
     # Add new files, including schema files
     def add_to_saved(self, file_id):
         # add file object to saved manifest.files
@@ -211,6 +223,9 @@ def update_in_saved(self, file_id):
     # Updated schema files should have been processed already.
     def update_mssat_in_saved(self, new_source_file, old_source_file):
 
+        if self.already_scheduled_for_parsing(old_source_file):
+            return
+
         # These files only have one node.
         unique_id = old_source_file.nodes[0]
 
@@ -251,12 +266,16 @@ def remove_node_in_saved(self, source_file, unique_id):
                         schema_file.node_patches.remove(unique_id)
 
     def update_macro_in_saved(self, new_source_file, old_source_file):
+        if self.already_scheduled_for_parsing(old_source_file):
+            return
         self.handle_macro_file_links(old_source_file, follow_references=True)
         file_id = new_source_file.file_id
         self.saved_files[file_id] = new_source_file
         self.add_to_pp_files(new_source_file)
 
     def update_doc_in_saved(self, new_source_file, old_source_file):
+        if self.already_scheduled_for_parsing(old_source_file):
+            return
         self.delete_doc_node(old_source_file)
         self.saved_files[new_source_file.file_id] = new_source_file
         self.add_to_pp_files(new_source_file)


Test patch:
diff --git a/test/integration/068_partial_parsing_tests/extra-files/model_four1.sql b/test/integration/068_partial_parsing_tests/extra-files/model_four1.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/068_partial_parsing_tests/extra-files/model_four1.sql
@@ -0,0 +1 @@
+select * from {{ ref('model_three') }}
diff --git a/test/integration/068_partial_parsing_tests/extra-files/model_four2.sql b/test/integration/068_partial_parsing_tests/extra-files/model_four2.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/068_partial_parsing_tests/extra-files/model_four2.sql
@@ -0,0 +1 @@
+select fun from {{ ref('model_one') }}
diff --git a/test/integration/068_partial_parsing_tests/test_partial_parsing.py b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
--- a/test/integration/068_partial_parsing_tests/test_partial_parsing.py
+++ b/test/integration/068_partial_parsing_tests/test_partial_parsing.py
@@ -59,7 +59,8 @@ def test_postgres_pp_models(self):
         results = self.run_dbt(["--partial-parse", "test"], expect_pass=False)
         self.assertEqual(len(results), 1)
         manifest = get_manifest()
-        self.assertEqual(len(manifest.files), 33)
+        project_files = [f for f in manifest.files if f.startswith('test://')]
+        self.assertEqual(len(project_files), 4)
         model_3_file_id = 'test://' + normalize('models-a/model_three.sql')
         self.assertIn(model_3_file_id, manifest.files)
         model_three_file = manifest.files[model_3_file_id]
@@ -104,7 +105,8 @@ def test_postgres_pp_models(self):
         manifest = get_manifest()
         schema_file_id = 'test://' + normalize('models-a/schema.yml')
         self.assertNotIn(schema_file_id, manifest.files)
-        self.assertEqual(len(manifest.files), 32)
+        project_files = [f for f in manifest.files if f.startswith('test://')]
+        self.assertEqual(len(project_files), 3)
 
         # Put schema file back and remove a model
         # referred to in schema file
@@ -118,12 +120,31 @@ def test_postgres_pp_models(self):
         results = self.run_dbt(["--partial-parse", "run"])
         self.assertEqual(len(results), 3)
 
+        # Add model four refing model three
+        shutil.copyfile('extra-files/model_four1.sql', 'models-a/model_four.sql')
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 4)
+
+        # Remove model_three and change model_four to ref model_one
+        # and change schema file to remove model_three
+        os.remove(normalize('models-a/model_three.sql'))
+        shutil.copyfile('extra-files/model_four2.sql', 'models-a/model_four.sql')
+        shutil.copyfile('extra-files/models-schema1.yml', 'models-a/schema.yml')
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 3)
+
+        # Remove model four, put back model three, put back schema file
+        shutil.copyfile('extra-files/model_three.sql', 'models-a/model_three.sql')
+        shutil.copyfile('extra-files/models-schema2.yml', 'models-a/schema.yml')
+        os.remove(normalize('models-a/model_four.sql'))
+        results = self.run_dbt(["--partial-parse", "run"])
+        self.assertEqual(len(results), 3)
+
         # Add a macro
         shutil.copyfile('extra-files/my_macro.sql', 'macros/my_macro.sql')
         results = self.run_dbt(["--partial-parse", "run"])
         self.assertEqual(len(results), 3)
         manifest = get_manifest()
-        self.assertEqual(len(manifest.macros), 148)
         macro_id = 'macro.test.do_something'
         self.assertIn(macro_id, manifest.macros)
 
@@ -152,6 +173,8 @@ def tearDown(self):
             os.remove(normalize('models-a/model_two.sql'))
         if os.path.exists(normalize('models-a/model_three.sql')):
             os.remove(normalize('models-a/model_three.sql'))
+        if os.path.exists(normalize('models-a/model_four.sql')):
+            os.remove(normalize('models-a/model_four.sql'))
         if os.path.exists(normalize('models-a/schema.yml')):
             os.remove(normalize('models-a/schema.yml'))
         if os.path.exists(normalize('target/partial_parse.msgpack')):