Repository: dbt-labs/dbt-core

Issue:
[CT-3145] [Bug] partial parsing issue when adding private models to a new group
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

[loom explanation](https://www.loom.com/share/9c61cf7d28ff41348ca7a4bba98ac6f1)

adding multiple nodes that have a relationships to the same group with private access results in a parsing error:

```
Parsing Error
  Node model.mega_corp_big_co_inc.int_returns__unioned attempted to reference node model.mega_corp_big_co_inc.stg_tpcds_core__catalog_returns, which is not allowed because the referenced node is private to the 'sales_analytics' group.
```

in this case, `int_returns__unioned` is a child of `stg_tpcds_core__catalog_returns`, both were added to a newly-created group called `sales_analytics` and had access set to `private`, all of which happened between invocations of dbt. 

### Expected Behavior

everything should parse just fine! these are both private members of the same group and we love that

### Steps To Reproduce

see loom!

### Relevant log output

_No response_

### Environment

```markdown
- OS: mac
- Python: Python 3.9.16
- dbt: 1.6.0 and 1.6.3 (see loom)
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

i love dbt


Discussion:
### Reprex (using the Loom as inspiration)

`models/my_model_a`
```sql
select 1 as id
```

`models/my_model_b`
```sql
select 2 as id
```

`models/my_model_c`
```sql
select * from {{ ref("my_model_a") }} union all
select * from {{ ref("my_model_b") }}
```

`models/_models.yml`
```yaml
models:
  - name: my_model_a
  - name: my_model_b
  - name: my_model_c
```

Clean out any previous artifacts and compile in that fresh environment (everything should work fine):
```shell
dbt clean
dbt compile
```

Now update your `models/_models.yml` file like this:
```yaml
groups:
  - name: sales_analytics
    owner:
      name: Sales Analytics
      email: sales@jaffleshop.com

models:
  - name: my_model_a
    access: private
    group: sales_analytics
  - name: my_model_b
    access: private
    group: sales_analytics
  - name: my_model_c
    access: private
    group: sales_analytics
```

Doing this will yield an error:
```shell
dbt compile
```

But doing this instead would have not yielded an error:
```
dbt --no-partial-parse compile
```

Either way, re-running this command a 2nd time worked for me:
```
dbt compile
```

Resolving patch:
diff --git a/core/dbt/parser/partial.py b/core/dbt/parser/partial.py
--- a/core/dbt/parser/partial.py
+++ b/core/dbt/parser/partial.py
@@ -338,7 +338,10 @@ def remove_node_in_saved(self, source_file, unique_id):
             file_id = node.patch_path
             # it might be changed...  then what?
             if file_id not in self.file_diff["deleted"] and file_id in self.saved_files:
-                # schema_files should already be updated
+                # Schema files should already be updated if this comes from a node,
+                # but this code is also called when updating groups and exposures.
+                # This might save the old schema file element, so when the schema file
+                # is processed, it should overwrite it by passing True to "merge_patch"
                 schema_file = self.saved_files[file_id]
                 dict_key = parse_file_type_to_key[source_file.parse_file_type]
                 # look for a matching list dictionary
@@ -615,13 +618,13 @@ def handle_schema_file_changes(self, schema_file, saved_yaml_dict, new_yaml_dict
             if key_diff["changed"]:
                 for elem in key_diff["changed"]:
                     self.delete_schema_mssa_links(schema_file, dict_key, elem)
-                    self.merge_patch(schema_file, dict_key, elem)
+                    self.merge_patch(schema_file, dict_key, elem, True)
             if key_diff["deleted"]:
                 for elem in key_diff["deleted"]:
                     self.delete_schema_mssa_links(schema_file, dict_key, elem)
             if key_diff["added"]:
                 for elem in key_diff["added"]:
-                    self.merge_patch(schema_file, dict_key, elem)
+                    self.merge_patch(schema_file, dict_key, elem, True)
             # Handle schema file updates due to env_var changes
             if dict_key in env_var_changes and dict_key in new_yaml_dict:
                 for name in env_var_changes[dict_key]:
@@ -630,7 +633,7 @@ def handle_schema_file_changes(self, schema_file, saved_yaml_dict, new_yaml_dict
                     elem = self.get_schema_element(new_yaml_dict[dict_key], name)
                     if elem:
                         self.delete_schema_mssa_links(schema_file, dict_key, elem)
-                        self.merge_patch(schema_file, dict_key, elem)
+                        self.merge_patch(schema_file, dict_key, elem, True)
 
         # sources
         dict_key = "sources"
@@ -640,7 +643,7 @@ def handle_schema_file_changes(self, schema_file, saved_yaml_dict, new_yaml_dict
                 if "overrides" in source:  # This is a source patch; need to re-parse orig source
                     self.remove_source_override_target(source)
                 self.delete_schema_source(schema_file, source)
-                self.merge_patch(schema_file, dict_key, source)
+                self.merge_patch(schema_file, dict_key, source, True)
         if source_diff["deleted"]:
             for source in source_diff["deleted"]:
                 if "overrides" in source:  # This is a source patch; need to re-parse orig source
@@ -650,7 +653,7 @@ def handle_schema_file_changes(self, schema_file, saved_yaml_dict, new_yaml_dict
             for source in source_diff["added"]:
                 if "overrides" in source:  # This is a source patch; need to re-parse orig source
                     self.remove_source_override_target(source)
-                self.merge_patch(schema_file, dict_key, source)
+                self.merge_patch(schema_file, dict_key, source, True)
         # Handle schema file updates due to env_var changes
         if dict_key in env_var_changes and dict_key in new_yaml_dict:
             for name in env_var_changes[dict_key]:
@@ -661,7 +664,7 @@ def handle_schema_file_changes(self, schema_file, saved_yaml_dict, new_yaml_dict
                     if "overrides" in source:
                         self.remove_source_override_target(source)
                     self.delete_schema_source(schema_file, source)
-                    self.merge_patch(schema_file, dict_key, source)
+                    self.merge_patch(schema_file, dict_key, source, True)
 
         def handle_change(key: str, delete: Callable):
             self._handle_element_change(
@@ -681,13 +684,13 @@ def _handle_element_change(
         if element_diff["changed"]:
             for element in element_diff["changed"]:
                 delete(schema_file, element)
-                self.merge_patch(schema_file, dict_key, element)
+                self.merge_patch(schema_file, dict_key, element, True)
         if element_diff["deleted"]:
             for element in element_diff["deleted"]:
                 delete(schema_file, element)
         if element_diff["added"]:
             for element in element_diff["added"]:
-                self.merge_patch(schema_file, dict_key, element)
+                self.merge_patch(schema_file, dict_key, element, True)
         # Handle schema file updates due to env_var changes
         if dict_key in env_var_changes and dict_key in new_yaml_dict:
             for name in env_var_changes[dict_key]:
@@ -696,7 +699,7 @@ def _handle_element_change(
                 elem = self.get_schema_element(new_yaml_dict[dict_key], name)
                 if elem:
                     delete(schema_file, elem)
-                    self.merge_patch(schema_file, dict_key, elem)
+                    self.merge_patch(schema_file, dict_key, elem, True)
 
     # Take a "section" of the schema file yaml dictionary from saved and new schema files
     # and determine which parts have changed
@@ -739,8 +742,10 @@ def get_diff_for(self, key, saved_yaml_dict, new_yaml_dict):
         }
         return diff
 
-    # Merge a patch file into the pp_dict in a schema file
-    def merge_patch(self, schema_file, key, patch):
+    # Merge a patch file into the pp_dict in a schema file. The "new_patch"
+    # flag indicates that we're processing a schema file, so if a matching
+    # patch has already been scheduled, replace it.
+    def merge_patch(self, schema_file, key, patch, new_patch=False):
         if schema_file.pp_dict is None:
             schema_file.pp_dict = {}
         pp_dict = schema_file.pp_dict
@@ -748,12 +753,17 @@ def merge_patch(self, schema_file, key, patch):
             pp_dict[key] = [patch]
         else:
             # check that this patch hasn't already been saved
-            found = False
+            found_elem = None
             for elem in pp_dict[key]:
                 if elem["name"] == patch["name"]:
-                    found = True
-            if not found:
+                    found_elem = elem
+            if not found_elem:
                 pp_dict[key].append(patch)
+            elif found_elem and new_patch:
+                # remove patch and replace with new one
+                pp_dict[key].remove(found_elem)
+                pp_dict[key].append(patch)
+
         schema_file.delete_from_env_vars(key, patch["name"])
         self.add_to_pp_files(schema_file)
 


Test patch:
diff --git a/tests/functional/partial_parsing/test_partial_parsing.py b/tests/functional/partial_parsing/test_partial_parsing.py
--- a/tests/functional/partial_parsing/test_partial_parsing.py
+++ b/tests/functional/partial_parsing/test_partial_parsing.py
@@ -54,22 +54,15 @@
     gsm_override_sql,
     gsm_override2_sql,
     orders_sql,
-    orders_downstream_sql,
     snapshot_sql,
     snapshot2_sql,
     generic_schema_yml,
     generic_test_sql,
     generic_test_schema_yml,
     generic_test_edited_sql,
-    groups_schema_yml_one_group,
-    groups_schema_yml_two_groups,
-    groups_schema_yml_two_groups_edited,
-    groups_schema_yml_one_group_model_in_group2,
-    groups_schema_yml_two_groups_private_orders_valid_access,
-    groups_schema_yml_two_groups_private_orders_invalid_access,
 )
 
-from dbt.exceptions import CompilationError, ParsingError
+from dbt.exceptions import CompilationError
 from dbt.contracts.files import ParseFileType
 from dbt.contracts.results import TestStatus
 from dbt.plugins.manifest import PluginNodes, ModelNodeArgs
@@ -653,93 +646,6 @@ def test_pp_generic_tests(self, project):
         assert expected_nodes == list(manifest.nodes.keys())
 
 
-class TestGroups:
-    @pytest.fixture(scope="class")
-    def models(self):
-        return {
-            "orders.sql": orders_sql,
-            "orders_downstream.sql": orders_downstream_sql,
-            "schema.yml": groups_schema_yml_one_group,
-        }
-
-    def test_pp_groups(self, project):
-
-        # initial run
-        results = run_dbt()
-        assert len(results) == 2
-        manifest = get_manifest(project.project_root)
-        expected_nodes = ["model.test.orders", "model.test.orders_downstream"]
-        expected_groups = ["group.test.test_group"]
-        assert expected_nodes == sorted(list(manifest.nodes.keys()))
-        assert expected_groups == sorted(list(manifest.groups.keys()))
-
-        # add group to schema
-        write_file(groups_schema_yml_two_groups, project.project_root, "models", "schema.yml")
-        results = run_dbt(["--partial-parse", "run"])
-        assert len(results) == 2
-        manifest = get_manifest(project.project_root)
-        expected_nodes = ["model.test.orders", "model.test.orders_downstream"]
-        expected_groups = ["group.test.test_group", "group.test.test_group2"]
-        assert expected_nodes == sorted(list(manifest.nodes.keys()))
-        assert expected_groups == sorted(list(manifest.groups.keys()))
-
-        # edit group in schema
-        write_file(
-            groups_schema_yml_two_groups_edited, project.project_root, "models", "schema.yml"
-        )
-        results = run_dbt(["--partial-parse", "run"])
-        assert len(results) == 2
-        manifest = get_manifest(project.project_root)
-        expected_nodes = ["model.test.orders", "model.test.orders_downstream"]
-        expected_groups = ["group.test.test_group", "group.test.test_group2_edited"]
-        assert expected_nodes == sorted(list(manifest.nodes.keys()))
-        assert expected_groups == sorted(list(manifest.groups.keys()))
-
-        # delete group in schema
-        write_file(groups_schema_yml_one_group, project.project_root, "models", "schema.yml")
-        results = run_dbt(["--partial-parse", "run"])
-        assert len(results) == 2
-        manifest = get_manifest(project.project_root)
-        expected_nodes = ["model.test.orders", "model.test.orders_downstream"]
-        expected_groups = ["group.test.test_group"]
-        assert expected_nodes == sorted(list(manifest.nodes.keys()))
-        assert expected_groups == sorted(list(manifest.groups.keys()))
-
-        # add back second group
-        write_file(groups_schema_yml_two_groups, project.project_root, "models", "schema.yml")
-        results = run_dbt(["--partial-parse", "run"])
-        assert len(results) == 2
-
-        # remove second group with model still configured to second group
-        write_file(
-            groups_schema_yml_one_group_model_in_group2,
-            project.project_root,
-            "models",
-            "schema.yml",
-        )
-        with pytest.raises(ParsingError):
-            results = run_dbt(["--partial-parse", "run"])
-
-        # add back second group, make orders private with valid ref
-        write_file(
-            groups_schema_yml_two_groups_private_orders_valid_access,
-            project.project_root,
-            "models",
-            "schema.yml",
-        )
-        results = run_dbt(["--partial-parse", "run"])
-        assert len(results) == 2
-
-        write_file(
-            groups_schema_yml_two_groups_private_orders_invalid_access,
-            project.project_root,
-            "models",
-            "schema.yml",
-        )
-        with pytest.raises(ParsingError):
-            results = run_dbt(["--partial-parse", "run"])
-
-
 class TestExternalModels:
     @pytest.fixture(scope="class")
     def external_model_node(self):
diff --git a/tests/functional/partial_parsing/test_pp_groups.py b/tests/functional/partial_parsing/test_pp_groups.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/partial_parsing/test_pp_groups.py
@@ -0,0 +1,157 @@
+import pytest
+from dbt.tests.util import run_dbt, get_manifest, write_file
+
+from dbt.exceptions import ParsingError
+
+from tests.functional.partial_parsing.fixtures import (
+    orders_sql,
+    orders_downstream_sql,
+    groups_schema_yml_one_group,
+    groups_schema_yml_two_groups,
+    groups_schema_yml_two_groups_edited,
+    groups_schema_yml_one_group_model_in_group2,
+    groups_schema_yml_two_groups_private_orders_valid_access,
+    groups_schema_yml_two_groups_private_orders_invalid_access,
+)
+
+
+class TestGroups:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "orders.sql": orders_sql,
+            "orders_downstream.sql": orders_downstream_sql,
+            "schema.yml": groups_schema_yml_one_group,
+        }
+
+    def test_pp_groups(self, project):
+
+        # initial run
+        results = run_dbt()
+        assert len(results) == 2
+        manifest = get_manifest(project.project_root)
+        expected_nodes = ["model.test.orders", "model.test.orders_downstream"]
+        expected_groups = ["group.test.test_group"]
+        assert expected_nodes == sorted(list(manifest.nodes.keys()))
+        assert expected_groups == sorted(list(manifest.groups.keys()))
+
+        # add group to schema
+        write_file(groups_schema_yml_two_groups, project.project_root, "models", "schema.yml")
+        results = run_dbt(["--partial-parse", "run"])
+        assert len(results) == 2
+        manifest = get_manifest(project.project_root)
+        expected_nodes = ["model.test.orders", "model.test.orders_downstream"]
+        expected_groups = ["group.test.test_group", "group.test.test_group2"]
+        assert expected_nodes == sorted(list(manifest.nodes.keys()))
+        assert expected_groups == sorted(list(manifest.groups.keys()))
+
+        # edit group in schema
+        write_file(
+            groups_schema_yml_two_groups_edited, project.project_root, "models", "schema.yml"
+        )
+        results = run_dbt(["--partial-parse", "run"])
+        assert len(results) == 2
+        manifest = get_manifest(project.project_root)
+        expected_nodes = ["model.test.orders", "model.test.orders_downstream"]
+        expected_groups = ["group.test.test_group", "group.test.test_group2_edited"]
+        assert expected_nodes == sorted(list(manifest.nodes.keys()))
+        assert expected_groups == sorted(list(manifest.groups.keys()))
+
+        # delete group in schema
+        write_file(groups_schema_yml_one_group, project.project_root, "models", "schema.yml")
+        results = run_dbt(["--partial-parse", "run"])
+        assert len(results) == 2
+        manifest = get_manifest(project.project_root)
+        expected_nodes = ["model.test.orders", "model.test.orders_downstream"]
+        expected_groups = ["group.test.test_group"]
+        assert expected_nodes == sorted(list(manifest.nodes.keys()))
+        assert expected_groups == sorted(list(manifest.groups.keys()))
+
+        # add back second group
+        write_file(groups_schema_yml_two_groups, project.project_root, "models", "schema.yml")
+        results = run_dbt(["--partial-parse", "run"])
+        assert len(results) == 2
+
+        # remove second group with model still configured to second group
+        write_file(
+            groups_schema_yml_one_group_model_in_group2,
+            project.project_root,
+            "models",
+            "schema.yml",
+        )
+        with pytest.raises(ParsingError):
+            results = run_dbt(["--partial-parse", "run"])
+
+        # add back second group, make orders private with valid ref
+        write_file(
+            groups_schema_yml_two_groups_private_orders_valid_access,
+            project.project_root,
+            "models",
+            "schema.yml",
+        )
+        results = run_dbt(["--partial-parse", "run"])
+        assert len(results) == 2
+
+        write_file(
+            groups_schema_yml_two_groups_private_orders_invalid_access,
+            project.project_root,
+            "models",
+            "schema.yml",
+        )
+        with pytest.raises(ParsingError):
+            results = run_dbt(["--partial-parse", "run"])
+
+
+my_model_c = """
+select * from {{ ref("my_model_a") }} union all
+select * from {{ ref("my_model_b") }}
+"""
+
+models_yml = """
+models:
+  - name: my_model_a
+  - name: my_model_b
+  - name: my_model_c
+"""
+
+models_and_groups_yml = """
+groups:
+  - name: sales_analytics
+    owner:
+      name: Sales Analytics
+      email: sales@jaffleshop.com
+
+models:
+  - name: my_model_a
+    access: private
+    group: sales_analytics
+  - name: my_model_b
+    access: private
+    group: sales_analytics
+  - name: my_model_c
+    access: private
+    group: sales_analytics
+"""
+
+
+class TestAddingModelsToNewGroups:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {
+            "my_model_a.sql": "select 1 as id",
+            "my_model_b.sql": "select 2 as id",
+            "my_model_c.sql": my_model_c,
+            "models.yml": models_yml,
+        }
+
+    def test_adding_models_to_new_groups(self, project):
+        run_dbt(["compile"])
+        # This tests that the correct patch is added to my_model_c. The bug
+        # was that it was using the old patch, so model_c didn't have the
+        # correct group and access.
+        write_file(models_and_groups_yml, project.project_root, "models", "models.yml")
+        run_dbt(["compile"])
+        manifest = get_manifest(project.project_root)
+        model_c_node = manifest.nodes["model.test.my_model_c"]
+        assert model_c_node.group == "sales_analytics"
+        assert model_c_node.access == "private"