Repository: dbt-labs/dbt-core

Issue:
[CT-646] [Proposal] Streamline Incremental Strategies
When defining an incremental model, there are several incremental strategies that could be used to execute the incremental model, (namely: "append," "delete+insert," "merge," and "insert_overwrite.") but not all of them are supported by every warehouse. Working with these incremental strategies as an adapter maintainer and as an advanced user is difficult. This ticket aims to improve the experience for both of these groups of people.

Functional Requirements
1. Adapter maintainers need to be able to specify which incremental strategies are supported by the warehouse and be able to supply the correct sql statement to execute if the default is not sufficient.
2. Advanced users that wish to specify a custom incremental strategy must be able to do so.

Behavior Today
1. To specify which incremental strategies are supported by the warehouse, the adapter maintainer must write a macro that returns a list of strings that name the supported strategies, and include the jinja macro `adapter__get_merge_sql` that takes a fixed set of parameters and returns the materialization sql. This macro is often copy-pasted from [core](https://github.com/dbt-labs/dbt-core/blob/ea5a9da71ef71cc13414cd8c2f8097448add5edb/core/dbt/include/global_project/macros/materializations/models/incremental/merge.sql#L5-L48), and is error prone when editing.
2. Advanced dbt users who wish to specify a custom incremental strategy must override the same boiler plate jinja macro by copy pasting it into their dbt project.
3. When an unsupported incremental strategy is specified, the user is notified via an exception at during the first `dbt run`.

Desired Behavior
1. To specify which incremental strategies are supported by the warehouse, the adapter maintainer must write a macro that returns a list of strings that name the supported strategies. If the warehouse only supports a subset of the default incremental strategies and the default implementation in the global project produces sql that works on the target warehouse, no additional work needs to be done. If any of the default behavior needs to be overridden, it can be done by writing a macro that conforms to the naming convention "get_incremental_NAME_sql" where "NAME" is the string used to configure the incremental strategy (e.g. "get incremental_append_sql"). If the warehouse has an additional incremental strategy, defining a new macro using the same "get_incremental_NAME_sql" convention will make it available to users of the adapter.
2. Advanced dbt users who wish to specify a custom incremental strategy will only need to create a macro that conforms to the naming convention "get_incremental_NAME_sql" that produces the correct sql for the target warehouse.
3. When an unsupported incremental strategy is designated, raise an error to users before runtime.
4. Existing projects will experience no breaking changes when they upgrade to the version of dbt that includes this new behavior.

Implementation
- [ ] For backwards compatibility, create new macros in `core/dbt/include/global_project/macros/materializations/models/incremental` that follow the naming convention for the existing incremental strategies and pass through to the existing current macros. The new macros will take a dictionary, and pass the key-value pairs explicitly to the wrapped macro call to return the sql string that executes the materialization. Concretely:
    - [ ] create a new macro `get_insert_into_sql() -> str` that returns `"insert into"` 
    - [ ] create a new macro `get_incremental_append_sql(dict)` that wraps `get_insert_into_sql`
    - [ ] create a new macro `get_incremental_delete_insert_sql(dict)` that wraps `get_delete_insert_merge_sql`
    - [ ] create a new macro `get_incremental_merge_sql(dict)` that wraps `get_merge_sql`
    - [ ] create a new macro `get_incremental_insert_overwrite_sql(dict)` that wraps `get_insert_overwrite_sql`
    - [ ] create a new macro `get_incremental_default_sql(dict)` that wraps `get_incremental_append_sql`. This is where adapters can override the default incremental strategy when append isn't appropriate.
    - Note: this dictionary will always pass the same key-value pairs from core. The dictionary value allows core to add new key-value pairs in the future without breaking existing adapters. It is very possible that a custom incremental strategy may require different parameters than the ones core provides. These can be added by users via context-global i.e. `config.get('...')`.
- [ ] Add `incremental_strategy` to `NodeConfig` with a default value of `"default"`.
- [ ] Create a new Python method on `core/dbt/adapters/sql/impl.py::AdapterConfig` called `validate_incremental_strategy` that takes no parameters and returns a list of strings. Adapter maintainers are expected to implement this so that the returned strings name the supported incremental strategies. This could have been a macro, but it's much easier for core to call this as a python function to validate incremental strategies in projects. (TODO what if an adapter supports a different set of incremental strategies for sql and python models? where would they indicate that?)
- [ ] Write a new Python function `get_incremental_strategy_macro` that takes in a string and returns the macro from the context. (e.g. "insert_overwrite" -> callable get_incremental_insert_overwrite_sql). This function will call `validate_incremental_strategy` and throw an exception if the requested strategy is not returned from `validate_incremental_strategy` or if it does not exist in the context. This function will need to be called in the [materialization macro for models](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/include/global_project/macros/materializations/models/incremental/incremental.sql)
- [ ] create a "functional" test for each incremental strategy in `tests/adapter` including tests for several configs for each strategy. This will allow each adapter to import the tests it needs as well as make their own as needed. (Thanks for pointing this requirement out, @dataders)

Docs
This will require a significant change to the documentation on incremental materializations. 


Discussion:
**Update: everything in this comment has been resolved, and incorporated into the main issue**

@nathaniel-may This is really clearly stated, and I think you captured the many nuances well!

> TODO name the macro

We were discussing `validate_incremental_strategy`, and that this may want to be an adapter (Python) method rather than a macro. It would work similar to ["snapshot strategy dispatch"](https://github.com/dbt-labs/dbt-core/blob/0d8e061a3d8d726bb10ff767d8f276439cc472b2/core/dbt/include/global_project/macros/materializations/snapshots/strategies.sql#L1-L32), but we agreed the Jinja here is pretty jank.

> TODO should this actually be a compile check?

I think this may get tricky with partial parsing: We'd need to recalculate any time a new `incremental_strategy` is defined, and any time a macro named `get_incremental_X_sql` is created/deleted. While it's definitely _preferable_ to raise this error before the model's actual runtime, it would be worse to cause a buggy experience.

It's also possible that users completely override the incremental materialization—a thing we want them to need to do less of, but still—in which case, the `validate_incremental_strategy` macro/method may not be used at all.

> TODO CONTINUE THIS LIST

Other things I recall from our conversation:
- The `get_incremental_X_sql` macros should all have the same macro (function) signature, but they may need different sorts of arguments to be passed in. Rather than developing macro signatures of 10+ arguments, and requiring all macros to evolve their signature when any one of them does, we might use a single dictionary argument instead. The materialization would pass this dictionary into the macro, and the macro would return a string (the templated SQL to be run).
- If a user defines their own custom strategy, that requires their own custom config, they can just pull that custom config into `get_incremental_X_sql` via context-global `config.get('...')`. That's not a good habit for _us_, as maintainers, but it's a valuable escape hatch for users, and avoids the need to copy-paste-edit the whole incremental materialization in their projects.
- There are several other complexities with the incremental materialization (full refresh, `on_schema_change`, whether to create a temp table or not), and this change won't solve for all of them. It should enable us to delete a bunch of boilerplate code, though, and clear the way for examining those other discrepancies that exist today.
one quick thought as a requirement. will come back to this later
- [ ] there should be a "functional" test for at least each incremental strategy in `core/tests` (arguably each possible config of each strategy), from which each adapter can import the tests it needs (as well as make their own as needed.
> TODO what is the default implementation?

The lowest-level default implementation should be `append` (as defined in `dbt-core`). That should also be the default when no `unique_key` is specified. For backwards compatibility with current behavior, each adapter should be able to specify its default strategy when a `unique_key` _is_ supplied, namely `delete+insert` on Postgres/Redshift and `merge` on Snowflake/BigQuery/Databricks.
top-level ticket description reflects the previous conversation.

A couple of nits:  I think the name "get_incremental_delete+insert_sql(dict)" would need to use an underscore instead of a plus.  I believe that the adapter tests should go in the adapter zone (tests/adapter) and not in core/tests. I *think* that's what was intended from the wording.
I'm not finding get_insert_into_sql or get_incremental_append_sql.
- Yeah the `+` isn't going to work. I'll change that to an `_`.
- Yeah adapter zone sounds like the right place for these tests I'll change that as well.

- `get_incremental_append_sql` is one of the macros that needs to be created. It just gets wrapped by another new "default" function in core. 
- `get_insert_into_sql` def isn't right though. I bet it was supposed to be `get_merge_sql` where `predicates=None` from [merge.sql](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/include/global_project/macros/materializations/models/incremental/merge.sql). Does that sound right to you?
`get_insert_into_sql` is the cleanest way to represent true "append" behavior, when a `unique_key` isn't specified. This is an attempt to clean up a few different things that exist today:
- [`get_insert_into_sql`](https://github.com/dbt-labs/dbt-spark/blob/bc9fc0baaacce28706d0542391704d9ea971cee4/dbt/include/spark/macros/materializations/incremental/strategies.sql#L12-L19), defined in `dbt-spark` only. This is explicit "append" behavior on Spark today.
- [`get_delete_insert_merge_sql`](https://github.com/dbt-labs/dbt-core/blob/eea872c31994c90e995405c74496c1ce20e07025/core/dbt/include/global_project/macros/materializations/models/incremental/merge.sql#L51-L86) when no `unique_key` is specified. This is implicit "append" behavior on Postgres + Redshift + [**Snowflake (!!)**](https://github.com/dbt-labs/dbt-snowflake/blob/e8124491efb9750b811bdef14ec4d1ecbebac5e8/dbt/include/snowflake/macros/materializations/merge.sql#L3-L23) today
- [`get_merge_sql`](https://github.com/dbt-labs/dbt-core/blob/eea872c31994c90e995405c74496c1ce20e07025/core/dbt/include/global_project/macros/materializations/models/incremental/merge.sql#L25-L27), when no `unique_key` (or other custom predicates) are specified, as it ends up using `ON FALSE`, a.k.a. [constant false predicate](https://cloud.google.com/bigquery/docs/using-dml-with-partitioned-tables). Turns out, that achieves the same result (= insert only, no update/delete). This is implicit "append" behavior on BigQuery + Databricks (Delta) today.

**Why (!!) for Snowflake**? Even though Snowflake uses the `merge` strategy by default, its `MERGE` statement does NOT support a constant false predicate, so [we've hard-coded `insert into` here](https://github.com/dbt-labs/dbt-snowflake/blob/e8124491efb9750b811bdef14ec4d1ecbebac5e8/dbt/include/snowflake/macros/materializations/merge.sql#L3-L23)!

Since all of these databases support good-old-fashioned `insert into`, I'd like to make that the explicit behavior of all of them when the `append` strategy is specified, i.e. the standard behavior of `get_incremental_append_sql`. Doesn't mean that `append` should be the default, and we definitely shouldn't break the current behavior (implicit append) when folks specify `delete+insert` or `merge` without a `unique_key` specified, but it's the direction we should move in.

Happy to spend more time talking about this, since it's pretty tricky stuff!
updated description to add the new macro `get_insert_into_sql`.

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -1162,6 +1162,43 @@ def get_rows_different_sql(
 
         return sql
 
+    def valid_incremental_strategies(self):
+        """The set of standard builtin strategies which this adapter supports out-of-the-box.
+        Not used to validate custom strategies defined by end users.
+        """
+        return ["append"]
+
+    def builtin_incremental_strategies(self):
+        return ["append", "delete+insert", "merge", "insert_overwrite"]
+
+    @available.parse_none
+    def get_incremental_strategy_macro(self, model_context, strategy: str):
+        # Construct macro_name from strategy name
+        if strategy is None:
+            strategy = "default"
+
+        # validate strategies for this adapter
+        valid_strategies = self.valid_incremental_strategies()
+        valid_strategies.append("default")
+        builtin_strategies = self.builtin_incremental_strategies()
+        if strategy in builtin_strategies and strategy not in valid_strategies:
+            raise RuntimeException(
+                f"The incremental strategy '{strategy}' is not valid for this adapter"
+            )
+
+        strategy = strategy.replace("+", "_")
+        macro_name = f"get_incremental_{strategy}_sql"
+        # The model_context should have MacroGenerator callable objects for all macros
+        if macro_name not in model_context:
+            raise RuntimeException(
+                'dbt could not find an incremental strategy macro with the name "{}" in {}'.format(
+                    macro_name, self.config.project_name
+                )
+            )
+
+        # This returns a callable macro
+        return model_context[macro_name]
+
 
 COLUMNS_EQUAL_SQL = """
 with diff_count as (
diff --git a/core/dbt/contracts/graph/model_config.py b/core/dbt/contracts/graph/model_config.py
--- a/core/dbt/contracts/graph/model_config.py
+++ b/core/dbt/contracts/graph/model_config.py
@@ -433,6 +433,7 @@ class NodeConfig(NodeAndTestConfig):
     # Note: if any new fields are added with MergeBehavior, also update the
     # 'mergebehavior' dictionary
     materialized: str = "view"
+    incremental_strategy: Optional[str] = None
     persist_docs: Dict[str, Any] = field(default_factory=dict)
     post_hook: List[Hook] = field(
         default_factory=list,
diff --git a/plugins/postgres/dbt/adapters/postgres/impl.py b/plugins/postgres/dbt/adapters/postgres/impl.py
--- a/plugins/postgres/dbt/adapters/postgres/impl.py
+++ b/plugins/postgres/dbt/adapters/postgres/impl.py
@@ -132,3 +132,9 @@ def _relations_cache_for_schemas(self, manifest, cache_schemas=None):
 
     def timestamp_add_sql(self, add_to: str, number: int = 1, interval: str = "hour") -> str:
         return f"{add_to} + interval '{number} {interval}'"
+
+    def valid_incremental_strategies(self):
+        """The set of standard builtin strategies which this adapter supports out-of-the-box.
+        Not used to validate custom strategies defined by end users.
+        """
+        return ["append", "delete+insert"]


Test patch:
diff --git a/tests/functional/artifacts/expected_manifest.py b/tests/functional/artifacts/expected_manifest.py
--- a/tests/functional/artifacts/expected_manifest.py
+++ b/tests/functional/artifacts/expected_manifest.py
@@ -31,6 +31,7 @@ def get_rendered_model_config(**updates):
         "meta": {},
         "unique_key": None,
         "grants": {},
+        "incremental_strategy": None,
     }
     result.update(updates)
     return result
@@ -59,6 +60,7 @@ def get_rendered_seed_config(**updates):
         "meta": {},
         "unique_key": None,
         "grants": {},
+        "incremental_strategy": None,
     }
     result.update(updates)
     return result
@@ -91,6 +93,7 @@ def get_rendered_snapshot_config(**updates):
         "target_schema": None,
         "meta": {},
         "grants": {},
+        "incremental_strategy": None,
     }
     result.update(updates)
     return result
diff --git a/tests/functional/list/test_list.py b/tests/functional/list/test_list.py
--- a/tests/functional/list/test_list.py
+++ b/tests/functional/list/test_list.py
@@ -91,6 +91,7 @@ def expect_snapshot_output(self, project):
                     "on_schema_change": "ignore",
                     "meta": {},
                     "grants": {},
+                    "incremental_strategy": None,
                 },
                 "unique_id": "snapshot.test.my_snapshot",
                 "original_file_path": normalize("snapshots/snapshot.sql"),
@@ -127,6 +128,7 @@ def expect_analyses_output(self):
                     "meta": {},
                     "unique_key": None,
                     "grants": {},
+                    "incremental_strategy": None,
                 },
                 "unique_id": "analysis.test.a",
                 "original_file_path": normalize("analyses/a.sql"),
@@ -164,6 +166,7 @@ def expect_model_output(self):
                         "alias": None,
                         "meta": {},
                         "grants": {},
+                        "incremental_strategy": None,
                     },
                     "original_file_path": normalize("models/ephemeral.sql"),
                     "unique_id": "model.test.ephemeral",
@@ -190,12 +193,12 @@ def expect_model_output(self):
                         "full_refresh": None,
                         "unique_key": None,
                         "on_schema_change": "ignore",
-                        "incremental_strategy": "delete+insert",
                         "database": None,
                         "schema": None,
                         "alias": None,
                         "meta": {},
                         "grants": {},
+                        "incremental_strategy": "delete+insert",
                     },
                     "original_file_path": normalize("models/incremental.sql"),
                     "unique_id": "model.test.incremental",
@@ -224,6 +227,7 @@ def expect_model_output(self):
                         "alias": None,
                         "meta": {},
                         "grants": {},
+                        "incremental_strategy": None,
                     },
                     "original_file_path": normalize("models/sub/inner.sql"),
                     "unique_id": "model.test.inner",
@@ -252,6 +256,7 @@ def expect_model_output(self):
                         "alias": None,
                         "meta": {},
                         "grants": {},
+                        "incremental_strategy": None,
                     },
                     "original_file_path": normalize("models/outer.sql"),
                     "unique_id": "model.test.outer",
@@ -295,6 +300,7 @@ def expect_model_ephemeral_output(self):
                         "alias": None,
                         "meta": {},
                         "grants": {},
+                        "incremental_strategy": None,
                     },
                     "unique_id": "model.test.ephemeral",
                     "original_file_path": normalize("models/ephemeral.sql"),
@@ -357,6 +363,7 @@ def expect_seed_output(self):
                     "alias": None,
                     "meta": {},
                     "grants": {},
+                    "incremental_strategy": None,
                 },
                 "unique_id": "seed.test.seed",
                 "original_file_path": normalize("seeds/seed.csv"),
diff --git a/tests/functional/materializations/test_incremental.py b/tests/functional/materializations/test_incremental.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/materializations/test_incremental.py
@@ -0,0 +1,52 @@
+import pytest
+from dbt.tests.util import run_dbt, get_manifest
+from dbt.exceptions import RuntimeException
+from dbt.context.providers import generate_runtime_model_context
+
+
+my_model_sql = """
+  select 1 as fun
+"""
+
+
+@pytest.fixture(scope="class")
+def models():
+    return {"my_model.sql": my_model_sql}
+
+
+def test_basic(project):
+    results = run_dbt(["run"])
+    assert len(results) == 1
+
+    manifest = get_manifest(project.project_root)
+    model = manifest.nodes["model.test.my_model"]
+
+    # Normally the context will be provided by the macro that calls the
+    # get_incrmental_strategy_macro method, but for testing purposes
+    # we create a runtime_model_context.
+    context = generate_runtime_model_context(
+        model,
+        project.adapter.config,
+        manifest,
+    )
+
+    macro_func = project.adapter.get_incremental_strategy_macro(context, "default")
+    assert macro_func
+    assert type(macro_func).__name__ == "MacroGenerator"
+
+    macro_func = project.adapter.get_incremental_strategy_macro(context, "append")
+    assert macro_func
+    assert type(macro_func).__name__ == "MacroGenerator"
+
+    macro_func = project.adapter.get_incremental_strategy_macro(context, "delete+insert")
+    assert macro_func
+    assert type(macro_func).__name__ == "MacroGenerator"
+
+    # These two incremental strategies are not valid for Postgres
+    with pytest.raises(RuntimeException) as excinfo:
+        macro_func = project.adapter.get_incremental_strategy_macro(context, "merge")
+    assert "merge" in str(excinfo.value)
+
+    with pytest.raises(RuntimeException) as excinfo:
+        macro_func = project.adapter.get_incremental_strategy_macro(context, "insert_overwrite")
+    assert "insert_overwrite" in str(excinfo.value)