Repository: dbt-labs/dbt-core

Issue:
[CT-1880] [Feature] Enable "merge" incremental strategy for postgres 15
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Merge is available since Postgres 15. Would be fantastic to enable it as an incremental strategy.

https://www.postgresql.org/docs/current/sql-merge.html


```
{{
    config(
        materialized='incremental',
        unique_key="'ID'", 
        incremental_strategy='merge'
    )
}}

SELECT * FROM {{ source('stage', 'table') }}
```

### Describe alternatives you've considered

delete+insert does not produce the desired behavior that I would have with merge. 
For example.
I have a table core.table with historical data that is to be updated from a table stage.table.
I want to merge the new data into the historical table. So it should update existing records or add new ones.
Postgres only supports delete+insert and append. Since I want to update existing records, I need to use delete+insert (default). However, the problem is that is seems to simply delete ALL records in the historical table, instead of just those that are to be updated…

I am not doing a --full-refresh every time…
```
{{
    config(
        materialized='incremental',
        unique_key="'GUID'", 
    )
}}

SELECT * FROM {{ source('stage', 'table') }}
```

Snapshot is not needed, since I do not need to track the changes. I just want to overwrite.


Discussion:
Thanks for opening this up @rainermensing !

Love the idea of enabling the `merge` strategy for incremental models for dbt-postgres since `MERGE` was released within [PostgreSQL 15](https://www.postgresql.org/about/news/postgresql-15-released-2526/) on 2022-10-13. 🤩 

See below for:
1. [Implementation hints](https://github.com/dbt-labs/dbt-core/issues/6696#issuecomment-1400874954)
1. [An alternative in the meantime](https://github.com/dbt-labs/dbt-core/issues/6696#issuecomment-1400875616)
1. [A key outstanding question we'll need to answer](https://github.com/dbt-labs/dbt-core/issues/6696#issuecomment-1400877777)
### Implementation hints

I didn't confirm or deny, but the implementation _might_ be as simple as:
1. Adding `merge` to the list within dbt-core [here](https://github.com/dbt-labs/dbt-core/blob/eb200b46877c31469f410b7d47dfd245473ed7cb/plugins/postgres/dbt/adapters/postgres/impl.py#L129-L133)
1. Adding an explicit implementation of `valid_incremental_strategies` to dbt-redshift (see details below)

#### dbt-core details
```python
    def valid_incremental_strategies(self):
        """The set of standard builtin strategies which this adapter supports out-of-the-box.
        Not used to validate custom strategies defined by end users.
        """
        return ["append", "delete+insert", "merge"]
```

#### dbt-redshift details
It would be necessary to override the definition of `valid_incremental_strategies` that `RedshiftAdapter` inherits from  `PostgresAdapter`. I _think_ (but didn't confirm) that the explicit implementation for dbt-redshift would be adding the following to the bottom of [this](https://github.com/dbt-labs/dbt-redshift/blob/main/dbt/adapters/redshift/impl.py) file:
```python
    def valid_incremental_strategies(self):
        """The set of standard builtin strategies which this adapter supports out-of-the-box.
        Not used to validate custom strategies defined by end users.
        """
        return ["append", "delete+insert"]
```

Whether this works or not will be entirely dependent upon the syntax of the default merge strategy being compatible with Postgres 15. If not, a custom definition of `postgres__get_merge_sql` will be needed.
### An alternative, in the meantime

Something [not yet documented](https://github.com/dbt-labs/docs.getdbt.com/issues/1761) is that:
> Users can also define their own "custom" strategies, by defining a macro get_incremental_silly_sql, and then configuring incremental_strategy: silly. dbt won't validate this, it will just look for the macro by that name, and raise an error if it can't find one.

So in this case, if you could use the built-in `merge` logic with Postgres 15 if you do the following (which I didn't test out at all!):

1) Define the `get_incremental_pgmerge_sql` macro in your dbt project like this:

```sql
{% macro get_incremental_pgmerge_sql(arg_dict) %}

  {% do return(get_merge_sql(arg_dict["target_relation"], arg_dict["temp_relation"], arg_dict["unique_key"], arg_dict["dest_columns"])) %}

{% endmacro %}
```

2) Configure your incremental model using the name of your "custom" incremental strategy like described [here](https://docs.getdbt.com/docs/build/incremental-models#configuring-incremental-strategy):
```sql
{{
  config(
    materialized='incremental',
    incremental_strategy='pgmerge',
    ...
  )
}}

select ...
```

Replace `pgmerge` above with whatever you want to call your custom strategy. Heck, calling it `merge` might even work (but maybe it won't due to it be reserved keyword somehow).
### Outstanding question
What should happen for folks using Postgres 14 or earlier?

### Options
1. Don't actually support the `merge` incremental strategy for dbt-postgres since so many Postgres versions wouldn't support it
    - Rather, document how a user can configure a custom strategy to get this functionality (possibly using a dbt package that has the relevant logic)
1. Make no attempt to determine the Postgres version, and just let `merge` fail whatever error message the driver spits out
1. Make no attempt to determine the Postgres version, and but catch the relevant exception and convert the error to to say something like "`merge` incremental strategy isn't supported until PostgreSQL 15"
1. Make some attempt to determine the Postgres version, and raise an exception with a helpful error message for versions prior to PostgreSQL 15

@jtcohen6 what are your thoughts about 1) supporting `merge` for dbt-postgres, and 2) the options for what should happen for Postgres 14 and lower?
You can run SQL command `show server_version;` to get the version.
`show server_version;` --> `13.7` (on one of my Postgres installations)
> @jtcohen6 what are your thoughts about 1) supporting merge for dbt-postgres, and 2) the options for what should happen for Postgres 14 and lower?

So long as `merge` isn't the default strategy, I think it's fine to start supporting it. We'd document that the `merge` strategy (optional) is only supported on Postgres 15+. It would be the responsibility of `dbt-postgres` users to know which version of Postgres they're running, if/when they choose to opt into the `merge` strategy.
Awesome @jtcohen6

So here's a summary of how to implement this feature:
1. Follow the hints [here](https://github.com/dbt-labs/dbt-core/issues/6696#issuecomment-1400874954)
1. Keep the current default specified in `get_incremental_default_sql` (which I believe is [append](https://github.com/dbt-labs/dbt-core/pull/5359#discussion_r925614129))
1. Update docs [here](https://docs.getdbt.com/docs/build/incremental-models#about-incremental_strategy) to include the [suggested messaging](https://github.com/dbt-labs/dbt-core/issues/6696#issuecomment-1405032995) for PostgreSQL

A bonus for us to consider:
- Would we accept an implementation that is able to dynamically infer the Postgres version and proactively raise a "NotSupported" exception (using a method like [this](https://github.com/dbt-labs/dbt-core/issues/6696#issuecomment-1405005911))?
> ### Implementation hints
> I didn't confirm or deny, but the implementation _might_ be as simple as:
> 
> 1. Adding `merge` to the list within dbt-core [here](https://github.com/dbt-labs/dbt-core/blob/eb200b46877c31469f410b7d47dfd245473ed7cb/plugins/postgres/dbt/adapters/postgres/impl.py#L129-L133)
> 2. Adding an explicit implementation of `valid_incremental_strategies` to dbt-redshift (see details below)
> 
> #### dbt-core details
> ```python
>     def valid_incremental_strategies(self):
>         """The set of standard builtin strategies which this adapter supports out-of-the-box.
>         Not used to validate custom strategies defined by end users.
>         """
>         return ["append", "delete+insert", "merge"]
> ```
> 
> #### dbt-redshift details
> It would be necessary to override the definition of `valid_incremental_strategies` that `RedshiftAdapter` inherits from `PostgresAdapter`. I _think_ (but didn't confirm) that the explicit implementation for dbt-redshift would be adding the following to the bottom of [this](https://github.com/dbt-labs/dbt-redshift/blob/main/dbt/adapters/redshift/impl.py) file:
> 
> ```python
>     def valid_incremental_strategies(self):
>         """The set of standard builtin strategies which this adapter supports out-of-the-box.
>         Not used to validate custom strategies defined by end users.
>         """
>         return ["append", "delete+insert"]
> ```
> 
> Whether this works or not will be entirely dependent upon the syntax of the default merge strategy being compatible with Postgres 15. If not, a custom definition of `postgres__get_merge_sql` will be needed.

## @dbeatty10 Thank you for this hint. I just tested it and it works!!
As a small detail, I have to give the unique_key column name in double quotes wrapped in single quotes, like unique_key ='"ID"'. Not sure if this is a desired behavior?  Else it will not compile correctly as DBT_INTERNAL_SOURCE."ID", but as DBT_INTERNAL_SOURCE.ID or DBT_INTERNAL_SOURCE.'ID' (both produce errors):
![image](https://user-images.githubusercontent.com/31634583/215443812-3b2cdf7d-a972-4b29-b4df-45b0939e76f3.png)

> @dbeatty10 Thank you for this hint. I just tested it and it works!!

🙌 @rainermensing That's awesome that you got it to work!! 🏆

### Quoting issues
Oh, the "joys" of how each database handles unquoted identifiers differently :sob: + :joy:

Here's a portion of the [PostgreSQL docs](https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS) that might be relevant:

> Quoting an identifier also makes it case-sensitive, whereas unquoted names are always folded to lower case

Could you provide a representative sample of your incremental model? I wonder if you are using a quoted identifier `"ID"` either with your incremental model or upstream?

### Next steps
Are you interested in opening a PR for this? If so, you can mark it as a draft until details like the column quoting are ironed out. Having a draft PR could help us work out those details too.
> 
> ### Quoting issues
> Oh, the "joys" of how each database handles unquoted identifiers differently 😭 + 😂
> 
@dbeatty10 Yes I am only using quoted identifiers. What is your specific concern? I was mainly wondering if it is desired behavior that one has to double quote when referencing a quoted identified. 

And yes, I could try open a pull request some time this week! 

@dbeatty10 I just set up my dbt for local development. Two problems:
1. Even in the unchanged dbt core 1.5.0-a1, the integration tests are not fully passing. I guess this should not be the case. It could be that this is because I run them on WSL (which often is the perpetrator), but literally 99% of the other tests are passing. So I am not sure.
![image](https://user-images.githubusercontent.com/31634583/216846769-1cc1a147-5668-4da8-ad92-f78b79b5a6f7.png)
(This is on Python 3.9.5)

2. You said that I also need to make the appropriate edit in dbt-redshift, but I do not have a redshift cluster ready to run the tests. 

So my question is whether I should just make the changes you suggested and make the pull request anyway. 
Also, I am not very familiar with the dbt repo, but I don't think the tests for the merge run automatically becauseI included "merge" in the accepted strategies? To what extend can they be copied from snowflake/bigquery?

Thank you in advance! 
@rainermensing I'm going to address the two problems you mentioned but in the opposite order (from easier to harder).

But first, a [question you asked](https://github.com/dbt-labs/dbt-core/issues/6696#issuecomment-1409226587) that I accidentally overlooked:

> What is your specific concern? I was mainly wondering if it is desired behavior that one has to double quote when referencing a quoted identifier.

I wouldn't say it's _desirable_ to require multiple layers of quoting like `unique_key ='"ID"'`, but it might be _necessary_ in your case. There's some good info on the default quoting policies that dbt uses for Postgres vs. Snowflake [here](https://docs.getdbt.com/reference/project-configs/quoting#default) and [here](https://docs.getdbt.com/reference/project-configs/quoting#explanation).

### Appropriate edit in dbt-redshift
2. It's okay that you don't have a cluster to test the dbt-redshift changes -- if you're willing, just go ahead and submit the PR that makes the change and we'll run it through GitHub Actions (GHA). If you're uncomfortable with that, just let me know, and I'll open up the PR instead.

When it comes time to merge PRs, they should be merged in this order:
1. dbt-redshift
2. dbt-core

This is because the changes in dbt-redshift will automatically inherit the new valid incremental strategies from dbt-postgres unless we assertively change dbt-redshift first.

In fact, we _could_ do the dbt-redshift one end-to-end even before the dbt-core is completely ready.

### Integration tests are not fully passing in dbt-core

1. Just go ahead and submit the pull request anyways. We'll run continuous integration (CI) tests against Postgres using GitHub Actions (GHA) and go from there. I haven't checked, but we _might_ need to make sure GHA uses an applicable version of the PostgreSQL database (i.e., >=15) -- it wouldn't surprise me if it is explicitly using some version older than 15.

#### Some things I don't know the answer to

Here's some relevant incremental tests in dbt-snowflake:
- https://github.com/dbt-labs/dbt-snowflake/tree/054422d/tests/functional/adapter/incremental

We can see that [test_incremental_on_schema_change.py](https://github.com/dbt-labs/dbt-snowflake/blob/0a9894b2de96e823b93ea5535095042682c73580/tests/functional/adapter/incremental/test_incremental_on_schema_change.py) inherits from [here](https://github.com/dbt-labs/dbt-core/blob/f841a7ca76b44429eb94174bc5d0c2fecbf2763b/tests/adapter/dbt/tests/adapter/incremental/test_incremental_on_schema_change.py#L81).

What isn't clear to me though:
- Which merge strategy is dbt-core using when it runs that group of tests?
- Which merge strategy is dbt-snowflake using when it runs that group of tests? Is is the same as dbt-core?

Are they both using `default` strategy (which then resolves to `merge` for Snowflake and `append` for Postgres)?

I don't know the answers to those questions yet -- we'll have to discover as we go along.

Once you submit your PR and the CI runs, we can use that as a jumping off point to determine if the `merge` strategy is being tested at all automatically or if we'll need to make some additions or changes to accomplish that.

Resolving patch:
diff --git a/plugins/postgres/dbt/adapters/postgres/impl.py b/plugins/postgres/dbt/adapters/postgres/impl.py
--- a/plugins/postgres/dbt/adapters/postgres/impl.py
+++ b/plugins/postgres/dbt/adapters/postgres/impl.py
@@ -140,7 +140,7 @@ def valid_incremental_strategies(self):
         """The set of standard builtin strategies which this adapter supports out-of-the-box.
         Not used to validate custom strategies defined by end users.
         """
-        return ["append", "delete+insert"]
+        return ["append", "delete+insert", "merge"]
 
     def debug_query(self):
         self.execute("select 1 as id")


Test patch:
diff --git a/tests/functional/materializations/test_incremental.py b/tests/functional/materializations/test_incremental.py
--- a/tests/functional/materializations/test_incremental.py
+++ b/tests/functional/materializations/test_incremental.py
@@ -42,11 +42,12 @@ def test_basic(project):
     assert macro_func
     assert type(macro_func).__name__ == "MacroGenerator"
 
-    # These two incremental strategies are not valid for Postgres
-    with pytest.raises(DbtRuntimeError) as excinfo:
-        macro_func = project.adapter.get_incremental_strategy_macro(context, "merge")
-    assert "merge" in str(excinfo.value)
+    # This incremental strategy only works for Postgres >= 15
+    macro_func = project.adapter.get_incremental_strategy_macro(context, "merge")
+    assert macro_func
+    assert type(macro_func).__name__ == "MacroGenerator"
 
+    # This incremental strategy is not valid for Postgres
     with pytest.raises(DbtRuntimeError) as excinfo:
         macro_func = project.adapter.get_incremental_strategy_macro(context, "insert_overwrite")
     assert "insert_overwrite" in str(excinfo.value)