Repository: dbt-labs/dbt-core

Issue:
[CT-96] Allow unique_key for incremental materializations to take a list
### Describe the feature
Right now when creating a model in dbt with materialization set to `incremental` you can pass in a single column to `unique key` that will act as the `key` for merging.

In an ideal world you would be able to pass in multiple columns as there are many cases where a table has more than one column that defines it's primary key.

The simplest solution would be to change to `unique_key` to take in a list (in addition to a string for backwards compatibility) and create the `predicates` for the merge based on the list vs. just the single column.

This might not be ideal as the param name `unique_key` implies a single key. Alternatives would be adding a new optional parameter `unique_key_list` or `unique_keys` that always take a list and eventually deprecate the `unique_key` parameter.

### Describe alternatives you've considered
Not necessarily other alternatives but another thing to consider is the use of `unique_key` throughout the dbt project. It would stand to reason that whatever change is made here would apply to all other usages of `unique_key`. This can be done in one large roll-out or in stages such as with merges first, then upserts, then snapshots, etc. 

### Additional context
This feature should work across all databases.

### Who will this benefit?
Hopefully most dbt users. Currently the only workaround for this is using dbt_utils.surrogate_key which a) doesn't work for BigQuery and b) should ideally be an out of the box dbt feature.



Discussion:
Thanks for the solid proposal, @azhard. I've now heard this from several members of the community. Though it's what we recommend, I understand that creating a single hashed surrogate key for each model is against some folks' warehousing paradigms. Currently, that requires overriding a complex merge macro, or the entire incremental materialization.

> a) doesn't work for BigQuery

Could you clarify what you mean here? Is the BQ implementation of `dbt_utils.surrogate_key` broken?
Hey @jtcohen6 the `surrogate_key` function is broken for this particular case but not sure if the fault lies on the `dbt` side or the `dbt_utils` side.

Essentially in merge.sql L25 dbt does this to generate the match predicate
```DBT_INTERNAL_SOURCE.{{ unique_key }} = DBT_INTERNAL_DEST.{{ unique_key }}```

While this normally would work, the unique_key is substituted with `to_hex(md5(cast(concat(coalesce(cast(id as ...` which means the generated SQL becomes `DBT_INTERNAL_SOURCE.to_hext(md5...)` which is incorrect as the `DBT_INTERNAL_SOURCE.` should be preceding `id`.

I've taken a look at implementing this change to use unique_key as a list in the `merge.sql` file and it doesn't seem super involved. I'd be happy to put up a PR if there is consensus on the best long-term approach.
@jtcohen6 any idea on the `surrogate_key` issue? I'd be happy to submit a PR to try and fix it, just not sure which side it counts as being broken on
I see what you mean. When I think of creating a surrogate key for an incremental model, I'm thinking of creating a column within that model, to be stored in the resulting table and passed as the `unique_key` for subsequent incremental runs:

```sql
{{ config(
    materialized = 'incremental',
    unique_key = 'unique_id'
) }}

select
    date_day,
    user_id,
    {{ dbt_utils.surrogate_key('date_day', 'user_id') }} as unique_id

...
```

You're right that, as a result of the way that merge macros are implemented on BigQuery, you cannot create the surrogate key directly within the config like so:
```sql
{{ config(
    materialized = 'incremental',
    unique_key = dbt_utils.surrogate_key('date_day', 'user_id')
) }}
```

I've now heard this change requested from several folks now, including (if I recall correctly) some Snowflake users who have found that merging on cluster keys improves performance somewhat. So I'm not opposed to passing an array of column names. I'm worried that `unique_keys` is ambiguous; following the lead of the [dbt-utils test](https://github.com/fishtown-analytics/dbt-utils#unique_combination_of_columns-source), I'm thinking along the lines of `unique_combination_of_columns`.

@drewbanin What do you think? Is that too much config-arg creep?
@jtcohen6 I'm into the idea of supporting an array of values for `unique_key`. I do think our best practices should state that a `unique_key` should be a single field name, but i think it should be relatively easy to build the correct merge statement from a list of fields.

Either way though, I do think we should mandate that `unique_key` args are simple field names, and we should try to support expressions in the `unique_key` arg.

So, this would not be supported:
```
{{ config(
    materialized = 'incremental',
    unique_key = dbt_utils.surrogate_key('date_day', 'user_id')
) }}
```

but this could be:
```
{{ config(
    materialized = 'incremental',
    unique_key = ['date_day', 'user_id']
) }}
```

I would like to think a little harder about how this would work for databases that _do not_ support `merge` statements. On those databases, we run a query like:

```
    delete
    from {{ target_relation }}
    where ({{ unique_key }}) in (
        select ({{ unique_key }})
        from {{ tmp_relation }}
    );
```

It's not clear to me yet how we should build this delete statement with multiple keys, or if we should just not support multiple keys on pg/redshift/etc
To the last point, [Postgres](https://www.postgresql.org/docs/10/sql-delete.html), [Redshift](https://docs.aws.amazon.com/redshift/latest/dg/r_DELETE.html), and [Snowflake](https://docs.snowflake.com/en/sql-reference/sql/delete.html) all support `using` syntax with `delete` statements, so we could write:
```sql
    delete
    from {{ target_relation }}
    using {{ tmp_relation }}
    where
    {% for column_name in unique_combination_of_columns %}
        {{ target_relation }}.{{ column_name }} = {{ tmp_relation }}.{{ column_name }}
        {{- "and" if not loop.last }}
    {% endfor %}
```
Any advice on how I can use multiple key columns locally with BigQuery right now? 
If need be, you could override the `get_merge_sql` macro, specifically to change these lines:
https://github.com/fishtown-analytics/dbt/blob/c3c99f317e044a333d20766cc59c523769a4204f/core/dbt/include/global_project/macros/materializations/common/merge.sql#L23-L28
to 
```sql
    {% if unique_key %}
        {% set unique_key_list = unique_key.split(",") %}
        {% for key in unique_key_list %}
            {% set this_key_match %}
                DBT_INTERNAL_SOURCE.{{ key }} = DBT_INTERNAL_DEST.{{ key }}
            {% endset %}
            {% do predicates.append(this_key_match) %}
        {% endfor %}
    {% else %}
```
and then pass a comma-separated list to `unique_key` in your model:
```sql
{{ config(
    materialized = 'incremental',
    unique_key = 'date_day, user_id'
) }}
```

That's obviously not the eventual implementation we're going for, but I think that would work for your specific use case.
Hey @jtcohen6 that works perfectly, thanks for the help!
Seeing the discussion and the proposed solution, could we rename this issue to "Allow unique_key to take a list"? To make it more generic. 

And I am ok with using `unique_key` for both a `str` and `List[str]` even though `unique_keys` is more accurate for the latter. This happens in `pandas` sometimes too, e.g. `drop` and `pivot` have a `columns` parameter that can be both `str` and `List[str]`. In my opinion this is better than adding a new parameter. 
I've had a read through both of the above threads and I'm happy to give this a go.

In summary:

* `unique_key` should support `unique_key='column'` or `unique_key=['column1', 'column2']`.
* As part of this work we should split out ModelConfig specific attributes from NodeConfig.
* `unique_key` should be validated in python.

Considerations:

I would like to introduce a jinja test for lists if everyone is in agreement.

So we can do things like
`{% if unique_key is list %}`
rather than
`{% if unique_key is sequence and unique_key is not mapping and unique_key is not string %}`

This will probably benefit a number of macros with list or string behaviour.
@triedandtested-dev Thanks for your interest! I'd welcome a PR for this :)

In terms of your three points:
1. Yes, let's have the same config `unique_key` accept either a string column name, or a list of string column names
2. While the split-apart of ModelConfig and NodeConfig is highly desirable, we can leave that out of scope for this effort. That can come instead in work for #3557.
3. The python validation will be a nice side-effect of defining `unique_key: Optional[Union[str, List[str]]]` within the [`NodeConfig` dataclass](https://github.com/dbt-labs/dbt/blob/ea07729bbf0a3481e220615644b7955659001529/core/dbt/contracts/graph/model_config.py#L387-L412)

The `is list` Jinja test is a neat idea! Even if that's not strictly necessary here, I'm hardly opposed.
Happy to keep the ModelConfig out of scope for this issue. 

I'll get cracking on a PR for this!

The jinja "is list" addition should help with readability going forward but completely agree its not technically necessary.

Resolving patch:
diff --git a/core/dbt/contracts/graph/model_config.py b/core/dbt/contracts/graph/model_config.py
--- a/core/dbt/contracts/graph/model_config.py
+++ b/core/dbt/contracts/graph/model_config.py
@@ -409,6 +409,7 @@ class NodeConfig(NodeAndTestConfig):
         metadata=MergeBehavior.Update.meta(),
     )
     full_refresh: Optional[bool] = None
+    unique_key: Optional[Union[str, List[str]]] = None
     on_schema_change: Optional[str] = 'ignore'
 
     @classmethod
@@ -494,6 +495,7 @@ def same_contents(
 @dataclass
 class EmptySnapshotConfig(NodeConfig):
     materialized: str = 'snapshot'
+    unique_key: Optional[str] = None  # override NodeConfig unique_key definition
 
 
 @dataclass


Test patch:
diff --git a/test/integration/029_docs_generate_tests/test_docs_generate.py b/test/integration/029_docs_generate_tests/test_docs_generate.py
--- a/test/integration/029_docs_generate_tests/test_docs_generate.py
+++ b/test/integration/029_docs_generate_tests/test_docs_generate.py
@@ -461,6 +461,7 @@ def rendered_model_config(self, **updates):
             'full_refresh': None,
             'on_schema_change': 'ignore',
             'meta': {},
+            'unique_key': None,
         }
         result.update(updates)
         return result
@@ -485,6 +486,7 @@ def rendered_seed_config(self, **updates):
             'schema': None,
             'alias': None,
             'meta': {},
+            'unique_key': None,
         }
         result.update(updates)
         return result
diff --git a/test/integration/047_dbt_ls_tests/test_ls.py b/test/integration/047_dbt_ls_tests/test_ls.py
--- a/test/integration/047_dbt_ls_tests/test_ls.py
+++ b/test/integration/047_dbt_ls_tests/test_ls.py
@@ -128,6 +128,7 @@ def expect_analyses_output(self):
                     'schema': None,
                     'alias': None,
                     'meta': {},
+                    'unique_key': None
                 },
                 'unique_id': 'analysis.test.a',
                 'original_file_path': normalize('analyses/a.sql'),
@@ -158,6 +159,7 @@ def expect_model_output(self):
                         'column_types': {},
                         'persist_docs': {},
                         'full_refresh': None,
+                        'unique_key': None,
                         'on_schema_change': 'ignore',
                         'database': None,
                         'schema': None,
@@ -184,6 +186,7 @@ def expect_model_output(self):
                         'column_types': {},
                         'persist_docs': {},
                         'full_refresh': None,
+                        'unique_key': None,
                         'on_schema_change': 'ignore',
                         'incremental_strategy': 'delete+insert',
                         'database': None,
@@ -211,6 +214,7 @@ def expect_model_output(self):
                         'column_types': {},
                         'persist_docs': {},
                         'full_refresh': None,
+                        'unique_key': None,
                         'on_schema_change': 'ignore',
                         'database': None,
                         'schema': None,
@@ -237,6 +241,7 @@ def expect_model_output(self):
                         'column_types': {},
                         'persist_docs': {},
                         'full_refresh': None,
+                        'unique_key': None,
                         'on_schema_change': 'ignore',
                         'database': None,
                         'schema': None,
@@ -332,6 +337,7 @@ def expect_seed_output(self):
                     'persist_docs': {},
                     'quote_columns': False,
                     'full_refresh': None,
+                    'unique_key': None,
                     'on_schema_change': 'ignore',
                     'database': None,
                     'schema': None,
diff --git a/test/unit/test_contracts_graph_parsed.py b/test/unit/test_contracts_graph_parsed.py
--- a/test/unit/test_contracts_graph_parsed.py
+++ b/test/unit/test_contracts_graph_parsed.py
@@ -302,6 +302,7 @@ def complex_parsed_model_object():
 def test_model_basic(basic_parsed_model_object, base_parsed_model_dict, minimal_parsed_model_dict):
     node = basic_parsed_model_object
     node_dict = base_parsed_model_dict
+    compare_dicts(node.to_dict(), node_dict)
     assert_symmetric(node, node_dict)
     assert node.empty is False
     assert node.is_refable is True
@@ -921,6 +922,7 @@ def test_basic_parsed_hook(minimal_parsed_hook_dict, base_parsed_hook_dict, base
 def test_complex_parsed_hook(complex_parsed_hook_dict, complex_parsed_hook_object):
     node = complex_parsed_hook_object
     node_dict = complex_parsed_hook_dict
+    # what's different?
     assert_symmetric(node, node_dict)
     assert node.empty is False
     assert node.is_refable is False
@@ -1494,6 +1496,7 @@ def basic_intermediate_timestamp_snapshot_object():
         tags=[],
         config=cfg,
         checksum=FileHash.from_contents(''),
+        created_at = 1,
         unrendered_config={
             'strategy': 'timestamp',
             'unique_key': 'id',
@@ -1596,7 +1599,7 @@ def basic_check_snapshot_object():
 
 
 @pytest.fixture
-def basic_intermedaite_check_snapshot_object():
+def basic_intermediate_check_snapshot_object():
     cfg = EmptySnapshotConfig()
     cfg._extra.update({
         'unique_key': 'id',
@@ -1626,6 +1629,7 @@ def basic_intermedaite_check_snapshot_object():
         tags=[],
         config=cfg,
         checksum=FileHash.from_contents(''),
+        created_at = 1.0,
         unrendered_config={
             'target_database': 'some_snapshot_db',
             'target_schema': 'some_snapshot_schema',
@@ -1642,20 +1646,20 @@ def test_timestamp_snapshot_ok(basic_timestamp_snapshot_dict, basic_timestamp_sn
     inter = basic_intermediate_timestamp_snapshot_object
 
     assert_symmetric(node, node_dict, ParsedSnapshotNode)
-    assert_symmetric(inter, node_dict, IntermediateSnapshotNode)
+#   node_from_dict = ParsedSnapshotNode.from_dict(inter.to_dict(omit_none=True))
+#   node_from_dict.created_at = 1
     assert ParsedSnapshotNode.from_dict(inter.to_dict(omit_none=True)) == node
     assert node.is_refable is True
     assert node.is_ephemeral is False
     pickle.loads(pickle.dumps(node))
 
 
-def test_check_snapshot_ok(basic_check_snapshot_dict, basic_check_snapshot_object, basic_intermedaite_check_snapshot_object):
+def test_check_snapshot_ok(basic_check_snapshot_dict, basic_check_snapshot_object, basic_intermediate_check_snapshot_object):
     node_dict = basic_check_snapshot_dict
     node = basic_check_snapshot_object
-    inter = basic_intermedaite_check_snapshot_object
+    inter = basic_intermediate_check_snapshot_object
 
     assert_symmetric(node, node_dict, ParsedSnapshotNode)
-    assert_symmetric(inter, node_dict, IntermediateSnapshotNode)
     assert ParsedSnapshotNode.from_dict(inter.to_dict(omit_none=True)) == node
     assert node.is_refable is True
     assert node.is_ephemeral is False
diff --git a/test/unit/utils.py b/test/unit/utils.py
--- a/test/unit/utils.py
+++ b/test/unit/utils.py
@@ -207,6 +207,8 @@ def assert_to_dict(obj, dct):
         obj_to_dict['created_at'] = 1
     if 'created_at' in dct:
         dct['created_at'] = 1
+    if obj_to_dict != dct:
+        compare_dicts(obj_to_dict, dct)
     assert obj_to_dict == dct