Repository: dbt-labs/dbt-core

Issue:
Allow require_partition_filter=true for BigQuery table creation
### Describe the feature

I would like to have enable require_partition_filter=true in table and incremental mode.
https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language#table_option_list

for example, we could config model as 
```
{{
  config (
    materialized='table',
    partition_by='DATE(stamp)',
    partition_by_required='true',
  )
}}
```

Currently, in 0.14.3,  bigquery/adapters.sql, only persist_docs and temporary options is enable. 
Could we make  bigquery_table_options macro below flexible to take in any other options, so that we could pass a generic key/value list in model config to create table options
``` {% macro bigquery_table_options(persist_docs, temporary) %} ```

for example, we could config model as 
```
{{
  config (
    materialized='table',
    partition_by='DATE(stamp)',
    options = '{ partition_by_required=true, partition_expiration_days=7 }'
  )
}}
```


### Additional context
This feature is BigQuery-specific. 

### Who will this benefit?
This feature will benefit anyone who use BigQuery partition tables, and users of BigQuery in general.


Discussion:
cool idea @hui-zheng! See also #1829 -- both of these issues require dbt to set `options` appropriately in the `create table as` statements that are run for table/incremental models.

I think we could support this pretty readily:
1. Add `require_partition_filter ` as a "known" config [in the plugin code](https://github.com/fishtown-analytics/dbt/blob/dev/louisa-may-alcott/plugins/bigquery/dbt/adapters/bigquery/impl.py#L48)
2. Add some logic in the [bigquery specific plugin macros](https://github.com/fishtown-analytics/dbt/blob/dev/louisa-may-alcott/plugins/bigquery/dbt/include/bigquery/macros/adapters.sql#L29-L60) which checks for a `require_partition_filter ` config, adding it to the list of supplied options

And that's it! What do you think about that?
yes, I agree with this approach. 
I've just done a minimal-effort implementation by overwriting the essential macros in my own macros folder and just want to add what I found out on the way: You will also need to touch the `merge` statements as they will not execute with `require_partition_filter=true` unless you specify the filter in the `on` statement of the merge. In other words, my `common_get_merge_sql` now looks something like this:
```
{% macro common_get_merge_sql(target, source, unique_key, dest_columns) -%}
...
    merge into {{ target }} as DBT_INTERNAL_DEST
    using {{ source }} as DBT_INTERNAL_SOURCE
    {% if unique_key %}
        on DBT_INTERNAL_SOURCE.{{ unique_key }} = DBT_INTERNAL_DEST.{{ unique_key }}
        {# The if statement below is new #}
        {% if config.get('part_filter_required', True) %}
          and DBT_INTERNAL_DEST.{{ config.get('part_date_column', default='time') }} > {{some_logic}}
        {% endif %}
    {% else %}
        on FALSE
    {% endif %}
...
{%- endmacro %}
```
This is (purposefully) hacky but works for me, I'm not recommending going this route, just wanted to make you aware of the limitation. Note that you will also need to supply the argument as a literal, something like `SELECT MAX(column) FROM {{this}}` as `{{some_logic}}` is not going to work. I solved it by calling a macro which calls statements which get me a datetime literal that works for my purposes.

The compiled and ready to run sql then looks something like this:
```
    merge into `[project]`.`[dataset].`[relation]` as DBT_INTERNAL_DEST
    using (
SELECT
	GENERATE_UUID() AS id
	, time
...
FROM `[project]`.`[source_schema]`.`[source_relation]`
WHERE time > '2019-10-26 20:55:40' 
GROUP BY
    2, ...
    ) as DBT_INTERNAL_SOURCE
        on DBT_INTERNAL_SOURCE.id = DBT_INTERNAL_DEST.id
          AND DBT_INTERNAL_DEST.time > '2019-10-26 20:55:40'
    when matched then update set
        id = DBT_INTERNAL_SOURCE.id,time = DBT_INTERNAL_SOURCE.time,...
    when not matched then insert
        (id, time, ...)
    values
        (id, time, ...)
```

Edit 2019-11-04: I wanted to add that if the merge joins `on FALSE` because there is no `unique_key` specified, there is also no need to include the partition column in the `on` statement. In other words, this would also work:
```
merge into `[project]`.`[dataset].`[relation]` as DBT_INTERNAL_DEST
    using (
SELECT
	GENERATE_UUID() AS id
	, time
...
FROM `[project]`.`[source_schema]`.`[source_relation]`
WHERE time > '2019-10-26 20:55:40' 
GROUP BY
    2, ...
    ) as DBT_INTERNAL_SOURCE
        on FALSE
    when matched then update set
        id = DBT_INTERNAL_SOURCE.id,time = DBT_INTERNAL_SOURCE.time,...
    when not matched then insert
        (id, time, ...)
    values
        (id, time, ...)
```
So the one thing that doesn't work is having *both* a `unique_key` and required partitioning filter.
@soltanianalytics thanks so much for the heads up! The logic you implemented reminds me a lot of https://github.com/fishtown-analytics/dbt/issues/1034. I think this is a problem we'll generally need to solve on BQ sometime soon - it's great to know that these two pieces of functionality are related.
We changed some of the relevant code around. The eventual implementation will need to:
1. Add an adapter-specific config [here](https://github.com/fishtown-analytics/dbt/blob/05634e61a70f4977726acbf05232a63899e62442/plugins/bigquery/dbt/adapters/bigquery/connections.py#L39-L46)
2. Return the `require_partiton_filter` option for table creation [here](https://github.com/fishtown-analytics/dbt/blob/05634e61a70f4977726acbf05232a63899e62442/plugins/bigquery/dbt/adapters/bigquery/impl.py#L656-L676)


Hey @drewbanin just wanted to check if this something that will be supported sometime soon? Would be a super useful config variable to have available.
Agree that this would be a great first issue for an external contributor!

One small thing to note about tables that require partition filters, very much outside the scope of the actual code change required here:

Built-in schema tests will _not_ work on these models. You'll need to instead use tests like [`unique_where`](https://github.com/fishtown-analytics/dbt-utils#unique_where-source), [`not_null_where`](https://github.com/fishtown-analytics/dbt-utils#not_null_where-source), [`relationships_where`](https://github.com/fishtown-analytics/dbt-utils#relationships_where-source). It could be cool to play around with behavior like:
```yml
models:
  - name: my_model
    columns:
      - name: id
        tests:
          - dbt_utils.unique_where:
              where: "{{ config.get('partition_by') ~ '= current_date' if config.get('partition_by_required') else '1=1' }}"
```

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/impl.py b/plugins/bigquery/dbt/adapters/bigquery/impl.py
--- a/plugins/bigquery/dbt/adapters/bigquery/impl.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/impl.py
@@ -111,6 +111,8 @@ class BigqueryConfig(AdapterConfig):
     partitions: Optional[List[str]] = None
     grant_access_to: Optional[List[Dict[str, str]]] = None
     hours_to_expiration: Optional[int] = None
+    require_partition_filter: Optional[bool] = None
+    partition_expiration_days: Optional[int] = None
 
 
 class BigQueryAdapter(BaseAdapter):
@@ -788,6 +790,14 @@ def get_table_options(
             labels = config.get('labels', {})
             opts['labels'] = list(labels.items())
 
+        if config.get('require_partition_filter'):
+            opts['require_partition_filter'] = config.get(
+                'require_partition_filter')
+
+        if config.get('partition_expiration_days') is not None:
+            opts['partition_expiration_days'] = config.get(
+                'partition_expiration_days')
+
         return opts
 
     @available.parse_none


Test patch:
diff --git a/test/integration/022_bigquery_test/partition-models/my_model.sql b/test/integration/022_bigquery_test/partition-models/my_model.sql
--- a/test/integration/022_bigquery_test/partition-models/my_model.sql
+++ b/test/integration/022_bigquery_test/partition-models/my_model.sql
@@ -4,7 +4,9 @@
     config(
         materialized="table",
         partition_by=var('partition_by'),
-        cluster_by=var('cluster_by')
+        cluster_by=var('cluster_by'),
+        partition_expiration_days=var('partition_expiration_days'),
+        require_partition_filter=var('require_partition_filter')
     )
 }}
 
diff --git a/test/integration/022_bigquery_test/test_bigquery_adapter_specific.py b/test/integration/022_bigquery_test/test_bigquery_adapter_specific.py
--- a/test/integration/022_bigquery_test/test_bigquery_adapter_specific.py
+++ b/test/integration/022_bigquery_test/test_bigquery_adapter_specific.py
@@ -1,4 +1,6 @@
 """"Test adapter specific config options."""
+from pprint import pprint
+
 from test.integration.base import DBTIntegrationTest, use_profile
 import textwrap
 import yaml
@@ -32,7 +34,7 @@ def project_config(self):
     @use_profile('bigquery')
     def test_bigquery_hours_to_expiration(self):
         _, stdout = self.run_dbt_and_capture(['--debug', 'run'])
-        
+
         self.assertIn(
             'expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL '
             '4 hour)', stdout)
diff --git a/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py b/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py
--- a/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py
+++ b/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py
@@ -29,45 +29,91 @@ def test_partitions(self, expected):
 
     @use_profile('bigquery')
     def test_bigquery_add_partition(self):
-        before = {"partition_by": None, "cluster_by": None}
-        after = {"partition_by": {'field': 'cur_time',
-                                  'data_type': 'timestamp'}, "cluster_by": None}
+        before = {"partition_by": None,
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp'},
+                 "cluster_by": None,
+                 'partition_expiration_days': 7,
+                 'require_partition_filter': True}
         self.run_changes(before, after)
         self.test_partitions({"expected": 1})
 
     @use_profile('bigquery')
     def test_bigquery_add_partition_year(self):
-        before = {"partition_by": None, "cluster_by": None}
-        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'year'}, "cluster_by": None}
+        before = {"partition_by": None,
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'year'},
+                 "cluster_by": None,
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)
         self.test_partitions({"expected": 1})
 
     @use_profile('bigquery')
     def test_bigquery_add_partition_month(self):
-        before = {"partition_by": None, "cluster_by": None}
-        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'month'}, "cluster_by": None}
+        before = {"partition_by": None,
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'month'},
+                 "cluster_by": None,
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)
         self.test_partitions({"expected": 1})
 
     @use_profile('bigquery')
     def test_bigquery_add_partition_hour(self):
-        before = {"partition_by": None, "cluster_by": None}
-        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'hour'}, "cluster_by": None}
+        before = {"partition_by": None,
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'hour'},
+                 "cluster_by": None,
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
+        self.run_changes(before, after)
+        self.test_partitions({"expected": 1})
+
+    @use_profile('bigquery')
+    def test_bigquery_add_partition_hour(self):
+        before = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'day'},
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'hour'},
+                 "cluster_by": None,
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)
         self.test_partitions({"expected": 1})
 
     @use_profile('bigquery')
     def test_bigquery_remove_partition(self):
-        before = {"partition_by": {'field': 'cur_time',
-                                   'data_type': 'timestamp'}, "cluster_by": None}
-        after = {"partition_by": None, "cluster_by": None}
+        before = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp'},
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": None,
+                 "cluster_by": None,
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)
 
     @use_profile('bigquery')
     def test_bigquery_change_partitions(self):
-        before = {"partition_by": {'field': 'cur_time',
-                                   'data_type': 'timestamp'}, "cluster_by": None}
-        after = {"partition_by": {'field': "cur_date"}, "cluster_by": None}
+        before = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp'},
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': "cur_date"},
+                 "cluster_by": None,
+                 'partition_expiration_days': 7,
+                 'require_partition_filter': True}
         self.run_changes(before, after)
         self.test_partitions({"expected": 1})
         self.run_changes(after, before)
@@ -75,10 +121,14 @@ def test_bigquery_change_partitions(self):
 
     @use_profile('bigquery')
     def test_bigquery_change_partitions_from_int(self):
-        before = {"partition_by": {"field": "id", "data_type": "int64", "range": {
-            "start": 0, "end": 10, "interval": 1}}, "cluster_by": None}
-        after = {"partition_by": {"field": "cur_date",
-                                  "data_type": "date"}, "cluster_by": None}
+        before = {"partition_by": {"field": "id", "data_type": "int64", "range": {"start": 0, "end": 10, "interval": 1}},
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {"field": "cur_date", "data_type": "date"},
+                 "cluster_by": None,
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)
         self.test_partitions({"expected": 1})
         self.run_changes(after, before)
@@ -86,29 +136,48 @@ def test_bigquery_change_partitions_from_int(self):
 
     @use_profile('bigquery')
     def test_bigquery_add_clustering(self):
-        before = {"partition_by": {'field': 'cur_time',
-                                   'data_type': 'timestamp'}, "cluster_by": None}
-        after = {"partition_by": {'field': "cur_date"}, "cluster_by": "id"}
+        before = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp'},
+                  "cluster_by": None,
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': "cur_date"},
+                 "cluster_by": "id",
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)
 
     @use_profile('bigquery')
     def test_bigquery_remove_clustering(self):
-        before = {"partition_by": {'field': 'cur_time',
-                                   'data_type': 'timestamp'}, "cluster_by": "id"}
-        after = {"partition_by": {'field': "cur_date"}, "cluster_by": None}
+        before = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp'},
+                  "cluster_by": "id",
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': "cur_date"},
+                 "cluster_by": None,
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)
 
     @use_profile('bigquery')
     def test_bigquery_change_clustering(self):
-        before = {"partition_by": {'field': 'cur_time',
-                                   'data_type': 'timestamp'}, "cluster_by": "id"}
-        after = {"partition_by": {'field': "cur_date"}, "cluster_by": "name"}
+        before = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp'},
+                  "cluster_by": "id",
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {"partition_by": {'field': "cur_date"},
+                 "cluster_by": "name",
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)
 
     @use_profile('bigquery')
     def test_bigquery_change_clustering_strict(self):
-        before = {'partition_by': {'field': 'cur_time',
-                                   'data_type': 'timestamp'}, 'cluster_by': 'id'}
-        after = {'partition_by': {'field': 'cur_date',
-                                  'data_type': 'date'}, 'cluster_by': 'name'}
+        before = {'partition_by': {'field': 'cur_time', 'data_type': 'timestamp'},
+                  'cluster_by': 'id',
+                  'partition_expiration_days': None,
+                  'require_partition_filter': None}
+        after = {'partition_by': {'field': 'cur_date', 'data_type': 'date'},
+                 'cluster_by': 'name',
+                 'partition_expiration_days': None,
+                 'require_partition_filter': None}
         self.run_changes(before, after)