Repository: dbt-labs/dbt-core

Issue:
Add support for BigQuery hourly partitioned tables
### Describe the feature
BigQuery [hourly partitioned tables](https://cloud.google.com/bigquery/docs/creating-column-partitions#daily_partitioning_vs_hourly_partitioning) are now in beta.

```sql
 CREATE TABLE
   mydataset.newtable (transaction_id INT64,
     transaction_ts TIMESTAMP)
 PARTITION BY
   TIMESTAMP_TRUNC(transaction_ts, HOUR)
```

Benefits:
* Greater control over partition pruning when querying timeseries data
* The `insert_overwrite` incremental strategy could replace partitions at a finer level of precision

### Required change

We need to:
* Find a decent way to reflect hourly partitioning in the `partition_by` spec
* Update `partition_by.render()` to avoid wrapping the partition column in `date()` if the user has configured hourly partitioning

Since 0.16.0, the `partition_by` spec has looked like:

```sql
{{ config(
      partition_by = {
        'field': 'my_date_or_ts_column',
        'data_type': 'date' or 'timestamp'
      }
}}
```

The `bq` API will reflect hourly partitioning like:
```json
"timePartitioning": {
        "field": "ts_column",
        "type": "HOUR"
      },
```

Quick implementation thoughts:
* I'm not a _huge_ fan of adding a `type` attribute separate from `data_type`
* If dbt accepted `'data_type': 'hour'`, would it assume that the `field` has _already_ been truncated, or that `field` is a timestamp in need of truncation?

### Who will this benefit?
* BigQuery users



Discussion:
+1 This support would be great to have!
+1 This would be great, especially if it extended to support month and year partitioning as well.
This is GA now, and I would love support for hour, day, month, year as well. What about adding `granularity` option and then default is day just like BQ API? BQ also supports `datetime` now.
> What about adding `granularity` option and then default is day just like BQ API?

I like it! This could look like:
```
{{ config(
    materialized = 'table',
    partition_by = {
      'field': 'my_column',
      'data_type': 'datetime',
      'granularity': 'hour'
    }
}}
```
Then dbt would know to return `create table ... partition by datetime_trunc(my_column, hour) as ...`.

Is anyone interested in contributing this change? I think it could be a fairly straightforward update to this code:

https://github.com/fishtown-analytics/dbt/blob/72038258ed4b5cb7830734ffee3f43376075d9f3/plugins/bigquery/dbt/adapters/bigquery/impl.py#L49-L63

Plus adding an integration test [here](https://github.com/fishtown-analytics/dbt/blob/dev/kiyoshi-kuromiya/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py) :)

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/impl.py b/plugins/bigquery/dbt/adapters/bigquery/impl.py
--- a/plugins/bigquery/dbt/adapters/bigquery/impl.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/impl.py
@@ -50,6 +50,7 @@ def sql_escape(string):
 class PartitionConfig(JsonSchemaMixin):
     field: str
     data_type: str = 'date'
+    granularity: str = 'day'
     range: Optional[Dict[str, Any]] = None
 
     def render(self, alias: Optional[str] = None):
@@ -57,10 +58,11 @@ def render(self, alias: Optional[str] = None):
         if alias:
             column = f'{alias}.{self.field}'
 
-        if self.data_type in ('timestamp', 'datetime'):
-            return f'date({column})'
-        else:
+        if self.data_type.lower() == 'date' and \
+                self.granularity.lower() == 'day':
             return column
+        else:
+            return f'{self.data_type}_trunc({column}, {self.granularity})'
 
     @classmethod
     def parse(cls, raw_partition_by) -> Optional['PartitionConfig']:
@@ -547,7 +549,9 @@ def _partitions_match(
             return True
         elif conf_partition and table.time_partitioning is not None:
             table_field = table.time_partitioning.field
-            return table_field == conf_partition.field
+            table_granularity = table.partitioning_type
+            return table_field == conf_partition.field \
+                and table_granularity == conf_partition.granularity
         elif conf_partition and table.range_partitioning is not None:
             dest_part = table.range_partitioning
             conf_part = conf_partition.range or {}


Test patch:
diff --git a/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py b/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py
--- a/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py
+++ b/test/integration/022_bigquery_test/test_bigquery_changing_partitions.py
@@ -34,7 +34,35 @@ def test_bigquery_add_partition(self):
         after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp'}, "cluster_by": None}
         self.run_changes(before, after)
         self.test_partitions({"expected": 1})
-        
+
+    @use_profile('bigquery')
+    def test_bigquery_add_partition_year(self):
+        before = {"partition_by": None, "cluster_by": None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'year'}, "cluster_by": None}
+        self.run_changes(before, after)
+        self.test_partitions({"expected": 1})
+
+    @use_profile('bigquery')
+    def test_bigquery_add_partition_month(self):
+        before = {"partition_by": None, "cluster_by": None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'month'}, "cluster_by": None}
+        self.run_changes(before, after)
+        self.test_partitions({"expected": 1})
+
+    @use_profile('bigquery')
+    def test_bigquery_add_partition_hour(self):
+        before = {"partition_by": None, "cluster_by": None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'hour'}, "cluster_by": None}
+        self.run_changes(before, after)
+        self.test_partitions({"expected": 1})
+
+    @use_profile('bigquery')
+    def test_bigquery_add_partition_hour(self):
+        before = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'day'}, "cluster_by": None}
+        after = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp', 'granularity': 'hour'}, "cluster_by": None}
+        self.run_changes(before, after)
+        self.test_partitions({"expected": 1})
+
     @use_profile('bigquery')
     def test_bigquery_remove_partition(self):
         before = {"partition_by": {'field': 'cur_time', 'data_type': 'timestamp'}, "cluster_by": None}
diff --git a/test/unit/test_bigquery_adapter.py b/test/unit/test_bigquery_adapter.py
--- a/test/unit/test_bigquery_adapter.py
+++ b/test/unit/test_bigquery_adapter.py
@@ -652,7 +652,19 @@ def test_parse_partition_by(self):
                 "field": "ts",
             }).to_dict(), {
                 "field": "ts",
-                "data_type": "date"
+                "data_type": "date",
+                "granularity": "day"
+            }
+        )
+
+        self.assertEqual(
+            adapter.parse_partition_by({
+                "field": "ts",
+                "data_type": "date",
+            }).to_dict(), {
+                "field": "ts",
+                "data_type": "date",
+                "granularity": "day"
             }
         )
 
@@ -660,9 +672,103 @@ def test_parse_partition_by(self):
             adapter.parse_partition_by({
                 "field": "ts",
                 "data_type": "date",
+                "granularity": "MONTH"
+
+            }).to_dict(), {
+                "field": "ts",
+                "data_type": "date",
+                "granularity": "MONTH"
+            }
+        )
+        
+        self.assertEqual(
+            adapter.parse_partition_by({
+                "field": "ts",
+                "data_type": "date",
+                "granularity": "YEAR"
+
+            }).to_dict(), {
+                "field": "ts",
+                "data_type": "date",
+                "granularity": "YEAR"
+            }
+        )
+
+        self.assertEqual(
+            adapter.parse_partition_by({
+                "field": "ts",
+                "data_type": "timestamp",
+                "granularity": "HOUR"
+
+            }).to_dict(), {
+                "field": "ts",
+                "data_type": "timestamp",
+                "granularity": "HOUR"
+            }
+        )
+
+        self.assertEqual(
+            adapter.parse_partition_by({
+                "field": "ts",
+                "data_type": "timestamp",
+                "granularity": "MONTH"
+
+            }).to_dict(), {
+                "field": "ts",
+                "data_type": "timestamp",
+                "granularity": "MONTH"
+            }
+        )
+
+        self.assertEqual(
+            adapter.parse_partition_by({
+                "field": "ts",
+                "data_type": "timestamp",
+                "granularity": "YEAR"
+
+            }).to_dict(), {
+                "field": "ts",
+                "data_type": "timestamp",
+                "granularity": "YEAR"
+            }
+        )
+
+        self.assertEqual(
+            adapter.parse_partition_by({
+                "field": "ts",
+                "data_type": "datetime",
+                "granularity": "HOUR"
+
+            }).to_dict(), {
+                "field": "ts",
+                "data_type": "datetime",
+                "granularity": "HOUR"
+            }
+        )
+
+        self.assertEqual(
+            adapter.parse_partition_by({
+                "field": "ts",
+                "data_type": "datetime",
+                "granularity": "MONTH"
+
+            }).to_dict(), {
+                "field": "ts",
+                "data_type": "datetime",
+                "granularity": "MONTH"
+            }
+        )
+
+        self.assertEqual(
+            adapter.parse_partition_by({
+                "field": "ts",
+                "data_type": "datetime",
+                "granularity": "YEAR"
+
             }).to_dict(), {
                 "field": "ts",
-                "data_type": "date"
+                "data_type": "datetime",
+                "granularity": "YEAR"
             }
         )
 
@@ -683,6 +789,7 @@ def test_parse_partition_by(self):
             }).to_dict(), {
                 "field": "id",
                 "data_type": "int64",
+                "granularity": "day",
                 "range": {
                     "start": 1,
                     "end": 100,