Repository: dbt-labs/dbt-core

Issue:
Support TTL for BigQuery tables
### Describe the feature
Add a BigQuery adapter specific config option `ttl` (or similar name) which denotes how long the produced table will exist, before being cleaned up automatically by BigQuery. This will take advantage of the `expiration_timestamp` and the BQ functions `TIMESTAMP_ADD()` `CURRENT_TIMESTAMP()`, and `INTERVAL` (with `ttl` as an argument)

### Describe alternatives you've considered
- Do this as a post-processing step
  - Would work, but this gives better guarantees that any table you create, if configured properly, will not stick around beyond it's TTL.

- #1472 was the idea of having dbt keep track of how long models have existed, and only update once expired, and called itself `TTL`, but is pretty different than this proposal.


### Additional context
AFAIK this is a BigQuery only feature (did some brief googling around and didn't find anything). This feature is already used for `temporary` tables in the BQ adapter, which are somewhat arbitrarily set to 12 hour TTLs. The same feature is not used for the other support databases, which have a `TEMPORARY` option which deletes tables at the end of a session, instead.


### Who will this benefit?
- Anyone who has requirements on how long their data can be kept around.
- Developers configuring projects which may produce a lot of clutter (and can set it to expire in a few hours while developing).

### Are you interested in contributing this feature?
This can be upstreamed from a locally maintained fork without too many changes (so definitely).


Discussion:
@kconvey Thanks for the writeup! I'm supportive of adding this as a model config. What do you think about `time_to_expiration`?

As you rightly observe, we already sort of have it, in that temporary tables are actually non-temporary tables with an arbitrary 12-hour TTL. I'd actually like to change that, and to use real temporary tables (i.e. BigQuery scripts) wherever we currently use 12-hours-to-live tables. We tried to do it in #2140, but we found we would have needed to rework snapshots in a way that was out of scope. It's definitely out of scope for this feature, too, but it's still on my mind :)
Planning to create the PR for this pretty soon, but I'm curious how you feel about adding support in `profiles.yml` @jtcohen6 ? An obvious use case for this would be when someone is developing, but doesn't want to manually clean up their tables / have them persist, they can set a dev profile to have a `time_to_expiration`
That's a good use case. I'd be more inclined to support it, still with `time_to_expiration` as a model config, via `dbt_project.yml` code like:
```yml
models:
  +time_to_expiration: "{{ ('8' if target.name == 'dev' else '1000') | as_number }}"
```

I think configs belong in `profiles.yml` if:
* they relate to the specifics mechanisms by which dbt connects to the database
* they are likely to differ for each individual developer
* they should not be checked into version control

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/impl.py b/plugins/bigquery/dbt/adapters/bigquery/impl.py
--- a/plugins/bigquery/dbt/adapters/bigquery/impl.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/impl.py
@@ -104,6 +104,7 @@ class BigqueryConfig(AdapterConfig):
     labels: Optional[Dict[str, str]] = None
     partitions: Optional[List[str]] = None
     grant_access_to: Optional[List[Dict[str, str]]] = None
+    hours_to_expiration: Optional[int] = None
 
 
 class BigQueryAdapter(BaseAdapter):
@@ -745,6 +746,12 @@ def get_table_options(
             expiration = 'TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 12 hour)'
             opts['expiration_timestamp'] = expiration
 
+        if (config.get('hours_to_expiration') is not None) and (not temporary):
+            expiration = (
+                'TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL '
+                '{} hour)').format(config.get('hours_to_expiration'))
+            opts['expiration_timestamp'] = expiration
+
         if config.persist_relation_docs() and 'description' in node:
             description = sql_escape(node['description'])
             opts['description'] = '"""{}"""'.format(description)


Test patch:
diff --git a/test/integration/022_bigquery_test/adapter-specific-models/expiring_table.sql b/test/integration/022_bigquery_test/adapter-specific-models/expiring_table.sql
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/adapter-specific-models/expiring_table.sql
@@ -0,0 +1 @@
+select 1 as id
diff --git a/test/integration/022_bigquery_test/test_bigquery_adapter_specific.py b/test/integration/022_bigquery_test/test_bigquery_adapter_specific.py
new file mode 100644
--- /dev/null
+++ b/test/integration/022_bigquery_test/test_bigquery_adapter_specific.py
@@ -0,0 +1,38 @@
+""""Test adapter specific config options."""
+from test.integration.base import DBTIntegrationTest, use_profile
+import textwrap
+import yaml
+
+
+class TestBigqueryAdapterSpecific(DBTIntegrationTest):
+
+    @property
+    def schema(self):
+        return "bigquery_test_022"
+
+    @property
+    def models(self):
+        return "adapter-specific-models"
+
+    @property
+    def profile_config(self):
+        return self.bigquery_profile()
+
+    @property
+    def project_config(self):
+        return yaml.safe_load(textwrap.dedent('''\
+        config-version: 2
+        models:
+            test:
+                materialized: table
+                expiring_table:
+                    hours_to_expiration: 4    
+        '''))
+
+    @use_profile('bigquery')
+    def test_bigquery_hours_to_expiration(self):
+        _, stdout = self.run_dbt_and_capture(['--debug', 'run'])
+        
+        self.assertIn(
+            'expiration_timestamp=TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL '
+            '4 hour)', stdout)
diff --git a/test/unit/test_bigquery_adapter.py b/test/unit/test_bigquery_adapter.py
--- a/test/unit/test_bigquery_adapter.py
+++ b/test/unit/test_bigquery_adapter.py
@@ -4,7 +4,7 @@
 import unittest
 from contextlib import contextmanager
 from requests.exceptions import ConnectionError
-from unittest.mock import patch, MagicMock, Mock
+from unittest.mock import patch, MagicMock, Mock, create_autospec
 
 import hologram
 
@@ -571,6 +571,35 @@ def test_parse_partition_by(self):
             }
         )
 
+    def test_hours_to_expiration(self):
+        adapter = self.get_adapter('oauth')
+        mock_config = create_autospec(
+            dbt.context.providers.RuntimeConfigObject)
+        config = {'hours_to_expiration': 4}
+        mock_config.get.side_effect = lambda name: config.get(name)
+
+        expected = {
+            'expiration_timestamp': 'TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 4 hour)',
+        }
+        actual = adapter.get_table_options(mock_config, node={}, temporary=False)
+        self.assertEqual(expected, actual)
+
+
+    def test_hours_to_expiration_temporary(self):
+        adapter = self.get_adapter('oauth')
+        mock_config = create_autospec(
+            dbt.context.providers.RuntimeConfigObject)
+        config={'hours_to_expiration': 4}
+        mock_config.get.side_effect = lambda name: config.get(name)
+
+        expected = {
+            'expiration_timestamp': (
+                'TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL 12 hour)'),
+        }
+        actual = adapter.get_table_options(mock_config, node={}, temporary=True)
+        self.assertEqual(expected, actual)
+
+
 
 class TestBigQueryFilterCatalog(unittest.TestCase):
     def test__catalog_filter_table(self):