Repository: dbt-labs/dbt-core

Issue:
Add maximum_bytes_billed option for BigQuery
### Describe the feature
BigQuery has the query option maximum_bytes_billed, which sets a limit for the maximum acceptable cost for individual queries. Any query that exceeds this cost will automatically fail. It would be nice if one could use this feature in DBT to guard against potentially expensive mistakes.

I suggest adding this option in the profile settings, where one can already set the BigQuery query priority and other BigQuery specific settings.

Let me know if you would like to get a pull request for this. It is a small change and I'd be happy to implement it.

### Describe alternatives you've considered
I have researched ways to try and set this option via environment variables or ~/.bigqueryrc. While one can set the option in bigqueryrc, that does not influence the BigQuery client SDK, only the BigQuery commandline tools (which does not help for DBT).

### Additional context
This is BigQuery specific.

### Who will this benefit?
Consider a table with a million rows that you join with another table with a million rows. You mess up the join condition and accidentally create a full cross join. Since BigQuery has huge scaling capabilities, it will happily create the resulting table with 10^11 rows. Any scan of that table will be hugely expensive. This option would protect against such mistakes.



Discussion:
hey @haukeduden - cool idea! Sure, please feel free to send through a PR for this change.

I think `maximum_bytes_billed` would be a good profile config name. If the value is not provided, or if it is `0`, then dbt should not configure a value for `maximum_bytes_billed` in a query.

Thanks for opening this issue!
@drewbanin Which branch should I use as the base for the PR? It seems that master is quite outdated ... is feature/dbt-project-v2 the right starting point maybe?
@haukeduden `dev/octavius-catto` should be the "default branch" on github - please branch off of that!

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/connections.py b/plugins/bigquery/dbt/adapters/bigquery/connections.py
--- a/plugins/bigquery/dbt/adapters/bigquery/connections.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/connections.py
@@ -43,6 +43,7 @@ class BigQueryCredentials(Credentials):
     location: Optional[str] = None
     priority: Optional[Priority] = None
     retries: Optional[int] = 1
+    maximum_bytes_billed: Optional[int] = None
     _ALIASES = {
         'project': 'database',
         'dataset': 'schema',
@@ -54,7 +55,7 @@ def type(self):
 
     def _connection_keys(self):
         return ('method', 'database', 'schema', 'location', 'priority',
-                'timeout_seconds')
+                'timeout_seconds', 'maximum_bytes_billed')
 
 
 class BigQueryConnectionManager(BaseConnectionManager):
@@ -216,6 +217,10 @@ def raw_execute(self, sql, fetch=False):
             job_params[
                 'priority'] = google.cloud.bigquery.QueryPriority.INTERACTIVE
 
+        maximum_bytes_billed = conn.credentials.maximum_bytes_billed
+        if maximum_bytes_billed is not None and maximum_bytes_billed != 0:
+            job_params['maximum_bytes_billed'] = maximum_bytes_billed
+
         def fn():
             return self._query_and_results(client, sql, conn, job_params)
 


Test patch:
diff --git a/test/unit/test_bigquery_adapter.py b/test/unit/test_bigquery_adapter.py
--- a/test/unit/test_bigquery_adapter.py
+++ b/test/unit/test_bigquery_adapter.py
@@ -58,6 +58,7 @@ def setUp(self):
                     'threads': 1,
                     'location': 'Luna Station',
                     'priority': 'batch',
+                    'maximum_bytes_billed': 0,
                 },
             },
             'target': 'oauth',
@@ -147,6 +148,21 @@ def test_acquire_connection_priority(self, mock_open_connection):
         connection.handle
         mock_open_connection.assert_called_once()
 
+    @patch('dbt.adapters.bigquery.BigQueryConnectionManager.open', return_value=_bq_conn())
+    def test_acquire_connection_maximum_bytes_billed(self, mock_open_connection):
+        adapter = self.get_adapter('loc')
+        try:
+            connection = adapter.acquire_connection('dummy')
+            self.assertEqual(connection.type, 'bigquery')
+            self.assertEqual(connection.credentials.maximum_bytes_billed, 0)
+
+        except dbt.exceptions.ValidationException as e:
+            self.fail('got ValidationException: {}'.format(str(e)))
+
+        mock_open_connection.assert_not_called()
+        connection.handle
+        mock_open_connection.assert_called_once()
+
     def test_cancel_open_connections_empty(self):
         adapter = self.get_adapter('oauth')
         self.assertEqual(adapter.cancel_open_connections(), None)