Repository: dbt-labs/dbt-core

Issue:
[BigQuery] Raw token authentication method
### Describe the feature
The existing auth methods for BigQuery provide a great experience for interactive users, allowing them to transparently provision access tokens in a variety of ways.  However, as an automation implementer on a team trying to programmatically wield dbt on behalf of customers, I'd like to be able to bypass these niceties and just inject an externally provisioned token for access.

In our typical non-dbt scenarios, we create service accounts for customers, have them grant specific limited permissions to those accounts for our service's operations, then use our master account to issue scoped tokens for those accounts/operations when they need to execute.  When running dbt operations, I'd like to avoid writing our master account credentials file to disk (as required by the `service-account`/`service-account-json` methods) to protect against reflected file attacks and/or potential vulnerabilities in dbt itself, which could potentially exfiltrate these creds.  While I don't see these as particularly likely scenarios, I'd love to have the ability to directly control the blast radius via smaller scoped credentials.

I propose a new BigQuery auth method named 'service-token' and an additional field named `service_token` to provide its value, e.g.:
```
my-bigquery-db:
  target: dev
  outputs:
    dev:
      type: bigquery

      # something like this?
      method: service-token
      service_token: 'ya29.c.KqwC3gfRlXO...'

      project: [GCP project id]
      dataset: [the name of your dbt dataset]
      threads: [1 or more]
      timeout_seconds: 300
      priority: interactive
      retries: 1
```

### Describe alternatives you've considered
With the addition of the `impersonate_service_account` field added in 0.18.0, I can connect as delegated services, but still need to provide the full master service keyfile rather than a more limited credential.

### Additional context
This is specific to BigQuery database connections and automation use-cases.  I think it's fair to say that standard interactive users would almost never want to use this authentication mode directly.

### Who will this benefit?
* Security-minded SaaS automators who are targeting multi-warehouse and connecting to BQ on behalf of customers
* IT/Dataeng staff using automation to provision limited credentials as part of employee access control

### Are you interested in contributing this feature?
Yes, happy to contribute to making this a reality.  I have a simple (if slightly hacky) proof-of-concept that I can polish into a real PR if this seems interesting.



Discussion:
@davehughes I'm broadly supportive of the need you're describing. To clarify what you mean by service token: it sounds like this is a temporary access token generated via oauth, as in [generateAccessToken](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/generateAccessToken), and that you handle generating and refreshing tokens through a separate automated process. Is that right?

The reason I ask is because we're planning to work later this year on supporting BigQuery oauth as a connection mechanism (https://github.com/fishtown-analytics/dbt/issues/2344), with the goal of hooking into GSuite SSO in dbt Cloud. I could be wrong, but it sounds like there's potential overlap between the core work we'd need there and an even more robust version of the auth mechanism you're interested in.
Yes, we use [generateAccessToken](https://cloud.google.com/iam/docs/reference/credentials/rest/v1/projects.serviceAccounts/generateAccessToken) to generate the tokens that we'd want to pass in here, and your general understanding is correct.

In #2344, dbt still has creds to connect to the remote service and generate/fetch tokens, each of which is used for service auth until the token expires (at which point it can be refreshed with another call to the token_uri).  Here, the token is completely opaque - not refreshable, may not have the right scopes, may already be expired, etc. - and it's the job of the external automated process to make sure scopes and expiry are properly set up.  dbt would just use the token directly for service auth or die trying.
> or die trying

😇 

@davehughes I checked out the docs that you linked to, and it sure seems like this one is doable. I see:
> The desired lifetime duration of the access token in seconds.
>
> By default, the maximum allowed value is 1 hour. To set a lifetime of up to 12 hours, you can add the service account as an allowed value in an Organization Policy that enforces the constraints/iam.allowServiceAccountCredentialLifetimeExtension constraint. See detailed instructions at https://cloud.google.com/iam/help/credentials/lifetime

The thing to watch out here would be the failure mode where a `dbt run` takes longer than the lifetime of the token. Sometimes access tokens have _very_ short TTLs, but 1-12 hours seems super appropriate to me.

> I have a simple (if slightly hacky) proof-of-concept that I can polish into a real PR if this seems interesting.

Can you just share a tiny bit about how you're creating a Credentials object in your proof-of-concept code? Are you creating a [google.oauth2.credentials.Credentials](https://google-auth.readthedocs.io/en/stable/reference/google.oauth2.credentials.html) object directly, and if so, are you just specifying a `token` to that constructor?

If so, then I think this change will indeed pair really well with #2344  (though there are two distinct code changes!). If that sounds about right to you, then we'd be really happy to accept a PR for this one :)
Hmm...I hadn't seen the doc on the max access token lifetime, which might throw a bit of a wrench in here, though setting up the lifetime extension and choosing an appropriately long lifetime won't be a problem.  We don't usually see `dbt run` invocations that take longer than an hour, but I can definitely imagine they exist (and even 12 hours+).  

As for creating a Credentials object, I resorted to just creating a small class that quacks like it should (this is the hacky bit!):
```
class RawServiceCredentials(credentials.Credentials)
    def __init__(self, token, expiry=None):
        self.token = token
        # This keeps users of the credential happy that it's not expired and doesn't need a refresh() call
        self.expiry = expiry or datetime.datetime.now() + datetime.timedelta(days=1)

    # This is here to fulfill an interface check I think, and should maybe raise NotImplementedError
    def refresh():
        pass
```
Yeah - I think that's fair. I do think that dbt runs which take > 1 hour are not-uncommon, but I think you get to add constraints around things like this for security-minded features!

I bet we can swing this implementation using the `google.oauth2.credentials.Credentials` object. Hopefully you can just pass in a token and everything else will work from there! I'm going to PR a change for #2344 which might serve as a good template for a change like this one if that's helpful at all - happy to link to it here when it's up!

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/connections.py b/plugins/bigquery/dbt/adapters/bigquery/connections.py
--- a/plugins/bigquery/dbt/adapters/bigquery/connections.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/connections.py
@@ -9,7 +9,10 @@
 import google.cloud.exceptions
 from google.api_core import retry, client_info
 from google.auth import impersonated_credentials
-from google.oauth2 import service_account
+from google.oauth2 import (
+    credentials as GoogleCredentials,
+    service_account as GoogleServiceAccountCredentials
+)
 
 from dbt.utils import format_bytes, format_rows_number
 from dbt.clients import agate_helper, gcloud
@@ -51,19 +54,30 @@ class BigQueryConnectionMethod(StrEnum):
     OAUTH = 'oauth'
     SERVICE_ACCOUNT = 'service-account'
     SERVICE_ACCOUNT_JSON = 'service-account-json'
+    OAUTH_SECRETS = 'oauth-secrets'
 
 
 @dataclass
 class BigQueryCredentials(Credentials):
     method: BigQueryConnectionMethod
-    keyfile: Optional[str] = None
-    keyfile_json: Optional[Dict[str, Any]] = None
     timeout_seconds: Optional[int] = 300
     location: Optional[str] = None
     priority: Optional[Priority] = None
     retries: Optional[int] = 1
     maximum_bytes_billed: Optional[int] = None
     impersonate_service_account: Optional[str] = None
+
+    # Keyfile json creds
+    keyfile: Optional[str] = None
+    keyfile_json: Optional[Dict[str, Any]] = None
+
+    # oauth-secrets
+    token: Optional[str] = None
+    refresh_token: Optional[str] = None
+    client_id: Optional[str] = None
+    client_secret: Optional[str] = None
+    token_uri: Optional[str] = None
+
     _ALIASES = {
         'project': 'database',
         'dataset': 'schema',
@@ -111,12 +125,13 @@ def exception_handler(self, sql):
             message = "Access denied while running query"
             self.handle_error(e, message)
 
-        except google.auth.exceptions.RefreshError:
+        except google.auth.exceptions.RefreshError as e:
             message = "Unable to generate access token, if you're using " \
                       "impersonate_service_account, make sure your " \
                       'initial account has the "roles/' \
                       'iam.serviceAccountTokenCreator" role on the ' \
-                      'account you are trying to impersonate.'
+                      'account you are trying to impersonate.\n\n' \
+                      f'{str(e)}'
             raise RuntimeException(message)
 
         except Exception as e:
@@ -152,7 +167,7 @@ def commit(self):
     @classmethod
     def get_bigquery_credentials(cls, profile_credentials):
         method = profile_credentials.method
-        creds = service_account.Credentials
+        creds = GoogleServiceAccountCredentials.Credentials
 
         if method == BigQueryConnectionMethod.OAUTH:
             credentials, project_id = google.auth.default(scopes=cls.SCOPE)
@@ -166,6 +181,16 @@ def get_bigquery_credentials(cls, profile_credentials):
             details = profile_credentials.keyfile_json
             return creds.from_service_account_info(details, scopes=cls.SCOPE)
 
+        elif method == BigQueryConnectionMethod.OAUTH_SECRETS:
+            return GoogleCredentials.Credentials(
+                token=profile_credentials.token,
+                refresh_token=profile_credentials.refresh_token,
+                client_id=profile_credentials.client_id,
+                client_secret=profile_credentials.client_secret,
+                token_uri=profile_credentials.token_uri,
+                scopes=cls.SCOPE
+            )
+
         error = ('Invalid `method` in profile: "{}"'.format(method))
         raise FailedToConnectException(error)
 


Test patch:
diff --git a/test/unit/test_bigquery_adapter.py b/test/unit/test_bigquery_adapter.py
--- a/test/unit/test_bigquery_adapter.py
+++ b/test/unit/test_bigquery_adapter.py
@@ -71,6 +71,31 @@ def setUp(self):
                     'threads': 1,
                     'impersonate_service_account': 'dummyaccount@dbt.iam.gserviceaccount.com'
                 },
+                'oauth-credentials-token': {
+                    'type': 'bigquery',
+                    'method': 'oauth-secrets',
+                    'token': 'abc',
+                    'project': 'dbt-unit-000000',
+                    'schema': 'dummy_schema',
+                    'threads': 1,
+                    'location': 'Luna Station',
+                    'priority': 'batch',
+                    'maximum_bytes_billed': 0,
+                },
+                'oauth-credentials': {
+                    'type': 'bigquery',
+                    'method': 'oauth-secrets',
+                    'client_id': 'abc',
+                    'client_secret': 'def',
+                    'refresh_token': 'ghi',
+                    'token_uri': 'jkl',
+                    'project': 'dbt-unit-000000',
+                    'schema': 'dummy_schema',
+                    'threads': 1,
+                    'location': 'Luna Station',
+                    'priority': 'batch',
+                    'maximum_bytes_billed': 0,
+                },
             },
             'target': 'oauth',
         }
@@ -145,6 +170,40 @@ def test_acquire_connection_service_account_validations(self, mock_open_connecti
         connection.handle
         mock_open_connection.assert_called_once()
 
+    @patch('dbt.adapters.bigquery.BigQueryConnectionManager.open', return_value=_bq_conn())
+    def test_acquire_connection_oauth_token_validations(self, mock_open_connection):
+        adapter = self.get_adapter('oauth-credentials-token')
+        try:
+            connection = adapter.acquire_connection('dummy')
+            self.assertEqual(connection.type, 'bigquery')
+
+        except dbt.exceptions.ValidationException as e:
+            self.fail('got ValidationException: {}'.format(str(e)))
+
+        except BaseException as e:
+            raise
+
+        mock_open_connection.assert_not_called()
+        connection.handle
+        mock_open_connection.assert_called_once()
+
+    @patch('dbt.adapters.bigquery.BigQueryConnectionManager.open', return_value=_bq_conn())
+    def test_acquire_connection_oauth_credentials_validations(self, mock_open_connection):
+        adapter = self.get_adapter('oauth-credentials')
+        try:
+            connection = adapter.acquire_connection('dummy')
+            self.assertEqual(connection.type, 'bigquery')
+
+        except dbt.exceptions.ValidationException as e:
+            self.fail('got ValidationException: {}'.format(str(e)))
+
+        except BaseException as e:
+            raise
+
+        mock_open_connection.assert_not_called()
+        connection.handle
+        mock_open_connection.assert_called_once()
+
     @patch('dbt.adapters.bigquery.BigQueryConnectionManager.open', return_value=_bq_conn())
     def test_acquire_connection_impersonated_service_account_validations(self, mock_open_connection):
         adapter = self.get_adapter('impersonate')