Repository: dbt-labs/dbt-core

Issue:
Use default google cloud project if not supplied?
### Describe the feature

Currently using BigQuery requires defining your project in `profiles.yml`: https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile/

Google Cloud APIs generally fall back to the default project when one isn't specified. This is helpful for code that runs in multiple project environments — it'll reference the datasets in whatever project it's running in.

So the feature would be to align `dbt` with that standard, and allow for:

```yaml
my-bigquery-db:
  target: dev
  outputs:
    dev:
      type: bigquery
      method: oauth
      # project: [GCP project id] <- uses the current project
      dataset: [the name of your dbt dataset] # You can also use "schema" here
      threads: [1 or more]
      timeout_seconds: 300
      location: US # Optional, one of US or EU
      priority: interactive
      retries: 1
```

### Describe alternatives you've considered

Currently we have something like:

```yaml
nimbus:
  target: main
  outputs:
    user: main:
      type: bigquery
      method: oauth
      project: "{{ env_var('PROJECT', 'project_foo') }}"
```

...and set `$PROJECT` to the result of `gcloud config get-value project`. This is OK, but some cruft.

(and if I'm missing something and there's any easy solution to this, that would be gratefully received!)

### Who will this benefit?
BigQuery users, particularly those running across dev and prod environments

### Are you interested in contributing this feature?
Not right now, given my other OSS work, but would be keen to contribute to dbt at some point!

Thank you!


Discussion:
Thanks for the detailed proposal, @max-sixty. I think it makes sense to fall back to the default project configured by the gcloud user / service account if it's not specified in `profiles.yml`.

This isn't a change we would prioritize, and I'm glad to see you have a workaround in the meantime. I imagine it could be quite straightforward. I'll mark this a good first issue, for whenever you (or another community member) has the time.
I'm happy to have a look into this — any ideas on where to start? I'm not familiar with the code base.
- Should the default be set when `profiles.yml` is parsed? Or when it's used? — Probably when it's used, and then caching it — would reduce the performance cost of shelling out; but maybe it's simplest to do at the parsing stage?
- Where is `profiles.yml` parsed? What's the python object that holds the results?
- dbt already pulls the project [here](https://github.com/fishtown-analytics/dbt/blob/e945bca1d9ea4a7e32144bba462bf61655537264/plugins/bigquery/dbt/adapters/bigquery/connections.py#L173) — could we use that?

You're looking in the right place: `/plugins/bigquery/dbt/adapters/bigquery/connections.py`.

Here is where dbt parses the `database` (with `project` as [alias](https://github.com/fishtown-analytics/dbt/blob/e945bca1d9ea4a7e32144bba462bf61655537264/plugins/bigquery/dbt/adapters/bigquery/connections.py#L81-L84)) out of the profile:
https://github.com/fishtown-analytics/dbt/blob/e945bca1d9ea4a7e32144bba462bf61655537264/plugins/bigquery/dbt/adapters/bigquery/connections.py#L90-L92

And here is where dbt uses that `database` value to generate a connection:
https://github.com/fishtown-analytics/dbt/blob/e945bca1d9ea4a7e32144bba462bf61655537264/plugins/bigquery/dbt/adapters/bigquery/connections.py#L214

In between those two is the code you linked to. I think it'd be a fairly straightforward change to add some logic that checks if database is none, and sets it to the user's default `project_id` accordingly.

Resolving patch:
diff --git a/plugins/bigquery/dbt/adapters/bigquery/connections.py b/plugins/bigquery/dbt/adapters/bigquery/connections.py
--- a/plugins/bigquery/dbt/adapters/bigquery/connections.py
+++ b/plugins/bigquery/dbt/adapters/bigquery/connections.py
@@ -1,7 +1,8 @@
 from contextlib import contextmanager
 from dataclasses import dataclass
+from functools import lru_cache
 from requests.exceptions import ConnectionError
-from typing import Optional, Any, Dict
+from typing import Optional, Any, Dict, Tuple
 
 import google.auth
 import google.auth.exceptions
@@ -45,6 +46,17 @@
 )
 
 
+@lru_cache()
+def get_bigquery_defaults() -> Tuple[Any, Optional[str]]:
+    """
+    Returns (credentials, project_id)
+
+    project_id is returned available from the environment; otherwise None
+    """
+    # Cached, because the underlying implementation shells out, taking ~1s
+    return google.auth.default()
+
+
 class Priority(StrEnum):
     Interactive = 'interactive'
     Batch = 'batch'
@@ -60,6 +72,9 @@ class BigQueryConnectionMethod(StrEnum):
 @dataclass
 class BigQueryCredentials(Credentials):
     method: BigQueryConnectionMethod
+    # BigQuery allows an empty database / project, where it defers to the
+    # environment for the project
+    database: Optional[str]
     timeout_seconds: Optional[int] = 300
     location: Optional[str] = None
     priority: Optional[Priority] = None
@@ -91,6 +106,16 @@ def _connection_keys(self):
         return ('method', 'database', 'schema', 'location', 'priority',
                 'timeout_seconds', 'maximum_bytes_billed')
 
+    def __post_init__(self):
+        # We need to inject the correct value of the database (aka project) at
+        # this stage, ref
+        # https://github.com/fishtown-analytics/dbt/pull/2908#discussion_r532927436.
+
+        # `database` is an alias of `project` in BigQuery
+        if self.database is None:
+            _, database = get_bigquery_defaults()
+            self.database = database
+
 
 class BigQueryConnectionManager(BaseConnectionManager):
     TYPE = 'bigquery'
@@ -170,7 +195,7 @@ def get_bigquery_credentials(cls, profile_credentials):
         creds = GoogleServiceAccountCredentials.Credentials
 
         if method == BigQueryConnectionMethod.OAUTH:
-            credentials, project_id = google.auth.default(scopes=cls.SCOPE)
+            credentials, _ = get_bigquery_defaults()
             return credentials
 
         elif method == BigQueryConnectionMethod.SERVICE_ACCOUNT:


Test patch:
diff --git a/test/unit/test_bigquery_adapter.py b/test/unit/test_bigquery_adapter.py
--- a/test/unit/test_bigquery_adapter.py
+++ b/test/unit/test_bigquery_adapter.py
@@ -96,6 +96,13 @@ def setUp(self):
                     'priority': 'batch',
                     'maximum_bytes_billed': 0,
                 },
+                'oauth--no-project': {
+                    'type': 'bigquery',
+                    'method': 'oauth',
+                    'schema': 'dummy_schema',
+                    'threads': 1,
+                    'location': 'Solar Station',
+                },
             },
             'target': 'oauth',
         }