Repository: astronomer/astronomer-cosmos (Python)

Issue:
[Bug]: BigQuery profile target 'dev' invalid: 'schema' is a required property
### Astronomer Cosmos Version

Other Astronomer Cosmos version (please specify below)

### If "Other Astronomer Cosmos version" selected, which one?

1.4.2

### dbt-core version

1.5.0

### Versions of dbt adapters

_No response_

### LoadMode

AUTOMATIC

### ExecutionMode

LOCAL

### InvocationMode

None

### airflow version

2.9.0

### Operating System

Linux

### If a you think it's an UI issue, what browsers are you seeing the problem on?

_No response_

### Deployment

Official Apache Airflow Helm Chart

### Deployment details

_No response_

### What happened?

my unit tests detected some issues in v1.4.2 that did not occur in v1.4.1 or v1.4.0
from CI:
```python
def test_validate_dags(mocker, config_instance: Config):
    mocker.patch("warnings.simplefilter")

    assert os.environ["AIRFLOW__DATAHUB__ENVIRONMENT"] == "production"
    assert os.environ["AIRFLOW_BUSINESS_UNIT"] == AIRFLOW_BUSINESS_UNIT
    assert (
        config_instance.config.get("bigquery").get("dataset").get("raw")
        == f"raw_{AIRFLOW_BUSINESS_UNIT}"
    )

    mock_get_connection = mocker.patch("airflow.hooks.base.BaseHook.get_connection")
    mock_get_connection.return_value = MockConnection()

    directories_to_scan = get_dag_directories()
    for dir in directories_to_scan:
        dag_bag = DagBag(dir)
        assert len(dag_bag.import_errors) == 0
``` 


### Relevant log output


Error Log:
```
    raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/opt/hostedtoolcache/Python/3.10.14/x64/bin/dbt', 'ls', '--output', 'json', '--select', 'tag:refresh@12', '--vars', "today: '{{ tomorrow_ds }}'\nyesterday: '{{ ds }}'\n", '--project-dir', '/tmp/tmp8_ka9hno', '--profiles-dir', '/tmp/tmpyyfkdhiu', '--profile', 'bigquery_profile', '--target', 'dev'] due to the error:
04:09:09  Running with dbt=1.5.0
04:09:10  Encountered an error:
Runtime Error
  Credentials in profile "bigquery_profile", target "dev" invalid: 'schema' is a required property
```

From local container

```
  File "/home/airflow/.local/lib/python3.10/site-packages/cosmos/dbt/graph.py", line 216, in run_dbt_ls
    stdout = run_command(ls_command, tmp_dir, env_vars)
  File "/home/airflow/.local/lib/python3.10/site-packages/cosmos/dbt/graph.py", line 96, in run_command
    raise CosmosLoadDbtException(f"Unable to run {command} due to the error:\n{details}")
cosmos.dbt.graph.CosmosLoadDbtException: Unable to run ['/home/airflow/.local/bin/dbt', 'ls', '--output', 'json', '--select', 'tag:refresh@6', '--vars', "today: '{{ tomorrow_ds }}'\nyesterday: '{{ ds }}'\n", '--project-dir', '/tmp/tmpa9cdtc8x', '--profiles-dir', '/tmp/tmp5xp_iwzf', '--profile', 'bigquery_profile', '--target', 'prod'] due to the error:
```
```


### How to reproduce

the above test is failing.
if you're wondering, the error message did not print fully, i'm not sure if it's from the package or truncated for some other reason






### Anything else :)?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Contact Details

_No response_


Discussion: @pankajastro I think this bug relates to this change: https://github.com/astronomer/astronomer-cosmos/pull/1017

It seems we are not adding `schema` to the `profiles.yml` file created with the `ProfileMapping`, but that dbt-bigquery adapter expects it to be there:

```
Runtime Error
  Credentials in profile "bigquery_profile", target "dev" invalid: 'schema' is a required property
```
I wonder if it is a specific version of `dbt-bigquery` that requires `schema` to be in profiles. Could we confirm if this happens for newer versions of `dbt-core` and `dbt-bigquery`?

It seems that the issue with the 'schema' in profiles was introduced by the changes made in PR #1017, which was a continuation of the work done in PR #839.

These PRs assumed that the BQ profile should not have to define the dataset and that this value could come from other dbt project files. 

@oliverrmaa @pankajastro , could you help us confirm the original use case and scenario so we can try to accommodate both cases?

The `dbt-bigquery` docs don't seem to be very explicit about which properties should be required and which should be optional: https://docs.getdbt.com/docs/core/connect-data-platform/bigquery-setup

While we don't release the fix, if you want to use 1.4.2, BQ users will have to add the following to their `ProfileMapping`.
```
        profile_args={
            "dataset": "<your-dataset>",
```



Fix (patch):
diff --git a/cosmos/profiles/bigquery/service_account_keyfile_dict.py b/cosmos/profiles/bigquery/service_account_keyfile_dict.py
index 17858d7bb..480db669b 100644
--- a/cosmos/profiles/bigquery/service_account_keyfile_dict.py
+++ b/cosmos/profiles/bigquery/service_account_keyfile_dict.py
@@ -20,8 +20,11 @@ class GoogleCloudServiceAccountDictProfileMapping(BaseProfileMapping):
     dbt_profile_type: str = "bigquery"
     dbt_profile_method: str = "service-account-json"
 
+    # Do not remove dataset as a required field form the below list. Although it's observed that it's not a required
+    # field for some databases like Postgres, it's required for BigQuery.
     required_fields = [
         "project",
+        "dataset",
         "keyfile_json",
     ]