Repository: dbt-labs/dbt-core

Issue:
Add AWS profile option to profiles.yml
### Describe the feature
New option to be allowed on Redshift targets in profiles.yml, specifying the AWS profile to use when getting credentials via IAM.

### Describe alternatives you've considered
Allow specification of full awscli command used to fetch credentials? Not sure

### Additional context
Maybe relevant only for Redshift, but I think other cloud providers would have similar features.

### Who will this benefit?
Increasing security regarding credentials when using DBT.



Discussion:
To allow the complete flexibility for IAM auth it may also be worth also adding the equivalent of:

* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_SESSION_TOKEN` (only valid if the two prior variables are specified)
* `AWS_DEFAULT_REGION`

EC2/ECS credentials will automatically be picked up by the provider chain so don't need any extra setup.

The alternative is to ignore this entirely and point to the boto3 documentation, as all of these can be used by setting the above environment variables before running DBT.

Reference https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html
> The alternative is to ignore this entirely and point to the boto3 documentation, as all of these can be used by setting the above environment variables before running DBT.

I _think_ this is how we've gone about this in the past, when it's been necessary.

* Could you say a bit more about which specific values you'd want to add as options for `method: iam` in the [Redshift profile](https://docs.getdbt.com/docs/supported-databases/profile-redshift/#iam-authentication)?
* How are you deploying dbt?
We have several AWS accounts, however the access key/credentials at the user level are only defined in one account, whereas redshift sits in a different account. Thus I always need to use credentials from account X to access account Y. The way I do that in the awscli is just specifying a profile with `--profile`.

We deploy DBT as an ECS task that runs on a schedule, and that ECS task is invoked using an exclusive IAM role that exists on the same environment as Redshift, and I haven't tried, but I think DBT would work fine with IAM validation when deploying since the credentials and redshift are on the same account.
I want to give it a shot at implementing this!

My idea is to add a new config option 'aws_profile' to profiles.yml, when the method is IAM, and pass that config to the boto3 call that gets the authentication.

Does that sound reasonable?
To be clear, your proposal is that we _shouldn't_ add additional parameters for all of:
* `AWS_ACCESS_KEY_ID`
* `AWS_SECRET_ACCESS_KEY`
* `AWS_SESSION_TOKEN`
* `AWS_DEFAULT_REGION`

Instead, we should expect users to configure those via boto3, and to identify a set of credentials by its profile name. Does that sound right?

If so, that sounds like a quick addition to me. It's all yours @landbay-brunomurino. Let me know if you want help getting started.

You should also look into adding to/updating the Redshift unit test [`test_iam_conn_optionals`](https://github.com/fishtown-analytics/dbt/blob/dev/marian-anderson/test/unit/test_redshift_adapter.py#L88).
Think so, yeah. I think (and correct me if I'm wrong) most users of AWS would have the .aws folder with their credentials and profiles, so simply pointing to the correct profile would enable all sorts of connections.

Thanks! Actually I'm struggling with something @jtcohen6  I've set up the environment but am struggling to run the unit tests specific to redshift, it doesn't connect successfully and I'm not sure I should play too much with how things run.
@landbay-brunomurino You shouldn't need any database connection to run unit tests. Are you able to run `make test-unit`?

If you're trying to run integration tests, you _will_ need to define Redshift credentials as environment variables in a local file called `test.env`. You can copy [`test.env.sample`](https://github.com/fishtown-analytics/dbt/blob/dev/marian-anderson/test.env.sample) as a starting point.
Yea, sorry, I didn't mean unit tests, I meant the integration tests. I was able to run `make test-unit` successfully.

I did define the credentials in the `test.env` but that's not working.

May not be relevant, but I connect to redshift via SSH tunnel, and when running DBT via docker I need to specify an internal docker host ip output of `dig +short host.docker.internal`, instead of `localhost` when not using docker. Since the tests run via docker, I tried putting that IP in the `test.env` file and now I'm getting a different error (I think the connection worked?):
```
No module named 'dbt.deps.local'
```

Got it, that is tricky. I haven't ever tried connecting to Redshift via Docker + SSH; I agree your error message is promising as far as the connection, but unexpected otherwise.

For what it's worth, when you open a PR, CI tests against a sandboxed Redshift cluster will run automatically. As long as you can do some end-to-end testing of your changes locally, I think it's fine to open a fork PR and see.

Resolving patch:
diff --git a/plugins/redshift/dbt/adapters/redshift/connections.py b/plugins/redshift/dbt/adapters/redshift/connections.py
--- a/plugins/redshift/dbt/adapters/redshift/connections.py
+++ b/plugins/redshift/dbt/adapters/redshift/connections.py
@@ -44,6 +44,7 @@ class RedshiftCredentials(PostgresCredentials):
         default=None,
         metadata={'description': 'If using IAM auth, the name of the cluster'},
     )
+    iam_profile: Optional[str] = None
     iam_duration_seconds: int = 900
     search_path: Optional[str] = None
     keepalives_idle: int = 240
@@ -56,7 +57,12 @@ def type(self):
 
     def _connection_keys(self):
         keys = super()._connection_keys()
-        return keys + ('method', 'cluster_id', 'iam_duration_seconds')
+        return keys + (
+            'method',
+            'cluster_id',
+            'iam_profile',
+            'iam_duration_seconds'
+        )
 
 
 class RedshiftConnectionManager(PostgresConnectionManager):
@@ -87,10 +93,20 @@ def fresh_transaction(self, name=None):
 
     @classmethod
     def fetch_cluster_credentials(cls, db_user, db_name, cluster_id,
-                                  duration_s, autocreate, db_groups):
+                                  iam_profile, duration_s, autocreate,
+                                  db_groups):
         """Fetches temporary login credentials from AWS. The specified user
         must already exist in the database, or else an error will occur"""
-        boto_client = boto3.client('redshift')
+
+        if iam_profile is None:
+            boto_client = boto3.client('redshift')
+        else:
+            logger.debug("Connecting to Redshift using 'IAM'" +
+                         f"with profile {iam_profile}")
+            boto_session = boto3.Session(
+                profile_name=iam_profile
+            )
+            boto_client = boto_session.client('redshift')
 
         try:
             return boto_client.get_cluster_credentials(
@@ -123,6 +139,7 @@ def get_tmp_iam_cluster_credentials(cls, credentials):
             credentials.user,
             credentials.database,
             credentials.cluster_id,
+            credentials.iam_profile,
             iam_duration_s,
             credentials.autocreate,
             credentials.db_groups,


Test patch: