Repository: dbt-labs/dbt-core

Issue:
dbt compile fails against redshift when using multi-threading
### Describe the bug
When I run `dbt compile` against our redshift data warehouse the command fails with the error 
```
KeyError: 'endpoint_resolver'
```
The error only occurs when threads > 1 and `method: iam` is used.

From what I can gather this is happening because the boto session object is not thread safe and it is being accessed from multiple threads without being protected, and this happens during the call to get_tmp_iam_cluster_credentials.

### Steps To Reproduce
* Create a dbt project containing a significant number of models.
* Configure the target in the profiles.yml file to point to a redshift dwh, with threads > 1 and method iam:
```
      type: redshift
      method: iam
      threads: 8
      host: xxxx.redshift.amazonaws.com
      cluster_id: xxx
      port: 5439
      user: xxx
      dbname: xxx
      schema: xxx
```
* Run `dbt compile`

### Expected behavior
That `dbt compile` succeeds.

### Screenshots and log output

```
2020-09-14 11:15:23.743840 (MainThread): Traceback (most recent call last):
  File "/Users/xxx/venv/lib/python3.7/site-packages/dbt/adapters/postgres/connections.py", line 46, in exception_handler
    yield
  File "/Users/xxx/venv/lib/python3.7/site-packages/dbt/adapters/sql/connections.py", line 76, in add_query
    cursor = connection.handle.cursor()
  File "/Users/xxx/venv/lib/python3.7/site-packages/dbt/contracts/connection.py", line 69, in handle
    self._handle.resolve(self)
  File "/Users/xxx/venv/lib/python3.7/site-packages/dbt/contracts/connection.py", line 90, in resolve
    return self.opener(connection)
  File "/Users/xxx/venv/lib/python3.7/site-packages/dbt/adapters/postgres/connections.py", line 77, in open
    credentials = cls.get_credentials(connection.credentials)
  File "/Users/xxx/venv/lib/python3.7/site-packages/dbt/adapters/redshift/connections.py", line 152, in get_credentials
    return cls.get_tmp_iam_cluster_credentials(credentials)
  File "/Users/xxx/venv/lib/python3.7/site-packages/dbt/adapters/redshift/connections.py", line 128, in get_tmp_iam_cluster_credentials
    credentials.db_groups,
  File "/Users/xxx/venv/lib/python3.7/site-packages/dbt/adapters/redshift/connections.py", line 93, in fetch_cluster_credentials
    boto_client = boto3.client('redshift')
  File "/Users/xxx/venv/lib/python3.7/site-packages/boto3/__init__.py", line 91, in client
    return _get_default_session().client(*args, **kwargs)
  File "/Users/xxx/venv/lib/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "/Users/xxx/venv/lib/python3.7/site-packages/botocore/session.py", line 828, in create_client
    endpoint_resolver = self._get_internal_component('endpoint_resolver')
  File "/Users/xxx/venv/lib/python3.7/site-packages/botocore/session.py", line 695, in _get_internal_component
    return self._internal_components.get_component(name)
  File "/Users/xxx/venv/lib/python3.7/site-packages/botocore/session.py", line 907, in get_component
    del self._deferred[name]
KeyError: 'endpoint_resolver'
```

Sometimes the error returned is 
```
KeyError: 'credential_provider'
```
but the stack trace is identical.

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [x] redshift
- [ ] bigquery
- [ ] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
installed version: 0.17.2
   latest version: 0.18.0

Your version of dbt is out of date! You can find instructions for upgrading here:
https://docs.getdbt.com/docs/installation

Plugins:
  - bigquery: 0.17.2
  - snowflake: 0.17.2
  - redshift: 0.17.2
  - postgres: 0.17.2
```

**The operating system you're using:**
macOS Catalina

**The output of `python --version`:**
Python 3.7.3

### Additional context
The error surfaced after I bumped dbt from version 0.14.2 to 0.17.2



Discussion:
Thanks for the detailed report @jweibel22. What you're saying seems in line with the [boto3 docs](https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html#:~:text=Multithreading%20and%20multiprocessing,import%20boto3%20import%20boto3.&text=Resources%20are%20not%20thread%20safe.). If I understand it right, this indicates that each dbt thread (`open`) should instantiate its own boto3 session, rather than instantiating the boto3 client once to get a set of temporary credentials.

> The error surfaced after I bumped dbt from version 0.14.2 to 0.17.2

Confirming that you were able to connect to Redshift via IAM with multiple threads when running with v0.14.2?

While this is tricky stuff, I think the change here should be fairly self-contained. Is this a fix you'd be interested in contributing?
Yes I can confirm that it was indeed working with version 0.14.2.

Sure, I'll give it a shot and create a PR. Thank you so far :-)


Resolving patch:
diff --git a/plugins/redshift/dbt/adapters/redshift/connections.py b/plugins/redshift/dbt/adapters/redshift/connections.py
--- a/plugins/redshift/dbt/adapters/redshift/connections.py
+++ b/plugins/redshift/dbt/adapters/redshift/connections.py
@@ -99,7 +99,8 @@ def fetch_cluster_credentials(cls, db_user, db_name, cluster_id,
         must already exist in the database, or else an error will occur"""
 
         if iam_profile is None:
-            boto_client = boto3.client('redshift')
+            session = boto3.Session()
+            boto_client = session.client("redshift")
         else:
             logger.debug("Connecting to Redshift using 'IAM'" +
                          f"with profile {iam_profile}")


Test patch:
diff --git a/test/unit/test_redshift_adapter.py b/test/unit/test_redshift_adapter.py
--- a/test/unit/test_redshift_adapter.py
+++ b/test/unit/test_redshift_adapter.py
@@ -1,6 +1,9 @@
 import unittest
 from unittest import mock
+from unittest.mock import Mock
+
 import agate
+import boto3
 
 import dbt.adapters  # noqa
 import dbt.flags as flags
@@ -128,6 +131,23 @@ def test_invalid_iam_no_cluster_id(self):
 
         self.assertTrue("'cluster_id' must be provided" in context.exception.msg)
 
+    def test_default_session_is_not_used_when_iam_used(self):
+        boto3.DEFAULT_SESSION = Mock()
+        self.config.credentials = self.config.credentials.replace(method='iam')
+        self.config.credentials.cluster_id = 'clusterid'
+        with mock.patch('dbt.adapters.redshift.connections.boto3.Session'):
+            RedshiftAdapter.ConnectionManager.get_credentials(self.config.credentials)
+            self.assertEquals(boto3.DEFAULT_SESSION.client.call_count, 0,
+                              "The redshift client should not be created using the default session because the session object is not thread-safe")
+
+    def test_default_session_is_not_used_when_iam_not_used(self):
+        boto3.DEFAULT_SESSION = Mock()
+        self.config.credentials = self.config.credentials.replace(method=None)
+        with mock.patch('dbt.adapters.redshift.connections.boto3.Session'):
+            RedshiftAdapter.ConnectionManager.get_credentials(self.config.credentials)
+            self.assertEquals(boto3.DEFAULT_SESSION.client.call_count, 0,
+                              "The redshift client should not be created using the default session because the session object is not thread-safe")
+
     def test_cancel_open_connections_empty(self):
         self.assertEqual(len(list(self.adapter.cancel_open_connections())), 0)