Repository: dbt-labs/dbt-core

Issue:
[CT-1233] [Bug] Race condition in lib.py causes concurrent API calls to fail
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

The `lib.py` file in Core contains logic to "reset adapters" in the process of generating a RuntimeConfig object. While this logic is appropriate for single-threaded usage of dbt Core, we've seen that concurrent api calls to this method can result in race conditions:

https://github.com/dbt-labs/dbt-core/blob/56783446dbc765d93c43afad2b763302f034ef6b/core/dbt/lib.py#L27-L32

The `reset_adapters` call will clear out the set of adapters registered in the dbt Core adapter factory. If a concurrent api call accesses adapters (eg. to compile sql, or run dbt, or similar) before the `register_adapters` call is complete, there's a chance it will fail with a `KeyError` eg. [here](https://github.com/dbt-labs/dbt-core/blob/main/core/dbt/adapters/factory.py#L101) because the adapter dictionary was cleared in another thread.

### Expected Behavior

The `get_dbt_config` should not mutate global state to generate a RuntimeConfig.

### Steps To Reproduce

Here's a minimal repro case:

```python
import dbt.lib
import threading
import time
import random
import dbt.adapters.factory

PROJECT_PATH = "/Users/drew/dev/internal-analytics"


def repro_func(thread_no):
    time.sleep(random.random())

    try:
        # Simulate concurrent use of adapters
        config = dbt.lib.get_dbt_config(PROJECT_PATH)
        adapter = dbt.adapters.factory.get_adapter_by_type(config.credentials.type)
        adapter.acquire_connection()
        adapter.execute('select 1 as id')

        print(f"Thread {thread_no} OK")
    except KeyError:
        print("ERROR: KeyError")


threads = []
for i in range(25):
    thread = threading.Thread(target=repro_func, args=(i,))
    thread.start()
    threads.append(thread)

for thread in threads:
    thread.join()
```

<img width="1552" alt="Screen Shot 2022-09-23 at 9 51 27 AM" src="https://user-images.githubusercontent.com/1541139/191976057-18a94daf-4330-4a0a-b9c0-5b5298c57570.png">



### Relevant log output

_No response_

### Environment

```markdown
- OS: macOS
- Python: 3.9.14
- dbt: 1.2.1
```


### Which database adapter are you using with dbt?

postgres, redshift, snowflake, bigquery, spark

### Additional Context

I am interested in submitting a patch for this issue


Discussion:
Thanks for opening @drewbanin!

I think you have a short-term patch for this, which feels appropriate for now.

@iknox-fa and I have been chatting about this a bit, most recently over in #5533. De-globalizing adapters, while not in the immediate scope of API-ification, is something I'd like to tackle in the next few months. I think we can disambiguate between:
- "plugin": a driver, appropriate to register globally at application startup, and then treat as library code for use in specific invocations
- "adapter": an object initialized using plugin (library) code, using config/credentials that might be specific to one invocation — and should not be treated as a singleton registered/cleared globally

> The `get_dbt_config` should not mutate global state to generate a `RuntimeConfig`.

Aside from registering/clearing adapters, there are two other pieces of global state mutation in `get_dbt_config`:
- Resetting the `flags` module (global)
- Resetting the `invocation_id` (global in the `events` module)

Those are both things we'll want to "de-globalize," in addition to adapters, and I sense they would be smaller lifts.

Resolving patch:
diff --git a/core/dbt/lib.py b/core/dbt/lib.py
--- a/core/dbt/lib.py
+++ b/core/dbt/lib.py
@@ -1,10 +1,16 @@
-# TODO: this file is one big TODO
 import os
 from dbt.exceptions import RuntimeException
 from dbt import flags
-from collections import namedtuple
+from dataclasses import dataclass
 
-RuntimeArgs = namedtuple("RuntimeArgs", "project_dir profiles_dir single_threaded profile target")
+
+@dataclass
+class RuntimeArgs:
+    project_dir: str
+    profiles_dir: str
+    single_threaded: bool
+    profile: str
+    target: str
 
 
 def get_dbt_config(project_dir, args=None, single_threaded=False):
@@ -17,27 +23,30 @@ def get_dbt_config(project_dir, args=None, single_threaded=False):
     else:
         profiles_dir = flags.DEFAULT_PROFILES_DIR
 
-    profile = args.profile if hasattr(args, "profile") else None
-    target = args.target if hasattr(args, "target") else None
-
-    # Construct a phony config
-    config = RuntimeConfig.from_args(
-        RuntimeArgs(project_dir, profiles_dir, single_threaded, profile, target)
+    runtime_args = RuntimeArgs(
+        project_dir=project_dir,
+        profiles_dir=profiles_dir,
+        single_threaded=single_threaded,
+        profile=getattr(args, "profile", None),
+        target=getattr(args, "target", None),
     )
-    # Clear previously registered adapters--
-    # this fixes cacheing behavior on the dbt-server
+
+    # Construct a RuntimeConfig from phony args
+    config = RuntimeConfig.from_args(runtime_args)
+
+    # Set global flags from arguments
     flags.set_from_args(args, config)
-    dbt.adapters.factory.reset_adapters()
-    # Load the relevant adapter
+
+    # This is idempotent, so we can call it repeatedly
     dbt.adapters.factory.register_adapter(config)
-    # Set invocation id
+
+    # Make sure we have a valid invocation_id
     dbt.events.functions.set_invocation_id()
 
     return config
 
 
 def get_task_by_type(type):
-    # TODO: we need to tell dbt-server what tasks are available
     from dbt.task.run import RunTask
     from dbt.task.list import ListTask
     from dbt.task.seed import SeedTask
@@ -70,16 +79,13 @@ def create_task(type, args, manifest, config):
     def no_op(*args, **kwargs):
         pass
 
-    # TODO: yuck, let's rethink tasks a little
     task = task(args, config)
-
-    # Wow! We can monkeypatch taskCls.load_manifest to return _our_ manifest
     task.load_manifest = no_op
     task.manifest = manifest
     return task
 
 
-def _get_operation_node(manifest, project_path, sql):
+def _get_operation_node(manifest, project_path, sql, node_name):
     from dbt.parser.manifest import process_node
     from dbt.parser.sql import SqlBlockParser
     import dbt.adapters.factory
@@ -92,26 +98,28 @@ def _get_operation_node(manifest, project_path, sql):
     )
 
     adapter = dbt.adapters.factory.get_adapter(config)
-    # TODO : This needs a real name?
-    sql_node = block_parser.parse_remote(sql, "name")
+    sql_node = block_parser.parse_remote(sql, node_name)
     process_node(config, manifest, sql_node)
     return config, sql_node, adapter
 
 
-def compile_sql(manifest, project_path, sql):
+def compile_sql(manifest, project_path, sql, node_name="query"):
     from dbt.task.sql import SqlCompileRunner
 
-    config, node, adapter = _get_operation_node(manifest, project_path, sql)
+    config, node, adapter = _get_operation_node(manifest, project_path, sql, node_name)
+
     runner = SqlCompileRunner(config, adapter, node, 1, 1)
+
     return runner.safe_run(manifest)
 
 
-def execute_sql(manifest, project_path, sql):
+def execute_sql(manifest, project_path, sql, node_name="query"):
     from dbt.task.sql import SqlExecuteRunner
 
-    config, node, adapter = _get_operation_node(manifest, project_path, sql)
+    config, node, adapter = _get_operation_node(manifest, project_path, sql, node_name)
+
     runner = SqlExecuteRunner(config, adapter, node, 1, 1)
-    # TODO: use same interface for runner
+
     return runner.safe_run(manifest)
 
 
@@ -128,5 +136,4 @@ def deserialize_manifest(manifest_msgpack):
 
 
 def serialize_manifest(manifest):
-    # TODO: what should this take as an arg?
     return manifest.to_msgpack()


Test patch: