Repository: dbt-labs/dbt-core

Issue:
Evaluate schema discovery queries that happen at startup
dbt runs a bunch of introspective queries at startup:
 - finding the schemas that exist in the database
 - finding the tables in those schemas
 - creating schemas if they don't exist

On some databases (postgres/redshift) these information schemas are pretty quick. On other databases (snowflake, spark) these queries can (and do) run for 10s of seconds! This means that the typical flow for running dbt looks like:
```
$ dbt run
<wait a not-insignificant number of seconds>
12:00:00 | Running with 8 threads
....
```

These queries are unavoidable. We experimented with using faster alternatives to the information schema (#1877) and ultimately, we must reckon with the reality that hitting the information schema can be slow. The next best thing that we can do is understand and optimize when these queries happen in order to provide the best possible experience for users running dbt.


Discussion:
There are really only two metadata queries that occur before a run starts on snowflake:
 - we list all the schemas from the information schema. We do this to avoid costly unnecessary 'create schema if not exists ...' calls.
    - One alternative here is to always try to create the schema if it doesn't exist. Very unscientifically: against `DBT_TEST`, the `list_schemas` call took 2.21s. A `create schema if not exists` for an existing schema took 2.24s. If that's at least close to normal, changing this will hurt performance in the >1 schemas case.
    - We could also try to use `show schemas in {database}`. The downsides are:
         1) You have to account for the >10k schemas case and fall back (probably fine!)
         2) You have to account for the database case sensitivity, likely with an extra `show databases` call
         3) It _seems_ like you have to use `table(result_scan(last_query_id()))` or similar to get useful data out of `show` commands, which doesn't seem substantially faster than `information_schema`. Maybe I'm way off here, if I am this might be worth doing.
    - We could parallelize this, using one thread per database.
  - We fill the cache with information about relations, by querying `information_schema.tables`. This area is probably the only hope for improving performance on this front.
    - Previous work on making catalogs faster didn't seem to indicate that `describe` or `show` had any significant performance improvements over querying the `information_schema`, and has similar limits around 10k. I'm happy to create a branch that does it this way and we can try it out on some large snowflake environments and see what happens.could create some frustrating performance side-effects.
    - We could do this up front, but parallelize this part across all the databases we're querying. This will probably get slower in the one-db case (paying the threading overhead costs) and faster everywhere else.
    - Now that we fully lazy-load our connections, it's possible that we could lazy-fill the cache on a schema by schema basis for non-postgres databases (postgres/redshift need to create the metadata links for `drop ... cascades` into views, so I think they'd still have to do this up-front). I don't think that's the worst idea ever, though there will be some thread coordination issues we'll have to consider, bottlenecking there could easily tank performance. This idea is the most likely to cause weird annoying deadlock-y bugs and hard-to-predict performance behavior. The graph queue would probably have to get smarter about ordering and schedule tasks in the same database more carefully.


dbt also makes a number of unnecessary `BEGIN` and `COMMIT` calls, each of which takes ~0.5s on my machine. We can save a couple seconds by skipping those, there seems to be no good reason for these queries to run in a transaction, beyond dbt's innate eagerness to wrap everything in transactions.

Resolving patch:
diff --git a/core/dbt/adapters/base/impl.py b/core/dbt/adapters/base/impl.py
--- a/core/dbt/adapters/base/impl.py
+++ b/core/dbt/adapters/base/impl.py
@@ -1,6 +1,5 @@
 import abc
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from concurrent.futures import Future  # noqa - we use this for typing only
+from concurrent.futures import as_completed, Future
 from contextlib import contextmanager
 from datetime import datetime
 from typing import (
@@ -27,7 +26,7 @@
 from dbt.exceptions import warn_or_error
 from dbt.node_types import NodeType
 from dbt.logger import GLOBAL_LOGGER as logger
-from dbt.utils import filter_null_values
+from dbt.utils import filter_null_values, executor
 
 from dbt.adapters.base.connections import BaseConnectionManager, Connection
 from dbt.adapters.base.meta import AdapterMeta, available
@@ -358,6 +357,12 @@ def _get_cache_schemas(
         # databases
         return info_schema_name_map
 
+    def _list_relations_get_connection(
+        self, db: BaseRelation, schema: str
+    ) -> List[BaseRelation]:
+        with self.connection_named(f'list_{db.database}_{schema}'):
+            return self.list_relations_without_caching(db, schema)
+
     def _relations_cache_for_schemas(self, manifest: Manifest) -> None:
         """Populate the relations cache for the given schemas. Returns an
         iterable of the schemas populated, as strings.
@@ -365,16 +370,22 @@ def _relations_cache_for_schemas(self, manifest: Manifest) -> None:
         if not dbt.flags.USE_CACHE:
             return
 
-        info_schema_name_map = self._get_cache_schemas(manifest,
-                                                       exec_only=True)
-        for db, schema in info_schema_name_map.search():
-            for relation in self.list_relations_without_caching(db, schema):
-                self.cache.add(relation)
+        schema_map = self._get_cache_schemas(manifest, exec_only=True)
+        with executor(self.config) as tpe:
+            futures: List[Future[List[BaseRelation]]] = [
+                tpe.submit(self._list_relations_get_connection, db, schema)
+                for db, schema in schema_map.search()
+            ]
+            for future in as_completed(futures):
+                # if we can't read the relations we need to just raise anyway,
+                # so just call future.result() and let that raise on failure
+                for relation in future.result():
+                    self.cache.add(relation)
 
         # it's possible that there were no relations in some schemas. We want
         # to insert the schemas we query into the cache's `.schemas` attribute
         # so we can check it later
-        self.cache.update_schemas(info_schema_name_map.schemas_searched())
+        self.cache.update_schemas(schema_map.schemas_searched())
 
     def set_relations_cache(
         self, manifest: Manifest, clear: bool = False
@@ -1047,13 +1058,11 @@ def _get_one_catalog(
     def get_catalog(
         self, manifest: Manifest
     ) -> Tuple[agate.Table, List[Exception]]:
-        # snowflake is super slow. split it out into the specified threads
-        num_threads = self.config.threads
         schema_map = self._get_cache_schemas(manifest)
 
-        with ThreadPoolExecutor(max_workers=num_threads) as executor:
-            futures = [
-                executor.submit(self._get_one_catalog, info, schemas, manifest)
+        with executor(self.config) as tpe:
+            futures: List[Future[agate.Table]] = [
+                tpe.submit(self._get_one_catalog, info, schemas, manifest)
                 for info, schemas in schema_map.items() if len(schemas) > 0
             ]
             catalogs, exceptions = catch_as_completed(futures)
diff --git a/core/dbt/task/runnable.py b/core/dbt/task/runnable.py
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -1,8 +1,9 @@
 import os
 import time
+from concurrent.futures import as_completed
 from datetime import datetime
 from multiprocessing.dummy import Pool as ThreadPool
-from typing import Optional, Dict, List, Set, Tuple
+from typing import Optional, Dict, List, Set, Tuple, Iterable
 
 from dbt.task.base import ConfiguredTask
 from dbt.adapters.factory import get_adapter
@@ -374,7 +375,9 @@ def interpret_results(self, results):
         failures = [r for r in results if r.error or r.fail]
         return len(failures) == 0
 
-    def get_model_schemas(self, selected_uids):
+    def get_model_schemas(
+        self, selected_uids: Iterable[str]
+    ) -> Set[Tuple[str, str]]:
         if self.manifest is None:
             raise InternalException('manifest was None in get_model_schemas')
 
@@ -387,19 +390,46 @@ def get_model_schemas(self, selected_uids):
 
         return schemas
 
-    def create_schemas(self, adapter, selected_uids):
+    def create_schemas(self, adapter, selected_uids: Iterable[str]):
         required_schemas = self.get_model_schemas(selected_uids)
         required_databases = set(db for db, _ in required_schemas)
 
         existing_schemas_lowered: Set[Tuple[str, str]] = set()
-        for db in required_databases:
-            existing_schemas_lowered.update(
-                (db.lower(), s.lower()) for s in adapter.list_schemas(db))
 
-        for db, schema in required_schemas:
-            if (db.lower(), schema.lower()) not in existing_schemas_lowered:
+        def list_schemas(db: str) -> List[Tuple[str, str]]:
+            with adapter.connection_named(f'list_{db}'):
+                return [
+                    (db.lower(), s.lower())
+                    for s in adapter.list_schemas(db)
+                ]
+
+        def create_schema(db: str, schema: str) -> None:
+            with adapter.connection_named(f'create_{db}_{schema}'):
                 adapter.create_schema(db, schema)
 
+        list_futures = []
+        create_futures = []
+
+        with dbt.utils.executor(self.config) as tpe:
+            list_futures = [
+                tpe.submit(list_schemas, db) for db in required_databases
+            ]
+
+            for ls_future in as_completed(list_futures):
+                existing_schemas_lowered.update(ls_future.result())
+
+            for db, schema in required_schemas:
+                db_schema = (db.lower(), schema.lower())
+                if db_schema not in existing_schemas_lowered:
+                    existing_schemas_lowered.add(db_schema)
+                    create_futures.append(
+                        tpe.submit(create_schema, db, schema)
+                    )
+
+            for create_future in as_completed(create_futures):
+                # trigger/re-raise any excceptions while creating schemas
+                create_future.result()
+
     def get_result(self, results, elapsed_time, generated_at):
         return ExecutionResult(
             results=results,
diff --git a/core/dbt/utils.py b/core/dbt/utils.py
--- a/core/dbt/utils.py
+++ b/core/dbt/utils.py
@@ -1,4 +1,5 @@
 import collections
+import concurrent.futures
 import copy
 import datetime
 import decimal
@@ -8,6 +9,7 @@
 import json
 import os
 from enum import Enum
+from typing_extensions import Protocol
 from typing import (
     Tuple, Type, Any, Optional, TypeVar, Dict, Union, Callable
 )
@@ -489,3 +491,48 @@ def format_bytes(num_bytes):
         num_bytes /= 1024.0
 
     return "> 1024 TB"
+
+
+# a little concurrent.futures.Executor for single-threaded mode
+class SingleThreadedExecutor(concurrent.futures.Executor):
+    def submit(*args, **kwargs):
+        # this basic pattern comes from concurrent.futures.Executor itself,
+        # but without handling the `fn=` form.
+        if len(args) >= 2:
+            self, fn, *args = args
+        elif not args:
+            raise TypeError(
+                "descriptor 'submit' of 'SingleThreadedExecutor' object needs "
+                "an argument"
+            )
+        else:
+            raise TypeError(
+                'submit expected at least 1 positional argument, '
+                'got %d' % (len(args) - 1)
+            )
+        fut = concurrent.futures.Future()
+        try:
+            result = fn(*args, **kwargs)
+        except Exception as exc:
+            fut.set_exception(exc)
+        else:
+            fut.set_result(result)
+        return fut
+
+
+class ThreadedArgs(Protocol):
+    single_threaded: bool
+
+
+class HasThreadingConfig(Protocol):
+    args: ThreadedArgs
+    threads: Optional[int]
+
+
+def executor(config: HasThreadingConfig) -> concurrent.futures.Executor:
+    if config.args.single_threaded:
+        return SingleThreadedExecutor()
+    else:
+        return concurrent.futures.ThreadPoolExecutor(
+            max_workers=config.threads
+        )


Test patch:
diff --git a/test/integration/001_simple_copy_test/test_simple_copy.py b/test/integration/001_simple_copy_test/test_simple_copy.py
--- a/test/integration/001_simple_copy_test/test_simple_copy.py
+++ b/test/integration/001_simple_copy_test/test_simple_copy.py
@@ -71,7 +71,6 @@ def test__postgres__simple_copy_with_materialized_views(self):
                 select * from {schema}.unrelated_materialized_view
             )
         '''.format(schema=self.unique_schema()))
-
         results = self.run_dbt(["seed"])
         self.assertEqual(len(results),  1)
         results = self.run_dbt()
diff --git a/test/unit/utils.py b/test/unit/utils.py
--- a/test/unit/utils.py
+++ b/test/unit/utils.py
@@ -25,6 +25,7 @@ def normalize(path):
 
 class Obj:
     which = 'blah'
+    single_threaded = False
 
 
 def mock_connection(name):