Repository: dbt-labs/dbt-core

Issue:
Implement the relation cache for `dbt compile` and `dbt ls`
### Describe the feature
In the `dbt run` command, dbt builds a [relation cache](https://github.com/fishtown-analytics/dbt/blob/dev/louisa-may-alcott/core/dbt/task/run.py#L61-L62). dbt should also build this cache when the `dbt compile` and `dbt ls` commands are invoked.

Bonus: It would be even better if dbt lazily populated this cache. TBD if that takes place in this issue, or if we should make a separate issue for that.

Edited: Removed references to the rpc server


Discussion:
Kicking this out of the LMA milestone. The better version of this will:
 - lazily populate the relation cache
 - lock around calls to populate the cache 
   - lock per schema?
   - lock around the whole cache?
Are contributions welcome on this? We're experiencing punishing compile times on Snowflake that would be alleviated by fixing this. 
hey @mcannamela - sure, we'd love a PR for this one! I think the big gain here is going to be lazily populating the relation cache.

We want to make these introspective queries run when they're needed, not at startup. Right now, dbt is going to scan every schema referenced in your project, even if you're just doing something like:
```
dbt compile -m my_model
```

In this case, we'd want to:
 - wait until `my_model` is actively compiled to query the information schema
 - only query the information schema for the relations in the relevant schema for `my_model`, not all of the schemas referenced in the project

The locking piece comes in if you have more than one model running concurrently that's hitting the same information schema:

```
dbt compile -m my_first_model my_second_model
```

If both of these models are materialized in the same information schema namespace[1], dbt should only run the information schema query _once_, then return the results to both models.

@beckjake there was some good reason why we _didn't_ implement the original feature request (populating the relation cache) -- do you remember why that was?

[1] this varies by warehouse. On postgres/redshift/snowflake, the information schema is accessed at the database level. On BigQuery, the information schema is accessed at the dataset level.
>there was some good reason why we didn't implement the original feature request (populating the relation cache) -- do you remember why that was?

I _think_ it's because populating the cache is expensive, and many `compile`/`ls` invocations don't require it. We didn't do lazy loading the cache because it seems hard and our time is finite (we wrote this issue instead). But that was a long time ago, my memory could be wrong.

Could you clarify which compile invocations would not require it? From my reading, it seems like it's going to be hit any time `BaseAdapter.get_relation` is called, so compiling any model with a `ref` would need to invoke it at least once. At least for us if you picked a model at random then more likely than not it would contain a `ref`. 

I came to this issue via https://github.com/fishtown-analytics/dbt/pull/1737 and that seemed like a win to me, since you could just pass the `--bypass-cache` flag  and be no worse off than you are currently. 

Is there some reason why populating the cache up front for all schemas is beneficial for the `run` command but would be detrimental for `compile`? Or would they ideally have the same lazy behavior?

Not trying to be contrarian here just want to snuff out my own ignorance here before I or my team undertakes this. 


The trick is that not everything has to call `get_relation` - `ref` doesn't actually require it.

`BaseAdapter.get_relation` is (generally!) only called during materialization execution, which compilation doesn't look at - compilation only generates the `sql` variable for materializations, and then at runtime those materializations are evaluated.

I think ideally they'd have the same lazy behavior.
Got it, thanks. 
Adding this to the Octavius Catto milestone (for now, might not make it into that release). Folks can disable this if they're experiencing negative performance characteristics with `--bypass-cache`.

Resolving patch:
diff --git a/core/dbt/task/run.py b/core/dbt/task/run.py
--- a/core/dbt/task/run.py
+++ b/core/dbt/task/run.py
@@ -88,9 +88,6 @@ def index_offset(self, value: int) -> int:
     def raise_on_first_error(self):
         return False
 
-    def populate_adapter_cache(self, adapter):
-        adapter.set_relations_cache(self.manifest)
-
     def get_hook_sql(self, adapter, hook, idx, num_hooks, extra_context):
         compiled = compile_node(adapter, self.config, hook, self.manifest,
                                 extra_context)
@@ -195,9 +192,8 @@ def print_results_line(self, results, execution_time):
             .format(stat_line=stat_line, execution=execution))
 
     def before_run(self, adapter, selected_uids):
+        super().before_run(adapter, selected_uids)
         with adapter.connection_named('master'):
-            self.create_schemas(adapter, selected_uids)
-            self.populate_adapter_cache(adapter)
             self.safe_run_hooks(adapter, RunHookType.Start, {})
 
     def after_run(self, adapter, results):
diff --git a/core/dbt/task/runnable.py b/core/dbt/task/runnable.py
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -340,11 +340,16 @@ def _mark_dependent_errors(self, node_id, result, cause):
         for dep_node_id in self.linker.get_dependent_nodes(node_id):
             self._skipped_children[dep_node_id] = cause
 
+    def populate_adapter_cache(self, adapter):
+        adapter.set_relations_cache(self.manifest)
+
     def before_hooks(self, adapter):
         pass
 
     def before_run(self, adapter, selected_uids):
-        pass
+        with adapter.connection_named('master'):
+            self.create_schemas(adapter, selected_uids)
+            self.populate_adapter_cache(adapter)
 
     def after_run(self, adapter, results):
         pass


Test patch: