Repository: dbt-labs/dbt-core

Issue:
Source schema caching + generate_schema_name macro
## Issue

### Issue description
This one has a few layers. I have two dbt projects, one is a dependency of the other. Both have this macro switched on:
```
{% macro generate_schema_name(schema_name) -%}
    {{ generate_schema_name_for_env(schema_name) }}
{%- endmacro %}
```
The dependent package defines `sources` from a schema that has a bit of noise in it. Namely, the schema appears to contain some cascading views that depend (in the postgres sense, not the dbt sense) on other relations. dbt knows and should care about neither these cascading views nor their dependencies.

### Results
When I try to run models from the dependent package from the "main" project, with `--target prod`, I get this error:
```
Encountered an error:
Cache inconsistency detected: in add_link, dependent link key _ReferenceKey(database='metrics', schema='source_schema', identifier='relation_that_dbt_does_not_care_about') not in cache!
```

### System information
The output of `dbt --version`:
```
installed version: 0.13.0
   latest version: 0.13.0

Up to date!
```

### Steps to reproduce

I haven't had a chance to try this out, but after speaking with @beckjake, I _think_ you should be able to reproduce this in the following steps:
* Create a bound view `my_view` in schema `my_schema` that selects from another relation (e.g. `my_dependency`)
* Define a source in dbt with schema `my_schema` and `tables` that _do not_ include `my_view` or `my_dependency`
* Activate the env-based `generate_schema_name` macro
* `dbt run --target prod`

Right now, I'm under the impression that the package dependency setup is a complicating factor but not a necessary condition for this error.


Discussion:
dbt does relation caching at the schema-level, and my guess is that the addition of Sources changed some of dbt's invariants. Whereas previously dbt didn't know anything about models outside of the dbt schema(s) in the database, it now knows about source schemas too.

Is it possible that one of the tables in the source schema was dropped during the dbt run? 

You pasted in this error message:
```
Cache inconsistency detected: in add_link, dependent link key _ReferenceKey(database='metrics', schema='source_schema', identifier='relation_that_dbt_does_not_care_about') not in cache!
```

In this example, `source_schema.relation_that_dbt_does_not_care_about` is a source table, right? So in fact, it looks like that was _created_ during the dbt run? Can you elaborate on the nature of the actual tables represented in this error message? I'm less interested in things like projects/packages and `generate_schema_name` macros, as I can't imagine they're super relevant to the incidence of this bug.

Thanks for the report!
`source_schema.relation_that_dbt_does_not_care_about` is not a source table that dbt knows about, but `source_schema` is a (noisy) schema that dbt knows to contain _other_ source tables. I believe `relation_that_dbt_does_not_care_about` is actually a view with postgres-y dependencies on external views/tables that dbt doesn't know about.

I have been able to reproduce this error reliably over the course of a few days. Also, when I query `source_schema.relation_that_dbt_does_not_care_about`, I get results. So I don't think it's a matter of that view being added or dropped.

I have confirmed that the multiple projects/packages doesn't matter, since I can replicate this error when running the dependent package alone. However, I can _only_ replicate this error when running with `--target prod` and the `generate_schema_name_for_env` macro enabled.

I have been able to resolve this error by removing the sources from dbt with schema `source_schema`.
@jtcohen6 thanks for the additional info. If you're able, can you try:
1. delete (or rename) your `logs/dbt.log` file
2. invoke dbt with `dbt run --log-cache-events`
3. DM me the resulting logs after you're able to reproduce this

I think these additional cache log lines will help us get to the bottom of what's going on here.

I tried reproducing this on postgres but couldn't quite do it. dbt does have logic to ignore cache misses for relations that it shouldn't care about, but it looks like something funky is happening here to make this slip through the cracks.

Resolving patch:
diff --git a/plugins/postgres/dbt/adapters/postgres/impl.py b/plugins/postgres/dbt/adapters/postgres/impl.py
--- a/plugins/postgres/dbt/adapters/postgres/impl.py
+++ b/plugins/postgres/dbt/adapters/postgres/impl.py
@@ -39,6 +39,11 @@ def verify_database(self, database):
         return ''
 
     def _link_cached_database_relations(self, schemas):
+        """
+
+        :param Set[str] schemas: The set of schemas that should have links
+            added.
+        """
         database = self.config.credentials.database
         table = self.execute_macro(GET_RELATIONS_MACRO_NAME)
 
@@ -74,8 +79,11 @@ def _get_cache_schemas(self, manifest, exec_only=False):
 
     def _link_cached_relations(self, manifest):
         schemas = set()
-        for db, schema in manifest.get_used_schemas():
-            self.verify_database(db)
+        # only link executable nodes
+        info_schema_name_map = self._get_cache_schemas(manifest,
+                                                       exec_only=True)
+        for db, schema in info_schema_name_map.search():
+            self.verify_database(db.database)
             schemas.add(schema)
 
         try:


Test patch:
diff --git a/test/integration/007_graph_selection_tests/test_graph_selection.py b/test/integration/007_graph_selection_tests/test_graph_selection.py
--- a/test/integration/007_graph_selection_tests/test_graph_selection.py
+++ b/test/integration/007_graph_selection_tests/test_graph_selection.py
@@ -234,6 +234,6 @@ def test__snowflake__skip_intermediate(self):
         user_last_end = users.timing[1]['completed_at']
         dep_first_start = dep.timing[0]['started_at']
         self.assertTrue(
-            user_last_end < dep_first_start,
+            user_last_end <= dep_first_start,
             'dependency started before its transitive parent ({} > {})'.format(user_last_end, dep_first_start)
         )
diff --git a/test/integration/042_sources_test/models/schema.yml b/test/integration/042_sources_test/models/schema.yml
--- a/test/integration/042_sources_test/models/schema.yml
+++ b/test/integration/042_sources_test/models/schema.yml
@@ -53,3 +53,7 @@ sources:
     tables:
       - name: test_table
         identifier: other_source_table
+  - name: external_source
+    schema: "{{ var('test_run_alt_schema', var('test_run_schema')) }}"
+    tables:
+      - name: table
diff --git a/test/integration/042_sources_test/test_sources.py b/test/integration/042_sources_test/test_sources.py
--- a/test/integration/042_sources_test/test_sources.py
+++ b/test/integration/042_sources_test/test_sources.py
@@ -40,6 +40,33 @@ def run_dbt_with_vars(self, cmd, *args, **kwargs):
 
 
 class TestSources(BaseSourcesTest):
+    def _create_schemas(self):
+        super(TestSources, self)._create_schemas()
+        self._create_schema_named(self.default_database,
+                                  self.alternative_schema())
+
+    def alternative_schema(self):
+        return self.unique_schema() + '_other'
+
+    def setUp(self):
+        super(TestSources, self).setUp()
+        self.run_sql(
+            'create table {}.dummy_table (id int)'.format(self.unique_schema())
+        )
+        self.run_sql(
+            'create view {}.external_view as (select * from {}.dummy_table)'
+            .format(self.alternative_schema(), self.unique_schema())
+        )
+
+    def run_dbt_with_vars(self, cmd, *args, **kwargs):
+        cmd.extend([
+            '--vars',
+            '{{test_run_schema: {}, test_run_alt_schema: {}}}'.format(
+                self.unique_schema(), self.alternative_schema()
+            )
+        ])
+        return self.run_dbt(cmd, *args, **kwargs)
+
     @use_profile('postgres')
     def test_postgres_basic_source_def(self):
         results = self.run_dbt_with_vars(['run'])
@@ -113,6 +140,7 @@ def test_source_childrens_parents(self):
         )
         self.assertTableDoesNotExist('nonsource_descendant')
 
+
 class TestSourceFreshness(BaseSourcesTest):
     def setUp(self):
         super(TestSourceFreshness, self).setUp()
@@ -256,3 +284,4 @@ def test_postgres_malformed_schema_nonstrict_will_not_break_run(self):
     def test_postgres_malformed_schema_strict_will_break_run(self):
         with self.assertRaises(CompilationException):
             self.run_dbt_with_vars(['run'], strict=True)
+