Repository: dbt-labs/dbt-core

Issue:
Defer refers to outdated seed 
### Describe the bug
I'm running the following on CI:

```bash
aws s3 cp $(MANIFEST_REMOTE_PATH) target/prod/manifest.json
dbt seed --select "state:modified+" --full-refresh --state target/prod/ --target dev-master
dbt run --models "state:modified+" --defer --state target/prod/ --target dev-master
...
```

Say we have a seed `countries.csv`:
```csv
name
Portugal
Ireland
Switzerland
```

And a model `dim_countries.sql`:

```sql
select to_upper(name) as name
from {{ ref("countries") }}
```

Now in one PR we change the seed to
```csv
id,name
1,Portugal
2,Ireland
3,Switzerland
```

And we change the model to:
```sql
select
  id, 
  to_upper(name) as name
from {{ ref("countries") }}
```


dbt correctly identifies that `countries.csv` changed, and that `dim_countries.sql` changed. However when the model runs, it fails with "column id does not exist" because the model tries to read the seed from the "main run" (it defers the seed) instead of identifying that it re-ran.

Note that I tried copying the manifest produced by `dbt seed` into `target/prod/` but what happens then is that dbt does not identify the model as "modified" 

### dbt version

`0.18.1`



Discussion:
Thanks for the writeup @dmateusp. The issue here is that `--defer` simply switches the reference of any resource  not included in the selection criteria. As part of node selection, dbt excludes all resources from the selection criteria that are not relevant to the invocation type, e.g. a seed from `dbt run`. So today there's no _real_ way to _not_ defer a seed.

I think this fix is going to be tricky: we'd need dbt to defer on the basis of an intermediate set of selected nodes, rather than the final set. This is part of what made test deferral so cumbersome (#2701), and ultimately not worthwhile. In this case, though, I think we need to figure out an answer.

A much longer-term resolution here would be a generalized command that can operate on both seeds and models (#2743).

In the meantime, the only workarounds I can think of are quite hacky. E.g. if you have a staging model (`models/staging/seeds/*.sql`) that selects from the seed, and `ref()` that staging model instead of the seed downstream, you could try to run that view once (without deferral), before running the rest of the DAG _with_ deferral...

```bash
dbt seed -s state:modified --state .state
dbt run -m state:modified+1,staging.seeds --state .state # no deferral
dbt run -m state:modified+ --exclude state:modified+1,staging.seeds --defer --state .state
```
Thank you for the details @jtcohen6, the staging model is an interesting solution but not something practical to enforce in our project right now unfortunately.

For a short term solution would it be possible to extend dbt to allow deferring only models ? That way I could change my CI to always re-create all the seeds (which is not a super big deal since seeds should be small)

`dbt run -m state:modified+ --defer models <...>` ?
@dmateusp That's a good thought, and if it proves much more straightforward, that may be the move. In either case, it will require tweaking some of the logic of deferral—which isn't something we can squeeze in for the next minor version (v0.19), but possibly for the one after.

In the meantime, I'll plan to document this as a known caveat to state comparison + deferral.
One approach that's occurred to me, though I haven't thought through all the implications:

Today, we defer _all_ models that are not included in the node selection criteria. That means we defer:
- seeds (always)
- modified models that are included, then excluded, from shifting criteria, e.g. `dbt run -m state:modified && dbt run -m state:modified,config.materialized:incremental` (see [slack thread](https://getdbt.slack.com/archives/CMZ2V0X8V/p1607008955284500))

Perhaps we shouldn't use selection criteria as the basis for deferral. Instead, during compilation, we could check to see if a referent's "new" representation (`ci_schema.identifier`) exists in the database (via cache lookup). If it doesn't exist,  we "fall back" to the comparison manifest's representation (`prod_schema.identifier`) of a node with the same `unique_id`.

If we took this more-naive approach to deferral:
- Would it also help us avoid tricky issues around manifest construction when previous source or ephemeral model parents have gone missing (#2875)?
- Could we even revisit the question of how deferral might work for tests (#2701)?
I like that a lot!

Resolving patch:
diff --git a/core/dbt/contracts/graph/manifest.py b/core/dbt/contracts/graph/manifest.py
--- a/core/dbt/contracts/graph/manifest.py
+++ b/core/dbt/contracts/graph/manifest.py
@@ -887,6 +887,7 @@ def resolve_doc(
 
     def merge_from_artifact(
         self,
+        adapter,
         other: 'WritableManifest',
         selected: AbstractSet[UniqueID],
     ) -> None:
@@ -898,10 +899,14 @@ def merge_from_artifact(
         refables = set(NodeType.refable())
         merged = set()
         for unique_id, node in other.nodes.items():
-            if (
+            current = self.nodes.get(unique_id)
+            if current and (
                 node.resource_type in refables and
                 not node.is_ephemeral and
-                unique_id not in selected
+                unique_id not in selected and
+                not adapter.get_relation(
+                    current.database, current.schema, current.identifier
+                )
             ):
                 merged.add(unique_id)
                 self.nodes[unique_id] = node.replace(deferred=True)
diff --git a/core/dbt/task/run.py b/core/dbt/task/run.py
--- a/core/dbt/task/run.py
+++ b/core/dbt/task/run.py
@@ -372,7 +372,7 @@ def _get_deferred_manifest(self) -> Optional[WritableManifest]:
             )
         return state.manifest
 
-    def defer_to_manifest(self, selected_uids: AbstractSet[str]):
+    def defer_to_manifest(self, adapter, selected_uids: AbstractSet[str]):
         deferred_manifest = self._get_deferred_manifest()
         if deferred_manifest is None:
             return
@@ -382,6 +382,7 @@ def defer_to_manifest(self, selected_uids: AbstractSet[str]):
                 'manifest to defer from!'
             )
         self.manifest.merge_from_artifact(
+            adapter=adapter,
             other=deferred_manifest,
             selected=selected_uids,
         )
@@ -389,10 +390,10 @@ def defer_to_manifest(self, selected_uids: AbstractSet[str]):
         self.write_manifest()
 
     def before_run(self, adapter, selected_uids: AbstractSet[str]):
-        self.defer_to_manifest(selected_uids)
         with adapter.connection_named('master'):
             self.create_schemas(adapter, selected_uids)
             self.populate_adapter_cache(adapter)
+            self.defer_to_manifest(adapter, selected_uids)
             self.safe_run_hooks(adapter, RunHookType.Start, {})
 
     def after_run(self, adapter, results):


Test patch:
diff --git a/test/integration/062_defer_state_test/test_defer_state.py b/test/integration/062_defer_state_test/test_defer_state.py
--- a/test/integration/062_defer_state_test/test_defer_state.py
+++ b/test/integration/062_defer_state_test/test_defer_state.py
@@ -102,6 +102,24 @@ def run_switchdirs_defer(self):
             expect_pass=False,
         )
 
+    def run_defer_iff_not_exists(self):
+        results = self.run_dbt(['seed', '--target', 'otherschema'])
+        assert len(results) == 1
+        results = self.run_dbt(['run', '--target', 'otherschema'])
+        assert len(results) == 2
+
+        # copy files over from the happy times when we had a good target
+        self.copy_state()
+        
+        results = self.run_dbt(['seed'])
+        assert len(results) == 1
+        results = self.run_dbt(['run', '--state', 'state', '--defer'])
+        assert len(results) == 2
+        
+        # because the seed now exists in our schema, we shouldn't defer it
+        assert self.other_schema not in results[0].node.compiled_sql
+        assert self.unique_schema() in results[0].node.compiled_sql
+
     @use_profile('postgres')
     def test_postgres_state_changetarget(self):
         self.run_and_defer()
@@ -118,8 +136,12 @@ def test_postgres_state_changetarget(self):
             self.run_dbt(['snapshot', '--defer'])
 
     @use_profile('postgres')
-    def test_postgres_stat_changedir(self):
+    def test_postgres_state_changedir(self):
         self.run_switchdirs_defer()
+        
+    @use_profile('postgres')
+    def test_postgres_state_defer_iffnotexists(self):
+        self.run_defer_iff_not_exists()
 
     @use_profile('snowflake')
     def test_snowflake_state_changetarget(self):