Repository: dbt-labs/dbt-core

Issue:
[Feature] dbt snapshot --defer
### Is there an existing feature request for this?

- [X] I have searched the existing issues

### Describe the Feature

Implement "dbt snapshot --defer", which would behave similar to "dbt run --defer".

### Describe alternatives you've considered

_No response_

### Who will this benefit?

When trying to set up "slim ci", one step in my case would be to run snapshots based on other dbt models.

`dbt snapshot --select 'dbt.*,state:modified' --state manifests/staging`

When running this command, generated references point to current schema, which doesn't have these tables, as they weren't generated during preceding run command:
`dbt run --models state:modified --defer --state manifests/airflow/staging`

### Are you interested in contributing this feature?

_No response_

### Anything else?

https://docs.getdbt.com/reference/node-selection/defer
https://docs.getdbt.com/docs/guides/best-practices#run-only-modified-models-to-test-changes-slim-ci

dbt version:  0.20.1


Discussion:
@iljau Thanks for opening!

The steps here are quite straightforward. I'm going to mark this a `good first issue`, and I'd welcome a PR from a community member.

1. Include snapshot in the subcommands that get the `--defer` flag:
https://github.com/dbt-labs/dbt-core/blob/f79a968a0908add9de1b6acfdb75a6b205134ebd/core/dbt/main.py#L1091-L1092

2. Remove `defer_to_manifest` override in `SnapshotTask`, so as to inherit the implementation from `RunTask`:
https://github.com/dbt-labs/dbt-core/blob/f79a968a0908add9de1b6acfdb75a6b205134ebd/core/dbt/task/snapshot.py#L25-L27

3. Then, we'd want to add/adjust a test case combining snapshots + deferral:
https://github.com/dbt-labs/dbt-core/blob/f79a968a0908add9de1b6acfdb75a6b205134ebd/test/integration/062_defer_state_test/test_defer_state.py#L154-L161

Is this a change you'd be interested in contributing?

### one step further

At this point, there's no good reason not to support `--defer` for `compile`, too, and thereby `docs generate` as well (https://github.com/dbt-labs/dbt-core/issues/3741). (The only reason I can come up with: we might eventually want to rename/rework the UX of deferral (#2968)).

For that support:
- Add the `--defer` flag to the `compile` and `docs` subparsers, too
- Move `defer_to_manifest` up from `RunTask` into [`CompileTask`](https://github.com/dbt-labs/dbt-core/blob/f79a968a0908add9de1b6acfdb75a6b205134ebd/core/dbt/task/compile.py#L36) (its parent)

### alternative

The `build` task, new in v0.21, supports both snapshots and deferral:
```
$ dbt build --resource-type snapshot --defer --state manifests/airflow/staging`
```

So, that's something you can use in the meantime.
Hello @jtcohen6 @iljau  :wave:,

Thank you for opening this issue and providing a step by step solution :slightly_smiling_face:.
Happy to work on this feature and to raise a PR for it. Ok for you?

:pray: 
I switched to using `dbt build` and don't have immediate plans to work on a fix.
Hi @jtcohen6 - just checking in.
Is there another issue you're aware of that covers your "one step further" note from above? We're missing deferral in catalog generation (dbt docs generate); followed the link here from #3741 but the PR for this does not appear to close that issue.

Resolving patch:
diff --git a/core/dbt/main.py b/core/dbt/main.py
--- a/core/dbt/main.py
+++ b/core/dbt/main.py
@@ -561,6 +561,7 @@ def _build_docs_generate_subparser(subparsers, base_subparser):
         Do not run "dbt compile" as part of docs generation
         """,
     )
+    _add_defer_argument(generate_sub)
     return generate_sub
 
 
@@ -1166,7 +1167,7 @@ def parse_args(args, cls=DBTArgumentParser):
     # list_sub sets up its own arguments.
     _add_selection_arguments(run_sub, compile_sub, generate_sub, test_sub, snapshot_sub, seed_sub)
     # --defer
-    _add_defer_argument(run_sub, test_sub, build_sub, snapshot_sub)
+    _add_defer_argument(run_sub, test_sub, build_sub, snapshot_sub, compile_sub)
     # --full-refresh
     _add_table_mutability_arguments(run_sub, compile_sub, build_sub)
 
diff --git a/core/dbt/task/compile.py b/core/dbt/task/compile.py
--- a/core/dbt/task/compile.py
+++ b/core/dbt/task/compile.py
@@ -1,10 +1,12 @@
 import threading
+from typing import AbstractSet, Optional
 
 from .runnable import GraphRunnableTask
 from .base import BaseRunner
 
+from dbt.contracts.graph.manifest import WritableManifest
 from dbt.contracts.results import RunStatus, RunResult
-from dbt.exceptions import InternalException
+from dbt.exceptions import InternalException, RuntimeException
 from dbt.graph import ResourceTypeSelector
 from dbt.events.functions import fire_event
 from dbt.events.types import CompileComplete
@@ -54,3 +56,33 @@ def get_runner_type(self, _):
 
     def task_end_messages(self, results):
         fire_event(CompileComplete())
+
+    def _get_deferred_manifest(self) -> Optional[WritableManifest]:
+        if not self.args.defer:
+            return None
+
+        state = self.previous_state
+        if state is None:
+            raise RuntimeException(
+                "Received a --defer argument, but no value was provided " "to --state"
+            )
+
+        if state.manifest is None:
+            raise RuntimeException(f'Could not find manifest in --state path: "{self.args.state}"')
+        return state.manifest
+
+    def defer_to_manifest(self, adapter, selected_uids: AbstractSet[str]):
+        deferred_manifest = self._get_deferred_manifest()
+        if deferred_manifest is None:
+            return
+        if self.manifest is None:
+            raise InternalException(
+                "Expected to defer to manifest, but there is no runtime " "manifest to defer from!"
+            )
+        self.manifest.merge_from_artifact(
+            adapter=adapter,
+            other=deferred_manifest,
+            selected=selected_uids,
+        )
+        # TODO: is it wrong to write the manifest here? I think it's right...
+        self.write_manifest()
diff --git a/core/dbt/task/run.py b/core/dbt/task/run.py
--- a/core/dbt/task/run.py
+++ b/core/dbt/task/run.py
@@ -18,7 +18,6 @@
 from dbt.clients.jinja import MacroGenerator
 from dbt.context.providers import generate_runtime_model_context
 from dbt.contracts.graph.compiled import CompileResultNode
-from dbt.contracts.graph.manifest import WritableManifest
 from dbt.contracts.graph.model_config import Hook
 from dbt.contracts.graph.parsed import ParsedHookNode
 from dbt.contracts.results import NodeStatus, RunResult, RunStatus, RunningStatus
@@ -408,36 +407,6 @@ def print_results_line(self, results, execution_time):
             HookFinished(stat_line=stat_line, execution=execution, execution_time=execution_time)
         )
 
-    def _get_deferred_manifest(self) -> Optional[WritableManifest]:
-        if not self.args.defer:
-            return None
-
-        state = self.previous_state
-        if state is None:
-            raise RuntimeException(
-                "Received a --defer argument, but no value was provided " "to --state"
-            )
-
-        if state.manifest is None:
-            raise RuntimeException(f'Could not find manifest in --state path: "{self.args.state}"')
-        return state.manifest
-
-    def defer_to_manifest(self, adapter, selected_uids: AbstractSet[str]):
-        deferred_manifest = self._get_deferred_manifest()
-        if deferred_manifest is None:
-            return
-        if self.manifest is None:
-            raise InternalException(
-                "Expected to defer to manifest, but there is no runtime " "manifest to defer from!"
-            )
-        self.manifest.merge_from_artifact(
-            adapter=adapter,
-            other=deferred_manifest,
-            selected=selected_uids,
-        )
-        # TODO: is it wrong to write the manifest here? I think it's right...
-        self.write_manifest()
-
     def before_run(self, adapter, selected_uids: AbstractSet[str]):
         with adapter.connection_named("master"):
             required_schemas = self.get_model_schemas(adapter, selected_uids)


Test patch:
diff --git a/test/integration/062_defer_state_tests/test_defer_state.py b/test/integration/062_defer_state_tests/test_defer_state.py
--- a/test/integration/062_defer_state_tests/test_defer_state.py
+++ b/test/integration/062_defer_state_tests/test_defer_state.py
@@ -47,6 +47,25 @@ def copy_state(self):
         os.makedirs('state')
         shutil.copyfile('target/manifest.json', 'state/manifest.json')
 
+    def run_and_compile_defer(self):
+        results = self.run_dbt(['seed'])
+        assert len(results) == 1
+        assert not any(r.node.deferred for r in results)
+        results = self.run_dbt(['run'])
+        assert len(results) == 2
+        assert not any(r.node.deferred for r in results)
+        results = self.run_dbt(['test'])
+        assert len(results) == 2
+
+        # copy files
+        self.copy_state()
+
+        # defer test, it succeeds
+        results, success = self.run_dbt_and_check(['compile', '--state', 'state', '--defer'])
+        self.assertEqual(len(results.results), 6)
+        self.assertEqual(results.results[0].node.name, "seed")
+        self.assertTrue(success)        
+
     def run_and_snapshot_defer(self):
         results = self.run_dbt(['seed'])
         assert len(results) == 1
@@ -195,3 +214,7 @@ def test_postgres_state_defer_deleted_upstream(self):
     @use_profile('postgres')
     def test_postgres_state_snapshot_defer(self):
         self.run_and_snapshot_defer()
+
+    @use_profile('postgres')
+    def test_postgres_state_compile_defer(self):
+        self.run_and_compile_defer()
diff --git a/tests/functional/artifacts/test_docs_generate_defer.py b/tests/functional/artifacts/test_docs_generate_defer.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/artifacts/test_docs_generate_defer.py
@@ -0,0 +1,33 @@
+import os
+import shutil
+import pytest
+from dbt.tests.util import run_dbt
+
+model_sql = """
+select 1 as id
+"""
+
+
+class TestDocsGenerateDefer:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {"model.sql": model_sql}
+
+    def copy_state(self):
+        assert not os.path.exists("state")
+        os.makedirs("state")
+        shutil.copyfile("target/manifest.json", "state/manifest.json")
+
+    def test_generate_defer(
+        self,
+        project,
+    ):
+        results = run_dbt(["run"])
+        assert len(results) == 1
+
+        # copy state files
+        self.copy_state()
+
+        # defer test, it succeeds
+        results = run_dbt(["docs", "generate", "--state", "./state", "--defer"])
+        assert results.nodes["model.test.model"]