Repository: dbt-labs/dbt-core

Issue:
[CT-1386] [Bug] `dbt docs generate` with `--defer`
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

It's great that dbt 1.3 makes it possible to use `dbt docs generate` with `--defer`, but it appears to be not quite working, unless I'm not understanding something.

When running `dbt docs generate (--select state:modified+) --defer --state /path/to/manifest` I'm getting an error `Object X does not exist or not authorized.` The cause for this error is that X indeed does not exist, so the `ref` referring to it should have been resolved using the manifest.

In my situation, the faulty `ref` actually happens in a macro, perhaps that's why it wasn't resolved properly? What is strange is that this doesn't seem to be a problem for `dbt run`.



### Expected Behavior

When using `dbt docs generate` with `--defer`, all `ref` calls should be resolved as described in the [dbt doc](https://docs.getdbt.com/reference/node-selection/defer).

By the way, it's unclear to me whether I'm supposed to use a `--select` when doing `dbt docs generate` with `--defer`. I would assume so, since it's supposed to play a role in resolving  the `ref`s according to the doc, but it doesn't seem to make a difference.

### Steps To Reproduce

1. Create a macro that `ref`s some model X.
2. Run `dbt docs generate` with `--defer` (and possibly other flags).
3. Observe that the `ref` in the macro is not resolved properly and produces an error.

### Relevant log output

_No response_

### Environment

```markdown
- OS:
- Python:
- dbt:
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

_No response_


Discussion:
Thanks for opening @seub! After a little bit of digging, I think I figured out what's going on here.

Even though https://github.com/dbt-labs/dbt-core/pull/4514 did succeed in moving the `defer_to_manifest` method into the `compile` task (from the `run` task), and adding `--defer` as a supported argument to `docs generate`, the call to `defer_to_manifest` is still missing from the `before_run` steps for `compile` / `docs generate`.

https://github.com/dbt-labs/dbt-core/blob/8145eed603266951ce35858f7eef3836012090bd/core/dbt/task/runnable.py#L420-L422

Needs to become:

```python
    def before_run(self, adapter, selected_uids: AbstractSet[str]):
        with adapter.connection_named("master"):
            self.populate_adapter_cache(adapter)
            self.defer_to_manifest(adapter, selected_uids)
```
And then it works!

Marking this one as a `good_first_issue`, if you're up to make the contribution :) Might be a teensy bit trickier to add an automated test for this, though I'm hopeful that it could be a simple extension of the test added in #4514 ([`test_docs_generate_defer`](https://github.com/dbt-labs/dbt-core/blob/main/tests/functional/artifacts/test_docs_generate_defer.py)).
Thank you very much @jtcohen6 . I appreciate the encouragement to contribute.

I went through the documentation to contribute and followed all the steps. I just created a [new PR ](https://github.com/dbt-labs/dbt-core/pull/4514) with the change that you suggested. 

However that's the only change I made (besides running `changie`), I did not add any new tests. Note: I was able to see that `make test` succeeds, however `make integration` failed, but it already failed before I made any changes.

Resolving patch:
diff --git a/core/dbt/task/freshness.py b/core/dbt/task/freshness.py
--- a/core/dbt/task/freshness.py
+++ b/core/dbt/task/freshness.py
@@ -147,6 +147,10 @@ def node_is_match(self, node):
 
 
 class FreshnessTask(GraphRunnableTask):
+    def defer_to_manifest(self, adapter, selected_uids):
+        # freshness don't defer
+        return
+
     def result_path(self):
         if self.args.output:
             return os.path.realpath(self.args.output)
diff --git a/core/dbt/task/list.py b/core/dbt/task/list.py
--- a/core/dbt/task/list.py
+++ b/core/dbt/task/list.py
@@ -179,6 +179,10 @@ def selection_arg(self):
         else:
             return self.args.select
 
+    def defer_to_manifest(self, adapter, selected_uids):
+        # list don't defer
+        return
+
     def get_node_selector(self):
         if self.manifest is None or self.graph is None:
             raise InternalException("manifest and graph must be set to get perform node selection")
diff --git a/core/dbt/task/runnable.py b/core/dbt/task/runnable.py
--- a/core/dbt/task/runnable.py
+++ b/core/dbt/task/runnable.py
@@ -152,6 +152,10 @@ def get_selection_spec(self) -> SelectionSpec:
     def get_node_selector(self) -> NodeSelector:
         raise NotImplementedException(f"get_node_selector not implemented for task {type(self)}")
 
+    @abstractmethod
+    def defer_to_manifest(self, adapter, selected_uids: AbstractSet[str]):
+        raise NotImplementedException(f"defer_to_manifest not implemented for task {type(self)}")
+
     def get_graph_queue(self) -> GraphQueue:
         selector = self.get_node_selector()
         spec = self.get_selection_spec()
@@ -419,6 +423,7 @@ def populate_adapter_cache(self, adapter, required_schemas: Set[BaseRelation] =
     def before_run(self, adapter, selected_uids: AbstractSet[str]):
         with adapter.connection_named("master"):
             self.populate_adapter_cache(adapter)
+            self.defer_to_manifest(adapter, selected_uids)
 
     def after_run(self, adapter, results):
         pass


Test patch:
diff --git a/test/integration/062_defer_state_tests/test_defer_state.py b/test/integration/062_defer_state_tests/test_defer_state.py
--- a/test/integration/062_defer_state_tests/test_defer_state.py
+++ b/test/integration/062_defer_state_tests/test_defer_state.py
@@ -109,12 +109,22 @@ def run_and_defer(self):
         # no state, wrong schema, failure.
         self.run_dbt(['test', '--target', 'otherschema'], expect_pass=False)
 
+        # test generate docs
+        # no state, wrong schema, empty nodes 
+        catalog = self.run_dbt(['docs','generate','--target', 'otherschema'])
+        assert not catalog.nodes
+
         # no state, run also fails
         self.run_dbt(['run', '--target', 'otherschema'], expect_pass=False)
 
         # defer test, it succeeds
         results = self.run_dbt(['test', '-m', 'view_model+', '--state', 'state', '--defer', '--target', 'otherschema'])
 
+        # defer docs generate with state, catalog refers schema from the happy times
+        catalog = self.run_dbt(['docs','generate', '-m', 'view_model+', '--state', 'state', '--defer','--target', 'otherschema'])
+        assert self.other_schema not in catalog.nodes["seed.test.seed"].metadata.schema
+        assert self.unique_schema() in catalog.nodes["seed.test.seed"].metadata.schema
+
         # with state it should work though
         results = self.run_dbt(['run', '-m', 'view_model', '--state', 'state', '--defer', '--target', 'otherschema'])
         assert self.other_schema not in results[0].node.compiled_code