Repository: dbt-labs/dbt-core

Issue:
[CT-2124] [Feature] `--no-connection` flag for `dbt docs generate`
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

Provide an optional flag to be used with doc generation that skips the metadata queries and generates `catalog.json` based only off what it can get from compilation (and/or the existing `manifest.json` if `--no-compile` is also passed).

The problem I am solving for is around enabling the docs to be used locally for exploratory purposes during local development in large projects where building the catalog would otherwise take a long time — in the case of our project building the catalog without compile takes >30 minutes (11k nodes, Snowflake, 8 threads, 2XL).

This problem and solution is actually described exactly in https://github.com/dbt-labs/dbt-core/issues/3947#issuecomment-1349013372:

> I work with a few teams in the company and, as a developer, I would love to see their DAGs and understand better how they are developing/deploying their environment. How do other teams generate certain metrics, how do they use those metrics, and what kinds of tests do they typically perform across their metrics? I know that some teams use pre-hooks and post-hooks to determine things like row counts before and after runs to ensure things look right. I'd love to be able to explore this in the `dbt docs serve` format.
> 
> I can manually parse through this info on my own but digging through dozens of sql files across multiple folders is... a bit intense. That said, I'm not as interested in the actual data and don't have access anyways.
> 
> I understand the huge benefit to having connectivity to the database when generating docs but it would be great if you could put a flag on the command that allows you to forego any benefits of connectivity. Something like `dbt docs generate --no_connection` so that I could create enough of the docs to view their processes but knowing that I wouldn't have access to some of the metadata that I might get if I didn't use the flag.

The intention would be that it's not recommended to use this flag for building production-quality docs, but it would unlock quick exploration in large projects.

### Describe alternatives you've considered

1.  Using dbt Cloud's in-editor DAG
2.  Parsing the manifest manually to create an explorable DAG (maybe this could be bundled as part of a vscode extension 😉)

### Who will this benefit?

Developers of large projects.

### Are you interested in contributing this feature?

Provided it is within my capabilities, then yes!

### Anything else?

I imagine this feature would be (ab)used by many folks who want to build their docs in CI without a DB connection.  I'm not sure that this is behaviour we should actually be preventing though, as long as they understand the tradeoffs.


Discussion:
+1 to this feature request... there was a similar discussion [as back as 2021](https://github.com/dbt-labs/dbt-core/issues/3947).
@jaypeedevlin Thanks for opening! This feels totally reasonable to me. I agree with your take that:
- We could be clear about the intended use case
- Even if someone were to misuse this, I don't think it's something we should actively prevent, as much as discourage / make sure we clearly communicate the trade-offs

### Implementations

I believe such a flag could conditionally wrap [the logic here](https://github.com/dbt-labs/dbt-core/blob/0ef9931d197c996ccd4e3caf641729176eceda99/core/dbt/task/generate.py#L230-L240) in the `GenerateTask`. For example:
```python
        # this should go at the top of the file
        import agate
        
        ...
        
        # initialize empty typed objects, based on return types of get_catalog
        catalog_table: agate.Table = agate.Table([])
        exceptions: List[Exception] = []

        if not self.args.no_connection: # or whatever we call this -- see below
            adapter = get_adapter(self.config)
            with adapter.connection_named("generate_catalog"):
                fire_event(BuildingCatalog())
                catalog_table, exceptions = adapter.get_catalog(self.manifest)
        
        # otherwise, use the empty objects, and proceed
```

What should the flag be called?
- `--no-connection`
- `--no-catalog`, `--empty-catalog`, `--skip-catalog-generation`
- ...?

Then:
```
$ dbt docs generate --no-compile --no-catalog
```
Would produce a `manifest.json` without any "compiled" SQL (since that could require introspective database queries), and without any catalog entries pulled from database metadata.

Going to mark this as a good first issue!

---

Incidentally, I think we could also solve for this via:
- https://github.com/dbt-labs/dbt-core/issues/6014
```
$ dbt docs generate --exclude fqn:* source:*
```
Just sharing as an observation — I don't think that's a reason to not also offer this flag!

---

In the meantime, here's a fun hack, as a treat:
```bash
# write manifest.json -- starting in v1.5, no need to pass this flag, 'parse' always writes
$ dbt parse --write-manifest

# write an empty catalog.json
$ echo '{"metadata": {}, "nodes": {}, "sources": {}, "errors": null}' > target/catalog.json

$ dbt docs serve
```
~Okay, I realize the last step required an Internet connection — but otherwise it was a totally didn't!~ You could even pull down whatever docs site you wanted to use, instead of the one distributed by default with `dbt-core`, if you were someone who wanted to build/use your own :)
FYI, providing the ability to bypass connection would also be very helpful for linting with sqlfluff, right now, in order to lint and test the sql/macros before actually running them it requires an active connection (which is an antipattern in my opinion) https://github.com/sqlfluff/sqlfluff/issues/4397
@manugarri this issue is about generating artifacts for docs, not for compilation.

At the risk of derailing this issue, the reason dbt needs a connection to compile is because jinja-sql supports functions like `run_query()` which call to the database at compile-time. Even if there were a mechanism for compilation without a connection, sqlfluff would not be able to support projects with this (reasonably frequent) pattern.

_Disclosure, I don't work for dbt Labs and this is just my personal opinion_
@jaypeedevlin thanks for the clarification, I would argue that just because we want to support run_query (which might or might not be used in individual dbt projects) we are forcing everyone to setup a connection on their ci/cd. IMHO it should be the other way around, optional connection, and if a user wants to use something like `run_query` then that subset of users can choose to use a connection.
I would encourage you to open a issue (or maybe a discussion is more appropriate) around this and/or find an existing one to continue the conversation!

Resolving patch:
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -185,6 +185,7 @@ def docs(ctx, **kwargs):
 @p.project_dir
 @p.select
 @p.selector
+@p.empty_catalog
 @p.state
 @p.deprecated_state
 @p.target
diff --git a/core/dbt/cli/params.py b/core/dbt/cli/params.py
--- a/core/dbt/cli/params.py
+++ b/core/dbt/cli/params.py
@@ -369,6 +369,13 @@
     "--skip-profile-setup", "-s", envvar=None, help="Skip interactive profile setup.", is_flag=True
 )
 
+empty_catalog = click.option(
+    "--empty-catalog",
+    help="If specified, generate empty catalog.json file during the `dbt docs generate` command.",
+    default=False,
+    is_flag=True,
+)
+
 state = click.option(
     "--state",
     envvar="DBT_STATE",
diff --git a/core/dbt/task/generate.py b/core/dbt/task/generate.py
--- a/core/dbt/task/generate.py
+++ b/core/dbt/task/generate.py
@@ -2,6 +2,7 @@
 import shutil
 from datetime import datetime
 from typing import Dict, List, Any, Optional, Tuple, Set
+import agate
 
 from dbt.dataclass_schema import ValidationError
 
@@ -227,10 +228,14 @@ def run(self) -> CatalogArtifact:
         if self.manifest is None:
             raise DbtInternalError("self.manifest was None in run!")
 
-        adapter = get_adapter(self.config)
-        with adapter.connection_named("generate_catalog"):
-            fire_event(BuildingCatalog())
-            catalog_table, exceptions = adapter.get_catalog(self.manifest)
+        if self.args.empty_catalog:
+            catalog_table: agate.Table = agate.Table([])
+            exceptions: List[Exception] = []
+        else:
+            adapter = get_adapter(self.config)
+            with adapter.connection_named("generate_catalog"):
+                fire_event(BuildingCatalog())
+                catalog_table, exceptions = adapter.get_catalog(self.manifest)
 
         catalog_data: List[PrimitiveDict] = [
             dict(zip(catalog_table.column_names, map(dbt.utils._coerce_decimal, row)))


Test patch:
diff --git a/tests/functional/docs/test_generate.py b/tests/functional/docs/test_generate.py
--- a/tests/functional/docs/test_generate.py
+++ b/tests/functional/docs/test_generate.py
@@ -1,5 +1,5 @@
 import os
-
+import json
 from dbt.tests.util import run_dbt
 
 
@@ -7,3 +7,11 @@ class TestGenerate:
     def test_generate_no_manifest_on_no_compile(self, project):
         run_dbt(["docs", "generate", "--no-compile"])
         assert not os.path.exists("./target/manifest.json")
+
+    def test_generate_empty_catalog(self, project):
+        run_dbt(["docs", "generate", "--empty-catalog"])
+        with open("./target/catalog.json") as file:
+            catalog = json.load(file)
+        assert catalog["nodes"] == {}, "nodes should be empty"
+        assert catalog["sources"] == {}, "sources should be empty"
+        assert catalog["errors"] is None, "errors should be null"