Repository: dbt-labs/dbt-core

Issue:
[CT-1767] [Feature] `dbt parse` should return a manifest
_Prerequisite: `dbt parse` supported as top-level command in new API (https://github.com/dbt-labs/dbt-core/issues/5550)_

`dbt parse` should return the [`Manifest`](https://github.com/dbt-labs/dbt-core/blob/94d6d19fb43f1ed466c9049dea40e203b23f89da/core/dbt/contracts/graph/manifest.py#L590) as its result. This makes it clearer that _the output of parsing is the manifest*_, and it would also enable programmatic users to do handy things like:
```python
# just for type hint below
from dbt.contracts.graph.manifest import Manifest

# initialize
from dbt.cli.main import dbtRunner
dbt = dbtRunner()

# 'parse' command
results, success = dbt.invoke(['parse'])

# hand waving this part for now -- we'd want 'results' to have a consistent type for all 'invoke' calls
manifest: Manifest = results...['manifest']

# now I can programmatically inspect the project contents!
# pseudo code: assert every non-ephemeral model has an "owner"
for model in [n for n in manifest.nodes if n.resource_type == 'model']:
    if model.materialized != 'ephemeral':
        assert (model.meta['owner'] is not None)
```

_*There are caveats to this statement, of course—the manifest is modified during `compile`, and technically there is other config resolution (profile, project, packages) that happens during parsing and is not included in the manifest—but it still holds 85% true today, and I'd like to see us getting to a place where it's 95% true._

### Current behavior

Currently, `dbt parse` returns `None` as its results.

It's possible to call the (undocumented, internal) `get_full_manifest` method directly and return a manifest. This is what `dbt.lib.parse_to_manifest` does:

https://github.com/dbt-labs/dbt-core/blob/94d6d19fb43f1ed466c9049dea40e203b23f89da/core/dbt/lib.py#L186-L189

The proposal in this issue is to expose that capability within `dbt-core`'s documented, contracted, top-level API.

### Motivation

**Programmatic, read-only access to project manifests**, as part of `dbt-core`'s initial public Python API + "library" capabilities.

This would make it quite easy to write "project quality" rules, that run in `pre-commit` or CI. Because this would be the **actual Manifest Python object**, it would also makes available "helper" methods specific to each node class, which are lost during serialization to flat dictionaries (Jinja `model` + `graph` variables) and JSON (`manifest.json`).

With that great power, comes great responsibility: we'd need to be **very** clear about which classes, attributes, & methods can be considered **public** versus **private.** (To date, all `dbt-core` Python internals are considered private, and any direct use of them is considered undocumented & un-guaranteed functionality.) Is this an API we'd be ready & willing to stand behind? Related: https://github.com/dbt-labs/dbt-core/issues/6391. If we're saying that the real contract is a `proto` class, and any Python dataclasses used to generate those data structures are implementation details (liable to change) rather than contracted APIs in & of themselves, that's something we'd need to document **very clearly**.

### Alternatives

- Parsing `manifest.json` after it's written by `dbt parse` (see https://github.com/dbt-labs/dbt-core/issues/6534) — with the additional cost of writing to / reading from disk, and without the helper methods described above
- Writing these rules in Jinja today. Because it's Jinja, it's gnarlier to implement & harder to test, but it does have the appeal of keeping it all in "dbt code"—which also means it can be installed as a dbt package, and run in dbt Cloud—without requiring additional Python scripting / infrastructure to run. The dbt Labs ProServ team has shown it's possible: https://github.com/dbt-labs/dbt-project-evaluator


Discussion:
An alternative would be to return _just_ `Manifest.flat_graph`, which serializes everything to vanilla data structures (lists/dicts), and more closely matches the existing contents of `manifest.json` and the Jinja `{{ graph }}` context variable.

The downside is, we're not giving users access to the ergonomics of our built-in classes/methods. It's really no different from reading in `manifest.json`, just without the extra step of needing to write/read a file.

The upside would be, we wouldn't be exposing _any_ of the internals that we might not be comfortable committing to maintaining as a public interface.
We're already a bit inconsistent as far as the return type of various commands. The "building" commands (`run`, `test`, `snapshot`, `seed`, `build`) all return `RunExecutionResult` (&rarr; `run_results.json`), but not others:
- `source freshness` returns a `FreshnessResult` (&rarr; `sources.json`)
- `docs generate` returns a `CatalogArtifact` (&rarr; `catalog.json`)
```python
>>> from dbt.cli.main import dbtRunner
>>> dbt = dbtRunner()
>>> results, success = dbt.invoke(['docs', 'generate'])
07:44:09  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 401 macros, 0 operations, 0 seed files, 0 sources, 0 exposures, 0 metrics
07:44:09
07:44:09  Concurrency: 1 threads (target='dev')
07:44:09
07:44:09  Done.
07:44:09  Building catalog
07:44:09  Catalog written to /Users/jerco/dev/scratch/testy/target/catalog.json
>>> type(results)
<class 'dbt.contracts.results.CatalogArtifact'>
```
The `dbt parse` command should return a `Manifest` to unblock shorter-term needs. Let's wrap the `Manifest` in a `ParseResult`, and explicitly say that we're not guaranteeing any of its underlying methods or anything about its structure in the future.

Out of scope (but worth doing later): creating a base result class, and having all commands return a subclass of that base class.

Resolving patch:
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -53,7 +53,8 @@ class dbtRunnerResult:
         bool,  # debug
         CatalogArtifact,  # docs generate
         List[str],  # list/ls
-        None,  # clean, deps, init, parse, source
+        Manifest,  # parse
+        None,  # clean, deps, init, source
         RunExecutionResult,  # build, compile, run, seed, snapshot, test
         RunOperationResultsArtifact,  # run-operation
     ] = None
@@ -533,7 +534,8 @@ def list(ctx, **kwargs):
 def parse(ctx, **kwargs):
     """Parses the project and provides information on performance"""
     # manifest generation and writing happens in @requires.manifest
-    return None, True
+
+    return ctx.obj["manifest"], True
 
 
 # dbt run


Test patch: