Repository: dbt-labs/dbt-core

Issue:
Write out manifest.json file when the RPC server is HUP'd
### Describe the feature

Use case: view the documentation site rendered from the state in the rpc server. Desired flow:
- edit some model sql, documentation, tests, etc
- HUP the server
- reload the index.html file from the `target/` dir and see the new docs

Challenges:
- serializing the manifest can be quite slow. Is this an unavoidable cost, or is there something we can do to support this feature without adding a significant delay to server HUP time?
- this should work well with partial parsing
- starting or HUP'ing the server does not actually invoke a _compile_ -- in order to achieve the flow stated above, we would in fact need to invoke a whole `dbt compile`!

Tagging @beckjake to confirm these challenges above in case I got anything wrong.

### Describe alternatives you've considered
I think that the use case described above is _absolutely_ something we should work to support in the future, but it might require us to make server reloading work meaningfully differently. A shorter path to supporting this type of use case could just be to make dbt invocations (like `dbt compile`, `dbt run`) write out the manifest (and other artifacts) at the end of the run. This is already supported for `dbt docs generate`.

### Who will this benefit
This will allow us to support docs viewing in the dbt Cloud IDE, as well as a similar flow for anyone else hosting the RPC server in a dev environment.


Discussion:
>serializing the manifest can be quite slow. Is this an unavoidable cost, or is there something we can do to support this feature without adding a significant delay to server HUP time?

On reload, the manifest is built in a background thread. We could just serialize it there too. I'm not sure what we'd do to make serializing not slow - I guess we could replace our current `json.dumps` with something faster: `orjson` looks like it's the current state of the art. I think a lot of the time and effort is spent on jinja-rendering (required) and converting data into json (obviously required...). We could write less data to the manifest file, or present an alternative `manifest-slim.json`, etc. We could even just try calling the `to_dict` method with the flag that tells it to skip serializing `None` values!

> starting or HUP'ing the server does not actually invoke a compile -- in order to achieve the flow stated above, we would in fact need to invoke a whole dbt compile!

If you want the compiled SQL: Yeah.
If you don't care, we could serialize the parsed manifest before compiling (that's what `dbt compile` does).


can we try out orjson (specifically https://github.com/ijl/orjson#dataclass), and see how it performs for manifest serialization before we make any big decisions here? 

@beckjake if i remember correctly, the last time we tried to do this, it was specifically json serialization that was slow, and why we decided to use pickles for partial parsing. do you still have any JSON vs. pickle benchmark data we can look at?

if it works as advertised (40-50x speedup on dataclasses) then it seems like it could get us very far.

EDIT: I realize I may be confusing parseresult with compiled manifest... anyway still interested to see how orjson compares
Well, it says it's 40-50x faster than other json libraries that serialize dataclasses, who knows what those or or what they're up to. I'm not sure it supports all the stuff we use hologram for.

I don't have any benchmark data, I didn't save it. I'm pretty sure it was the json part is what was slow, though of course if you pickle you also get to bypass the `hologram` `to_dict` stuff, which is extra bonus. If memory serves, it was not even close for large-ish manifests. I think parse results will be comparable to manifests - the differences are pretty minor, really: parse results are just sort of proto-manifests.
Let's make this an RPC method that returns the manifest from memory. TODO:
 - Figure out what we do about catalog and run_results (@drewbanin) cc @cmcarthur 

Resolving patch:
diff --git a/core/dbt/contracts/rpc.py b/core/dbt/contracts/rpc.py
--- a/core/dbt/contracts/rpc.py
+++ b/core/dbt/contracts/rpc.py
@@ -9,6 +9,7 @@
 from hologram.helpers import StrEnum
 
 from dbt.contracts.graph.compiled import CompileResultNode
+from dbt.contracts.graph.manifest import WritableManifest
 from dbt.contracts.results import (
     TimingInfo,
     CatalogResults,
@@ -143,8 +144,13 @@ class RPCSourceFreshnessParameters(RPCParameters):
     select: Union[None, str, List[str]] = None
 
 
+@dataclass
+class GetManifestParameters(RPCParameters):
+    pass
+
 # Outputs
 
+
 @dataclass
 class RemoteResult(JsonSchemaMixin):
     logs: List[LogMessage]
@@ -322,6 +328,11 @@ class KillResult(RemoteResult):
     logs: List[LogMessage] = field(default_factory=list)
 
 
+@dataclass
+class GetManifestResult(RemoteResult):
+    manifest: Optional[WritableManifest]
+
+
 # this is kind of carefuly structured: BlocksManifestTasks is implied by
 # RequiresConfigReloadBefore and RequiresManifestReloadAfter
 class RemoteMethodFlags(enum.Flag):
@@ -526,8 +537,34 @@ class PollInProgressResult(PollResult):
     pass
 
 
+@dataclass
+class PollGetManifestResult(GetManifestResult, PollResult):
+    state: TaskHandlerState = field(
+        metadata=restrict_to(TaskHandlerState.Success,
+                             TaskHandlerState.Failed),
+    )
+
+    @classmethod
+    def from_result(
+        cls: Type['PollGetManifestResult'],
+        base: GetManifestResult,
+        tags: TaskTags,
+        timing: TaskTiming,
+        logs: List[LogMessage],
+    ) -> 'PollGetManifestResult':
+        return cls(
+            manifest=base.manifest,
+            logs=logs,
+            tags=tags,
+            state=timing.state,
+            start=timing.start,
+            end=timing.end,
+            elapsed=timing.elapsed,
+        )
+
 # Manifest parsing types
 
+
 class ManifestStatus(StrEnum):
     Init = 'init'
     Compiling = 'compiling'
diff --git a/core/dbt/rpc/builtins.py b/core/dbt/rpc/builtins.py
--- a/core/dbt/rpc/builtins.py
+++ b/core/dbt/rpc/builtins.py
@@ -10,6 +10,7 @@
     LastParse,
     GCParameters,
     GCResult,
+    GetManifestResult,
     KillParameters,
     KillResult,
     KillResultStatus,
@@ -27,6 +28,7 @@
     PollInProgressResult,
     PollKilledResult,
     PollExecuteCompleteResult,
+    PollGetManifestResult,
     PollRunCompleteResult,
     PollCompileCompleteResult,
     PollCatalogCompleteResult,
@@ -144,6 +146,7 @@ def poll_complete(
         PollCatalogCompleteResult,
         PollRemoteEmptyCompleteResult,
         PollRunOperationCompleteResult,
+        PollGetManifestResult
     ]]
 
     if isinstance(result, RemoteExecutionResult):
@@ -159,6 +162,8 @@ def poll_complete(
         cls = PollRemoteEmptyCompleteResult
     elif isinstance(result, RemoteRunOperationResult):
         cls = PollRunOperationCompleteResult
+    elif isinstance(result, GetManifestResult):
+        cls = PollGetManifestResult
     else:
         raise dbt.exceptions.InternalException(
             'got invalid result in poll_complete: {}'.format(result)
diff --git a/core/dbt/task/rpc/project_commands.py b/core/dbt/task/rpc/project_commands.py
--- a/core/dbt/task/rpc/project_commands.py
+++ b/core/dbt/task/rpc/project_commands.py
@@ -1,8 +1,10 @@
 from datetime import datetime
 from typing import List, Optional, Union
 
-
+from dbt.contracts.graph.manifest import WritableManifest
 from dbt.contracts.rpc import (
+    GetManifestParameters,
+    GetManifestResult,
     RPCCompileParameters,
     RPCDocsGenerateParameters,
     RPCRunOperationParameters,
@@ -188,3 +190,30 @@ def set_args(self, params: RPCSourceFreshnessParameters) -> None:
         if params.threads is not None:
             self.args.threads = params.threads
         self.args.output = None
+
+
+# this is a weird and special method.
+class GetManifest(
+    RemoteManifestMethod[GetManifestParameters, GetManifestResult]
+):
+    METHOD_NAME = 'get-manifest'
+
+    def set_args(self, params: GetManifestParameters) -> None:
+        self.args.models = None
+        self.args.exclude = None
+
+    def handle_request(self) -> GetManifestResult:
+        task = RemoteCompileProjectTask(self.args, self.config, self.manifest)
+        task.handle_request()
+
+        manifest: Optional[WritableManifest] = None
+        if task.manifest is not None:
+            manifest = task.manifest.writable_manifest()
+
+        return GetManifestResult(
+            logs=[],
+            manifest=manifest,
+        )
+
+    def interpret_results(self, results):
+        return results.manifest is not None


Test patch:
diff --git a/test/rpc/test_base.py b/test/rpc/test_base.py
--- a/test/rpc/test_base.py
+++ b/test/rpc/test_base.py
@@ -885,3 +885,32 @@ def test_rpc_vars(
         results = querier.async_wait_for_result(querier.cli_args('run --vars "{param: 100}"'))
         assert len(results['results']) == 1
         assert results['results'][0]['node']['compiled_sql'] == 'select 100 as id'
+
+
+def test_get_manifest(
+    project_root, profiles_root, postgres_profile, unique_schema
+):
+    project = ProjectDefinition(
+        models={
+            'my_model.sql': 'select 1 as id',
+        },
+    )
+    querier_ctx = get_querier(
+        project_def=project,
+        project_dir=project_root,
+        profiles_dir=profiles_root,
+        schema=unique_schema,
+        test_kwargs={},
+    )
+
+    with querier_ctx as querier:
+        results = querier.async_wait_for_result(querier.cli_args('run'))
+        assert len(results['results']) == 1
+        assert results['results'][0]['node']['compiled_sql'] == 'select 1 as id'
+        result = querier.async_wait_for_result(querier.get_manifest())
+        assert 'manifest' in result
+        manifest = result['manifest']
+        assert manifest['nodes']['model.test.my_model']['raw_sql'] == 'select 1 as id'
+        assert 'manifest' in result
+        manifest = result['manifest']
+        assert manifest['nodes']['model.test.my_model']['compiled_sql'] == 'select 1 as id'
diff --git a/test/rpc/util.py b/test/rpc/util.py
--- a/test/rpc/util.py
+++ b/test/rpc/util.py
@@ -369,6 +369,11 @@ def run_sql(
             method='run_sql', params=params, request_id=request_id
         )
 
+    def get_manifest(self, request_id=1):
+        return self.request(
+            method='get-manifest', params={}, request_id=request_id
+        )
+
     def is_result(self, data: Dict[str, Any], id=None) -> Dict[str, Any]:
         if id is not None:
             assert data['id'] == id