Repository: dbt-labs/dbt-core
Pull request: Generate API docs
resolves #5528

### Description

Allows for the auto generation of docs for the high level CLI API. This does not also generate those docs via Github Actions, only enables their generation.

### Checklist

- [X] I have read [the contributing guide](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md) and understand what's expected of me
- [X] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
- [X] I have run this code in development and it appears to resolve the stated issue
- [ ] This PR includes tests, or tests are not required/relevant for this PR
- [ ] I have [opened an issue to add/update docs](https://github.com/dbt-labs/docs.getdbt.com/issues/new/choose), or docs changes are not required/relevant for this PR
- [ ] I have run `changie new` to [create a changelog entry](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#adding-a-changelog-entry)


Problem statement:
[CT-902] CLI (Top level API) is automatically documented
**User Story:**
As a user I would like to have accurate and easy to read documentation for how to use `dbt-core` as a library.

**Acceptance Criteria:**
An auto-documentation product (probably Sphinx) is set up and documents the top-level api.  This runs as a GHA.

Hints from issue discussion:
Let's make sure we account for accurate help text for commands which is referenced in this issue:
https://github.com/dbt-labs/dbt-core/issues/5319
FYI-- I think there's been some confusion vis-a-vis the documentation.  This ticket is to generate sphinx docs as a reference, it doesn't refer to help text at all.

That said, after looking at #5319  we don't have to worry about it.  The behavior described there is the default  behavior for Click CLIs so we get that for "free"

Commit message: Add changie entry

Patch to review:
diff --git a/.changes/unreleased/Docs-20221007-090656.yaml b/.changes/unreleased/Docs-20221007-090656.yaml
new file mode 100644
index 00000000000..1159879a249
--- /dev/null
+++ b/.changes/unreleased/Docs-20221007-090656.yaml
@@ -0,0 +1,7 @@
+kind: Docs
+body: Generate API docs for new CLI interface
+time: 2022-10-07T09:06:56.446078-05:00
+custom:
+  Author: stu-k
+  Issue: "5528"
+  PR: "6022"
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
index 3f3b94ea9e3..a6030b8e4c8 100644
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -409,4 +409,5 @@ def test(ctx, **kwargs):
 
 # Support running as a module
 if __name__ == "__main__":
+    breakpoint()
     cli_runner()
diff --git a/core/dbt/docs/Makefile b/core/dbt/docs/Makefile
new file mode 100644
index 00000000000..d0c3cbf1020
--- /dev/null
+++ b/core/dbt/docs/Makefile
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/core/dbt/docs/README.rst b/core/dbt/docs/README.rst
new file mode 100644
index 00000000000..9deee7d0efd
--- /dev/null
+++ b/core/dbt/docs/README.rst
@@ -0,0 +1,5 @@
+dbt-core
+=======
+
+**dbt-core** An ELT tool for managing your SQL transformations and data models.
+For more documentation on functionality, visit: docs.getdbt.com
diff --git a/core/dbt/docs/make.bat b/core/dbt/docs/make.bat
new file mode 100644
index 00000000000..747ffb7b303
--- /dev/null
+++ b/core/dbt/docs/make.bat
@@ -0,0 +1,35 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd
diff --git a/core/dbt/docs/source/_ext/dbt_click.py b/core/dbt/docs/source/_ext/dbt_click.py
new file mode 100644
index 00000000000..ed62aa9fb0a
--- /dev/null
+++ b/core/dbt/docs/source/_ext/dbt_click.py
@@ -0,0 +1,116 @@
+import click
+import click.types as click_t
+import dbt.cli.option_types as dbt_t
+from docutils import nodes
+from docutils.parsers.rst import Directive
+import traceback
+import typing as t
+
+
+PARAM_TYPE_MAP = {
+    click_t.BoolParamType: lambda _: "boolean",
+    click_t.Choice: lambda c: f"choice: {c.choices}",
+    click_t.IntParamType: lambda _: "int",
+    click_t.Path: lambda _: "path",
+    click_t.StringParamType: lambda _: "string",
+    dbt_t.YAML: lambda _: "YAML",
+}
+
+
+def format_command(cmd) -> nodes.Section:
+    cmd_name = cmd.name.replace("-", "_")
+    section = nodes.section(
+        "",
+        nodes.title(text=f"Command: {cmd_name}"),
+        ids=[cmd_name],
+        names=[cmd_name],
+    )
+    section.extend(format_params(cmd))
+    return section
+
+
+def format_params(cmd) -> t.List[nodes.Section]:
+    lines = []
+    for param in cmd.params:
+        uid = f"{cmd.name}|{param.name}"
+        param_section = nodes.section(
+            "",
+            nodes.title(text=param.name),
+            ids=[uid],
+            names=[uid],
+        )
+
+        get_type_str = PARAM_TYPE_MAP.get(type(param.type), lambda _: "unknown")
+        type_str = get_type_str(param.type)
+
+        param_section.append(nodes.paragraph(text=f"Type: {type_str}"))
+        param_section.append(nodes.paragraph(text=param.help))
+        lines.append(param_section)
+    return lines
+
+
+def load_module(module_path: str, error) -> t.Union[click.Command, click.Group]:
+    try:
+        module_name, attr_name = module_path.split(":", 1)
+    except ValueError:  # noqa
+        raise error(f'"{module_path}" is not of format "module:parser"')
+
+    try:
+        mod = __import__(module_name, globals(), locals(), [attr_name])
+    except Exception:  # noqa
+        raise error(
+            f'Failed to import "{attr_name}" from "{module_name}". '
+            f"The following exception was raised:\n{traceback.format_exc()}"
+        )
+
+    if not hasattr(mod, attr_name):
+        raise error(f'Module "{module_name}" has no attribute "{attr_name}"')
+
+    parser = getattr(mod, attr_name)
+
+    if not isinstance(parser, (click.Command, click.Group)):
+        raise error(
+            f'"{type(parser)}" of type "{module_path}" is not click.Command'
+            ' or click.Group."click.BaseCommand"'
+        )
+    return parser
+
+
+class DBTClick(Directive):
+    has_content = False
+    required_arguments = 1
+
+    def run(self):
+        section = nodes.section(
+            "",
+            ids=["dbt-section"],
+            names=["dbt-section"],
+        )
+        cmds = self._get_commands(self.arguments[0])
+        for cmd in cmds:
+            command_section = format_command(cmd)
+            section.extend(command_section)
+        return [section]
+
+    def _get_commands(self, module: str) -> t.List[click.Command]:
+        click_group = load_module(module, self.error)
+        if type(click_group) is not click.Group:
+            raise self.error('Type "click.Group" not supported in dbt_click extension')
+        cmd_strs = [cmd for cmd in click_group.commands]
+        cmd_strs.sort()
+        cmds = []
+        for cmd_str in click_group.commands:
+            cmd = click_group.commands.get(cmd_str)
+            if cmd is not None:
+                cmds.append(cmd)
+        return cmds
+
+
+def setup(app) -> t.Dict[str, t.Any]:
+    app.add_directive("dbt_click", DBTClick)
+
+    return {
+        "version": "0.1",
+        "parallel_read_safe": True,
+        "parallel_write_safe": True,
+    }
diff --git a/core/dbt/docs/source/conf.py b/core/dbt/docs/source/conf.py
new file mode 100644
index 00000000000..25c0a3b4456
--- /dev/null
+++ b/core/dbt/docs/source/conf.py
@@ -0,0 +1,36 @@
+import sys
+import os
+import typing as t
+
+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+sys.path.insert(0, os.path.abspath("../.."))
+sys.path.insert(0, os.path.abspath("./_ext"))
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+project = "dbt-core"
+copyright = "2022, dbt Labs"
+author = "dbt Labs"
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = [
+    "sphinx.ext.autodoc",
+    "sphinx_click",
+    "dbt_click",
+]
+
+templates_path = ["_templates"]
+exclude_patterns: t.List[str] = []
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = "alabaster"
+html_static_path = ["_static"]
diff --git a/core/dbt/docs/source/index.rst b/core/dbt/docs/source/index.rst
new file mode 100644
index 00000000000..d5e3c6007af
--- /dev/null
+++ b/core/dbt/docs/source/index.rst
@@ -0,0 +1,4 @@
+dbt-core's API documentation
+============================
+
+.. dbt_click:: dbt.cli.main:cli
diff --git a/dev-requirements.txt b/dev-requirements.txt
index b8095db1cab..ec3bd0632e0 100644
--- a/dev-requirements.txt
+++ b/dev-requirements.txt
@@ -18,6 +18,7 @@ pytz
 tox>=3.13
 twine
 types-colorama
+types-docutils
 types-PyYAML
 types-freezegun
 types-Jinja2


Code review comments:
diff_hunk: @@ -0,0 +1,5 @@
+dbt-core
+=======
+
+**dbt-core** An ELT tool for managing your SQL transformations and data models.
path: core/dbt/docs/README.rst
text: @user1:
Just for understanding, is this file used to generate some header for the generated doc?

@author:
I'm actually not sure if this is necessary, this isn't part of what is generated. I'm attempting to remove it and test if everything still works as intended, but I'm having some issues with mypy stubs. I'll check once I resolve those.

@author:
Removed!

diff_hunk: @@ -409,4 +409,5 @@ def test(ctx, **kwargs):
 
 # Support running as a module
 if __name__ == "__main__":
+    breakpoint()
path: core/dbt/cli/main.py
text: @user1:
remove before merge?