Repository: dbt-labs/dbt-core

Issue:
Provide test name and kwargs in test nodes in the manifest
## Feature

### Feature description
In the documentation website, the columns box shows three different types of built-in schema tests: P, U, N (primary key, unique, not_null). But not for the fourth built-in schema test, `accepted_values`. It also doesn't show any custom schema tests. 

### Who will this benefit?
Anyone who uses `accepted_values` and was expecting to see it in their docs site! For custom schema tests, a single letter would need to be defined...or just stuff the name of it into the Tests column. 🙂 

Link to code: https://github.com/fishtown-analytics/dbt-docs/blob/master/src/app/docs/model.html#L126-L134


Discussion:
This is prioritized for our Wilt Chamberlain release.

If you want to feel sad, have a look at [the code responsible for rendering column tests](https://github.com/fishtown-analytics/dbt-docs/blob/master/src/app/services/project_service.js#L168-L208) in the docs site. The docs site needs to parse test attributes out of generated test names because dbt is not currently encoding this information in the manifest in a structured way.

In order to delete all of this code, as well as support accepted_values tests, we should add a structured dict to `test` nodes with the following information:
1. the name of the test macro
2. the arguments supplied to the test macro (represented as a dictionary)

The docs site is going to need to semantically know what `unique`, `not_null`, etc mean, but I think that's acceptable for now.

In the future, I'd like to make schema tests capable of generating their own descriptions. This would help us render custom schema test information in the docs site. Out of scope for this particular issue, but worth keeping in mind.

## Feature spec:
Include a `test_definition:` (or similar) field in `test` nodes which contains:
 - the test type
 - the kwargs provided to the test

eg:
```
{
  "name": "unique_stg_close_opportunities_opportunity_id"
  "raw_sql": "{{ test_unique(model=ref('stg_close_opportunities'), column_name='opportunity_id') }}",
  ...
  "test_definition": {
    "test_type": "schema",
    "test": "unique",
    "options": {
      "model": "ref('stg_close_opportunities')"
      "column_name": "opportunity_id"
    }
  }
}
```

I think ideally, the `model` option shown here would read as `stg_close_opportunities`, not `ref('stg_close_opportunities')`, but I suspect that might be difficult to implement at the present time. Either one is going to be usable (and much better than what we have now!) in the dbt Docs code.

This will enable us to add accepted values to the docs site, as well as clean up some really gnarly code around relationships tests.

For custom data tests, that field should look like:
```
{
  "name": "unique_stg_close_opportunities_opportunity_id"
  "raw_sql": "select 1 as id",
  ...
  "test_definition": {
    "test_type": "data"
  }
}
```
Kicking this out of Wilt Chamberlain and into Louisa May Alcott

Resolving patch:
diff --git a/core/dbt/contracts/graph/compiled.py b/core/dbt/contracts/graph/compiled.py
--- a/core/dbt/contracts/graph/compiled.py
+++ b/core/dbt/contracts/graph/compiled.py
@@ -9,6 +9,7 @@
     ParsedSourceDefinition,
     ParsedTestNode,
     TestConfig,
+    TestMetadata,
     PARSED_TYPES,
 )
 from dbt.node_types import NodeType
@@ -112,6 +113,7 @@ class CompiledTestNode(CompiledNode):
     resource_type: NodeType = field(metadata={'restrict': [NodeType.Test]})
     column_name: Optional[str] = None
     config: TestConfig = field(default_factory=TestConfig)
+    test_metadata: Optional[TestMetadata] = None
 
 
 def _inject_ctes_into_sql(sql: str, ctes: List[InjectedCTE]) -> str:
diff --git a/core/dbt/contracts/graph/parsed.py b/core/dbt/contracts/graph/parsed.py
--- a/core/dbt/contracts/graph/parsed.py
+++ b/core/dbt/contracts/graph/parsed.py
@@ -260,11 +260,19 @@ class TestConfig(NodeConfig):
     severity: Severity = 'error'
 
 
+@dataclass
+class TestMetadata(JsonSchemaMixin):
+    namespace: Optional[str]
+    name: str
+    kwargs: Dict[str, Any]
+
+
 @dataclass
 class ParsedTestNode(ParsedNode):
     resource_type: NodeType = field(metadata={'restrict': [NodeType.Test]})
     column_name: Optional[str] = None
     config: TestConfig = field(default_factory=TestConfig)
+    test_metadata: Optional[TestMetadata] = None
 
 
 @dataclass(init=False)
diff --git a/core/dbt/parser/schemas.py b/core/dbt/parser/schemas.py
--- a/core/dbt/parser/schemas.py
+++ b/core/dbt/parser/schemas.py
@@ -10,7 +10,11 @@
 from dbt.config.renderer import ConfigRenderer
 from dbt.contracts.graph.manifest import SourceFile
 from dbt.contracts.graph.parsed import (
-    ParsedNodePatch, ParsedTestNode, ParsedSourceDefinition, ColumnInfo, Docref
+    ParsedNodePatch,
+    ParsedSourceDefinition,
+    ColumnInfo,
+    Docref,
+    ParsedTestNode,
 )
 from dbt.contracts.graph.unparsed import (
     UnparsedSourceDefinition, UnparsedNodeUpdate, NamedTested,
@@ -255,6 +259,12 @@ def parse_node(self, block: SchemaTestBlock) -> ParsedTestNode:
 
         config = self.initial_config(fqn)
 
+        metadata = {
+            'namespace': builder.namespace,
+            'name': builder.name,
+            'kwargs': builder.args,
+        }
+
         node = self._create_parsetime_node(
             block=block,
             path=compiled_path,
@@ -263,6 +273,7 @@ def parse_node(self, block: SchemaTestBlock) -> ParsedTestNode:
             name=builder.fqn_name,
             raw_sql=builder.build_raw_sql(),
             column_name=block.column_name,
+            test_metadata=metadata,
         )
         self.render_update(node, config)
         self.add_result_node(block, node)


Test patch:
diff --git a/test/integration/029_docs_generate_tests/test_docs_generate.py b/test/integration/029_docs_generate_tests/test_docs_generate.py
--- a/test/integration/029_docs_generate_tests/test_docs_generate.py
+++ b/test/integration/029_docs_generate_tests/test_docs_generate.py
@@ -961,6 +961,11 @@ def expected_seeded_manifest(self, model_database=None):
                     'extra_ctes': [],
                     'injected_sql': AnyStringWith('count(*)'),
                     'wrapped_sql': AnyStringWith('count(*)'),
+                    'test_metadata': {
+                        'namespace': None,
+                        'name': 'not_null',
+                        'kwargs': {'column_name': 'id'},
+                    },
                 },
                 'test.test.test_nothing_model_': {
                     'alias': 'test_nothing_model_',
@@ -1003,6 +1008,11 @@ def expected_seeded_manifest(self, model_database=None):
                     'extra_ctes': [],
                     'injected_sql': AnyStringWith('select 0'),
                     'wrapped_sql': AnyStringWith('select 0'),
+                    'test_metadata': {
+                        'namespace': 'test',
+                        'name': 'nothing',
+                        'kwargs': {},
+                    },
                 },
                 'test.test.unique_model_id': {
                     'alias': 'unique_model_id',
@@ -1045,6 +1055,11 @@ def expected_seeded_manifest(self, model_database=None):
                     'extra_ctes': [],
                     'injected_sql': AnyStringWith('count(*)'),
                     'wrapped_sql': AnyStringWith('count(*)'),
+                    'test_metadata': {
+                        'namespace': None,
+                        'name': 'unique',
+                        'kwargs': {'column_name': 'id'},
+                    },
                 },
             },
             'parent_map': {
@@ -2455,7 +2470,12 @@ def expected_run_results(self, quote_schema=True, quote_model=False,
                     'database': self.default_database,
                     'tags': ['schema'],
                     'unique_id': 'test.test.not_null_model_id',
-                    'wrapped_sql': AnyStringWith('id is null')
+                    'wrapped_sql': AnyStringWith('id is null'),
+                    'test_metadata': {
+                        'namespace': None,
+                        'name': 'not_null',
+                        'kwargs': {'column_name': 'id'},
+                    },
                 },
                 'thread_id': ANY,
                 'timing': [ANY, ANY],
@@ -2508,6 +2528,11 @@ def expected_run_results(self, quote_schema=True, quote_model=False,
                     'tags': ['schema'],
                     'unique_id': 'test.test.test_nothing_model_',
                     'wrapped_sql': AnyStringWith('select 0'),
+                    'test_metadata': {
+                        'namespace': 'test',
+                        'name': 'nothing',
+                        'kwargs': {},
+                    },
                 },
                 'thread_id': ANY,
                 'timing': [ANY, ANY],
@@ -2559,7 +2584,12 @@ def expected_run_results(self, quote_schema=True, quote_model=False,
                     'sources': [],
                     'tags': ['schema'],
                     'unique_id': 'test.test.unique_model_id',
-                    'wrapped_sql': AnyStringWith('count(*)')
+                    'wrapped_sql': AnyStringWith('count(*)'),
+                    'test_metadata': {
+                        'namespace': None,
+                        'name': 'unique',
+                        'kwargs': {'column_name': 'id'},
+                    },
                 },
                 'thread_id': ANY,
                 'timing': [ANY, ANY],
diff --git a/test/unit/test_parser.py b/test/unit/test_parser.py
--- a/test/unit/test_parser.py
+++ b/test/unit/test_parser.py
@@ -306,6 +306,15 @@ def test__parse_basic_model_tests(self):
         self.assertTrue(tests[0].name.startswith('accepted_values_'))
         self.assertEqual(tests[0].fqn, ['snowplow', 'schema_test', tests[0].name])
         self.assertEqual(tests[0].unique_id.split('.'), ['test', 'snowplow', tests[0].name])
+        self.assertEqual(tests[0].test_metadata.name, 'accepted_values')
+        self.assertIsNone(tests[0].test_metadata.namespace)
+        self.assertEqual(
+            tests[0].test_metadata.kwargs,
+            {
+                'column_name': 'color',
+                'values': ['red', 'blue', 'green'],
+            }
+        )
 
         # foreign packages are a bit weird, they include the macro package
         # name in the test name
@@ -318,6 +327,15 @@ def test__parse_basic_model_tests(self):
         self.assertTrue(tests[1].name.startswith('foreign_package_test_case_'))
         self.assertEqual(tests[1].package_name, 'snowplow')
         self.assertEqual(tests[1].unique_id.split('.'), ['test', 'snowplow', tests[1].name])
+        self.assertEqual(tests[1].test_metadata.name, 'test_case')
+        self.assertEqual(tests[1].test_metadata.namespace, 'foreign_package')
+        self.assertEqual(
+            tests[1].test_metadata.kwargs,
+            {
+                'column_name': 'color',
+                'arg': 100,
+            },
+        )
 
         self.assertEqual(tests[2].config.severity, 'WARN')
         self.assertEqual(tests[2].tags, ['schema'])
@@ -327,6 +345,14 @@ def test__parse_basic_model_tests(self):
         self.assertTrue(tests[2].name.startswith('not_null_'))
         self.assertEqual(tests[2].fqn, ['snowplow', 'schema_test', tests[2].name])
         self.assertEqual(tests[2].unique_id.split('.'), ['test', 'snowplow', tests[2].name])
+        self.assertEqual(tests[2].test_metadata.name, 'not_null')
+        self.assertIsNone(tests[2].test_metadata.namespace)
+        self.assertEqual(
+            tests[2].test_metadata.kwargs,
+            {
+                'column_name': 'color',
+            },
+        )
 
         path = get_abs_os_path('./dbt_modules/snowplow/models/test_one.yml')
         self.assertIn(path, self.parser.results.files)