Repository: dbt-labs/dbt-core

Issue:
[CT-3078] Improve diff displayed in stdout when a test fails
### Description

Currently, a failing unit test simply logs both the actual and expected results to stdout:

```sh
❯ dbt unit-test
20:34:18  Running with dbt=1.7.0-b1
20:34:18  target not specified in profile 'postgres', using 'default'
20:34:18  Registered adapter: postgres=1.7.0-b1
20:34:19  Found 10 models, 3 seeds, 20 tests, 0 sources, 0 exposures, 0 metrics, 360 macros, 0 groups, 0 semantic models, 5 unit tests
20:34:19  Found 1 unit test, 2 models, 0 sources, 0 exposures, 0 metrics, 360 macros, 0 groups, 0 semantic models
20:34:19  
20:34:19  Concurrency: 1 threads (target='default')
20:34:19  
20:34:19  1 of 1 START unit_test my_model__test_my_model ................................. [RUN]
20:34:19  1 of 1 FAIL 1 my_model__test_my_model .......................................... [FAIL 1 in 0.11s]
20:34:19  
20:34:19  Finished running 1 unit_test in 0 hours 0 minutes and 0.53 seconds (0.53s).
20:34:19  
20:34:19  Completed with 1 error and 0 warnings:
20:34:19  
20:34:19  Failure in unit_test my_model__test_my_model (models/unit_testing/schema.yml)
20:34:19    

Actual:
| c |
| - |
| 3 |

Expected:
| c |
| - |
| 4 |

20:34:19  
20:34:19    compiled code at target/compiled/jaffle_shop/models/unit_testing/schema.yml/models/unit_testing/my_model__test_my_model.sql
20:34:19  
20:34:19  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
```

This may work fine for very small expected results (few rows & columns), but would be difficult to debug for larger results as the diff is not visualized. For example: 


```sh
❯ dbt unit-test
...
Found 1 differing row between actual and expected outputs for my_model:
Actual
- {'c': 3}
          ^
Expected
+ {'c': 4}
```

### Acceptance Criteria   

Test failure message displays diff in a readable format                                                                                  
                                                                                                                                                         
### Impact to other teams
                  
None                 
                                
### Will backports be required?
                                                                                                         
No              

### Context


Discussion:
Of the formats I've seen thus far, the [daff](https://paulfitz.github.io/daff/) format is the one I prefer since it makes it easier to see exactly where the differences are. The tabular diff specification for daff is described in more detail [here](https://paulfitz.github.io/daff-doc/spec.html).

<img width="345" alt="image" src="https://github.com/dbt-labs/dbt-core/assets/44704949/80bb8140-6e1b-4083-8bac-9c1de2cb05f2">
Since we're allowing folks to define fixtures in separate csv files, I think it would be good to include the input mock data in the logs as well so they don't have to go splunking through a ton of files.

Resolving patch:
diff --git a/core/dbt/clients/agate_helper.py b/core/dbt/clients/agate_helper.py
--- a/core/dbt/clients/agate_helper.py
+++ b/core/dbt/clients/agate_helper.py
@@ -3,6 +3,7 @@
 import agate
 import datetime
 import isodate
+import io
 import json
 import dbt.utils
 from typing import Iterable, List, Dict, Union, Optional, Any
@@ -137,6 +138,23 @@ def table_from_data_flat(data, column_names: Iterable[str]) -> agate.Table:
     )
 
 
+def json_rows_from_table(table: agate.Table) -> List[Dict[str, Any]]:
+    "Convert a table to a list of row dict objects"
+    output = io.StringIO()
+    table.to_json(path=output)  # type: ignore
+
+    return json.loads(output.getvalue())
+
+
+def list_rows_from_table(table: agate.Table) -> List[Any]:
+    "Convert a table to a list of lists, where the first element represents the header"
+    rows = [[col.name for col in table.columns]]
+    for row in table.rows:
+        rows.append(list(row.values()))
+
+    return rows
+
+
 def empty_table():
     "Returns an empty Agate table. To be used in place of None"
 
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -81,6 +81,7 @@
         "protobuf>=4.0.0",
         "pytz>=2015.7",
         "pyyaml>=6.0",
+        "daff>=1.3.46",
         "typing-extensions>=4.4",
         # ----
         # Match snowflake-connector-python, to ensure compatibility in dbt-snowflake


Test patch:
diff --git a/core/dbt/task/unit_test.py b/core/dbt/task/unit_test.py
--- a/core/dbt/task/unit_test.py
+++ b/core/dbt/task/unit_test.py
@@ -1,13 +1,16 @@
+import agate
 from dataclasses import dataclass
 from dbt.dataclass_schema import dbtClassMixin
+import daff
 import threading
-from typing import Dict, Any, Optional, AbstractSet
-import io
+import re
+from typing import Dict, Any, Optional, AbstractSet, List
 
 from .compile import CompileRunner
 from .run import RunTask
 
 from dbt.adapters.factory import get_adapter
+from dbt.clients.agate_helper import list_rows_from_table, json_rows_from_table
 from dbt.contracts.graph.nodes import UnitTestNode
 from dbt.contracts.graph.manifest import Manifest
 from dbt.contracts.results import TestStatus, RunResult
@@ -25,16 +28,26 @@
 )
 from dbt.node_types import NodeType
 from dbt.parser.unit_tests import UnitTestManifestLoader
+from dbt.ui import green, red
+
+
+@dataclass
+class UnitTestDiff(dbtClassMixin):
+    actual: List[Dict[str, Any]]
+    expected: List[Dict[str, Any]]
+    rendered: str
 
 
 @dataclass
 class UnitTestResultData(dbtClassMixin):
     should_error: bool
     adapter_response: Dict[str, Any]
-    diff: Optional[str] = None
+    diff: Optional[UnitTestDiff] = None
 
 
 class UnitTestRunner(CompileRunner):
+    _ANSI_ESCAPE = re.compile(r"\x1B(?:[@-Z\\-_]|\[[0-?]*[ -/]*[@-~])")
+
     def describe_node(self):
         return f"{self.node.resource_type} {self.node.name}"
 
@@ -95,15 +108,23 @@ def execute_unit_test(self, node: UnitTestNode, manifest: Manifest) -> UnitTestR
         result = context["load_result"]("main")
         adapter_response = result["response"].to_dict(omit_none=True)
         table = result["table"]
-        actual = self._get_unit_test_table(table, "actual")
-        expected = self._get_unit_test_table(table, "expected")
-        should_error = actual.rows != expected.rows
-        diff = None
-        if should_error:
-            actual_output = self._agate_table_to_str(actual)
-            expected_output = self._agate_table_to_str(expected)
-
-            diff = f"\n\nActual:\n{actual_output}\n\nExpected:\n{expected_output}\n"
+        actual = self._get_unit_test_agate_table(table, "actual")
+        expected = self._get_unit_test_agate_table(table, "expected")
+
+        # generate diff, if exists
+        should_error, diff = False, None
+        daff_diff = self._get_daff_diff(expected, actual)
+        if daff_diff.hasDifference():
+            should_error = True
+            rendered = self._render_daff_diff(daff_diff)
+            rendered = f"\n\n{red('expected')} differs from {green('actual')}:\n\n{rendered}\n"
+
+            diff = UnitTestDiff(
+                actual=json_rows_from_table(actual),
+                expected=json_rows_from_table(expected),
+                rendered=rendered,
+            )
+
         return UnitTestResultData(
             diff=diff,
             should_error=should_error,
@@ -119,7 +140,7 @@ def execute(self, node: UnitTestNode, manifest: Manifest):
         failures = 0
         if result.should_error:
             status = TestStatus.Fail
-            message = result.diff
+            message = result.diff.rendered if result.diff else None
             failures = 1
 
         return RunResult(
@@ -136,7 +157,7 @@ def execute(self, node: UnitTestNode, manifest: Manifest):
     def after_execute(self, result):
         self.print_result_line(result)
 
-    def _get_unit_test_table(self, result_table, actual_or_expected: str):
+    def _get_unit_test_agate_table(self, result_table, actual_or_expected: str) -> agate.Table:
         unit_test_table = result_table.where(
             lambda row: row["actual_or_expected"] == actual_or_expected
         )
@@ -144,14 +165,32 @@ def _get_unit_test_table(self, result_table, actual_or_expected: str):
         columns.remove("actual_or_expected")
         return unit_test_table.select(columns)
 
-    def _agate_table_to_str(self, table) -> str:
-        # Hack to get Agate table output as string
-        output = io.StringIO()
-        if self.config.args.output == "json":
-            table.to_json(path=output)
-        else:
-            table.print_table(output=output, max_rows=None)
-        return output.getvalue().strip()
+    def _get_daff_diff(
+        self, expected: agate.Table, actual: agate.Table, ordered: bool = False
+    ) -> daff.TableDiff:
+
+        expected_daff_table = daff.PythonTableView(list_rows_from_table(expected))
+        actual_daff_table = daff.PythonTableView(list_rows_from_table(actual))
+
+        alignment = daff.Coopy.compareTables(expected_daff_table, actual_daff_table).align()
+        result = daff.PythonTableView([])
+
+        flags = daff.CompareFlags()
+        flags.ordered = ordered
+
+        diff = daff.TableDiff(alignment, flags)
+        diff.hilite(result)
+        return diff
+
+    def _render_daff_diff(self, daff_diff: daff.TableDiff) -> str:
+        result = daff.PythonTableView([])
+        daff_diff.hilite(result)
+        rendered = daff.TerminalDiffRender().render(result)
+        # strip colors if necessary
+        if not self.config.args.use_colors:
+            rendered = self._ANSI_ESCAPE.sub("", rendered)
+
+        return rendered
 
 
 class UnitTestSelector(ResourceTypeSelector):