Repository: meltano/meltano

Issue:
Prettier exception prints
Migrated from GitLab: https://gitlab.com/meltano/meltano/-/issues/3037

Originally created by @aaronsteers on 2021-11-02 18:56:27

---

On the structlog site, there's some example screenshots of exceptions that are inherently very easy to read due to formatting and structure. 

Should we look at adopting something like [rich](https://rich.readthedocs.io/) or [better-exceptions](https://github.com/Qix-/better-exceptions) for Meltano and/or the SDK?

The docs for structlog seem to imply we only need to install the libraries, but I expect it may be slightly more work, and there could be reasons I'm not thinking of why we would not want to use them.

https://www.structlog.org/en/stable/development.html:

![image](http://gitlab.com/meltano/meltano/uploads/3e18f622a003275ef6358c18dab5977c/image.png)


Discussion:
[View 6 previous comments from the original issue on GitLab](https://gitlab.com/meltano/meltano/-/issues/3037)
@pandemicsyn, @edgarrmondragon, @kgpayne 

I'd like us to pick this up - specifically to add support for prettier exception prints - and to do so behind a feature flag.

When we originally discussed this, we didn't have feature flags yet so the change was deemed excessively risky given people's various preferences and given that different execution environments would have different rendering experiences.

Since we're already on `structlog`, it seems the overall dev cost is pretty low, and the potential improvement to user experience when doing local debug is very high.

@pandemicsyn, @edgarrmondragon - Anything we need to do to disambiguate this spec and size? Is `4` an appropriate weight estimate, given a minimal initial spec and given the expectation that we launch behind a feature flag, disabled by default?

🥷  Ninja edit: According to the docs, all we need to do is import the library. The effort would be (presumably) in _disabling_ that default behavior if the feature flag is not set.

Additional context:

To make a better experience for times when `structlog` is distracting and hard to read, I'd like to make a more viable experience when log level is set very high, such as to the `WARNING` level. If exceptions are clear and readable and containing the needed context, then we have less need for very verbose logging.

Not in any way a requirement at the current stage, but I think this investment starts us in the right direction of getting exception/failure messages to be "the thing" rather than always needing DEBUG logging turned on. Currently, there's an impression that our users have to make a choice every time they run meltano - "Do I think it might fail? If so, I need DEBUG logging turned on." But a more desirable situation would be that they almost never need DEBUG logging - instead relying on good exception message prints and then clicking the a link to a detailed log file for further analysis when needed.


> @pandemicsyn, @edgarrmondragon - Anything we need to do to disambiguate this spec and size? Is 4 an appropriate weight estimate, given a minimal initial spec and given the expectation that we launch behind a feature flag, disabled by default?

If the spec is just "enable pretty printing of exceptions via rich and the ff flag is turned on" the weights probably a 2 or 4.  I think they automatically turn on when rich is installed (which is actually super annoying), but when we talked about this last, I think I went and looked and pretty sure there's also an accessible flag that lets you explicitly control the exception formatter in structlog.  IMHO that would be the preferred path.  

 > But a more desirable situation would be that they almost never need DEBUG logging - instead relying on good exception message prints and then clicking the a link to a detailed log file for further analysis when needed.

110% agree, and something I think I've been complaining about since I joined (but probably not loudly enough). Our error handling, logging, and messaging is very inconsistent, sometimes we're too verbose, sometimes not verbose enough and all you get is an exception name raised to click etc. Its definitely messy.

My general rule of thumb has always been that, DEBUG level logging should almost never be **required** by end-users to troubleshoot user induced error. Broadly speaking, DEBUG level is what we should be using to debug ...bugs in meltano. 
@aaronsteers (and anyone who picks this up in future)

> 🥷 Ninja edit: According to the docs, all we need to do is import the library. The effort would be (presumably) in disabling that default behavior if the feature flag is not set.

@edgarrmondragon and I found a way to [disable the default behaviour](https://github.com/meltano/meltano/pull/6103/files#diff-24a50af78828a9d2013a489f195be20afb9a5d809a574aef2b6f6aeea09fb6a3R78) in #6103 by setting the `exception_formatter` back to `plain_traceback` in the stuctlog config 🙂
@kgpayne - with the learnings from #6103 it sounds like this is closer now to a `2` in weight. Would you agree?

Specifically:

1. We add a boolean like `ff.print_rich_exceptions` (defaulted to `false`).
   - Alternatively: something like `ff.exception_formatter=[rich|default|...]`
3. We keep existing behaviors unchanged by default.
4. We print rich exceptions if the boolean feature flag is `on`.
5. Future iterations consider toggling the default, especially when running in tty modes.

cc @pandemicsyn 

I think this would be good to add before #6103 since it will give some learnings on the library.


Resolving patch:
diff --git a/src/meltano/core/logging/formatters.py b/src/meltano/core/logging/formatters.py
--- a/src/meltano/core/logging/formatters.py
+++ b/src/meltano/core/logging/formatters.py
@@ -2,11 +2,16 @@
 
 from __future__ import annotations
 
-from typing import Sequence
+from typing import Sequence, TextIO
 
+import click
 import structlog
+from rich.console import Console
+from rich.traceback import Traceback, install
 from structlog.types import Processor
 
+install(suppress=[click])
+
 TIMESTAMPER = structlog.processors.TimeStamper(fmt="iso")
 
 LEVELED_TIMESTAMPED_PRE_CHAIN = frozenset(
@@ -19,30 +24,71 @@
 )
 
 
-def _process_formatter(processor: Processor):
+def plain_rich_traceback(sio: TextIO, exc_info: structlog.types.ExcInfo) -> None:
+    """Pretty-print `exc_info` to `sio` using the rich package, with colors disabled.
+
+    To be passed into `ConsoleRenderer`'s `exception_formatter` argument.
+
+    Args:
+        sio: Return of open() in text mode.
+        exc_info: Execution info.
+    """
+    sio.write("\n")
+    Console(file=sio, no_color=True).print(
+        Traceback.from_exception(*exc_info, show_locals=True)
+    )
+
+
+def _process_formatter(processor: Processor) -> structlog.stdlib.ProcessorFormatter:
     """Use _process_formatter to configure a structlog.stdlib.ProcessFormatter.
 
     It will automatically add log level and timestamp fields to any log entries not originating from structlog.
 
     Args:
         processor: A structlog message processor such as structlog.dev.ConsoleRenderer.
+
+    Returns:
+        A configured log processor.
     """
     return structlog.stdlib.ProcessorFormatter(
         processor=processor, foreign_pre_chain=LEVELED_TIMESTAMPED_PRE_CHAIN
     )
 
 
-def console_log_formatter(colors: bool = False) -> None:
-    """Create a logging formatter for console rendering that supports colorization."""
-    return _process_formatter(structlog.dev.ConsoleRenderer(colors=colors))
+def console_log_formatter(colors: bool = False) -> structlog.stdlib.ProcessorFormatter:
+    """Create a logging formatter for console rendering that supports colorization.
+
+    Args:
+        colors: Add color to output.
+
+    Returns:
+        A configured console log formatter.
+    """
+    exception_formatter = (
+        structlog.dev.rich_traceback if colors else plain_rich_traceback
+    )
+    return _process_formatter(
+        structlog.dev.ConsoleRenderer(
+            colors=colors, exception_formatter=exception_formatter
+        )
+    )
 
 
 def key_value_formatter(
     sort_keys: bool = False,
     key_order: Sequence[str] | None = None,
     drop_missing: bool = False,
-) -> None:
-    """Create a logging formatter that renders lines in key=value format."""
+) -> structlog.stdlib.ProcessorFormatter:
+    """Create a logging formatter that renders lines in key=value format.
+
+    Args:
+        sort_keys: Whether to sort keys when formatting.
+        key_order: List of keys that should be rendered in this exact order. Missing keys will be rendered as None, extra keys depending on *sort_keys* and the dict class.
+        drop_missing: When True, extra keys in *key_order* will be dropped rather than rendered as None.
+
+    Returns:
+        A configured key=value formatter.
+    """
     return _process_formatter(
         processor=structlog.processors.KeyValueRenderer(
             sort_keys=sort_keys, key_order=key_order, drop_missing=drop_missing
@@ -50,6 +96,10 @@ def key_value_formatter(
     )
 
 
-def json_formatter() -> None:
-    """Create a logging formatter that renders lines in JSON format."""
+def json_formatter() -> structlog.stdlib.ProcessorFormatter:
+    """Create a logging formatter that renders lines in JSON format.
+
+    Returns:
+        A configured JSON formatter.
+    """
     return _process_formatter(processor=structlog.processors.JSONRenderer())
diff --git a/src/meltano/core/logging/output_logger.py b/src/meltano/core/logging/output_logger.py
--- a/src/meltano/core/logging/output_logger.py
+++ b/src/meltano/core/logging/output_logger.py
@@ -1,3 +1,4 @@
+"""Output Logger."""
 from __future__ import annotations
 
 import asyncio
@@ -19,7 +20,14 @@
 
 
 class OutputLogger:
+    """Output Logger."""
+
     def __init__(self, file):
+        """Instantiate an Output Logger.
+
+        Args:
+            file: A file to output to.
+        """
         self.file = file
         self.stdout = sys.stdout
         self.stderr = sys.stderr
@@ -55,25 +63,66 @@ def out(
 
 
 class LineWriter:
+    """Line Writer."""
+
     def __init__(self, out):
+        """Instantiate a Line Writer.
+
+        Args:
+            out: The location to write to.
+        """
         self.__out = out
 
     def __getattr__(self, name):
+        """Get attribute.
+
+        Args:
+            name: The attribute name.
+
+        Returns:
+            The specified attributes value.
+        """
         return getattr(self.__out, name)
 
     def write(self, line):
+        """Write a line.
+
+        Args:
+            line: A line to write.
+        """
         self.__out.writeline(line.rstrip())
 
 
 class FileDescriptorWriter:
+    """File Descriptor Writer."""
+
     def __init__(self, out, fd):
+        """Instantiate File Descriptor Writer.
+
+        Args:
+            out: Output location.
+            fd: A file descriptor.
+        """
         self.__out = out
         self.__writer = os.fdopen(fd, "w")
 
     def __getattr__(self, name):
+        """Get attribute.
+
+        Args:
+            name: The attribute name.
+
+        Returns:
+            The specified attributes value.
+        """
         return getattr(self.__writer, name)
 
     def isatty(self):
+        """Is out location a tty.
+
+        Returns:
+            True if output location is a tty.
+        """
         return self.__out.isatty()
 
 
@@ -113,7 +162,9 @@ def redirect_log_handler(self) -> logging.Handler:
             logging.FileHandler using an uncolorized console formatter
         """
         formatter = structlog.stdlib.ProcessorFormatter(
-            processor=structlog.dev.ConsoleRenderer(colors=False),
+            processor=structlog.dev.ConsoleRenderer(
+                colors=False, exception_formatter=structlog.dev.plain_traceback
+            ),
             foreign_pre_chain=LEVELED_TIMESTAMPED_PRE_CHAIN,
         )
         handler = logging.FileHandler(self.file)
@@ -122,11 +173,23 @@ def redirect_log_handler(self) -> logging.Handler:
 
     @contextmanager
     def line_writer(self):
+        """Yield a line writer instance.
+
+        Yields:
+            A line writer instance
+        """
         yield LineWriter(self)
 
     @contextmanager
     def redirect_logging(self, ignore_errors=()):
-        """Redirect log entries to a temporarily added file handler."""
+        """Redirect log entries to a temporarily added file handler.
+
+        Args:
+            ignore_errors: A tuple of Error classes to ignore.
+
+        Yields:
+            With the side-effect of redirecting logging.
+        """  # noqa: DAR401
         logger = logging.getLogger()
         logger.addHandler(self.redirect_log_handler)
         ignored_errors = (
@@ -146,6 +209,11 @@ def redirect_logging(self, ignore_errors=()):
 
     @asynccontextmanager
     async def writer(self):
+        """Yield a writer.
+
+        Yields:
+            A writer.
+        """
         read_fd, write_fd = os.pipe()
 
         reader = asyncio.ensure_future(self._read_from_fd(read_fd))
@@ -161,18 +229,32 @@ async def writer(self):
 
     @asynccontextmanager
     async def redirect_stdout(self):
+        """Redirect STDOUT.
+
+        Yields:
+            A writer with redirected output.
+        """
         async with self.writer() as stdout:
             with redirect_stdout(stdout):
                 yield
 
     @asynccontextmanager
     async def redirect_stderr(self):
+        """Redirect STDERR.
+
+        Yields:
+            A writer with redirected output.
+        """
         async with self.writer() as stderr:
             with redirect_stderr(stderr):
                 yield
 
     def writeline(self, line: str) -> None:
-        """Write a line to the underlying structured logger, cleaning up any dangling control chars."""
+        """Write a line to the underlying structured logger, cleaning up any dangling control chars.
+
+        Args:
+            line: A line to write.
+        """
         self.last_line = line
         self.logger.log(self.write_level, line.rstrip(), name=self.name)
 
diff --git a/src/meltano/core/logging/utils.py b/src/meltano/core/logging/utils.py
--- a/src/meltano/core/logging/utils.py
+++ b/src/meltano/core/logging/utils.py
@@ -75,7 +75,9 @@ def default_config(log_level: str) -> dict:
         "formatters": {
             "colored": {
                 "()": structlog.stdlib.ProcessorFormatter,
-                "processor": structlog.dev.ConsoleRenderer(colors=True),
+                "processor": structlog.dev.ConsoleRenderer(
+                    colors=True, exception_formatter=structlog.dev.rich_traceback
+                ),
                 "foreign_pre_chain": LEVELED_TIMESTAMPED_PRE_CHAIN,
             },
         },


Test patch:
diff --git a/tests/meltano/cli/test_run.py b/tests/meltano/cli/test_run.py
--- a/tests/meltano/cli/test_run.py
+++ b/tests/meltano/cli/test_run.py
@@ -2,6 +2,7 @@
 
 import asyncio
 import json
+import re
 
 import pytest
 import structlog
@@ -10,6 +11,7 @@
 from meltano.cli import cli
 from meltano.core.block.ioblock import IOBlock
 from meltano.core.logging.formatters import LEVELED_TIMESTAMPED_PRE_CHAIN
+from meltano.core.logging.utils import default_config
 from meltano.core.plugin import PluginType
 from meltano.core.plugin.singer import SingerTap
 from meltano.core.plugin_invoker import PluginInvoker
@@ -1318,3 +1320,60 @@ def test_run_dry_run(
             assert not matcher.find_by_event("Block run completed.")
             assert create_subprocess_exec.call_count == 0
             assert asyncio_mock.call_count == 0
+
+    @pytest.mark.backend("sqlite")
+    @pytest.mark.parametrize(
+        "colors",
+        [True, False],
+    )
+    def test_color_console_exception_handler(
+        self,
+        colors,
+        cli_runner,
+        project,
+        tap,
+        target,
+        tap_process,
+        target_process,
+        project_plugins_service,
+    ):
+        # toggle color in logging configuration
+        logging_config = default_config(log_level="info")
+        if not colors:
+            logging_config["formatters"]["colored"] = {
+                "()": "meltano.core.logging.console_log_formatter",
+                "colors": colors,
+            }
+
+        # in this scenario, the tap fails on the third read. Target should still complete.
+        args = ["run", tap.name, target.name]
+
+        tap_process.wait.return_value = 1
+        tap_process.returncode = 1
+        tap_process.stderr.readline.side_effect = (
+            b"tap starting\n",
+            b"tap running\n",
+            b"tap failure\n",
+        )
+
+        invoke_async = AsyncMock(side_effect=(tap_process, target_process))
+
+        with mock.patch(
+            "meltano.core.logging.utils.default_config", return_value=logging_config
+        ), mock.patch.object(
+            PluginInvoker, "invoke_async", new=invoke_async
+        ), mock.patch(
+            "meltano.core.block.parser.ProjectPluginsService",
+            return_value=project_plugins_service,
+        ), mock.patch(
+            "meltano.core.transform_add_service.ProjectPluginsService",
+            return_value=project_plugins_service,
+        ):
+            result = cli_runner.invoke(cli, args)
+
+            ansi_color_escape = re.compile(r"\x1b\[[0-9;]+m")
+            match = ansi_color_escape.search(result.stderr)
+            if colors:
+                assert match
+            else:
+                assert not match