Repository: meltano/meltano

Issue:
bug: Secrets leak in logs if `asyncio.create_subprocess_exec()` fails (or if log-level=debug)
### Meltano Version

2.13.0

### Python Version

3.8

### Bug scope

CLI (options, error messages, logging, etc.)

### Operating System

Ubuntu

### Description

Meltano failed for me in production and during the printing of the huge line of stack traces (Which most of the traces are pretty useless to me while I"m in "user mode" unless I"m running in debug mode).


Ideally envs wouldn't print out. I was running `meltano run dashboard` which is a job name

```
 /venv/lib/python3.9/site-packages/meltano/core/plugin_invoker.py:453 in      │
│ invoke_async                                                                 │
│                                                                              │
│   450 │   │   │   popen_options,                                             │
│   451 │   │   │   popen_env,                                                 │
│   452 │   │   ):                                                             │
│ ❱ 453 │   │   │   return await asyncio.create_subprocess_exec(               │
│   454 │   │   │   │   *popen_args,                                           │
│   455 │   │   │   │   **popen_options,                                       │
│   456 │   │   │   │   env=popen_env,                                         │
│                                                                              │
│ ╭───────────────────────────────── locals ─────────────────────────────────╮ │
│ │          args = ('--discover',)                                          │ │
│ │        kwargs = {                                                        │ │
│ │                 │   'stdout': -1,                                        │ │
│ │                 │   'stderr': -1,                                        │ │
│ │                 │   'universal_newlines': False                          │ │
│ │                 }                                                        │ │
│ │    popen_args = [                                                        │ │
│ │                 │                                                        │ │
│ │                 '/builds/autoidm/prodder/.meltano/extractors/tap-gmail/… │ │
│ │                 │   '--config',                                          │ │
│ │                 │                                                        │ │
│ │                 '/builds/autoidm/prodder/.meltano/run/tap-gmail/tap.c25… │ │
│ │                 │   '--discover'                                         │ │
│ │                 ]                                                        │ │
│ │     popen_env = {                                                        │ │
│ │                 │                                                        │ │
│ │                 'TAP_GMAIL_PERSONAL_OAUTH_CREDENTIALS_REFRESH_TOKEN':    │ │
│ │                 'refreshtoken_listd_here… │ │
│ │                 │   'TAP_GMAIL_USER_ID': 'me',                           │ │
│ │                 │   'CI_PROJECT_NAMESPACE': 'autoidm',                   │ │
│ │                 │   'GITLAB_USER_ID': 'userid',                         │ │
│ │                 │   'CI_RUNNER_VERSION': '15.6.0~beta.186.ga889181a',    │ │
│ │                 │   'FF_SKIP_NOOP_BUILD_STAGES': 'true',                 │ │
│ │                 │   'CI_SERVER_NAME': 'GitLab',                          │ │
│ │                 │   'CI_RUNNER_DESCRIPTION':                             │ │
│ │                 '2-blue.shared.runners-manager.gitlab.com/default',      │ │
│ │                 │   'GITLAB_USER_EMAIL': 'dvisch@autoidm.com',           │ │
│ │                 │   'CI_SERVER_REVISION': 'fd1954e655c',   
```

### Code

_No response_


Discussion:
I've noticed this as well. It'd be preferable to have the value of environment variables excluded from logs. If I'm running locally it's not the end of the world, but this is enough to keep me from enabling debug logging in a remote execution environment.
> I've noticed this as well. It'd be preferable to have the value of environment variables excluded from logs. If I'm running locally it's not the end of the world, but this is enough to keep me from enabling debug logging in a remote execution environment.

I'm also not running with debug logging in this scenario so it impacts me in prod :/ 
@visch - Thanks for logging. I've updated this issue description to exactly describe this exception point.

@visch and @menzenski, if either of you find similar cases, I do appreciate you logging them as separate issues. I know this issue has other instances, and we'll try to tackle each as we identify them.

I've marked with the `security` tag, urgency `High` and weight `2` (very small).

The preferred handling here would be to make this a Meltano-handled error (subclassed from `MeltanoError`), with a custom error message that does not print the details of env vars or config.

Related:

- https://github.com/meltano/meltano/issues/7184

Still confirming there in the thread, but that looks like the same issue as this one, except related to how we treat the debug logs when launching the subprocess.
@aaronsteers I think https://github.com/meltano/meltano/issues/7184 is not a dupe

- `asyncio.create_subprocess_exec()`: env vars are leaked as part of the `env=popen_env` argument passed to the function
- debug logs: this is logged explicitly in https://github.com/meltano/meltano/blob/a13651fd7165c438d7093e5b58ac0fef3a3e8902/src/meltano/core/plugin_invoker.py#L429


@edgarrmondragon - Got it! Thanks for checking - and confirming. 👍 

Resolving patch:
diff --git a/src/meltano/core/logging/formatters.py b/src/meltano/core/logging/formatters.py
--- a/src/meltano/core/logging/formatters.py
+++ b/src/meltano/core/logging/formatters.py
@@ -2,7 +2,8 @@
 
 from __future__ import annotations
 
-from typing import Callable, Sequence, TextIO
+import sys
+from typing import TYPE_CHECKING, Callable, Sequence, TextIO
 
 import click
 import structlog
@@ -12,6 +13,13 @@
 
 from meltano.core.utils import get_no_color_flag
 
+if TYPE_CHECKING:
+    if sys.version_info >= (3, 8):
+        from typing import Literal
+    else:
+        from typing_extensions import Literal
+
+
 install(suppress=[click])
 
 TIMESTAMPER = structlog.processors.TimeStamper(fmt="iso")
@@ -27,8 +35,9 @@
 
 
 def rich_exception_formatter_factory(
-    color_system: str = "auto",
+    color_system: Literal["auto", "standard", "256", "truecolor", "windows"] = "auto",
     no_color: bool | None = None,
+    show_locals: bool = False,
 ) -> Callable[[TextIO, structlog.types.ExcInfo], None]:
     """Create an exception formatter for logging using the rich package.
 
@@ -39,6 +48,7 @@ def rich_exception_formatter_factory(
     Args:
         color_system: The color system supported by your terminal.
         no_color: Enabled no color mode, or None to auto detect. Defaults to None.
+        show_locals: Whether to show local variables in the traceback.
 
     Returns:
         Exception formatter function.
@@ -47,27 +57,15 @@ def rich_exception_formatter_factory(
     def _traceback(sio, exc_info) -> None:
         sio.write("\n")
         Console(file=sio, color_system=color_system, no_color=no_color).print(
-            Traceback.from_exception(*exc_info, show_locals=True)
+            Traceback.from_exception(
+                *exc_info,
+                show_locals=show_locals,
+            ),
         )
 
     return _traceback
 
 
-def plain_rich_traceback(sio: TextIO, exc_info: structlog.types.ExcInfo) -> None:
-    """Pretty-print `exc_info` to `sio` using the rich package, with colors disabled.
-
-    To be passed into `ConsoleRenderer`'s `exception_formatter` argument.
-
-    Args:
-        sio: Return of open() in text mode.
-        exc_info: Execution info.
-    """
-    sio.write("\n")
-    Console(file=sio, no_color=True).print(
-        Traceback.from_exception(*exc_info, show_locals=True)
-    )
-
-
 def _process_formatter(processor: Processor) -> structlog.stdlib.ProcessorFormatter:
     """Use _process_formatter to configure a structlog.stdlib.ProcessFormatter.
 
@@ -84,11 +82,15 @@ def _process_formatter(processor: Processor) -> structlog.stdlib.ProcessorFormat
     )
 
 
-def console_log_formatter(colors: bool = False) -> structlog.stdlib.ProcessorFormatter:
+def console_log_formatter(
+    colors: bool = False,
+    show_locals: bool = False,
+) -> structlog.stdlib.ProcessorFormatter:
     """Create a logging formatter for console rendering that supports colorization.
 
     Args:
         colors: Add color to output.
+        show_locals: Whether to show local variables in the traceback.
 
     Returns:
         A configured console log formatter.
@@ -96,9 +98,15 @@ def console_log_formatter(colors: bool = False) -> structlog.stdlib.ProcessorFor
     colors = colors and not get_no_color_flag()
 
     if colors:
-        exception_formatter = rich_exception_formatter_factory(color_system="truecolor")
+        exception_formatter = rich_exception_formatter_factory(
+            color_system="truecolor",
+            show_locals=show_locals,
+        )
     else:
-        exception_formatter = rich_exception_formatter_factory(no_color=True)
+        exception_formatter = rich_exception_formatter_factory(
+            no_color=True,
+            show_locals=show_locals,
+        )
 
     return _process_formatter(
         structlog.dev.ConsoleRenderer(
diff --git a/src/meltano/core/logging/output_logger.py b/src/meltano/core/logging/output_logger.py
--- a/src/meltano/core/logging/output_logger.py
+++ b/src/meltano/core/logging/output_logger.py
@@ -15,6 +15,8 @@
 
 import structlog
 
+from meltano.core.runner import RunnerError
+
 from .formatters import LEVELED_TIMESTAMPED_PRE_CHAIN
 from .utils import capture_subprocess_output
 
@@ -201,6 +203,9 @@ def redirect_logging(self, ignore_errors=()):
             yield
         except ignored_errors:  # noqa: WPS329
             raise
+        except RunnerError as err:
+            logger.error(str(err))
+            raise
         except Exception as err:
             logger.error(str(err), exc_info=True)
             raise


Test patch:
diff --git a/tests/meltano/core/logging/test_formatters.py b/tests/meltano/core/logging/test_formatters.py
--- a/tests/meltano/core/logging/test_formatters.py
+++ b/tests/meltano/core/logging/test_formatters.py
@@ -2,6 +2,9 @@
 
 import logging
 import re
+from pathlib import Path
+from types import TracebackType
+from typing import cast
 
 import pytest
 
@@ -10,6 +13,40 @@
 ANSI_RE = re.compile(r"\033\[[;?0-9]*[a-zA-Z]")
 
 
+class FakeCode:
+    """A fake code object that can be used to test log formatters."""
+
+    def __init__(self, co_filename, co_name):
+        self.co_filename = co_filename
+        self.co_name = co_name
+
+
+class FakeFrame:
+    """A fake traceback frame that can be used to test log formatters."""
+
+    def __init__(self, f_code, f_globals, f_locals=None):
+        self.f_code = f_code
+        self.f_globals = f_globals
+        self.f_locals = f_locals or {}
+
+
+class FakeTraceback:  # pragma: no cover
+    """A fake traceback that can be used to test log formatters."""
+
+    def __init__(self, frames, line_nums):
+        if len(frames) != len(line_nums):
+            raise ValueError("Ya messed up!")
+        self._frames = frames
+        self._line_nums = line_nums
+        self.tb_frame = frames[0]
+        self.tb_lineno = line_nums[0]
+
+    @property
+    def tb_next(self):
+        if len(self._frames) > 1:
+            return FakeTraceback(self._frames[1:], self._line_nums[1:])
+
+
 class TestLogFormatters:
     @pytest.fixture
     def record(self):
@@ -19,10 +56,43 @@ def record(self):
             pathname="test",
             lineno=1,
             msg="test",
-            args=[],
+            args=None,
             exc_info=None,
         )
 
+    @pytest.fixture
+    def record_with_exception(self, tmp_path: Path):
+        path = tmp_path / "my_module.py"
+        path.write_text("cause_an_error()\n")
+
+        tb = cast(
+            TracebackType,
+            FakeTraceback(
+                [
+                    FakeFrame(
+                        FakeCode(str(path), "test"),
+                        {"__name__": "__main__"},
+                        {"my_var": "my_value"},
+                    ),
+                ],
+                [1],
+            ),
+        )
+
+        return logging.LogRecord(
+            name="test",
+            level=logging.DEBUG,
+            pathname="test",
+            lineno=1,
+            msg="test",
+            args=None,
+            exc_info=(
+                Exception,
+                Exception("test"),
+                tb,
+            ),
+        )
+
     def test_console_log_formatter_colors(self, record):
         formatter = console_log_formatter(colors=True)
         assert ANSI_RE.match(formatter.format(record))
@@ -30,3 +100,15 @@ def test_console_log_formatter_colors(self, record):
     def test_console_log_formatter_no_colors(self, record):
         formatter = console_log_formatter(colors=False)
         assert not ANSI_RE.match(formatter.format(record))
+
+    def test_console_log_formatter_no_locals(self, record_with_exception):
+        formatter = console_log_formatter(show_locals=False)
+        output = formatter.format(record_with_exception)
+        assert "locals" not in output
+        assert "my_var = 'my_value'" not in output
+
+    def test_console_log_formatter_show_locals(self, record_with_exception):
+        formatter = console_log_formatter(show_locals=True)
+        output = formatter.format(record_with_exception)
+        assert "locals" in output
+        assert "my_var = 'my_value'" in output