Repository: meltano/meltano

Issue:
Update `meltano select` to print full error message on catalog discovery errors (and with other CLI commands if possible)
When a catalog discovery attempt fails, the error message does not give enough information about what happened:

```
$ meltano select tap-postgres --list --all
2022-06-10T14:46:22.487369Z [info     ] Environment 'dev' is active
Cannot list the selected attributes: Catalog discovery failed: command ['/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/bin/tap-postgres', '--config', '/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/run/tap-postgres/tap.c331e588-15d0-4ec6-aa62-90ec0c99c76f.config.json', '--discover'] returned 1
```

In fact, the intermediate `.meltano/run/tap-postgres/tap.c331e588-15d0-4ec6-aa62-90ec0c99c76f.config.json` config file gets automatically deleted, which makes debugging more cumbersome.

The current workaround is to:

1. Generate a config manually: `meltano config tap-postgres > config.json`
2. Rerun the command with the custom config, assuming it's the same as the temporary one generated on the fly: `/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/bin/tap-postgres --config config.json --discover`
3. See actual error

It would be nice if the "returned 1" message would also give the output of the command, for example:

```
$ /home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/bin/tap-postgres --config config.json --discover
time=2022-06-10 16:49:52 name=tap_postgres level=CRITICAL message=Config is missing required keys: ['dbname']
Traceback (most recent call last):
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/bin/tap-postgres", line 8, in <module>
    sys.exit(main())
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 435, in main
    raise exc
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 432, in main
    main_impl()
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 393, in main_impl
    args = parse_args(REQUIRED_CONFIG_KEYS)
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/tap_postgres/__init__.py", line 384, in parse_args
    utils.check_config(args.config, required_config_keys)
  File "/home/juanlu/Projects/Orchest/Singer/test-replication-none/.meltano/extractors/tap-postgres/venv/lib/python3.8/site-packages/singer/utils.py", line 190, in check_config
    raise Exception("Config is missing required keys: {}".format(missing_keys))
Exception: Config is missing required keys: ['dbname']
```


Discussion:
@astrojuanlu Good call, I think I fixed this in `meltano elt` a while ago (https://gitlab.com/meltano/meltano/-/issues/2340), but we should definitely do the same in `meltano select`.

@tayloramurphy I think we should prioritize this as part of the "new user experience" goal.
@cjohnhanson - Is it okay if I assign this to you? It might be a new part of the codebase for you to dive into, but would be a good exercise if so and valuable usability improvement either way. 😄 

Also - you may have some prior art in the above linked issue which could be helpful.
Related to https://gitlab.com/meltano/meltano/-/issues/2340.
@tayloramurphy, @pandemicsyn, @cjohnhanson, @visch  - This came up in office hours today related to other commands which might not be printing `stderr` for failed discovery operations.

@cjohnhanson - When you are back and ready to pick this up, you see about tackling discovery as part of all three of these code paths?:

- [ ] `meltano select`
- [ ] `meltano invoke`
- [ ] `meltano run`
- [x] `meltano elt` (already done?)

From that office hours conversation, it sounded as though perhaps we've tackled `elt` but maybe not the others. Also, as is standard, it's totally acceptable if you decide it is best to solve in multiple smaller iterations; just wanted to make sure we're thinking about the full long-term scope as you begin the work.

Thanks!

Another proposal that came up in that conversation was to record all STDERR from discovery into a log file so that it is available for inspection even if `--log-level=DEBUG` is not set. I'm not sure what's best here but wanted to log that idea here in case it's helpful.



Resolving patch:
diff --git a/src/meltano/core/plugin/singer/tap.py b/src/meltano/core/plugin/singer/tap.py
--- a/src/meltano/core/plugin/singer/tap.py
+++ b/src/meltano/core/plugin/singer/tap.py
@@ -9,6 +9,7 @@
 import sys
 from asyncio.streams import StreamReader
 from hashlib import sha1
+from io import StringIO
 from pathlib import Path
 from typing import Tuple
 
@@ -37,23 +38,24 @@
 
 
 async def _stream_redirect(
-    stream: asyncio.StreamReader, file_like_obj, write_str=False
+    stream: asyncio.StreamReader, *file_like_objs, write_str=False
 ):
     """Redirect stream to a file like obj.
 
     Args:
         stream: the stream to redirect
-        file_like_obj: the object to redirect the stream to
+        file_like_objs: the objects to redirect the stream to
         write_str: if True, stream is written as str
     """
     encoding = sys.getdefaultencoding()
     while not stream.at_eof():
         data = await stream.readline()
-        file_like_obj.write(data.decode(encoding) if write_str else data)
+        for file_like_obj in file_like_objs:
+            file_like_obj.write(data.decode(encoding) if write_str else data)
 
 
 def _debug_logging_handler(
-    name: str, plugin_invoker: PluginInvoker, stderr: StreamReader
+    name: str, plugin_invoker: PluginInvoker, stderr: StreamReader, *other_dsts
 ) -> asyncio.Task:
     """Route debug log lines to stderr or an OutputLogger if one is present in our invocation context.
 
@@ -61,20 +63,27 @@ def _debug_logging_handler(
         name: name of the plugin
         plugin_invoker: the PluginInvoker to route log lines for
         stderr: stderr StreamReader to route to
+        other_dsts: other destinations that the stream should be routed too along with logging output
 
     Returns:
         asyncio.Task which performs the routing of log lines
     """
     if not plugin_invoker.context or not plugin_invoker.context.base_output_logger:
         return asyncio.ensure_future(
-            _stream_redirect(stderr, sys.stderr, write_str=True)
+            _stream_redirect(
+                stderr, *(sys.stderr, *other_dsts), write_str=True  # noqa: WPS517
+            )
         )
 
     out = plugin_invoker.context.base_output_logger.out(
         name, logger.bind(type="discovery", stdio="stderr")
     )
     with out.line_writer() as outerr:
-        return asyncio.ensure_future(_stream_redirect(stderr, outerr, write_str=True))
+        return asyncio.ensure_future(
+            _stream_redirect(
+                stderr, *(outerr, *other_dsts), write_str=True  # noqa: WPS517
+            )
+        )
 
 
 def config_metadata_rules(config):
@@ -392,51 +401,58 @@ async def run_discovery(  # noqa: WPS238
             raise PluginLacksCapabilityError(
                 f"Extractor '{self.name}' does not support catalog discovery (the `discover` capability is not advertised)"
             )
-
-        try:
-            with catalog_path.open(mode="wb") as catalog:
-
-                # since we're using subproccess wait() - we need to ensure that that stderr's  buffer
-                # is drained regardless of whether or not we want the output.
-                if logger.isEnabledFor(logging.DEBUG):
-                    stderr_dst = asyncio.subprocess.PIPE
-                else:
-                    stderr_dst = asyncio.subprocess.DEVNULL
-
-                handle = await plugin_invoker.invoke_async(
-                    "--discover",
-                    stdout=asyncio.subprocess.PIPE,
-                    stderr=stderr_dst,
-                    universal_newlines=False,
-                )
-                invoke_futures = [
-                    asyncio.ensure_future(_stream_redirect(handle.stdout, catalog)),
-                    asyncio.ensure_future(handle.wait()),
-                ]
-
-                if logger.isEnabledFor(logging.DEBUG) and handle.stderr:
-                    invoke_futures.append(
-                        _debug_logging_handler(self.name, plugin_invoker, handle.stderr)
+        with StringIO("") as stderr_buff:
+            try:
+                with catalog_path.open(mode="wb") as catalog:
+                    handle = await plugin_invoker.invoke_async(
+                        "--discover",
+                        stdout=asyncio.subprocess.PIPE,
+                        stderr=asyncio.subprocess.PIPE,
+                        universal_newlines=False,
                     )
 
-                done, _ = await asyncio.wait(
-                    invoke_futures,
-                    return_when=asyncio.ALL_COMPLETED,
+                    invoke_futures = [
+                        asyncio.ensure_future(_stream_redirect(handle.stdout, catalog)),
+                        asyncio.ensure_future(handle.wait()),
+                    ]
+                    if logger.isEnabledFor(logging.DEBUG) and handle.stderr:
+                        invoke_futures.append(
+                            _debug_logging_handler(
+                                self.name,
+                                plugin_invoker,
+                                handle.stderr,
+                                stderr_buff,
+                            )
+                        )
+                    else:
+                        invoke_futures.append(
+                            asyncio.ensure_future(
+                                _stream_redirect(
+                                    handle.stderr, stderr_buff, write_str=True
+                                )
+                            )
+                        )
+                    done, _ = await asyncio.wait(
+                        invoke_futures,
+                        return_when=asyncio.ALL_COMPLETED,
+                    )
+                    failed = [
+                        future for future in done if future.exception() is not None
+                    ]
+                    if failed:
+                        failed_future = failed.pop()
+                        raise failed_future.exception()
+                exit_code = handle.returncode
+            except Exception:
+                catalog_path.unlink()
+                raise
+
+            if exit_code != 0:
+                catalog_path.unlink()
+                stderr_buff.seek(0)
+                raise PluginExecutionError(
+                    f"Catalog discovery failed: command {plugin_invoker.exec_args('--discover')} returned {exit_code} with stderr:\n {stderr_buff.read()}"
                 )
-                failed = [future for future in done if future.exception() is not None]
-                if failed:
-                    failed_future = failed.pop()
-                    raise failed_future.exception()
-            exit_code = handle.returncode
-        except Exception:
-            catalog_path.unlink()
-            raise
-
-        if exit_code != 0:
-            catalog_path.unlink()
-            raise PluginExecutionError(
-                f"Catalog discovery failed: command {plugin_invoker.exec_args('--discover')} returned {exit_code}"
-            )
 
     @hook("before_invoke")
     async def apply_catalog_rules_hook(


Test patch:
diff --git a/tests/meltano/core/plugin/singer/test_tap.py b/tests/meltano/core/plugin/singer/test_tap.py
--- a/tests/meltano/core/plugin/singer/test_tap.py
+++ b/tests/meltano/core/plugin/singer/test_tap.py
@@ -782,8 +782,7 @@ async def test_run_discovery_stderr_output(
             "meltano.core.plugin.singer.tap.logger.isEnabledFor", return_value=False
         ), mock.patch("meltano.core.plugin.singer.tap._stream_redirect") as stream_mock:
             await subject.run_discovery(invoker, catalog_path)
-
-            assert stream_mock.call_count == 1
+            assert stream_mock.call_count == 2
 
         with mock.patch(
             "meltano.core.plugin.singer.tap.logger.isEnabledFor", return_value=True
@@ -791,9 +790,7 @@ async def test_run_discovery_stderr_output(
             "meltano.core.plugin.singer.tap._stream_redirect"
         ) as stream_mock2:
             await subject.run_discovery(invoker, catalog_path)
-
             assert stream_mock2.call_count == 2
-            assert stream_mock2.call_args[1]["write_str"] is True
 
         # ensure stderr is redirected to devnull if we don't need it
         discovery_logger = logging.getLogger("meltano.core.plugin.singer.tap")
@@ -807,10 +804,8 @@ async def test_run_discovery_stderr_output(
             await subject.run_discovery(invoker, catalog_path)
 
             assert stream_mock3.call_count == 2
-            assert stream_mock3.call_args[1]["write_str"] is True
-
             call_kwargs = invoker.invoke_async.call_args_list[0][1]
-            assert call_kwargs.get("stderr") is subprocess.DEVNULL
+            assert call_kwargs.get("stderr") is subprocess.PIPE
 
         discovery_logger.setLevel(original_level)
 
@@ -869,12 +864,6 @@ async def test_run_discovery_utf8_output(
 
         assert sys.getdefaultencoding() == "utf-8"
 
-        with mock.patch(
-            "meltano.core.plugin.singer.tap.logger.isEnabledFor", return_value=True
-        ), mock.patch("sys.getdefaultencoding", return_value="ascii"):
-            with pytest.raises(UnicodeDecodeError):
-                await subject.run_discovery(invoker, catalog_path)
-
         with mock.patch(
             "meltano.core.plugin.singer.tap.logger.isEnabledFor", return_value=True
         ):