Repository: streamlink/streamlink

Issue:
output with --json flag is not parseable if handled error occurs
### Checklist

- [X] This is a bug report and not a different kind of issue
- [X] [I have read the contribution guidelines](https://github.com/streamlink/streamlink/blob/master/CONTRIBUTING.md#contributing-to-streamlink)
- [X] [I have checked the list of open and recently closed bug reports](https://github.com/streamlink/streamlink/issues?q=is%3Aissue+label%3A%22bug%22)
- [X] [I have checked the commit log of the master branch](https://github.com/streamlink/streamlink/commits/master)

### Streamlink version

Latest stable release

### Description

### Steps to reproduce
1. Run streamlink against a YouTube channel that is not live, e.g. `streamlink youtube.com/c/LudwigAhgren -j`
2. Output is unparseable as JSON:

### Expected Behavior
All output on stdout should be parseable when the  `--json` is enabled (barring an unrecoverable error)
```json
{
  "error": "No playable streams found on this URL: youtube.com/c/LudwigAhgren"
}
```
### Actual Behavior
Plugin errors are printed to stdout along with the parseable JSON output
```json
[plugins.youtube][error] Could not find videoId on channel page
{
  "error": "No playable streams found on this URL: youtube.com/c/LudwigAhgren"
}
```

### Possible Solutions
* Write error messages to stderr instead of stdout when `--json` flag is enabled (or always)
* Suppress error messages when `--json` flag is enabled, since the content is duplicated in the JSON output



### Debug log

```text
[plugins.youtube][error] Could not find videoId on channel page
{
  "error": "No playable streams found on this URL: youtube.com/c/LudwigAhgren"
}
```

*Note*:   
the debug log is with the proper `--loglevel debug` param, it seems `--json` suppresses debug messages already. System info is as follows:
```
[cli][debug] OS:         Windows 10
[cli][debug] Python:     3.9.8
[cli][debug] Streamlink: 3.0.3
[cli][debug] Requests(2.26.0), Socks(1.7.1), Websocket(1.2.1)
```


Discussion:
The problem here is how the logger and console output stream is set up when `--json` is set. It uses the same stream output, which is usually fine, but `console.exit` has special logic when `--json` is set, so any regular console/log message prior to the `console.exit` call gets printed to the same output stream. The logger and console output should not share the same stream output when `--json` is set, and the logger should print to `stderr` while `console` should print to `stdout`.

- https://github.com/streamlink/streamlink/blob/3.0.3/src/streamlink_cli/main.py#L1009
- https://github.com/streamlink/streamlink/blob/3.0.3/src/streamlink_cli/main.py#L967-L987
- https://github.com/streamlink/streamlink/blob/3.0.3/src/streamlink_cli/console.py#L85-L87
- https://github.com/streamlink/streamlink/blob/3.0.3/src/streamlink_cli/main.py#L571-L572
- https://github.com/streamlink/streamlink/blob/3.0.3/src/streamlink/plugins/youtube.py#L321
Had a quick look at this again. The way `--json` is implemented is not particularly great. And neither is the entire streamlink_cli, but that's not the point of this thread.

The JSON output is either done via `console.msg_json(data)` or `console.exit(data)` (if `ConsoleOutput` was initialized with `json=True`). As I said earlier, both the logger and the console instances share the same output stream. The output stream however varies depending on whether streamlink_cli outputs stream data to `stdout` when `--stdout` is set (or related arguments like `--output=-`, etc). This means that outputting the console's JSON data to `stdout` and the logger's log messages to `stderr` for example when `--json` is set is not always possible, so fixing this issue is not that simple.

`console.msg_json(data)` and `console.exit()` are meant as the last output method call for printing messages, because there can only be one JSON output. In order to respect the `--json` argument, all logger calls would either need to be suppressed when it is set (which is bad), or they need to be buffered and then included in the final JSON output, so that it can be parsed. That however does not guarantee that there will be a `console.msg_json(data)` or `console.exit()` call at the end, so the `ConsoleOutput` would need to be made aware of a `SystemExit`.

Resolving patch:
diff --git a/src/streamlink/logger.py b/src/streamlink/logger.py
--- a/src/streamlink/logger.py
+++ b/src/streamlink/logger.py
@@ -1,24 +1,41 @@
 import logging
+import sys
 from datetime import datetime
-from logging import CRITICAL, DEBUG, ERROR, INFO, NOTSET, WARNING
+from logging import CRITICAL, DEBUG, ERROR, INFO, WARNING
+from pathlib import Path
 from threading import Lock
+from typing import IO, List, Optional, Union
 
+
+FORMAT_STYLE = "{"
+FORMAT_BASE = "[{name}][{levelname}] {message}"
+FORMAT_DATE = "%H:%M:%S"
+REMOVE_BASE = ["streamlink", "streamlink_cli"]
+
+# Make NONE ("none") the highest possible level that suppresses all log messages:
+#  `logging.NOTSET` (equal to 0) can't be used as the "none" level because of `logging.Logger.getEffectiveLevel()`, which
+#  loops through the logger instance's ancestor chain and checks whether the instance's level is NOTSET. If it is NOTSET,
+#  then it continues with the parent logger, which means that if the level of `streamlink.logger.root` was set to "none" and
+#  its value NOTSET, then it would continue with `logging.root` whose default level is `logging.WARNING` (equal to 30).
+NONE = sys.maxsize
+# Add "trace" to Streamlink's log levels
 TRACE = 5
-_levelToName = {
+
+# Define Streamlink's log levels (and register both lowercase and uppercase names)
+_levelToNames = {
+    NONE: "none",
     CRITICAL: "critical",
     ERROR: "error",
     WARNING: "warning",
     INFO: "info",
     DEBUG: "debug",
     TRACE: "trace",
-    NOTSET: "none",
 }
-_nameToLevel = {name: level for level, name in _levelToName.items()}
 
-for level, name in _levelToName.items():
-    logging.addLevelName(level, name)
+for _level, _name in _levelToNames.items():
+    logging.addLevelName(_level, _name.upper())
+    logging.addLevelName(_level, _name)
 
-levels = [name for _, name in _levelToName.items()]
 _config_lock = Lock()
 
 
@@ -60,40 +77,50 @@ def format(self, record):
         return super().format(record)
 
 
-def basicConfig(**kwargs) -> logging.StreamHandler:
+# noinspection PyShadowingBuiltins,PyPep8Naming
+def basicConfig(
+    filename: Optional[Union[str, Path]] = None,
+    filemode: str = "a",
+    stream: Optional[IO] = None,
+    level: Optional[str] = None,
+    format: str = FORMAT_BASE,
+    style: str = FORMAT_STYLE,
+    datefmt: str = FORMAT_DATE,
+    remove_base: Optional[List[str]] = None
+) -> Union[logging.FileHandler, logging.StreamHandler]:
     with _config_lock:
-        filename = kwargs.get("filename")
-        if filename:
-            mode = kwargs.get("filemode", "a")
-            handler = logging.FileHandler(filename, mode)
+        if filename is not None:
+            handler = logging.FileHandler(filename, filemode)
         else:
-            stream = kwargs.get("stream")
             handler = logging.StreamHandler(stream)
-        fs = kwargs.get("format", BASIC_FORMAT)
-        style = kwargs.get("style", FORMAT_STYLE)
-        dfs = kwargs.get("datefmt", FORMAT_DATE)
-        remove_base = kwargs.get("remove_base", REMOVE_BASE)
 
-        formatter = StringFormatter(fs, dfs, style=style, remove_base=remove_base)
+        formatter = StringFormatter(
+            format,
+            datefmt,
+            style=style,
+            remove_base=remove_base or REMOVE_BASE
+        )
         handler.setFormatter(formatter)
 
         root.addHandler(handler)
-        level = kwargs.get("level")
         if level is not None:
             root.setLevel(level)
 
     return handler
 
 
-BASIC_FORMAT = "[{name}][{levelname}] {message}"
-FORMAT_STYLE = "{"
-FORMAT_DATE = "%H:%M:%S"
-REMOVE_BASE = ["streamlink", "streamlink_cli"]
-
-
 logging.setLoggerClass(StreamlinkLogger)
 root = logging.getLogger("streamlink")
 root.setLevel(WARNING)
 
+levels = list(_levelToNames.values())
+
 
-__all__ = ["StreamlinkLogger", "TRACE", "basicConfig", "root", "levels"]
+__all__ = [
+    "NONE",
+    "TRACE",
+    "StreamlinkLogger",
+    "basicConfig",
+    "root",
+    "levels",
+]
diff --git a/src/streamlink_cli/console.py b/src/streamlink_cli/console.py
--- a/src/streamlink_cli/console.py
+++ b/src/streamlink_cli/console.py
@@ -48,6 +48,8 @@ def askpass(self, prompt: str) -> Union[None, str]:
         return getpass(prompt, self.output)
 
     def msg(self, msg: str) -> None:
+        if self.json:
+            return
         self.output.write(f"{msg}\n")
 
     def msg_json(self, *objs: Any, **keywords: Any) -> None:
@@ -77,7 +79,7 @@ def msg_json(self, *objs: Any, **keywords: Any) -> None:
             out.update(**keywords)
 
         msg = dumps(out, cls=JSONEncoder, indent=2)
-        self.msg(msg)
+        self.output.write(f"{msg}\n")
 
         if type(out) is dict and out.get("error"):
             sys.exit(1)


Test patch:
diff --git a/tests/test_cli_main.py b/tests/test_cli_main.py
--- a/tests/test_cli_main.py
+++ b/tests/test_cli_main.py
@@ -1,4 +1,5 @@
 import datetime
+import logging
 import os
 import sys
 import unittest
@@ -526,7 +527,7 @@ def test_custom_multiple(self, mock_log):
 
 class _TestCLIMainLogging(unittest.TestCase):
     @classmethod
-    def subject(cls, argv):
+    def subject(cls, argv, **kwargs):
         session = Streamlink()
         session.load_plugins(os.path.join(os.path.dirname(__file__), "plugin"))
 
@@ -552,12 +553,16 @@ def tearDown(self):
         streamlink_cli.main.logger.root.handlers.clear()
 
     # python >=3.7.2: https://bugs.python.org/issue35046
-    _write_calls = (
-        ([call("[cli][info] foo\n")]
-         if sys.version_info >= (3, 7, 2)
-         else [call("[cli][info] foo"), call("\n")])
-        + [call("bar\n")]
+    _write_call_log_cli_info = (
+        [call("[cli][info] foo\n")]
+        if sys.version_info >= (3, 7, 2) else
+        [call("[cli][info] foo"), call("\n")]
     )
+    _write_call_console_msg = [call("bar\n")]
+    _write_call_console_msg_error = [call("error: bar\n")]
+    _write_call_console_msg_json = [call("{\n  \"error\": \"bar\"\n}\n")]
+
+    _write_calls = _write_call_log_cli_info + _write_call_console_msg
 
     def write_file_and_assert(self, mock_mkdir: Mock, mock_write: Mock, mock_stdout: Mock):
         streamlink_cli.main.log.info("foo")
@@ -567,7 +572,59 @@ def write_file_and_assert(self, mock_mkdir: Mock, mock_write: Mock, mock_stdout:
         self.assertFalse(mock_stdout.write.called)
 
 
-class TestCLIMainLogging(_TestCLIMainLogging):
+class TestCLIMainLoggingStreams(_TestCLIMainLogging):
+    # python >=3.7.2: https://bugs.python.org/issue35046
+    _write_call_log_testcli_err = (
+        [call("[test_cli_main][error] baz\n")]
+        if sys.version_info >= (3, 7, 2) else
+        [call("[test_cli_main][error] baz"), call("\n")]
+    )
+
+    def subject(self, argv, stream=None):
+        super().subject(argv)
+        childlogger = logging.getLogger("streamlink.test_cli_main")
+
+        with self.assertRaises(SystemExit):
+            streamlink_cli.main.log.info("foo")
+            childlogger.error("baz")
+            streamlink_cli.main.console.exit("bar")
+
+        self.assertIs(streamlink_cli.main.log.parent.handlers[0].stream, stream)
+        self.assertIs(childlogger.parent.handlers[0].stream, stream)
+        self.assertIs(streamlink_cli.main.console.output, stream)
+
+    @patch("sys.stderr")
+    @patch("sys.stdout")
+    def test_no_pipe_no_json(self, mock_stdout: Mock, mock_stderr: Mock):
+        self.subject(["streamlink"], mock_stdout)
+        self.assertEqual(mock_stdout.write.mock_calls,
+                         self._write_call_log_cli_info + self._write_call_log_testcli_err + self._write_call_console_msg_error)
+        self.assertEqual(mock_stderr.write.mock_calls, [])
+
+    @patch("sys.stderr")
+    @patch("sys.stdout")
+    def test_no_pipe_json(self, mock_stdout: Mock, mock_stderr: Mock):
+        self.subject(["streamlink", "--json"], mock_stdout)
+        self.assertEqual(mock_stdout.write.mock_calls, self._write_call_console_msg_json)
+        self.assertEqual(mock_stderr.write.mock_calls, [])
+
+    @patch("sys.stderr")
+    @patch("sys.stdout")
+    def test_pipe_no_json(self, mock_stdout: Mock, mock_stderr: Mock):
+        self.subject(["streamlink", "--stdout"], mock_stderr)
+        self.assertEqual(mock_stdout.write.mock_calls, [])
+        self.assertEqual(mock_stderr.write.mock_calls,
+                         self._write_call_log_cli_info + self._write_call_log_testcli_err + self._write_call_console_msg_error)
+
+    @patch("sys.stderr")
+    @patch("sys.stdout")
+    def test_pipe_json(self, mock_stdout: Mock, mock_stderr: Mock):
+        self.subject(["streamlink", "--stdout", "--json"], mock_stderr)
+        self.assertEqual(mock_stdout.write.mock_calls, [])
+        self.assertEqual(mock_stderr.write.mock_calls, self._write_call_console_msg_json)
+
+
+class TestCLIMainLoggingInfos(_TestCLIMainLogging):
     @unittest.skipIf(is_win32, "test only applicable on a POSIX OS")
     @patch("streamlink_cli.main.log")
     @patch("streamlink_cli.main.os.geteuid", Mock(return_value=0))
diff --git a/tests/test_console.py b/tests/test_console.py
--- a/tests/test_console.py
+++ b/tests/test_console.py
@@ -6,21 +6,17 @@
 
 
 class TestConsoleOutput(unittest.TestCase):
-    def test_msg_format(self):
+    def test_msg(self):
         output = StringIO()
         console = ConsoleOutput(output)
         console.msg("foo")
+        console.msg_json({"test": 1})
         self.assertEqual("foo\n", output.getvalue())
 
-    def test_msg_json_not_set(self):
-        output = StringIO()
-        console = ConsoleOutput(output)
-        self.assertEqual(None, console.msg_json({"test": 1}))
-        self.assertEqual("", output.getvalue())
-
     def test_msg_json(self):
         output = StringIO()
         console = ConsoleOutput(output, json=True)
+        console.msg("foo")
         console.msg_json({"test": 1})
         self.assertEqual('{\n  "test": 1\n}\n', output.getvalue())
 
diff --git a/tests/test_log.py b/tests/test_log.py
--- a/tests/test_log.py
+++ b/tests/test_log.py
@@ -15,6 +15,34 @@ def _new_logger(cls, format="[{name}][{levelname}] {message}", style="{", **para
         logger.basicConfig(stream=output, format=format, style=style, **params)
         return logging.getLogger("streamlink.test"), output
 
+    def test_level_names(self):
+        self.assertEqual(logger.levels, [
+            "none", "critical", "error", "warning", "info", "debug", "trace"
+        ])
+        self.assertEqual(logging.getLevelName(logger.NONE), "none")
+        self.assertEqual(logging.getLevelName(logger.CRITICAL), "critical")
+        self.assertEqual(logging.getLevelName(logger.ERROR), "error")
+        self.assertEqual(logging.getLevelName(logger.WARNING), "warning")
+        self.assertEqual(logging.getLevelName(logger.INFO), "info")
+        self.assertEqual(logging.getLevelName(logger.DEBUG), "debug")
+        self.assertEqual(logging.getLevelName(logger.TRACE), "trace")
+
+        self.assertEqual(logging.getLevelName("none"), logger.NONE)
+        self.assertEqual(logging.getLevelName("critical"), logger.CRITICAL)
+        self.assertEqual(logging.getLevelName("error"), logger.ERROR)
+        self.assertEqual(logging.getLevelName("warning"), logger.WARNING)
+        self.assertEqual(logging.getLevelName("info"), logger.INFO)
+        self.assertEqual(logging.getLevelName("debug"), logger.DEBUG)
+        self.assertEqual(logging.getLevelName("trace"), logger.TRACE)
+
+        self.assertEqual(logging.getLevelName("NONE"), logger.NONE)
+        self.assertEqual(logging.getLevelName("CRITICAL"), logger.CRITICAL)
+        self.assertEqual(logging.getLevelName("ERROR"), logger.ERROR)
+        self.assertEqual(logging.getLevelName("WARNING"), logger.WARNING)
+        self.assertEqual(logging.getLevelName("INFO"), logger.INFO)
+        self.assertEqual(logging.getLevelName("DEBUG"), logger.DEBUG)
+        self.assertEqual(logging.getLevelName("TRACE"), logger.TRACE)
+
     def test_level(self):
         log, output = self._new_logger()
         logger.root.setLevel("info")
@@ -25,6 +53,17 @@ def test_level(self):
         log.debug("test")
         self.assertNotEqual(output.tell(), 0)
 
+    def test_level_none(self):
+        log, output = self._new_logger()
+        logger.root.setLevel("none")
+        log.critical("test")
+        log.error("test")
+        log.warning("test")
+        log.info("test")
+        log.debug("test")
+        log.trace("test")
+        self.assertEqual(output.tell(), 0)
+
     def test_output(self):
         log, output = self._new_logger()
         logger.root.setLevel("debug")