Repository: dbt-labs/dbt-core

Issue:
Text vs JSON and File vs CLI logs
### Is there an existing feature request for this?

- [X] I have searched the existing issues

### Describe the Feature

re [this comment](https://github.com/dbt-labs/dbt-core/pull/4353#discussion_r758603782), text log lines might want to be different per destination. We may want text log lines to filter from json log lines, but I'm not sure about that. Just something to consider.

### Describe alternatives you've considered

_No response_

### Who will this benefit?

_No response_

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_


Discussion:
To summarize myself over in #4293:

> Broadly, I find that I want:
> 
> - color, concision, clean spacing on the CLI
> - colorless + compact verbosity in the File + JSON-formatted

**Edit since then**: Let's let users determine whether to use color or not based on the colorizing configs. There's good reasons to preserve color in JSON-formatted log messages, if they're going to be eventually displayed in a CLI-like setting.
@gshank This is the issue we can use for the last bit of cosmetic work.

Let's keep JSON-formatted logs as they are. They include _everything_, as they should. The intended consumer of these is a machine, and users can spin up their own scripts (or even just `jq`) to modify/filter down what's shown.

From my perspective, the big idea is differentiating between standard _text formatted_ logs for the CLI (info-level) and File (debug-level). These are both intended for use by human beings, and for fundamentally different purposes:
- CLI logs are interactive. They should show what's happening, while it's happening, and not too much more. (Conversely, we should avoid extended periods where we show nothing—this adds to frustration when things are slow, such as full re-parses of big projects.) CLI logs should also surface warnings and errors in a way that makes them easy to spot, and easy to take further action.
- Sometimes, that further action should look like diving into more detailed logs, stored in `logs/dbt.log`. These should have _everything_ that a human being might need to debug what went wrong with a dbt project. If we've failed to include useful information, it's lost to the sands of time—or until they re-run / reproduce the error with JSON-formatted logging.

So my inclination is that, relative to how this works presently (in `main`):
- **CLI:** Text-formatted log lines should include short timestamps and messages — that's all
- **File:** Text-formatted messages should include longer timestamps, log level (info/warn/error), thread name, and a way to visually see where one run stopped and the next one started. (Would it make sense to include `invocation_id` too, or would that add too much clutter?)
```
2021-12-01 15:55:36.159022 | info  | MainThread | Running with dbt=1.0.0-rc3
2021-12-01 15:55:36.159022 | debug | MainThread |

======================================

Running dbt with arguments:
- ...

======================================
```
```
2021-12-01 15:55:37.432664 | debug  | Thread-3  | Using postgres connection "model.testy.my_view"
```

Or something like that.

In practice, I think this will look like splitting up [`create_text_log_line`](https://github.com/dbt-labs/dbt-core/blob/e93ad5f118261f428bd24f052a780d7f72277207/core/dbt/events/functions.py#L162-L171) into two functions (one for CLI and one for File), adding an arg to `create_log_line` for CLI/File, and passing that arg from [these two sites](https://github.com/dbt-labs/dbt-core/blob/e93ad5f118261f428bd24f052a780d7f72277207/core/dbt/events/functions.py#L286-L310) accordingly.

Tricky things:
- I think we're going to struggle to get `MainReportVersion` + `MainReportArgs` into the File, because they're currently fired _before_ we've parsed `dbt_project.yml`, i.e. before we know the `log_path` (in case it's user-configured).
- We do have the ability to construct different messages whether an event is firing to the CLI or the File, by using/overriding the `file_msg` and `cli_msg` methods. (The base types are implemented to use the same `message` for both by default.)

More things to think about:
- Some logs are so verbose that we shouldn't include them by default, e.g. adapter cache events. Today, we turn those off by default, and allow users to turn them on with the `--log-cache-events` config. (In practice, we only do this to debug _very_ specific errors.) In the future, we could think about encoding these as "trace"-level, and allowing users to opt in.
- We should enable users to opt into / relevel logs from dependencies. This is especially important to see "trace-level" details of connection errors for specific database clients. We support this in [a pretty janky way today](https://github.com/dbt-labs/dbt-core/blob/e93ad5f118261f428bd24f052a780d7f72277207/core/dbt/logger.py#L625-L638) (hard-coded in dbt-core, ugh).

Open to hearing thoughts from everyone!

Resolving patch:
diff --git a/core/dbt/events/functions.py b/core/dbt/events/functions.py
--- a/core/dbt/events/functions.py
+++ b/core/dbt/events/functions.py
@@ -3,7 +3,7 @@
 from datetime import datetime
 import dbt.events.functions as this  # don't worry I hate it too.
 from dbt.events.base_types import Cli, Event, File, ShowException, NodeInfo, Cache
-from dbt.events.types import EventBufferFull, T_Event
+from dbt.events.types import EventBufferFull, T_Event, MainReportVersion
 import dbt.flags as flags
 # TODO this will need to move eventually
 from dbt.logger import SECRET_ENV_PREFIX, make_log_dir_if_missing, GLOBAL_LOGGER
@@ -17,6 +17,7 @@
 from logging.handlers import RotatingFileHandler
 import os
 import uuid
+import threading
 from typing import Any, Callable, Dict, List, Optional, Union
 import dataclasses
 from collections import deque
@@ -162,12 +163,31 @@ def event_to_serializable_dict(
 # translates an Event to a completely formatted text-based log line
 # you have to specify which message you want. (i.e. - e.message, e.cli_msg(), e.file_msg())
 # type hinting everything as strings so we don't get any unintentional string conversions via str()
-def create_text_log_line(e: T_Event, msg_fn: Callable[[T_Event], str]) -> str:
+def create_stdout_text_log_line(e: T_Event, msg_fn: Callable[[T_Event], str]) -> str:
     color_tag: str = '' if this.format_color else Style.RESET_ALL
     ts: str = e.get_ts().strftime("%H:%M:%S")
     scrubbed_msg: str = scrub_secrets(msg_fn(e), env_secrets())
+    log_line: str = f"{color_tag}{ts}  {scrubbed_msg}"
+    return log_line
+
+
+def create_file_text_log_line(e: T_Event, msg_fn: Callable[[T_Event], str]) -> str:
+    log_line: str = ''
+    # Create a separator if this is the beginning of an invocation
+    if type(e) == MainReportVersion:
+        separator = 30 * '='
+        log_line = f'\n\n{separator} {e.get_ts()} | {get_invocation_id()} {separator}\n'
+    color_tag: str = '' if this.format_color else Style.RESET_ALL
+    ts: str = e.get_ts().strftime("%H:%M:%S.%f")
+    scrubbed_msg: str = scrub_secrets(msg_fn(e), env_secrets())
     level: str = e.level_tag() if len(e.level_tag()) == 5 else f"{e.level_tag()} "
-    log_line: str = f"{color_tag}{ts} | [ {level} ] | {scrubbed_msg}"
+    thread = ''
+    if threading.current_thread().getName():
+        thread_name = threading.current_thread().getName()
+        thread_name = thread_name[:10]
+        thread_name = thread_name.ljust(10, ' ')
+        thread = f' [{thread_name}]:'
+    log_line = log_line + f"{color_tag}{ts} [{level}]{thread} {scrubbed_msg}"
     return log_line
 
 
@@ -179,13 +199,14 @@ def create_json_log_line(e: T_Event, msg_fn: Callable[[T_Event], str]) -> str:
     return scrub_secrets(raw_log_line, env_secrets())
 
 
-# calls create_text_log_line() or create_json_log_line() according to logger config
-def create_log_line(e: T_Event, msg_fn: Callable[[T_Event], str]) -> str:
-    return (
-        create_json_log_line(e, msg_fn)
-        if this.format_json else
-        create_text_log_line(e, msg_fn)
-    )
+# calls create_stdout_text_log_line() or create_json_log_line() according to logger config
+def create_log_line(e: T_Event, msg_fn: Callable[[T_Event], str], file_output=False) -> str:
+    if this.format_json:
+        return create_json_log_line(e, msg_fn)  # json output, both console and file
+    elif file_output is True:
+        return create_file_text_log_line(e, msg_fn)  # default file output
+    else:
+        return create_stdout_text_log_line(e, msg_fn)  # console output
 
 
 # allows for resuse of this obnoxious if else tree.
@@ -285,7 +306,7 @@ def fire_event(e: Event) -> None:
 
     # always logs debug level regardless of user input
     if isinstance(e, File):
-        log_line = create_log_line(e, msg_fn=lambda x: x.file_msg())
+        log_line = create_log_line(e, msg_fn=lambda x: x.file_msg(), file_output=True)
         # doesn't send exceptions to exception logger
         send_to_logger(FILE_LOG, level_tag=e.level_tag(), log_line=log_line)
 
diff --git a/core/dbt/main.py b/core/dbt/main.py
--- a/core/dbt/main.py
+++ b/core/dbt/main.py
@@ -221,24 +221,22 @@ def track_run(task):
 def run_from_args(parsed):
     log_cache_events(getattr(parsed, 'log_cache_events', False))
 
-    # we can now use the logger for stdout
-    # set log_format in the logger
-    # if 'list' task: set stdout to WARN instead of INFO
-    level_override = parsed.cls.pre_init_hook(parsed)
-
-    fire_event(MainReportVersion(v=str(dbt.version.installed)))
-
     # this will convert DbtConfigErrors into RuntimeExceptions
     # task could be any one of the task objects
     task = parsed.cls.from_args(args=parsed)
-    fire_event(MainReportArgs(args=parsed))
 
+    # Set up logging
     log_path = None
     if task.config is not None:
         log_path = getattr(task.config, 'log_path', None)
-    # we can finally set the file logger up
     log_manager.set_path(log_path)
+    # if 'list' task: set stdout to WARN instead of INFO
+    level_override = parsed.cls.pre_init_hook(parsed)
     setup_event_logger(log_path or 'logs', level_override)
+
+    fire_event(MainReportVersion(v=str(dbt.version.installed)))
+    fire_event(MainReportArgs(args=parsed))
+
     if dbt.tracking.active_user is not None:  # mypy appeasement, always true
         fire_event(MainTrackingUserState(dbt.tracking.active_user.state()))
 


Test patch: