Repository: dbt-labs/dbt-core

Issue:
Structured Logging Phase 2
### Describe the Feature

Individual tasks are ordered by expected release with high priority unfinished items at the top of the list.

### 1.0.0-RC2 (released 11/22)
- [x] (size: M) (@nathaniel-may) cleanup code #4266
- [x] (size: XL) (@iknox-fa) add codes to each event ~~#4268~~ #4319
- [x] (size: S) (@nathaniel-may) add versioning ~#4272~ #4289 
- [x] (size: M) (@nathaniel-may) fix adapter interface #4266
- [x] (size: S) (@emmyoop) include event dataclass attributes in json-formatted logs `{code: ..., msg: ...  data:{<goes here>}}` #4301  
- [x] (size: S) (@jtcohen6) fix log duplication in subsequent tests #4275 

### 1.0.0-RC3 (released 11/30)
- [x] (size: XXL) (@emmyoop) add additional log lines for node status reporting ([details](https://github.com/dbt-labs/dbt-core/issues/3359#issuecomment-934428619))  #4336
- [x] (size: M) (@iknox-fa) Does something graceful when it runs out of memory (small machines shouldn't explode because of logging), is queryable by a potential future server-wrapper #4358
- [x] (size: S) (@nathaniel-may) add thread name to each event (cheap way to distinguish dbt runs from each other, tracks dbt threads when we go multithreaded) (make sure windows is handled) #4353
- [x] (size: XL) (@nathaniel-may) stabilize JSON serialization.
- [x] (size: M) (@gshank) make sure we can distinguish logs from different runs Impl - Existing InvocationID. If tracking is off, it isn't created today so we would need to create it. (to add it to logging, add it as a [class property on the superclass](https://github.com/dbt-labs/dbt-core/blob/e1a2e8d9f542b15d283b3085ee867e882d85f5b7/core/dbt/events/base_types.py#L55-L59)) (previously considered: preserve process id like it is today, global run_id reported at start, or allow some sort of user markers)  #4340

### 1.0.0 (goal: merge by Fri 12/03)
**Cleanup:**
- [x] (size: S) (@jtcohen6) Find a way to configure logger _before_ firing any events, e.g. avoid text logs preceding JSON logs https://github.com/dbt-labs/dbt-core/pull/4341
- [x] (size: S) (@emmyoop) turn on verbose cache logging (every event fired by `cache.py`) if and only if the `--log-cache-events` flag is turned on #4369
- [x] (size: S) (@jtcohen6) include snowplow tracking events in file logging (not cli) when available (i.e. without breaking `dbt clean`) https://github.com/dbt-labs/dbt-core/pull/4341
- [x] (size: L?) (@iknox-fa) fix failing Windows test (#4261)
- [x] (size: S) (@jtcohen6) feature parity for `dbt ls` + workflow automation: set stdout logger to WARN level instead of INFO (see [here](https://github.com/dbt-labs/dbt-core/blob/22416980d19888d7bbbcfcda639c16d37a80b948/core/dbt/logger.py#L517-L521) + [here](https://github.com/dbt-labs/dbt-core/blob/22416980d19888d7bbbcfcda639c16d37a80b948/core/dbt/task/list.py#L55-L59)), or better yet replace [`print()` here](https://github.com/dbt-labs/dbt-core/blob/22416980d19888d7bbbcfcda639c16d37a80b948/core/dbt/task/list.py#L151) with an actual fired event that can be filtered on + piped out (??) https://github.com/dbt-labs/dbt-core/pull/4341
- [x] (size XL) (@nathaniel-may) black-box schema testing of all possible log events. https://github.com/dbt-labs/dbt-core/pull/4327
- [x] (size S) (@emmyoop) update `README` to reflect current logging state
- [x] (size M?) (@gshank ) make sure logs go to stdout not stderr #4367
- [x] (size S) (@iknox-fa) make event history size configurable #4416
- [x] (size XS) (@emmyoop) #4391
- [x] (size M) (@nathaniel-may) json serialization needs structural tweak #4392
- [x] (size S) (@jtcohen6 @nathaniel-may) Don't fire EmptyLine events to json https://github.com/dbt-labs/dbt-core/pull/4388
- [x] (size XS) (@jtcohen6) #4404 - https://github.com/dbt-labs/dbt-core/pull/4405
- [x] (size XS) (@emmyoop) rename `dbt.log.old` to `dbt.log.legacy` #4402 
- [x] (size S) (@jtcohen6) sketch out/create user facing docs (data, system for codes, etc) - possibly its own page dbt-labs/docs.getdbt.com#938



Discussion:
From my point of view, `extra.run_state` was pretty unintuitive, and it was the best we had to offer! I bet we can do something much better with node status reporting :) excited for us to take a more holistic view, and add in more events to fill in the gaps
It's unclear to me if this is already covered by "improve look and feel", but the new logs are much less visually parsable in some cases. 

e.g. this output from `dbt deps`:

```
(dbt-dev) joel@Joel-Labes internal-analytics % dbt deps 
{"level": "info ", "msg": "Running with dbt=1.0.0-rc1", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing dbt-labs/dbt_utils@0.7.4-b1", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.7.4-b1", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Updated version available: 0.7.4", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing dbt-labs/snowplow@0.13.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.13.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Updated version available: 0.13.1", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing dbt-labs/codegen@0.4.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.4.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Up to date!", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing tailsdotcom/dbt_artifacts@0.5.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.5.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Up to date!", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing fivetran/hubspot@0.4.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.4.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Updated version available: 0.4.1", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing fivetran/zendesk_source@0.4.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.4.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Updated version available: 0.4.1", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing dbt-labs/audit_helper@0.4.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.4.0", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Up to date!", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing fivetran/hubspot_source@0.4.3", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.4.3", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Up to date!", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "Installing fivetran/fivetran_utils@0.2.9", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Installed from version 0.2.9", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "  Up to date!", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
{"level": "info ", "msg": "\nUpdates available for packages: ['dbt-labs/dbt_utils', 'dbt-labs/snowplow', 'fivetran/hubspot', 'fivetran/zendesk_source']                 \nUpdate your versions in packages.yml, then run dbt deps", "pid": 38573, "ts": "2021-11-17T22:31:48.762544"}
```

![image](https://user-images.githubusercontent.com/7335046/142274672-234a4ff9-f142-46bf-9911-f0b0a39be7b1.png)

The old version:
```
(dbt-prod) joel@Joel-Labes internal-analytics % dbt deps
Running with dbt=0.21.0
Installing dbt-labs/dbt_utils@0.7.4
  Installed from version 0.7.4
  Up to date!
Installing dbt-labs/snowplow@0.13.0
  Installed from version 0.13.0
  Updated version available: 0.13.2
Installing dbt-labs/codegen@0.4.0
  Installed from version 0.4.0
  Up to date!
Installing tailsdotcom/dbt_artifacts@0.5.0
  Installed from version 0.5.0
  Up to date!
Installing fivetran/hubspot@0.4.0
  Installed from version 0.4.0
  Updated version available: 0.4.1
Installing fivetran/zendesk_source@0.4.0
  Installed from version 0.4.0
  Updated version available: 0.4.1
Installing dbt-labs/audit_helper@0.4.0
  Installed from version 0.4.0
  Up to date!
Installing fivetran/hubspot_source@0.4.3
  Installed from version 0.4.3
  Up to date!
Installing fivetran/fivetran_utils@0.2.9
  Installed from version 0.2.9
  Up to date!

Updates available for packages: ['dbt-labs/snowplow', 'fivetran/hubspot', 'fivetran/zendesk_source']                 
Update your versions in packages.yml, then run dbt deps
```
![image](https://user-images.githubusercontent.com/7335046/142275236-905916dc-3d8c-4562-baee-56e41439053c.png)

@joellabes This was a bug.  #4302 will resolve it.
Phenomenal :heart_eyes:  thank you! 

Resolving patch:
diff --git a/core/dbt/events/types.py b/core/dbt/events/types.py
--- a/core/dbt/events/types.py
+++ b/core/dbt/events/types.py
@@ -7,7 +7,7 @@
 )
 from dbt.events.format import format_fancy_output_line, pluralize
 from dbt.node_types import NodeType
-from typing import Any, Callable, cast, Dict, List, Optional, Set, Tuple, TypeVar
+from typing import Any, Callable, Dict, List, Optional, Set, Tuple, TypeVar
 
 
 # The classes in this file represent the data necessary to describe a
@@ -734,7 +734,7 @@ class DumpBeforeAddGraph(DebugLevel, Cli, File):
     def message(self) -> str:
         # workaround for https://github.com/python/mypy/issues/6910
         # TODO remove when we've upgraded to a mypy version without that bug
-        func_returns = cast(Callable[[], Dict[str, List[str]]], getattr(self, "graph_func"))
+        func_returns = self.graph_func()  # type: ignore
         return f"before adding : {func_returns}"
 
     # TODO should we manually cache the graph here so it doesn't get called for the message
@@ -755,7 +755,7 @@ class DumpAfterAddGraph(DebugLevel, Cli, File):
 
     def message(self) -> str:
         # workaround for https://github.com/python/mypy/issues/6910
-        func_returns = cast(Callable[[], Dict[str, List[str]]], getattr(self, "graph_func"))
+        func_returns = self.graph_func()  # type: ignore
         return f"after adding: {func_returns}"
 
     # TODO should we manually cache the graph here so it doesn't get called for the message
@@ -776,7 +776,7 @@ class DumpBeforeRenameSchema(DebugLevel, Cli, File):
 
     def message(self) -> str:
         # workaround for https://github.com/python/mypy/issues/6910
-        func_returns = cast(Callable[[], Dict[str, List[str]]], getattr(self, "graph_func"))
+        func_returns = self.graph_func()  # type: ignore
         return f"before rename: {func_returns}"
 
     def fields_to_json(self, val: Any) -> Any:
@@ -793,7 +793,7 @@ class DumpAfterRenameSchema(DebugLevel, Cli, File):
 
     def message(self) -> str:
         # workaround for https://github.com/python/mypy/issues/6910
-        func_returns = cast(Callable[[], Dict[str, List[str]]], getattr(self, "graph_func"))
+        func_returns = self.graph_func()  # type: ignore
         return f"after rename: {func_returns}"
 
     def fields_to_json(self, val: Any) -> Any:


Test patch: