Repository: dbt-labs/dbt-core

Issue:
[CT-1474] [Feature] Let users add fields to strucutred logs
### Is this your first time submitting a feature request?

No, I have made other feature requests.
- [x] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

At the moment, structured logs have a predefined set of JSON properties ([docs](https://docs.getdbt.com/reference/events-logging#structured-logging)). 

I would like to to be able to add fields to the structured logs to consume in downstream systems.

This ask was inspired by the [query-comment marco](https://docs.getdbt.com/reference/project-configs/query-comment#advanced-use-a-macro-to-generate-a-comment). By customizing the macro, I can "log" exactly the details I need, removing info I don't need and adding info I do need.

### Describe alternatives you've considered

Here's my work-around to augment logs with additional data. 

Instead of consuming logs as they are generated to stdout, I write the logs to file. Then I execute a Python script that adds data from `manifest.json` into the logs file (by `data.node_info.unique_id` in logs) and then re-emits the logs.

### Who will this benefit?

Use-case: add node tags to the `data.node_info` property in logs. By tagging a node with team ownership, the log consumer can alert different teams based on the failures.

Example:
```
{"level": "error", "data": { "node_info": {"tags": ["finance_team"], ...}, ...}, ...}
```
The consumer would see the `"level": "error"` and prepare to alert. Then the consumer would see that the node is tagged with `"finance_team"`, so instead of sending a general alert, a more narrow, finance alert is sent.

### Are you interested in contributing this feature?

Yes, I could help if we are interested in the feature.

### Anything else?

_No response_


Discussion:
@tmastny Thanks for opening! & long time no see :)

First off, you have good timing. We're doing work on structured logging right now, and one piece of it is ensuring that the `node_info` dict is available on as many relevant events as possible (at parity with the old logging system: https://github.com/dbt-labs/dbt-core/issues/6222).

Two thoughts:
1. I don't think we should put the full `node` object (manifest entry) on every single node-related. The motivation for `node_info` is, this should be enough information to answer 80-90% of real-ish-time questions you could ask about _dbt models running right now_. For everything else, including retrospective / longitudinal analysis, a join to the full manifest entry feels appropriate.
2. I don't think we'll be able to support this in the same sort of _totally flexible_ mechanism as the `query_comment` macro. We're doing a lot of work to make our events/log messages strongly typed, and that requires us to be able to say in advance the exact structure of their contents.

That said - I totally get the value in being able to annotate a model with some custom attributes, and see those attributes propagate to all log events related to that model, in a way that makes it super easy to `| jq` out just the log events you're interested in.

### Proposal

So: **What do you think about using the `meta` field for this?** That is: always `meta`, and only `meta`.

The big idea with `meta` is:
- it's a dictionary that can contain whatever key-value pairs you want
- it has no functional purpose within dbt — you put things in there, and they show up
- it's for metadata! and logs are metadata :)

Not too dissimilar from the ability to annotate all log events with env/run-level metadata (https://github.com/dbt-labs/dbt-core/issues/5228). It's explicit, and it's explicitly _not used_ for any actual dbt core functionality.

How is `meta` different from `tags`? Well... `tags` are a teensy bit more special, and originally designed for node selection. (You can use `config.meta.key:value` now for node selection, too, but that's not its original intent.) I'd vote for one or the other, and my vote goes to `meta`.

### Implementation

As far as implementing this, I think it could be a fairly simple change in two places. First, the proto type:
https://github.com/dbt-labs/dbt-core/blob/6c76137da4e72bbcad6e7789a78c31f1bf5cffac/core/dbt/events/types.proto#L27-L37
Adding something like: `map<string, string> meta = 9;`

And then the Python code to use it:
https://github.com/dbt-labs/dbt-core/blob/6c76137da4e72bbcad6e7789a78c31f1bf5cffac/core/dbt/contracts/graph/parsed.py#L186-L203

Adding something like: `"meta": getattr(self, "meta", {}),`
Comments from internal discussion:
- Is the distinction between `tags` and `meta` so meaningful? Should we just include both? I lean toward the simplicity of one or the other
- Should this be the entire `meta` dict, or a subset of `meta` properties based on (e.g.) a specific key prefix? I lean toward the simplicity of including the entire dict; this isn't like env vars, where there are lots of noisy / functional env vars that we want to exclude

Resolving patch:
diff --git a/core/dbt/contracts/graph/nodes.py b/core/dbt/contracts/graph/nodes.py
--- a/core/dbt/contracts/graph/nodes.py
+++ b/core/dbt/contracts/graph/nodes.py
@@ -46,6 +46,7 @@
 from dbt.events.contextvars import set_contextvars
 from dbt import flags
 from dbt.node_types import ModelLanguage, NodeType
+from dbt.utils import cast_dict_to_dict_of_strings
 
 
 from .model_config import (
@@ -206,6 +207,8 @@ class NodeInfoMixin:
 
     @property
     def node_info(self):
+        meta = getattr(self, "meta", {})
+        meta_stringified = cast_dict_to_dict_of_strings(meta)
         node_info = {
             "node_path": getattr(self, "path", None),
             "node_name": getattr(self, "name", None),
@@ -215,6 +218,7 @@ def node_info(self):
             "node_status": str(self._event_status.get("node_status")),
             "node_started_at": self._event_status.get("started_at"),
             "node_finished_at": self._event_status.get("finished_at"),
+            "meta": meta_stringified,
         }
         node_info_msg = NodeInfo(**node_info)
         return node_info_msg
diff --git a/core/dbt/events/proto_types.py b/core/dbt/events/proto_types.py
--- a/core/dbt/events/proto_types.py
+++ b/core/dbt/events/proto_types.py
@@ -47,6 +47,7 @@ class NodeInfo(betterproto.Message):
     node_status: str = betterproto.string_field(6)
     node_started_at: str = betterproto.string_field(7)
     node_finished_at: str = betterproto.string_field(8)
+    meta: str = betterproto.string_field(9)
 
 
 @dataclass
diff --git a/core/dbt/utils.py b/core/dbt/utils.py
--- a/core/dbt/utils.py
+++ b/core/dbt/utils.py
@@ -683,3 +683,10 @@ def cast_to_int(integer: Optional[int]) -> int:
         return 0
     else:
         return integer
+
+
+def cast_dict_to_dict_of_strings(dct):
+    new_dct = {}
+    for k, v in dct.items():
+        new_dct[str(k)] = str(v)
+    return new_dct


Test patch:
diff --git a/tests/functional/logging/test_meta_logging.py b/tests/functional/logging/test_meta_logging.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/logging/test_meta_logging.py
@@ -0,0 +1,43 @@
+import pytest
+from dbt.tests.util import run_dbt, read_file
+import json
+
+model1 = 'select 1 as fun'
+model2 = '{{ config(meta={"owners": ["team1", "team2"]})}} select 1 as fun'
+model3 = '{{ config(meta={"key": 1})}} select 1 as fun'
+
+@pytest.fixture(scope="class")  # noqa
+def models():
+    return {"model1.sql": model1, "model2.sql": model2, "model3.sql": model3}
+
+
+# This test checks that various events contain node_info,
+# which is supplied by the log_contextvars context manager
+def test_meta(project, logs_dir):
+    run_dbt(["--log-format=json", "run"])
+
+    # get log file
+    log_file = read_file(logs_dir, "dbt.log")
+    assert log_file
+
+    for log_line in log_file.split('\n'):
+        # skip empty lines
+        if len(log_line) == 0:
+            continue
+        # The adapter logging also shows up, so skip non-json lines
+        if "[debug]" in log_line:
+            continue
+
+        log_dct = json.loads(log_line)
+        if "node_info" not in log_dct["data"]:
+            continue
+
+        print(f"--- log_dct: {log_dct}")
+        node_info = log_dct["data"]["node_info"]
+        node_path = node_info['node_path']
+        if node_path == "model1.sql":
+            assert node_info['meta'] == {}
+        elif node_path == "model2.sql":
+            assert node_info['meta'] == {"owners": "['team1', 'team2']"}
+        elif node_path == "model3.sql":
+            assert node_info['meta'] == {"key": "1"}
diff --git a/tests/unit/test_proto_events.py b/tests/unit/test_proto_events.py
--- a/tests/unit/test_proto_events.py
+++ b/tests/unit/test_proto_events.py
@@ -87,6 +87,7 @@ def test_exception_events():
 
 
 def test_node_info_events():
+    meta_dict = {"string-key1": ["value1", 2], "string-key2": {"nested-dict-key": "value2"}, 1: "value-from-non-string-key", "string-key3": 1, "string-key4": ["string1", 1, "string2", 2]}
     node_info = {
         "node_path": "some_path",
         "node_name": "some_name",
@@ -96,6 +97,7 @@ def test_node_info_events():
         "node_status": "started",
         "node_started_at": "some_time",
         "node_finished_at": "another_time",
+        "meta": meta_dict,
     }
     event = LogStartLine(
         description="some description",
@@ -105,6 +107,7 @@ def test_node_info_events():
     )
     assert event
     assert event.node_info.node_path == "some_path"
+    assert event.node_info.meta == meta_dict
 
 
 def test_extra_dict_on_event(monkeypatch):