Repository: dbt-labs/dbt-core

Issue:
[CT-2264] Don't raise exception during event serialization failure, just log warning
Instead of raising as an `Exception` that stops the whole run, let's instead fire a different warn-level event to call out the event serialization failure. Ideally, this would include enough information about the event's contents (serialized as a string?) that would be helpful in debugging, but let's make extra sure that doesn't silently/loudly fail either.

In `main` branch:

https://github.com/dbt-labs/dbt-core/blob/72076b3fe5a9a97884bbc4b6951a984eb50556a9/core/dbt/events/functions.py#L205-L209

In `1.3.latest` (should be similar in `1.2.latest` and `1.1.latest`):

https://github.com/dbt-labs/dbt-core/blob/7f2e9f94c0dcb27feb4ac1f6b545140e2fd301b8/core/dbt/events/functions.py#L129-L135


Discussion:
Duh. An obvious improvement. Should have thought of this before...
Opened this up while chatting with @QMalcolm earlier! We were taking a look at:

- https://github.com/dbt-labs/dbt-core/issues/7108
- https://github.com/dbt-labs/dbt-core/issues/6568

Both of those have actually been fixed in `main` / `1.4.latest`, but it's worth:
- adding tests for the odd behavior to `main` (to ensure we don't regress on either one in the future)
- making this change, to prevent serialization errors from ever being a deal-breaker
- backporting those patches to older versions (v1.1, v1.2, v1.3)
Proposed catch-all fixes for:
- `1.0.latest`: https://github.com/dbt-labs/dbt-core/commit/fd83cc7f9ba37bf7d85eba6651356ae38fcfb84d
- `1.1.latest`, `1.2.latest`, `1.3.latest`: https://github.com/dbt-labs/dbt-core/commit/dd9bcddaf10b436d53322ed9c05a097008dea71b (need to handle errors dict construction and also during JSON serialization)

I tested those changes with the reproduction case from https://github.com/dbt-labs/dbt-core/issues/7108, and it gracefully handled the bad log message:
```json
{"code": "A001", "data": {"v": "=1.3.3"}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "info", "log_version": 2, "msg": "Running with dbt=1.3.3", "pid": 71932, "thread_name": "MainThread", "ts": "2023-03-03T21:16:43.492425Z", "type": "log_line"}
{"code": "W006", "data": {"stat_line": "1 model, 0 tests, 0 snapshots, 0 analyses, 290 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics"}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "info", "log_version": 2, "msg": "Found 1 model, 0 tests, 0 snapshots, 0 analyses, 290 macros, 0 operations, 1 seed file, 0 sources, 0 exposures, 0 metrics", "pid": 71932, "thread_name": "MainThread", "ts": "2023-03-03T21:16:43.574628Z", "type": "log_line"}
{"code": "Q026", "data": {"num_threads": 5, "target_name": "dev"}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "info", "log_version": 2, "msg": "Concurrency: 5 threads (target='dev')", "pid": 71932, "thread_name": "MainThread", "ts": "2023-03-03T21:16:43.778860Z", "type": "log_line"}
{"code": "Q033", "data": {"description": "sql view model dbt_jcohen.my_model", "index": 1, "node_info": {"materialized": "view", "node_finished_at": null, "node_name": "my_model", "node_path": "my_model.sql", "node_started_at": "2023-03-03T21:16:43.782645", "node_status": "started", "resource_type": "model", "unique_id": "model.test.my_model"}, "total": 1}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "info", "log_version": 2, "msg": "1 of 1 START sql view model dbt_jcohen.my_model ................................ [RUN]", "pid": 71932, "thread_name": "Thread-1", "ts": "2023-03-03T21:16:43.783106Z", "type": "log_line"}
{"code": "Z046", "data": {"log_fmt": "", "msg": "Error while serializing MacroEventInfo(msg=Undefined, code='M011')"}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "warn", "log_version": 2, "msg": "", "pid": 71932, "thread_name": "Thread-1", "ts": "2023-03-03T21:16:43.786917Z", "type": "log_line"}
{"code": "Q012", "data": {"description": "sql view model dbt_jcohen.my_model", "execution_time": 0, "index": 1, "node_info": {"materialized": "view", "node_finished_at": null, "node_name": "my_model", "node_path": "my_model.sql", "node_started_at": "2023-03-03T21:16:43.782645", "node_status": "compiling", "resource_type": "model", "unique_id": "model.test.my_model"}, "status": "CREATE VIEW", "total": 1}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "info", "log_version": 2, "msg": "1 of 1 OK created sql view model dbt_jcohen.my_model ........................... [\u001b[32mCREATE VIEW\u001b[0m in 0.08s]", "pid": 71932, "thread_name": "Thread-1", "ts": "2023-03-03T21:16:43.867398Z", "type": "log_line"}
{"code": "E040", "data": {"execution": " in 0 hours 0 minutes and 0.30 seconds", "execution_time": 0.3049299716949463, "stat_line": "1 view model"}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "info", "log_version": 2, "msg": "Finished running 1 view model in 0 hours 0 minutes and 0.30 seconds (0.30s).", "pid": 71932, "thread_name": "MainThread", "ts": "2023-03-03T21:16:43.880736Z", "type": "log_line"}
{"code": "Z030", "data": {"keyboard_interrupt": false, "num_errors": 0, "num_warnings": 0}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "info", "log_version": 2, "msg": "\u001b[32mCompleted successfully\u001b[0m", "pid": 71932, "thread_name": "MainThread", "ts": "2023-03-03T21:16:43.887088Z", "type": "log_line"}
{"code": "Z023", "data": {"stats": {"error": 0, "pass": 1, "skip": 0, "total": 1, "warn": 0}}, "invocation_id": "c3726ca0-9001-46d7-a554-21e5299f0333", "level": "info", "log_version": 2, "msg": "Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1", "pid": 71932, "thread_name": "MainThread", "ts": "2023-03-03T21:16:43.887329Z", "type": "log_line"}
```

We can do something nicer for `main` + `1.4.latest`, which are using the nice new event system
Also "tojson" can cause a similar error.

Resolving patch:
diff --git a/core/dbt/events/base_types.py b/core/dbt/events/base_types.py
--- a/core/dbt/events/base_types.py
+++ b/core/dbt/events/base_types.py
@@ -79,7 +79,7 @@ def __init__(self, *args, **kwargs):
             if "pytest" in sys.modules:
                 raise Exception(error_msg)
             else:
-                fire_event(Note(msg=error_msg))
+                fire_event(Note(msg=error_msg), level=EventLevel.WARN)
                 self.pb_msg = msg_cls()
 
     def __setattr__(self, key, value):
diff --git a/core/dbt/events/functions.py b/core/dbt/events/functions.py
--- a/core/dbt/events/functions.py
+++ b/core/dbt/events/functions.py
@@ -2,7 +2,7 @@
 from dbt.events.base_types import BaseEvent, EventLevel, EventMsg
 from dbt.events.eventmgr import EventManager, LoggerConfig, LineFormat, NoFilter
 from dbt.events.helpers import env_secrets, scrub_secrets
-from dbt.events.types import Formatting
+from dbt.events.types import Formatting, Note
 from dbt.flags import get_flags, ENABLE_LEGACY_LOGGER
 from dbt.logger import GLOBAL_LOGGER, make_log_dir_if_missing
 from functools import partial
@@ -219,7 +219,9 @@ def msg_to_dict(msg: EventMsg) -> dict:
         )
     except Exception as exc:
         event_type = type(msg).__name__
-        raise Exception(f"type {event_type} is not serializable. {str(exc)}")
+        fire_event(
+            Note(msg=f"type {event_type} is not serializable. {str(exc)}"), level=EventLevel.WARN
+        )
     # We don't want an empty NodeInfo in output
     if (
         "data" in msg_dict


Test patch:
diff --git a/tests/unit/test_base_context.py b/tests/unit/test_base_context.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_base_context.py
@@ -0,0 +1,22 @@
+import os
+
+from dbt.context.base import BaseContext
+from jinja2.runtime import Undefined
+
+
+class TestBaseContext:
+    def test_log_jinja_undefined(self):
+        # regression test for CT-2259
+        try:
+            os.environ["DBT_ENV_SECRET_LOG_TEST"] = "cats_are_cool"
+            BaseContext.log(msg=Undefined(), info=True)
+        except Exception as e:
+            assert False, f"Logging an jinja2.Undefined object raises an exception: {e}"
+
+    def test_log_with_dbt_env_secret(self):
+        # regression test for CT-1783
+        try:
+            os.environ["DBT_ENV_SECRET_LOG_TEST"] = "cats_are_cool"
+            BaseContext.log({"fact1": "I like cats"}, info=True)
+        except Exception as e:
+            assert False, f"Logging while a `DBT_ENV_SECRET` was set raised an exception: {e}"
diff --git a/tests/unit/test_events.py b/tests/unit/test_events.py
--- a/tests/unit/test_events.py
+++ b/tests/unit/test_events.py
@@ -1,3 +1,4 @@
+import pytest
 import re
 from typing import TypeVar
 
@@ -442,3 +443,21 @@ def test_date_serialization():
     ti_dict = ti.to_dict()
     assert ti_dict["started_at"].endswith("Z")
     assert ti_dict["completed_at"].endswith("Z")
+
+
+def test_bad_serialization():
+    """Tests that bad serialization enters the proper exception handling
+
+    When pytest is in use the exception handling of `BaseEvent` raises an
+    exception. When pytest isn't present, it fires a Note event. Thus to test
+    that bad serializations are properly handled, the best we can do is test
+    that the exception handling path is used.
+    """
+
+    with pytest.raises(Exception) as excinfo:
+        types.Note(param_event_doesnt_have="This should break")
+
+    assert (
+        str(excinfo.value)
+        == "[Note]: Unable to parse dict {'param_event_doesnt_have': 'This should break'}"
+    )
diff --git a/tests/unit/test_functions.py b/tests/unit/test_functions.py
--- a/tests/unit/test_functions.py
+++ b/tests/unit/test_functions.py
@@ -2,8 +2,8 @@
 import pytest
 
 import dbt.flags as flags
-from dbt.events.functions import warn_or_error
-from dbt.events.types import NoNodesForSelectionCriteria
+from dbt.events.functions import msg_to_dict, warn_or_error
+from dbt.events.types import InfoLevel, NoNodesForSelectionCriteria
 from dbt.exceptions import EventCompilationError
 
 
@@ -43,3 +43,19 @@ def test_warn_or_error_warn_error(warn_error, expect_compilation_exception):
             warn_or_error(NoNodesForSelectionCriteria())
     else:
         warn_or_error(NoNodesForSelectionCriteria())
+
+
+def test_msg_to_dict_handles_exceptions_gracefully():
+    class BadEvent(InfoLevel):
+        """A spoof Note event which breaks dictification"""
+
+        def __init__(self):
+            self.__class__.__name__ = "Note"
+
+    event = BadEvent()
+    try:
+        msg_to_dict(event)
+    except Exception as exc:
+        assert (
+            False
+        ), f"We expect `msg_to_dict` to gracefully handle exceptions, but it raised {exc}"