Repository: PrefectHQ/prefect

Issue:
Logging gives up too easily for non-JSON types
Very possible that I'm missing a good reason why it has to be this way, but now with #1961 it's more apparent how many logging messages that seem innocuous cause issues for the current Cloud logging setup.

For example, the following works fine with stdout:
```
logging.info("n = %d", np.int64(2))
INFO:root:n = 2
```
but shows up in Prefect Cloud as 
```
CRITICAL  Failed to write log with error: Object of type int64 is not JSON serializable
```
Same goes for anything of type `type` which is a pretty common thing to pass via `%s` in a debugging context.

Any reason we can't let `logging` do whatever it normally would to coerce these values into a printable string and then ship that?

cc @joshmeek 


Discussion:
Hmm I would need to take a deeper look into the logging configuration again. I see what you're saying, we should convert these appropriately to allow for the most amount of granular logs to appear as possible. We do something _similar_ in terms of the current result handlers where we strip off clients in order to make them serializable. Will take a look into this 👍 and any other eyes on this would be appreciated 👀 

Resolving patch:
diff --git a/src/prefect/utilities/logging.py b/src/prefect/utilities/logging.py
--- a/src/prefect/utilities/logging.py
+++ b/src/prefect/utilities/logging.py
@@ -130,6 +130,9 @@ def emit(self, record) -> None:  # type: ignore
             assert isinstance(self.client, Client)  # mypy assert
 
             record_dict = record.__dict__.copy()
+            ## remove potentially non-json serializable formatting args
+            record_dict.pop("args", None)
+
             log = dict()
             log["flow_run_id"] = prefect.context.get("flow_run_id", None)
             log["task_run_id"] = prefect.context.get("task_run_id", None)


Test patch:
diff --git a/tests/utilities/test_logging.py b/tests/utilities/test_logging.py
--- a/tests/utilities/test_logging.py
+++ b/tests/utilities/test_logging.py
@@ -135,6 +135,32 @@ def test_remote_handler_captures_tracebacks(caplog, monkeypatch):
         logger.handlers = []
 
 
+def test_cloud_handler_formats_messages_and_removes_args(caplog, monkeypatch):
+    monkeypatch.setattr("prefect.client.Client", MagicMock)
+    client = MagicMock()
+    try:
+        with utilities.configuration.set_temporary_config(
+            {"logging.log_to_cloud": True}
+        ):
+            logger = utilities.logging.configure_logging(testing=True)
+            assert hasattr(logger.handlers[-1], "client")
+            logger.handlers[-1].client = client
+
+            child_logger = logger.getChild("sub-test")
+            child_logger.info("Here's a number: %d", 42)
+
+            time.sleep(0.75)
+
+            cloud_logs = client.write_run_logs.call_args[0][0]
+            assert len(cloud_logs) == 1
+            assert cloud_logs[0]["message"] == "Here's a number: 42"
+            assert "args" not in cloud_logs[0]["info"]
+    finally:
+        # reset root_logger
+        logger = utilities.logging.configure_logging(testing=True)
+        logger.handlers = []
+
+
 def test_remote_handler_ships_json_payloads(caplog, monkeypatch):
     monkeypatch.setattr("prefect.client.Client", MagicMock)
     client = MagicMock()