Repository: elastic/apm-agent-python

Issue:
HTTP 400: data validation error: Problem validating JSON document against schema
Sometimes when sending an event to the APM instance a HTTP 400 response is returned:

    HTTP 400: data validation error: Problem validating JSON document against schema: 
    I[#] S[#] doesn't validate with "error#" I[#/errors/0/log/message] 
    S[#/properties/errors/items/properties/log/properties/message/type] 
    expected string, but got array

I've also seen a similar, but subtly different message:

    HTTP 400: data validation error: Problem validating JSON document against schema: 
    I[#] S[#] doesn't validate with "error#" I[#/errors/0/log/param_message] 
    S[#/properties/errors/items/properties/log/properties/param_message/type] 
    expected string or null, but got array

This is then followed by a log message:

    Failed to submit message: '<no message value>'

I'm using the latest elastic-apm client available from PyPI (3.0.1) with Python 3.6.5 and the APM server is using version 6.4.1.

(This seems similar to #135, but not quite the same).


Discussion:
Hi @jaap3, thanks for opening the issue! Do you use `capture_message` manually somewhere, or have you set up our logging handlers?
Good question, I might have omitted some relevant information. 

I've configured the client in a Django project (version 2.1.2) and pretty much use "all the things" (no manual calls to the client however), in short I've:

* appended `'elasticapm.contrib.django'` to `INSTALLED_APPS`:
  `INSTALLED_APPS.append('elasticapm.contrib.django')`
* inserted the `TracingMiddleware` (at the top):
 `MIDDLEWARE = ['elasticapm.contrib.django.middleware.TracingMiddleware'] + MIDDLEWARE`
* Configured the client:
   `ELASTIC_APM = {'SERVICE_NAME': APM_SERVICE_NAME, 'SECRET_TOKEN': APM_TOKEN, 'SERVER_URL': 'https://apm.example.com'}`
* Setup logging (as a root handler):
  `LOGGING['handlers']['elasticapm'] = {'level': 'WARNING', 'class': 'elasticapm.contrib.django.handlers.LoggingHandler'}`
  `LOGGING['loggers']['']['handlers'].append('elasticapm')`

Note that the error happens rarely (in 2 days ~10 times). If I look at the APM data it all looks normal (as far as I can tell, this is my first time trying APM).

The reason I know this error is happening is because I'm also using Sentry to monitor this project (using the Raven client version 6.9.0). 

For Sentry I got basically the same setup as APM, meaning `'raven.contrib.django.raven_compat'` is added to `INSTALLED_APPS` and a logging handler is set up (`'raven.handlers.logging.SentryHandler'`). Not sure if any of this is relevant.

Any update on this? I've now seen this error ~130 times and it's a reason for me to hold back a rollout of APM on other projects. It's unclear to me if this error is fatal, caused by a misconfiguration etc. I would really appreciate some insight.

I think there may be a bug in `elasticapm.handlers.logging.LoggingHandler` and/or `elasticapm.Message.capture`.

In the Message.capture, we set the message field like:

```python
message = param_message["message"] % params if params else param_message["message"]
```

So if you logged a message with no parameters, then it would take the message as it is, without any interpolation. This code assumes that `params_message["message"]` is a string, which is not necessarily true. If the message event originates from LoggingHandler, then that value comes from the log record's msg attribute, which is not necessarily a string: https://docs.python.org/3/howto/logging.html#using-arbitrary-objects-as-messages

I'm not sure what the best fix is, but I think it might just a matter of changing LoggingHandler's call to client_capture like this:

```diff
diff --git a/elasticapm/handlers/logging.py b/elasticapm/handlers/logging.py
index b2dbd4d6..15b04436 100644
--- a/elasticapm/handlers/logging.py
+++ b/elasticapm/handlers/logging.py
@@ -130,7 +130,7 @@ class LoggingHandler(logging.Handler):
 
         return self.client.capture(
             "Message",
-            param_message={"message": record.msg, "params": record.args},
+            param_message={"message": str(record.msg), "params": record.args},
             stack=stack,
             custom=custom,
             exception=exception,
```
A quick grep through the project didn't find any logging calls where the message param isn't a string. But who knows, maybe a library we're using is doing this.

The proposed fix looks fairly harmless (`str`does behave differently on Python 2, so that might be an issue), I could try it for a few days and see if the error still pops up. Is there a branch I can install?

Resolving patch:
diff --git a/elasticapm/handlers/logbook.py b/elasticapm/handlers/logbook.py
--- a/elasticapm/handlers/logbook.py
+++ b/elasticapm/handlers/logbook.py
@@ -16,6 +16,7 @@
 import logbook
 
 from elasticapm.base import Client
+from elasticapm.utils import compat
 from elasticapm.utils.encoding import to_unicode
 
 LOOKBOOK_LEVELS = {
@@ -80,7 +81,7 @@ def _emit(self, record):
             exception = None
 
         return self.client.capture_message(
-            param_message={"message": record.msg, "params": record.args},
+            param_message={"message": compat.text_type(record.msg), "params": record.args},
             exception=exception,
             level=LOOKBOOK_LEVELS[record.level],
             logger_name=record.channel,
diff --git a/elasticapm/handlers/logging.py b/elasticapm/handlers/logging.py
--- a/elasticapm/handlers/logging.py
+++ b/elasticapm/handlers/logging.py
@@ -130,7 +130,7 @@ def _emit(self, record, **kwargs):
 
         return self.client.capture(
             "Message",
-            param_message={"message": record.msg, "params": record.args},
+            param_message={"message": compat.text_type(record.msg), "params": record.args},
             stack=stack,
             custom=custom,
             exception=exception,


Test patch:
diff --git a/tests/handlers/logbook/logbook_tests.py b/tests/handlers/logbook/logbook_tests.py
--- a/tests/handlers/logbook/logbook_tests.py
+++ b/tests/handlers/logbook/logbook_tests.py
@@ -143,3 +143,12 @@ def test_logbook_handler_dont_emit_elasticapm(capsys, elasticapm_client):
     handler.emit(LogRecord("elasticapm.errors", 1, "Oops"))
     out, err = capsys.readouterr()
     assert "Oops" in err
+
+
+def test_arbitrary_object(elasticapm_client, logbook_logger, logbook_handler):
+    with logbook_handler.applicationbound():
+        logbook_logger.info(["a", "list", "of", "strings"])
+    assert len(logbook_handler.client.events) == 1
+    event = logbook_handler.client.events.pop(0)["errors"][0]
+    assert "param_message" in event["log"]
+    assert event["log"]["param_message"] == "['a', 'list', 'of', 'strings']"
diff --git a/tests/handlers/logging/logging_tests.py b/tests/handlers/logging/logging_tests.py
--- a/tests/handlers/logging/logging_tests.py
+++ b/tests/handlers/logging/logging_tests.py
@@ -203,3 +203,11 @@ def test_logging_handler_dont_emit_elasticapm(capsys, elasticapm_client):
     handler.emit(LogRecord("elasticapm.errors", 1, "/ab/c/", 10, "Oops", [], None))
     out, err = capsys.readouterr()
     assert "Oops" in err
+
+
+def test_arbitrary_object(logger):
+    logger.error(["a", "list", "of", "strings"])
+    assert len(logger.client.events) == 1
+    event = logger.client.events.pop(0)["errors"][0]
+    assert "param_message" in event["log"]
+    assert event["log"]["param_message"] == "['a', 'list', 'of', 'strings']"