Repository: elastic/apm-agent-python

Issue:
Transport.queue spams log when queue is full
In a busy application, the agent-internal event queue may fill up frequently, which will lead to events being dropped. We currently log a warning every time this occurs, which may lead to spamming the logs.

Ideally, the agent would keep a record of the number of events dropped, and periodically report this as a metric to the server, and logging would no longer be needed. In the meantime, we should at least throttle this log message to prevent filling up the logs.


Discussion:
Related to this, if the queue fills up is there a way to drain/purge it?
Turns out this is solved by adding more threads to gunicorn.
@axw Implementing throttling would probably need quite a bit of work, how about changing the log level from `WARNING` to `INFO` or `DEBUG` for now?

But to be honest, I'm not even sure that's a good idea. In normal operation, this should almost never happen. I think it's OK to be somewhat noisy if things are failing so badly that the queue is spilling over constantly.

Regarding metrics: yes, it makes sense to track this as a metric, but it's not a replacement for logging the error, as the metrics themselves would use the same queue and also be lost in a situation like this.

@fxdgear I'm still a bit confused why adding threads helped in your situation. Threads in the same process use the same queue, so in my mind it should have made the problem worse, not better. Sometimes I hate computers... 
> @axw Implementing throttling would probably need quite a bit of work, how about changing the log level from WARNING to INFO or DEBUG for now?

I think making it DEBUG would be OK for now.

> But to be honest, I'm not even sure that's a good idea. In normal operation, this should almost never happen. I think it's OK to be somewhat noisy if things are failing so badly that the queue is spilling over constantly.

The definition of "normal operation" will depend on the application. There will be applications where the load is always more than the agent can handle, so in that case it's normal for events to be dropped.

You could argue that the user should just decrease the sampling rate, but spamming logs can be pretty detrimental to performance, mostly due to I/O. For a busy application, that might be problematic.

> Regarding metrics: yes, it makes sense to track this as a metric, but it's not a replacement for logging the error, as the metrics themselves would use the same queue and also be lost in a situation like this.

I would consider that a bug. One which previously affected the Go agent: https://github.com/elastic/apm-agent-go/pull/320 :)
> Sometimes I hate computers...

Truer words have never been spoken!
TBH I’m not sure why adding threads helped. 

I also increased the mem limit on my container from 200 to 500mb
I also switched from sync workers to gevent async workers. 

But, it’s been over 24hr and I can’t remember if those changes were all at once or one at a time. 

I do remember that once I hit my mem limit on the container that’s when I started getting spammed with those queue full messages. So I chased a memory leak theory in gunicorn for a bit as well. 

Resolving patch:
diff --git a/elasticapm/transport/base.py b/elasticapm/transport/base.py
--- a/elasticapm/transport/base.py
+++ b/elasticapm/transport/base.py
@@ -109,7 +109,7 @@ def queue(self, event_type, data, flush=False):
             self._event_queue.put((event_type, data, flush), block=False, **kwargs)
 
         except compat.queue.Full:
-            logger.warning("Event of type %s dropped due to full event queue", event_type)
+            logger.debug("Event of type %s dropped due to full event queue", event_type)
 
     def _process_queue(self):
         buffer = self._init_buffer()


Test patch: