graphite-project/carbon — pull request #92 by pcn (2013-04-01 22:58:20+00:00)
Title: Performance/stability improvements in the relay

This issue started with https://github.com/graphite-project/carbon/issues/53. This resolves that issue by pushing all datapoint sends via a single queue per destination, and yielding the event loop and sends the backlog in discreet chunks. If one destination's queue fills, stats will still be sent to other destinations.

This also makes some of the parameters relating to clearing the queue adjustable, so instead of 80% of the queue needing to be clear for more datapoints to be accepted, config.QUEUE_LOW_WATERMARK_PCT can now be set, so if a large queue is in use, stats can be accepted when there is enough space, not only when there is waaaay too much space.

In addition, everything now uses the sendQueued method to send datapoints after a small delay in order to accumulate and send multiple datapoints in one batch. To do this, when a datapoint is seen, it will use a twisted.internet.reactor.callLater invocation to call sendQueued after a small delay. This delay is configurable, but the default delay (in config.TIME_TO_DEFER_SENDING) of 0.0001 seems to work very quickly. The sendQueued method also calls itself with callLater, but it doesn't have a tunable time.