heroku/logplex — pull request #89 by voidlock (2014-03-27 18:40:37+00:00)
Title: Add localized channel ring buffer monitoring

One of the main sources of load for Logplex is getting log messages into
the Redis ring buffer in order for some backlog to be visible when a
customer requests their log tail. For apps which produce a reasonable
amount of data this works fine, but for apps which produce an excessive
amount of log data, it is both unnecessary and time-consuming to store
these log messages in Redis, because there's no need for a backlog when
simply watching the tail for a few seconds will give you plenty of data.

This commit adds node localized monitoring of the messages being sent to
the Redis ring buffer. A new flag `no_redis_local` has been to control
the automated channel monitoring. Localized monitoring has been added in
such a way that it does not intefere with existing global channel flags.
In fact setting either `no_tail` or `no_redis` will take precedence and
remove any localized flags that existed on the channel.

There are two application environment variables that can be set to
control the behaviour of this monitoring.

- `channel_flood_msg_threshold` The messages per second required to
trigger flagging a channel as being flooded.
- `channel_flood_flap_threshold` The number of consecutive flooding
notices required before actually setting the `no_redis_local` flag on
the channel.

The latter environment variable helps control flapping situations and
has been inplemented in such a way that if it requires 3 consecutive
flood notices before taking action, then the system must also receive 3
consecutive notices of normal flow before disabling the `no_redis_local`
flag.

A couple tooling utilities were added to help with running the test
suites. These are only slightly modified versions of some tools Phil was
already using so credits for the bin scripts should go to him.

Additionally there are a few outstanding items that should be taken
care of or added. Those are:

- The `logplex_worker` and `logplex_message` modules were modified but
not directly tested.
- An additional stat record should be added to record the number of
times the msg threshold is crossed and the number of times the actual
`no_redis_local` flag is set on a channel. Some interesting stats could
be gathered to see how the automated system is performing and how it
could be tuned.