BrightcoveOS/Diamond — pull request #668 by prune998 (2014-07-08 21:31:14+00:00)
Title: added a timer to delay influxdb writes

Added a timer to wait for an incremental time in case influxdb is not responding.
This will stop the 100% CPU usage of diamond when influxdb is down.

Instead of calling the _send() each time the batch size is reached (as it will occure for EVERY new metric gathered in case of failure), the function is called only when :
1) the batch size is reached
2) the last _send() request was done X seconds before

X is 2s in normal operation, and is rooted by 2 each time, so first write is after 2s, then 4s, then 8s, 16s and 32s, which is the max.
During this time, the metrics will stay in memory. if more than self.metric_max_cache metrics are stored, stop recording them.
I did it this way as the array of metrics is multi-level, indexed by its path and we can't easily know how many values are in each path and which one to remove to trim the cache...
This will work AS LONG AS the batch size is less than the cache size, which should always be the case anyway...