cloudant/bucky — pull request #2 by mk-fg (2011-11-24 22:04:42+00:00)
Title: Aggregation methods dump

Getting a lot of metrics into graphite through bucky is easy, but in most cases all retentions but the most granular one *will be calculated incorrectly*.

Problem is that there are lot of counters in collectd, there is a counter type in statsd and metricsd and these can't be just "averaged": if you have "60s:1d,5m:7d" retentions, then counter increments like "1,2,1,2,1" will be averaged to "1" for 5m interval and this will be *totally wrong*, since 5-minute counter should show "7" - the sum of all counters for "60s" intervals, not the average.

Graphite 0.9.9 allows to specify aggregation methods via storage-aggregation.conf and manually via whisper-set-aggregation-method.py script.

Bucky *knows* exactly which aggregation type to use for each graphite name, but it has no way to pass this data to graphite directly afaik, hence this change.
--dump-aggregation-methods option make bucky dump lines like "host.postfix.connection.in.open sum", which can be then easily processed with stuff like awk to either run whisper-set-aggregation-method.py or build storage-aggregation.conf, or - which is probably the case - both.
Doing the same thing by hand for some tens / hundreds of metrics is just insane.

I considered writing this data to some dbm, so there will be strictly one record per name, but it introduces complexity which I just couldn't justify - it's much simpier to read plain file with common unix tools than write your own db-parser.
Although now I see that I was probably wrong there - since this data is a subject to change at runtime, restarting bucky with this flag just to collect new methods every once in a while is wrong, so I'll probably change target from stdout to dbm (with "name: first-seen-timestamp, aggregation method" format), so changes there can be tracked and applied to whisper files and/or graphite configuration.

Changes in the PR are quite intrusive and can potentially hurt performance, since checks are made on processing of every value, but I thought I'd still create this PR as a kind of RFC, feel free to just reject it if you have a better implementation in mind or consider it unnecessary or out of scope for the app, so I can either change direction here or won't bother you with this again.