opscode/estatsd — pull request #6 by christophermaier (2013-06-20 16:00:54+00:00)
Title: Statistics Fixes with Bear... plus more percentiles!

I wanted to add extra percentiles beyond the 90th that estatsd already computes. In the course of doing that, I discovered that the mean that estatsd computes is actually the mean of everything _below the 90th percentile_, rather than everything. A great weeping and gnashing of teeth ensued.

To preserve sanity, I chucked the existing stats computation code in favor of [bear](https://github.com/boundary/bear). I also added the 95th and 99th percentiles. These are hard-coded right now, but since we're talking about ditching estatsd soon anyway, and since bear only computes a small hard-coded set of percentiles anyway (we also have access to 50th, 75th, and 99.9th if we want them), I didn't bother exposing these in app configuration.

Using this will affect our metrics collection in a few ways.

First, all our `mean` metrics will change slightly, since they'll now be computed for all values in a given time window.

Second, we'll be generating a lot more metrics; these will be computed for _everything_ that flows through estatsd. We'll need to ensure that our Graphite infrastructure can handle it before this is deployed.

Finally, bear has a built-in threshold for metrics computation; if there are fewer than 5 values in a batch, all metrics will be reported as zeros. While I don't anticipate this will cause much trouble in prod due to the volume of requests it receives, it may impact preprod metric reporting, since we have much less traffic there.

cc: @seth @jamesc