opscode/estatsd — pull request #7 by stevendanna (2013-11-25 21:17:19+00:00)
Title: Use bear to calculate statistics

From the original pull request:

I wanted to add extra percentiles beyond the 90th that estatsd already computes. In the course of doing that, I discovered that the mean that estatsd computes is actually the mean of everything below the 90th percentile, rather than everything. A great weeping and gnashing of teeth ensued.

To preserve sanity, I chucked the existing stats computation code in favor of bear. I also added the 95th and 99th percentiles. These are hard-coded right now, but since we're talking about ditching estatsd soon anyway, and since bear only computes a small hard-coded set of percentiles anyway (we also have access to 50th, 75th, and 99.9th if we want them), I didn't bother exposing these in app configuration.

Using this will affect our metrics collection in a few ways.

First, all our mean metrics will change slightly, since they'll now be computed for all values in a given time window.

Second, we'll be generating a lot more metrics; these will be computed for everything that flows through estatsd. We'll need to ensure that our Graphite infrastructure can handle it before this is deployed.

Finally, bear has a built-in threshold for metrics computation; if there are fewer than 5 values in a batch, all metrics will be reported as zeros. While I don't anticipate this will cause much trouble in prod due to the volume of requests it receives, it may impact preprod metric reporting, since we have much less traffic there.

This pull requests includes the above changes and the addition of the 50th percentile