etsy/statsd — pull request #56 by wickedchicken (2012-01-25 01:39:32+00:00)
Title: Batched send, reliability fixes, multiple graphing backends, package.json

This is a rather large pull request, one we've split into multiple commits to help ease facilitation into the mainline Etsy codebase. The test infrastructure we've previously developed was intended to ensure these sets of changes don't include regressions.

Note that these patches are rebased and distilled code from the [librato-statsd](https://github.com/librato/statsd/) fork. Some of the changes were authored by @adelcambre and @till and integrated into our history. The process of rebasing for creation of Etsy-friendly patchsets has erased that authorship information, and it's not certain if it's possible to edit the history to say so.

The first two commits are cleanup patches from @till which clean up the main directory and remove spurious warnings respectively. The 3rd commit performs a restructure of statsd's sending logic. Specifically it enables syslog support if 'node-syslog' is installed and working, splits statsd's Graphite code into an 'output pipeline' which allows for batched sends, and ensures that new pipeline can submit to several graphing backends at once. Finally, the last patch inserts Librato metrics as a new graphing backend module.

The first two patches should be self-explanatory, but the 3rd should warrant some discussions of the engineering decisions behind its design. The current statsd code has a specific target in mind: a medium number of metrics being delivered to Graphite. Our graphing infrastructure requires handling tens of thousands of metrics simultaneously. As our API is JSON based, we needed a way to split large submissions into smaller, easier-to-handle chunks automatically. This also provides a layer of fault-tolerance and modularity for both Graphite and our API; smaller chunks can be processed and tracked independently of each other both on the network and in the daemon.

With respect to modularity, stats now go through a pipeline: initial data collection, batching, and postprocessing. Postprocessing at the innermost level allows graphing backend-specific code to be localized while the outer maps are common, keeping the codebase clean.

Finally, the last patch exploits the modularity of the 3rd patch to add [Librato Metrics](https://metrics.librato.com/) as a configurable graphing backend. Note that each graphing backend works independently of each other and can work simultaneously. We have included a test which verifies that this is in fact true. It is our hope that other graphing backends can be added and tested in a similar manner.