LockerProject/Locker — Issue #877: locker core should be (optionally) instrumented
Author: othiym23
Created: 2012-02-27 20:52:45+00:00
State: open

To better understand the behavior of the locker over time, it would be extremely useful to have some fine-grained instrumentation of many of the events inside the locker:

1. synclet starts
1. synclet stops
1. synclet sends event
1. collection receives event
1. collection processes event
1. datastore loads object
1. datastore saves object
1. locker starts a process
1. locker stops a process

Events for collections and synclets should include the collection or synclet name so we can do some correlation of load based on events.

To track these statistics, we need a small library that knows how (and whether) to send events to an events-gathering daemon (statsd for now), and to modify the various pieces of locker core to use said library to emit events.

The emitter functions in the instrumentation library will be synchronous for simplicity's sake – sending UDP packets is both nonblocking and very fast. On the receiving end, we'll see how well statsd deals with the level of traffic, and may need to put it on its own host / partition the account servers so that we don't saturate its bandwidth. For now, graphic.test.singly.com will do until we have evidence that it's dropping packets.