fourk/blueflood — pull request #1 by fourk (2013-11-27 02:14:06+00:00)
Title: Batch rollup writes part 2

This is the second part of batching rollup writes (part 1: https://github.com/rackerlabs/blueflood/pull/155)

It adds tests and makes everything not covered in part 1 (read: tests) use RollupWriteContexts to write rollups.

This also resolves https://github.com/rackerlabs/blueflood/issues/115 by rescheduling a slot that fails for any reason.

I'm a little concerned with the solution as implemented because this could result in running into two (based on locator fetch concurrency) terminally bad slots (i.e slots that for some reason fail every time we attempt to roll them up for some reason) would completely halt rollup activity on a rollup node.

I've thought of some ways to mitigate that but would like some feedback:
* keep a meter of how often we send slots back due to failure and stop sending any back after a certain threshold
* same idea, but slotKey-specific -- keep a guava cache, with a low expireAfterWrite threshold and do the above scoped to individual slotKeys
* instead of re-inserting at position 0 in orderedScheduledSlots, re-insert somewhere further down the list, like at the end of it