apache/incubator-spark — pull request #521 by bijaybisht (2014-01-28 18:24:35+00:00)
Title: Binning abstraction added via BinStream

For events which have duration spanning multiple batches, performing computation on a batch is inaccurate, the solution is to wait for the typical (or max) event duration and perform a delayed calculation with prorated data. This introduces delay in the output (for some of the shorter events). Binning wraps BinStream over DStream providing api to get incremental updates for bins per batch duration. It puts the event in the right BinStream and provides 3 different ways to get the BinStreams.
1. Incremental updates for last n Bins.
2. For a non associate computation latest update to the last n Bins.
3. Final BinStream in you want to generate just a final Bin.

A BinStreamer instance is constructed per event type which describes how the start time and the end time is accessed for an event. BinStreams are then generated using the DStreams for the event and using the methods of BinStreamer.