harlow/go-etl — pull request #1 by harlow (2014-07-25 07:21:11+00:00)
Title: Message buffer

@bernerdschaefer here is a first PR for the `MsgBuffer`.

This struct will act as a temporary data store as we're looping over records from Kinesis. It also make sure we don't store any duplicate records by keeping a tally of the sequence numbers.

When the buffer size reaches its `maxRecordsInBuffer` the `ShouldFlush()` method returns `true` and the app will know its time to do persist the data.

* Create a structure for collecting unique messages.
* Keep track of first checkpoint, and last checkpoint through sequence
numbers from Kinesis.
* Add specs for core functionality.