mcohen01/amazonica — pull request #62 by glangford (2014-08-07 15:49:50+00:00)
Title: Preliminary/not for merge: Kinesis and core.async comments requested

This is a preview of Kinesis providing data to applications via core.async channels (#55).

The goals are:

* Decouple checkpointing from record ingestion by using channels. Kinesis apps read records on a shard channel, process them asynchronously, and write sequence numbers to be checkpointed on a checkpoint channel.
* Allow transformation and bulk loading of data into storage services independent of the Kinesis arrival rate, without becoming entangled in the Java framework implied by IKinesisConnectorPipeline for transforming and buffering records
* Allow some automated testing of the machinery without running live on Kinesis
* Eventually allow Amazonica users a method for simulation and testing on a single machine with a mock Kinesis, for early prototyping and testing prior to deployment

All functional pieces are complete including channel delivery and checkpointing, but the user API is not defined. All tests in the mock Kinesis environment pass.

All feedback is appreciated!