quantifind/kafka-utils — pull request #1 by squito (2013-11-23 18:56:53+00:00)
Title: Safe kafka consumers that can handle batches

As discussed in the kafka mailing list http://mail-archives.apache.org/mod_mbox/kafka-users/ (threads "are kafka consumer apps guaranteed to see msgs at least once?" & "processing a batch of messages in a "transaction""), its very hard to write a kafka app that truly has "at least once" semantics. There is a small problem with processing single messages, and its extremely difficult to get batch processing correct.

This provides an API which solves those problems. `KafkaConsumer` can be used for apps which process msgs one at a time, and guarantees at least once *by the app*. `KafkaBatchConsumer` can be used for apps which batch a number of messages together, and then process them, to guarantee that every message will get fully processed in at least one batch.

Furthermore, this does NOT require all workers to be stopped in order to commit progress (for either basic consumer or batch consumer), which should increase output. This is possible by using `consumer.commitOffsets(myOffsets)` (which is not officially part of kafka yet).