wvanbergen/kafkacluster — pull request #1 by wvanbergen (2014-05-20 22:06:22+00:00)
Title: Initial stab at Zookeeper-managed topic consumer.

This needs some work, but is ready for a first look. This implements the zookeeper logic to have a nice consumer group:

- It discovers the Kafka cluster as it is defined in Zookeeper.
- It load balances based on the number of running consumers
- It commits the offset for every partition to Zookeeper every x seconds (current x is hardcoded to 1 second)
- It automatically resumes from the latest committed offset.

This is heavily based on https://github.com/bsm/sarama/tree/master/cluster, but large parts are rewritten because it wasn't working for me. The Zookeeper-backed consumer group management should be compatible with the JVM library, as documented here: https://cwiki.apache.org/confluence/display/KAFKA/Offset+Management

### TODO

- [ ] It currently only supports one concurrent goroutine. For concurrency, simply run the application multiple times. However, it is totally possible to have multiple partition consumers as separate goroutines.
- [ ] Error handling needs a close look.
- [ ] Make some hardcoded values configurable (e.g. the commit interval of 1 second)
- [ ] Remove a bunch of logging statements.
- [ ] KafkaConsumer and ConsumerGroup should probably be merged into one type.
- [ ] Some stuff in PartitionConsumer should move to Sarama to clean up the implementation.

@jnormore @berkcaputcu @manygrams /cc @eapache @bsm