wvanbergen/kafka — pull request #19 by wvanbergen (2014-10-20 23:47:36+00:00)
Title: Refactor consumer group code

This is basically a full rewrite.

Highlights:
- Support for multiple topics in a single consumer group.
- Instead of checking out one partition at the time, it will now have a goroutine for every partition is has claimed
- All the metadata for (re)balancing load is retrieved from Zookeeper (think currently running consumers, brokers, partitions). Sarama/Kafka is only accessed for when it actually wants to consume a topic. This limits the number of error scenarios significantly.
- Shutdown works properly, so it properly deregisters the consumer in Zookeeper, meaning it will trigger a rebalance in the other consumer immediately.

What is missing:
- Committing offsets to Zookeeper. Hopefully I can get this to work later today, although I am not quite sure yet about the API.
- Documentation - although the API hasn't really changed. :)

What needs improvement:
- Error handling in goroutines - right now I am panicking. Any suggestions on how I can let the parent goroutine know nicely that shit has hit the fan?
- Right now, all events from all topics are provided through a single channel. This keeps things simple, but it might be nice to have a channel per topic. What do you think?

@manygrams @eapache @cyprusad