wvanbergen/kafka — pull request #24 by darinkrauss (2014-12-29 19:25:06+00:00)
Title: Bug fixes

A number of fixes around initial state, error handling, and concurrency. Specifically, in order by commit:

7adee6d - If the ConsumerGroup is closed twice at about the same time (say from panics in more than one Consumer), then the stopper and events chans could be closed more than once, which causes subsequent unhandled panics. As the ConsumerGroup id is used as the indicator of whether the ConsumerGroup is closed or not, simply move clearing the id earlier in the close function. Still not perfect as there really should be a lock around reading/writing the id for complete concurrency safety.

626dd5c - Remove the offset check completely as offsets may not be sequential if the commit log has even been compacted. We could retain the offset check here to ensure if is at least increasing (not sequential, though), but the code currently just breaks from the partitionConsumerLoop which just stops the Consumer and leaves the rest of the ConsumerGroup running (not a good state).

7f9a01e - Allow client to know of and handle any errors from Kafka. The current error handling just stops the Consumer and leaves the rest of the ConsumerGroup running. This is an undesired state as the client has no idea that any error occurred and one topic partition is no longer been read from. So, remove the error check and allow client to handle error (presumably by closing the ConsumerGroup after a number of errors).

293ae90 - If the topic has never been read, the offset returned from Kafka will be zero and the commit will not occur. Set initial last offset to -1 so that it will commit even on the first event from the topic.