acrosa/kafka-rb — pull request #11 by ept (2012-08-09 01:16:36+00:00)
Title: Add consumer support for transparent gzip compression

The Scala Kafka producer implementation has support for gzip-compressing
individual messages or sets of consecutive messages, and the compression
is transparent to the Scala Kafka consumer implementation. It's very
convenient (just a matter of setting compression.codec=1 in the producer
config).

The compression codec of a message is indicated in the bottom two bits
of the 'attributes' byte of messages with magic == 1. This means it's
possible to add transparent compression support to consumers without any
configuration.

This commit adds compression support to the Ruby consumer. Because a
compressed message may actually contain more than one message inside it
(this makes compression more effective by grouping lots of small
messages into one big message), I had to move some of the parsing logic
from Kafka::Consumer to Kafka::Message.