kevinweil/elephant-bird — pull request #108 by rangadi (2011-12-01 17:47:43+00:00)
Title: RCFile storage file for Protobufs

[ this is NOT ready for merge yet. but usable and works well. I will send a different pull request for merge ]

This patch add support for RCFile support for storing and loading protobufs. The existing loaders deserialize the entire protobuf irrespective of which fields are required. Columnar storage like RCFile can help with avoiding loading unnecessary columns as well as better compression.

Mainly includes following four classes :

* RCFileProtobufInputFormat
* RCFileProtobufOutputFormat
* RCFileProtobufPigLoader
* RCFileProtobufPigStorage

Unknown fields are preserved while storing and are read while loading (if required).

results with a large protobuf with 70+ simple fields (compared with LzoProtobufBlockPigStorage, lzo compression level 3, 3.7M records, Pig local mode) :

- space savings : 30% with gzip, 15% with lzo
- load speed up : takes less than half the time: 125% speed up (both gzip and lzo)

current TODO:

- Add tests
- Thrift support