kevinweil/elephant-bird — pull request #144 by rangadi (2012-02-22 00:47:58+00:00)
Title: RCFile storage for Protobuf and Thrift

[ this is supersedes old pull #108 ]

The patch adds support for storing protobuf and thrift objects in RCFiles.

The top level fields in the message are stored as separate columns in RCFile.

Storing in RCFiles does better both with space required and more efficient retrieval when only partial set of columns are loaded.

The patch include InputFormat, OutputFormat, Pig loader and storage for both Protobufs and Thrift. Also includes tests.

Protobuf implementation preserves 'unknown' fields. It is simpler to preserve since Protobuf has native support unknown fields.

Thrift implementation does support this, yet. We need to deserialize objects ourselves if we need to preserve these (in stead of generated deserializer)