kevinweil/elephant-bird — pull request #92 by rangadi (2011-11-02 03:33:00+00:00)
Title: Projection support for protobuf loaders

[ Almost complete patch for review, but not ready for merge yet ]

so far protobuf and thrift loaders used "lazy tuples" to handle projections, expecting Pig to access only the fields it needs from the tuples. This seems to be have changed in Pig 0.8 and above.

For top level tuple, Pig always reads all the fields (invokes tuple.getAll()). we need to support projection pushdown.

It seems to be reading only the required fields for the inner tuples.

The current patch add projection support for Protobufs. will add Thrift as well.

E.g. runtime benift:
For a protobuf with 78 fields, the patch save about 40% time in local mode (input is a single with about 3M protobufs, 102 sec vs 60 sec):

u = load './users_snapshot' using LzoProtobufBlockPigLoader('com.twitter.data.proto.Tables.UserExtended');
uid = foreach u generate id;
c = foreach ( group uid all ) generate COUNT(uid); dump c;