apache/incubator-parquet-mr — pull request #12 by dcw-netflix (2014-07-08 18:06:29+00:00)
Title: Column index access support

This patch adds the ability to use column index based access to parquet files in pig, which allows for rename capability similar to other file formats. This is achieved by using the parametrized loader with an alternate schema.

Example:
# File Schema: {c1:int, c2:float, c3:chararray}
p = LOAD '/data/parquet/' USING parquet.pig.ParquetLoader('n1:int, n2:float, n3:chararray', 'true');

In this example, the names from the requested schema will be translated to the column positions from the file and will produce tuples based on the index position.

Two test cases are included that exercise index based access for both full file reads and column projected reads.

Note: This patch also disables the enforcer plugin on the pig project per discussion at the parquet meetup. The justification for this is that the enforcer is too strict for internal classes and results in dead code because duplicating methods is required to add parameters where there is only one usage of the constructor/method. The interface for the pig loader is imposed by LoadFunc and StoreFunc by the pig project and the implementations internals should not be used directly.