Parquet/parquet-mr — pull request #401 by tongjiechen (2014-05-30 20:27:13+00:00)
Title: push projection only needs to push the first level of the schema and not nested fields.

ParquetLoader fails to push projection for map column key field (push projection of whole map column is fine). During push projection, ParquetLoader recursively constructs subfields' schema; for map column key, it does not have a FieldSchema for specified key, hence result in NullPointException in FieldSchema's clone method.

The error message indicates to try "-t ColumnMapKeyPrune", although the script will succeed using that; however, this will turn off mapKeyPrune and columnPrune altogether, not a desirable thing as well.

ParquetLoader should not recursively push down subfields, it should only push down top level fields.

---------------------------------------- simple pig script -------------------------------------------
register '..../parquet-pig-bundle-1.5.0.jar';
a = load 'hdfs://parquet_file_directory' using parquet.pig.ParquetLoader();
b = foreach a generate
(chararray)other_properties#'chukwa_uid' as uid, ---- other_properties is a map column
hostname as hostname,
devtype_id as devtype_id;
c = limit b 10;
dump c;

------------------------------------------ ERROR message ------------------------------------------
Caused by: org.apache.pig.impl.logicalLayer.FrontendException: ERROR 2000: Error processing rule ColumnMapKeyPrune. Try -t ColumnMapKeyPrune
at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:122)
at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:281)
at org.apache.pig.PigServer.compilePp(PigServer.java:1351)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1263)
at org.apache.pig.PigServer.storeEx(PigServer.java:942)
... 14 more
Caused by: java.lang.NullPointerException
at parquet.pig.ParquetLoader.getSchemaFromRequiredFieldList(ParquetLoader.java:296)
at parquet.pig.ParquetLoader.getSchemaFromRequiredFieldList(ParquetLoader.java:303)
at parquet.pig.ParquetLoader.pushProjection(ParquetLoader.java:280)
at org.apache.pig.newplan.logical.rules.ColumnPruneVisitor.visit(ColumnPruneVisitor.java:155)
at org.apache.pig.newplan.logical.relational.LOLoad.accept(LOLoad.java:229)
at org.apache.pig.newplan.ReverseDependencyOrderWalker.walk(ReverseDependencyOrderWalker.java:70)
at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:52)
at org.apache.pig.newplan.logical.rules.ColumnMapKeyPrune$ColumnMapKeyPruneTransformer.transform(ColumnMapKeyPrune.java:141)
at org.apache.pig.newplan.optimizer.PlanOptimizer.optimize(PlanOptimizer.java:110)
... 18 more