Parquet/parquet-mr — pull request #132 by FelixNeutatz (2013-08-13 00:19:57+00:00)
Title: Extending Parquet to be able to use projection for multiple input files

I am big fan of Parquet and at the moment I am experimenting to join Parquet files. But I recognized that it is not possible yet to read two Parquet files with different schemas and apply a projection for each schema. Therefore I propose a minor change of the class ReadSupport (https://github.com/Parquet/parquet-mr/blob/master/parquet-hadoop/src/main/java/parquet/hadoop/api/ReadSupport.java) in the function getSchemaForRead(). At the moment it only reads one message from the configuration variable ReadSupport.PARQUET_READ_SCHEMA. But I would suggest to introduce a delimiter like i.e. "|" to be able to store there more messages. In ReadSupport we could check then whether there is one message which fits to our current parquet input file.

The way I implemented getSchemaForRead() in ReadSupport is attached.

To show that this is really working I wrote a test which is a mixture of https://github.com/Parquet/parquet-mr/blob/master/parquet-thrift/src/test/java/parquet/hadoop/thrift/TestParquetToThriftReadProjection.java and https://github.com/Parquet/parquet-mr/blob/master/parquet-thrift/src/test/java/parquet/hadoop/thrift/TestInputOutputFormat.java. You can find the sourcecode of this test also attached.

If you have another idea how to solve this issue, I am glad to help with the implementation.

You did a great job with Parquet. It is an awesome project :)

Kind regards,

Felix Neutatz