bigdatagenomics/adam — pull request #267 by tdanford (2014-06-17 20:41:10+00:00)
Title: Parquet and indexed Parquet RDD implementations, and indices.

[ Frank, Matt, this PR is woefully under-tested. I'm filing it now so you guys can start to look at the code in earnest -- one thing I'd like to know, early, is whether this is too big for a single PR and if I should break it up into two (or more) separate requests, or not. At any rate, it has several test suites commented out and several as-yet-unwritten. I'll be updating it in the coming (small-N number of) days with additional tests and header comments. Appreciate any feedback in the meantime.]

This commit includes several major pieces:

* Scala reimplementation of the Parquet footer parsing.

* Implementations of two Parquet-based RDDs

AvroParquetRDD.scala contains two RDD implementations around Parquet files. The first,
AvroParquetRDD, reads a single Parquet file (containing an Avro schema) and partitions the RDD
based on the row groups of the file.

The second, AvroIndexedParquetRDD, is an implementation that reads an index across a set of
parquet files, determines (according to the predicate contained in CombinedFilter) which row groups
in which files are relevant to the query, and produces an RDD whose partitions are in 1-1 correspondence
with the matching row groups from the index.

The parquet_reimpl/index package contains the code necessary for reading and writing indices.

* Adding two CLI commands, including an index generation command.

Adding two new CLI commands:
1. IndexFlatGenotype, which reads in a flat genotype Parquet file, and produces the index for that file.
2. the PrintParquet, which reads a Parquet file and prints out a limited number of entries form the file.

(at the moment, the latter only supports ADAMFlatGenotype)

* Implemented ID/Range indices on Parquet files.

Added an IDRangeIndexGenerator implementation.
Updating IndexFlatGenotype to generate the IDRangeIndex entries.
Updating S3AvroParquetRDD to use the IDRangeIndex
Fixed the tests for S3AvroIndexedParquetRDD