bigdatagenomics/adam — pull request #488 by fnothaft (2014-11-18 19:09:21+00:00)
Title: Added tests for multi Parquet rdds in S3.

@tdanford @carlyeks A little something something for you both to look at. I cleaned up the S3 tests a bit. You'll need to set `AWS_SECRET_ACCESS_KEY` and `AWS_ACCESS_KEY_ID` to run. The multi-parquet S3 tests fail due to:

```
- Retrieve records from a multi-partition Parquet directory through S3 *** FAILED ***
java.lang.IllegalArgumentException: requirement failed: File does not appear to be a Parquet file
at scala.Predef$.require(Predef.scala:233)
at org.bdgenomics.adam.rdd.ParquetCommon$.readFileMetadata(ParquetCommon.scala:70)
at org.bdgenomics.adam.parquet_reimpl.AvroMultiParquetRDD$$anonfun$3.apply(AvroParquetRDD.scala:128)
at org.bdgenomics.adam.parquet_reimpl.AvroMultiParquetRDD$$anonfun$3.apply(AvroParquetRDD.scala:126)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
...
- Retrieve records from a multi-partition Parquet directory through S3 and apply both a projection and a predicate *** FAILED ***
java.lang.IllegalArgumentException: requirement failed: File does not appear to be a Parquet file
at scala.Predef$.require(Predef.scala:233)
at org.bdgenomics.adam.rdd.ParquetCommon$.readFileMetadata(ParquetCommon.scala:70)
at org.bdgenomics.adam.parquet_reimpl.AvroMultiParquetRDD$$anonfun$3.apply(AvroParquetRDD.scala:128)
at org.bdgenomics.adam.parquet_reimpl.AvroMultiParquetRDD$$anonfun$3.apply(AvroParquetRDD.scala:126)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.TraversableLike$$anonfun$flatMap$1.apply(TraversableLike.scala:251)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:45)
at scala.collection.TraversableLike$class.flatMap(TraversableLike.scala:251)
...
```

So, something's not _quite_ right in the implementation.