hammerlab/guacamole — pull request #105 by timodonnell (2014-06-13 14:42:00+00:00)
Title: Performance improvements, code refactoring, and upgrade to Spark 1.0.

Initial goal here was just to be able to run on chr20 without specifying -loci argument and still have things work (i.e. don't have two tasks stuck looking at other contigs that don't have reads). That ended up expanding into a number performance improvements, refactorings, dependency upgrades, and code fixes.

Major changes:

* Much more efficient handling of the reads assigned to each task. We used to filter an unsorted iterator of reads into iterators over each contig, and then sort. This resulted in many traversals of the reads. Now we sort first and then walk through the reads only once.
* Upgrade to Spark 1.0, Java 7, and ADAM 0.11.1-SNAPSHOT. We can move off ADAM snapshot to a real release once ADAM has a release that uses Spark 1.0

Minor changes:

* Fix issues involved in Spark upgrade: move hadoop dependency to end of pom, call setReferences(true) in Kryo initialization to avoid infinite recursion.
* Introduce ReadSet for keeping track of read metadata (e.g. sequence dictionaries) with the RDD[Read] instance.
* paritionLociByApproximateReadDepth now has special handling of contigs with no reads mapped for better efficiency. Unit tests.
* pileupFlatMap and variants now have a 'skipEmpty' option, which, if set, will result in skipping loci where no reads align. This can be an important optimization when running over e.g. many contigs that have no reads mapped. Add 'nextStartLocus' method to SlidingReadWindow to enable an efficient implementation of this.
* SlidingReadWindow: more efficient setCurrentLocus implementation.
* Minor convenience functions added to LociSet.
* Add Read.InputFilters to have a single place to put filters for loading reads. Modified read loading functions to use it.
* Add hasMdTag as a read filter.
* Added -no-sequence-dictionary argument to specify that contigs and lengths should come directly from reads, not from the sequence dictionary.
* Added a multi-contig test set of BAMs (synth1 subsets).
* Added a simple does-it-run? unit test for SomaticThreshold caller.
* Dropped some for loops (converted into foreach or while loops) for supposed performance improvement (not measured).
* Specifying non-existent contigs / loci with the -loci argument results in an error. Closes #6.
* Added runOnly hack to SparkTest to give us a way to run a single unit test.

TODO:
* Modified Alex's caller so it compiles with Spark 1.0, but ended up commenting out the unit tests for it since they hit a runtime error. Alex, I'm hoping you can fix this quicker than me since you're familiar with your code.
* Disabled code in Arun's StrandBiasFilter. The ADAM API used here seems to have changed, and I think this is best fixed by Arun.