googlegenomics/spark-examples — pull request #49 by elmer-garduno (2014-11-01 14:54:06+00:00)
Title: Added a fixed bases per reference splitter.

This splitter provides a way to partition a reference on a constant number of bases. The main advantage of this method over splitting each reference on a constant number of parts, is that it makes it easy to reason about the size of the partitions, as queries across all references are now of the same size.

The other advantage is that is that it the default value (100K bases) provides a good balance for many use cases, smaller requests (< 100K bases) will usually be retrieved fast enough with 1 or 2 workers, and larger refequests like chr1 will be partitioned in 2.5K parts which provides enough material for all the workers to be busy.

For larger requests the drawback comes in the shuffle step as it will be merging a much larger number of partitions, but it can be fixed by tuning the parameter `--bases-per-partition` to a larger value, for example 1 million bases for a genome wide request.