bigdatagenomics/adam — pull request #508 by massie (2014-11-24 18:49:58+00:00)
Title: Upgrade from Parquet 1.4.3 to 1.6.0rc4

This upgrade adds dictionary support which dramatically reduces the
memory footprint.

For example, a simple program to load and count mouse mitochondrial DNA..

val reads = ac.loadAlignments("/workspace/data/mouse_chrM.adam").cache()
println(reads.count())

In 1.6.0rc4, this rdd was able to be cached in memory, e.g.

14/11/24 10:37:16 INFO BlockManagerInfo: Added rdd_1_0 in memory on zenfractal:60004 (size: 193.0 MB, free: 72.5 MB)

With 1.4.3, it was not, e.g.

14/11/24 10:33:07 WARN CacheManager: Not enough space to cache partition rdd_1_0 in memory! Free memory is 278185689 bytes.