apache/incubator-spark — pull request #6 by wannabeast (2013-09-18 01:56:29+00:00)
Title: ParallelCollectionRDD memory optimizations

Freeing the original `Seq` once it's been partitioned, when it's an `Array`, saves 8B per data element. For `SparkLR` in a 7GB heap, I'm able to generate about 10M records, so this saves about 80MB.

To be conservative, I've cached the partitions in `ParallelCollectionRDD`. But this is probably not needed, since they seem to cached in the `RDD` parent already.