mesos/spark — Issue #87: Ranges of Longs don't get split properly by SparkContext.parallelize
Author: mateiz
Created: 2011-11-02 17:49:32+00:00
State: open

We wanted Ranges to be split into smaller Range objects to avoid shipping the integers in them over the network, but Scala 2.9's Long ranges don't get matched as a Range by the ParallelCollection code and thus get converted to arrays, costing a lot of RAM.