apache/spark — pull request #389 by CrazyJvm (2014-04-11 06:55:54+00:00)
Title: update spark.default.parallelism

actually, the value 8 is only valid in mesos fine-grained mode :

override def defaultParallelism() = sc.conf.getInt("spark.default.parallelism", 8)

while in coarse-grained model including mesos coares-grained, the value of the property depending on core numbers!

override def defaultParallelism(): Int = {
conf.getInt("spark.default.parallelism", math.max(totalCoreCount.get(), 2))
}