mesos/spark — pull request #459 by stephenh (2013-02-10 08:30:45+00:00)
Title: Change defaultPartitioner to use upstream split size.

Previously it used the SparkContext.defaultParallelism, which occassionally
ended up being a very bad guess. Looking at upstream RDDs seems to make
better use of the context.

Also sorted the upstream RDDs by partition size first, as if we have
a hugely-partitioned RDD and tiny-partitioned RDD, it is unlikely
we want the resulting RDD to be tiny-partitioned.

AFAIK these changes make sense--let me know if I'm missing something.

Currently I have an explicit split size littering my code, and I think this would basically get rid of it.