apache/spark — pull request #1362 by staple (2014-07-10 20:37:14+00:00)
Title: [SPARK-695] In DAGScheduler's getPreferredLocs, track set of visited partitions.

What do you think of a method like this for avoiding exponential path exploration in DAGScheduler's getPreferredLocs, per SPARK-695?

Some minor cleanups are also included. The comment I removed in Dependency.scala seemed incorrect to me, per my commit comment:

Remove apparently incorrect comment describing NarrowDependency.

Each of a CartesianRDD's dependencies, for example, is a NarrowDependency, but child partitions of
these dependencies may depend on shared parent partitions. For example, the cartesian product of
RDDs a and b containing partitions [ a_0 ] and [ b_0, b_1 ] respectively will be partitioned as
[ ( a_0, b_0 ), ( a_0, b_1 ) ]. Each child partition depends on parent partition a_0.

I'm new to spark, scala (and even java) so a careful review may be in order.

Thanks,
Aaron