[SPARK-695] In DAGScheduler's getPreferredLocs, track set of visited partitions.

What do you think of a method like this for avoiding exponential path exploration in DAGScheduler's getPreferredLocs, per SPARK-695?

Some minor cleanups are also included.  The comment I removed in Dependency.scala seemed incorrect to me, per my commit comment:

    Remove apparently incorrect comment describing NarrowDependency.
  
    Each of a CartesianRDD's dependencies, for example, is a NarrowDependency, but child partitions of
    these dependencies may depend on shared parent partitions.  For example, the cartesian product of
    RDDs a and b containing partitions [ a_0 ] and [ b_0, b_1 ] respectively will be partitioned as
    [ ( a_0, b_0 ), ( a_0, b_1 ) ].  Each child partition depends on parent partition a_0.

I'm new to spark, scala (and even java) so a careful review may be in order.

Thanks,
Aaron
Maybe this is an opportunity for me to get some scala education.  I can see that, in this stageDependsOn function, the visitedRdds val is used to prevent re-visiting an RDD, but I only see two uses of _visitedStages_.  visitedStages is constructed, and then stages are added to it.  I’m not seeing how visitedStages is being used.  Is there some sort of callback when a stage is added to this HashSet?  I’m an admitted scala novice, so please let me know if there’s some magic going on or I’m being dense here :)

Because I didn’t see how visitedStages was being used, I removed this variable in my “Remove unused variable.” commit.  I’ll go ahead and withdraw that commit for now.  If the variable is in fact in use, I’d love to learn understand that better.