apache/spark — pull request #469 by andrewor14 (2014-04-21 22:41:00+00:00)
Title: [Spark-1538] Fix SparkUI incorrectly hiding persisted RDDs

*Bug*: After the following command `sc.parallelize(1 to 1000, 4).persist.map(_ + 1).count()` is run, the SparkUI does not show the persisted RDD.

*Cause*: The command creates two RDDs in one stage, a `ParallelCollectionRDD` and a `MappedRDD`. However, the existing `StageInfo` only keeps the `RDDInfo` of the last RDD associated with the stage (`MappedRDD`), and so all RDD information regarding the first RDD (`ParallelCollectionRDD`) is discarded. In this case, we persist the first RDD, but the `StorageTab` doesn't know about this RDD because it is not encoded in the `StageInfo`.

*Fix*: Record information of all RDDs in `StageInfo`, instead of just the last RDD (i.e. `stage.rdd`). Since stage boundaries are marked by shuffle dependencies, the solution is to traverse the last RDD's dependency tree, visiting only ancestor RDDs related through a sequence of narrow dependencies.

(This PR also moves `RDDInfo` to its own file.)