apache/spark — pull request #1566 by kayousterhout (2014-07-24 05:01:22+00:00)
Title: [SPARK-1726] [SPARK-2567] Eliminate zombie stages in UI.

Due to problems with when we update runningStages (in DAGScheduler.scala)
and how we decide to send a SparkListenerStageCompleted message to
SparkListeners, sometimes stages can be shown as "running" in the UI forever
(even after they have failed). This issue can manifest when stages are
resubmitted with 0 tasks, or when the DAGScheduler catches non-serializable
tasks. The problem also resulted in a (small) memory leak in the DAGScheduler,
where stages can stay in runningStages forever. This commit fixes
that problem and adds a unit test.

Thanks @tsudukim for helping to look into this issue!

cc @markhamstra @rxin