mesos/spark — pull request #891 by xiajunluan (2013-09-04 02:00:41+00:00)
Title: [SPARK-864]DAGScheduler Exception if we delete Worker and StandaloneExecutorBackend then add Worker

In fair scheduling mode, pool information of each stage(taskset) is stored in job(ActiveJob) that contain this stage. but in spark, each stage could be used more than one job, so we must not delete ActiveJob in idToActiveJob structure when job is ended, otherwise, once we need re-compute this stage , it will happens following exception

Exception in thread "DAGScheduler" java.util.NoSuchElementException: key not found: 2
at scala.collection.MapLike$class.default(MapLike.scala:225)
at scala.collection.mutable.HashMap.default(HashMap.scala:45)
at scala.collection.MapLike$class.apply(MapLike.scala:135)
at scala.collection.mutable.HashMap.apply(HashMap.scala:45)
at spark.scheduler.DAGScheduler.spark$scheduler$DAGScheduler$$submitMissingTasks(DAGScheduler.scala:515)
at spark.scheduler.DAGScheduler.spark$scheduler$DAGScheduler$$submitStage(DAGScheduler.scala:481)
at spark.scheduler.DAGScheduler$$anonfun$resubmitFailedStages$3.apply(DAGScheduler.scala:383)
at spark.scheduler.DAGScheduler$$anonfun$resubmitFailedStages$3.apply(DAGScheduler.scala:382)
at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:34)
at scala.collection.mutable.ArrayOps.foreach(ArrayOps.scala:38)
at spark.scheduler.DAGScheduler.resubmitFailedStages(DAGScheduler.scala:382)
at spark.scheduler.DAGScheduler.spark$scheduler$DAGScheduler$$run(DAGScheduler.scala:433)
at spark.scheduler.DAGScheduler$$anon$1.run(DAGScheduler.scala:135)