apache/spark — pull request #186 by CodingCat (2014-03-20 05:03:56+00:00)
Title: SPARK-1235: fail all jobs when DAGScheduler crashes for some reason

https://spark-project.atlassian.net/browse/SPARK-1235

In the current implementation, the running job will hang if the DAGScheduler crashes for some reason (eventProcessActor throws exception in receive() )

The reason is that the actor will automatically restart when the exception is thrown during the running but is not captured properly (Akka behaviour), and the JobWaiters are still waiting there for the completion of the tasks

In this patch, I override the preRestart hook of the actor, in which I fail all running jobs if the dagScheduler crashes and restart

thanks for @kayousterhout and @markhamstra to give the hints in JIRA