apache/spark — pull request #273 by scwf (2014-03-30 15:42:47+00:00)
Title: DAGScheduler NullPointerException bug fix

DAGScheduler throws this NullPointerException below occasionally when running spark apps.
【
java.lang.NullPointerException
at org.apache.spark.scheduler.DAGScheduler.executorAdded(DAGScheduler.scala:186)
at org.apache.spark.scheduler.TaskSchedulerImpl.executorAdded(TaskSchedulerImpl.scala:409)
at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$1.apply(TaskSchedulerImpl.scala:210)
at org.apache.spark.scheduler.TaskSchedulerImpl$$anonfun$resourceOffers$1.apply(TaskSchedulerImpl.scala:206)
at scala.collection.mutable.ArraySeq.foreach(ArraySeq.scala:73)
at org.apache.spark.scheduler.TaskSchedulerImpl.resourceOffers(TaskSchedulerImpl.scala:206)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverActor.makeOffers(CoarseGrainedSchedulerBackend.scala:130)
at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend$DriverActor$$anonfun$receive$1.applyOrEls
】
This exception actually happens in the situation that eventProcessActor in DAGScheduler has not being initialized when ExecutorBackend register with SchedulerBackend. so the executorAdded method in DAGScheduler will throw the exception. We can replay this exception by adding this line of code to start method of DAGScheduler

def start() {
Thread.sleep(10000) // add this line
eventProcessActor = env.actorSystem.actorOf(Props(new Actor {
...

So fix this problem by judging eventProcessActor initialized or not when adding executor.