apache/incubator-spark — pull request #446 by CodingCat (2014-01-16 00:36:42+00:00)
Title: Failed executor fix

the old PR is https://github.com/apache/incubator-spark/pull/440

this is a new PR to sync with the latest fix from @pwendell https://github.com/apache/incubator-spark/pull/442

prior to this PR, the failed creation of executor will make the worker restart with a new workerID and then failed into a cascading failure

worker restart->try to register with the same address with the new id -> failed to register with the new id -> send heartbeat to master -> master say "heartbeat from unregistered worker" -> old worker die -> application blocked

the empty SPARK_HOME may be the only reason of the failed creation for now (maybe, I'm not sure), but I think it is still worth to encapsulate the creation process in a try/catch