apache/spark — pull request #1383 by sarutak (2014-07-12 02:48:49+00:00)
Title: [SPARK-1667] Jobs never finish successfully once bucket file missing occurred

If jobs execute shuffle, bucket files are created in a temporary directory (named like spark-local-*).
When the bucket files are missing cased by disk failure or any reasons, jobs cannot execute shuffle which has same shuffle id for the bucket files.

I think when Executors cannot read bucket files from their local directory (spark-local-*), they should abort and marked as lost.
In this case, Executor which has bucket files throw FileNotFoundException, so I think, we should handle IOException as fatal in Utils.scala to abort.

After I modified the code as follows, an Executor which fetches bucket files from failed Executor could retry to fetch from another Executor.

def isFatalError(e: Throwable): Boolean = {
e match {
case _: IOException =>
true
case NonFatal(_) | _: InterruptedException | _: NotImplementedError | _: ControlThrowable =>
false
case _ =>
true
}
}