mesos/spark — pull request #935 by rxin (2013-09-17 06:11:36+00:00)
Title: job killing in Spark

This subsumes pull request #655 by @harsha2010

Different from #655, this PR is not using interrupts to kill tasks because interrupts would rely on the user code properly handling the interrupts. One big counter example is Hadoop client would mark a data node as dead when it sees an interrupt during reads. Instead, this pull requests add a new volatile variable "interrupted" to the TaskContext class, and rely on the upper level code to check that flag. This can be done by simply wrapping an existing iterator with an InterruptibleIterator.

Todos:

1. Use InterruptibleIterator in all "input" iterators, including the ones in ParallelCollectionsRDD, HadoopRDD, NewHadoopRDD, ShuffleFetcher. (It might be easier to create an InterruptibleRDD class and just make the various input RDDs the parent of this new RDD.)

2. Design an interface for the user to be able to kill jobs. One idea is to add a new submitJob function in DAGScheduler, and for that to return a Future-like object that has a "cancel" or "kill" method. Note that this can be reused in the future to implement progress reporting feature.