apache/spark — pull request #210 by CodingCat (2014-03-23 18:42:13+00:00)
Title: SPARK-1298: remove duplicate partitionID checking

In the current implementation, we check whether partitionIDs is >=0 & < maxPartitionNum in SparkContext.runJob()

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/SparkContext.scala#L896

However, immediately following, in DAGScheduler (calling path SparkContext.runJob -> DAGScheduler.runJob -> DAGScheduler.submitJob)

we check it again, (just missing a < 0 condition),

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala#L432

I propose to remove the SparkContext one and check it in DAGScheduler (which makes more sense, from my view, as DAGScheduler handles everything about job, it should check whether the input of the job is valid before running it. But SparkContext is something more like a user-facing interface)

at least we should check it in only one place....