apache/spark — pull request #1140 by sebastienrainville (2014-06-19 21:14:37+00:00)
Title: Launch tasks on the proper executors in mesos fine-grained mode

The scheduler for Mesos in fine-grained mode launches tasks on random executors. MesosSchedulerBackend.resourceOffers(SchedulerDriver, List[Offer]) is assuming that TaskSchedulerImpl.resourceOffers(Seq[WorkerOffer]) is returning task lists in the same order as the offers it was passed, but in the current implementation TaskSchedulerImpl.resourceOffers shuffles the offers to avoid assigning the tasks always to the same executors. The result is that the tasks are launched on random executors.

The solution is to not assume that the order in which the tasks are returned is the same as the offers. What I am not sure about is that I considered slaveId and executorId to be the same, which is true at least in my setup, but I don't know if that is always true.

I tested this and it seems to work fine on our cluster.