apache/spark — pull request #3779 by davies (2014-12-23 20:04:14+00:00)
Title: [SPARK-4939] move to next locality when no pending tasks for executors

Currently, if there are different locality in a task set, the tasks with NODE_LOCAL only get scheduled after all the PROCESS_LOCAL tasks are scheduled and timeout with spark.locality.wait.process (3 seconds by default). In local mode, the LocalScheduler will never call resourceOffer() again once it failed to get a task with same locality, then all the NODE_LOCAL tasks will be never scheduled.

This bug could be reproduced by run example python/streaming/stateful_network_wordcount.py, it will hang after finished a batch with some data.

This patch will try to remove the `execId` from `pendingTasksForExecutor` if no more tasks for this executor, it's cheched once for each launched PROCESS_LOCAL tasks. Then we can check whether there are PROCESS_LOCAL tasks or not by call `pendingTasksForExecutor.isEmpty`, which is cheap. Finally, we can change to next locality level once there is no more PROCESS_LOCAL tasks without waiting for `spark.locality.wait.process` seconds.

The tasks in pendingTasksForHost and pendingTasksForRack are the same set of tasks, so we don't need this logic for them.

cc @tdas