SPARK-2294: fix locality inversion bug in TaskManager

copied from original JIRA (https://issues.apache.org/jira/browse/SPARK-2294): 

If an executor E is free, a task may be speculatively assigned to E when there are other tasks in the job that have not been launched (at all) yet. Similarly, a task without any locality preferences may be assigned to E when there was another NODE_LOCAL task that could have been scheduled.
This happens because TaskSchedulerImpl calls TaskSetManager.resourceOffer (which in turn calls TaskSetManager.findTask) with increasing locality levels, beginning with PROCESS_LOCAL, followed by NODE_LOCAL, and so on until the highest currently allowed level. Now, supposed NODE_LOCAL is the highest currently allowed locality level. The first time findTask is called, it will be called with max level PROCESS_LOCAL; if it cannot find any PROCESS_LOCAL tasks, it will try to schedule tasks with no locality preferences or speculative tasks. As a result, speculative tasks or tasks with no preferences may be scheduled instead of NODE_LOCAL tasks.


----


I added an additional parameter in resourceOffer and findTask, maxLocality, indicating when we should consider the tasks without locality preference

pendingTasksWithNoPrefs should be run after NODE_LOCAL tasks have been picked, and before RACK_LOCAL tasks are picked for the current executor.
Under 'normal' conditions, this list contains tasks which do not have any locality preference (running on any node is fine) - so if there are no node specific tasks available for an executor, running these before RACK_LOCAL minimizes data transfer in the cluster.

RACK_LOCAL tasks get typically scheduled much later in typical jobs (via locality wait) to avoid inter node data transfer : so pushing tasks without pref to after that is inefficient (and results in simply idling executors).


Unfortunately, given recent scheduler changes, it is not so clear anymore what the population in pendingTasksWithNoPrefs would be : tasks which do not have any preference or tasks which have preference but those are not available right now. Is that the reason for the confusion ?