mesos/spark — pull request #828 by mateiz (2013-08-14 21:04:19+00:00)
Title: Scheduler improvements

This is a work in progress branch to simplify and improve a few things in the TaskSet scheduler and the way we handle locality. Here's a summary of changes (see also the commit messages):

- Eliminated most hostPort based data structures and just used executor IDs instead, since the hostPort vs host stuff was confusing (and not checkable with static typing, leading to ugly debug code), and hostPorts are not provided by Mesos. The hostPort stuff was there from before when we could easily pass executorIDs through.

- Removed non-local fallback stuff in ClusterScheduler that tried to launch less-local tasks on a node once the local ones were all assigned. This change didn't work because many cluster schedulers send offers for just one node at a time (even the standalone and YARN ones do so as nodes join the cluster one by one). Thus, lots of non-local tasks would be assigned even though a node with locality for them would be able to receive tasks just a short time later.

- Updated the way ClusterTaskSetManager handles racks: instead of enqueueing a task to a separate queue for all the hosts in the rack, which would create lots of large queues, have one queue per rack name.

- Added multi-level delay scheduling in ClusterTaskSetManager so you can have different waits for each locality level.

- Added periodic revival of offers in StandaloneSchedulerBackend (this was an option in the YARN patch but seems good to have by default)

- Simplified and cleaned up a bunch of the code.

The only thing missing is new unit tests for this stuff, which I'm working on, but the code appears to run fine so I wanted to put it up for others to try.