mesos/hadoop — pull request #33 by tarnfeld (2014-09-16 20:04:02+00:00)
Title: [WIP] Idle TaskTrackers

This pull request introduces the ability to revoke slots from a running TaskTracker once it becomes idle. It contributes to solving #32 (map/reduce slot deadlock) as the cluster is able to remove slots that are idle and launch more when needed, avoid a deadlock situation (when resources *are* available).

**TLDR; The JobTracker / Mesos Framework is able to launch and kill map and reduce slots in the cluster as they become idle, to make better use of unitised resources.**

Essentially, what we've done here is separate the TaskTracker process from the slots. This means launching two mesos tasks attached to the same executor, one for the TaskTracker (as a task with potentially no resources) and another task which can be killed to free up mesos resources while keeping the TaskTracker itself alive. We attach the resources for "revokable" slots to the second task, reserving the ability to free up resources later on.

*Note: This is kind of hacky, and is no where near worthy of testing in production yet. Working progress!*

### How does it work?

Given the use case, I am only dealing with the situation where a running TaskTracker is completely idle. For example, if we launch 10 task trackers with only map slots and 5 with only reduce slots, while the reduce phase is running the map slots (and resources associated with them) can become completely idle. These slots can be killed, as long as the TaskTracker is alive to serve map output to the reducer. It _seems_ hadoop copes perfectly fine with TaskTrackers that have zero slots, too.

If we kill all map slots, we introduce potential failure cases where a node serving map data fails, and there are no map slots to re-compute the data. This is skirted around by only revoking a percentage of map slots from each TaskTracker (`remaining = max(slots - (slots * 0.9), 1)` by default).

- [ ] Make it work properly
- [ ] Rebase all my nasty commits away
- [ ] Configuration, configuration, configuration
- [ ] Do all of the todos

I'd be very interested to hear what the community thinks of the solution. There's no doubt something obvious I have missed, but worth discussing the idea.