mesos/spark — pull request #533 by mridulm (2013-03-18 05:17:56+00:00)
Title: Fix performance issues and bugs in YARN branch.

Individual commits have more details on what is fixed in each of them (hopefully, they are logically grouped). These were the fixes we did to fix some severe bugs, make slaves more responsive as cluster increases (and there are more than 8 reducers !), reduce latency of computation, improve rack awareness, etc.

For some of our ML jobs, the time came down from 20+ mins to under 5 mins without any user code change (YMMV ofcourse !).

More conservatively, after all changes (bug fixes, etc) applied, the scheduling changes itself gave us a 2x-2.5x improvement on a small 40 container test (which increases as number of nodes increase) - by default, the scheduling changes are disabled - and are explicitly enabled by flags.

I am probably missing out other details, please query in case there are any issues, comments or notes !