mesos/spark — pull request #575 by mridulm (2013-04-18 19:20:39+00:00)
Title: Manual merge of yarn branch to trunk

Hi,

This is a manual merge of yarn branch to trunk.
All the tests have passed, and we have tested the resulting jar on a yarn cluster - and the performance is comparable to the yarn branch.

Note that there are a few todo's - which have been left for later.

a) Enhancement of yarn support - monitoring, management, hadoop security, and so on.
These can be seen as feature additions/enhancements - this patch (and yarn branch) just provides basic support.

b) Add support for mirror 'assembly' functionality from sbt.
I am not very comfortable with maven - so this is left as a todo for later.
Essentially, we need to mirror whatever is done as part of assembly for core while building the hadoop2-yarn maven profile.

c) Using multiple cores per container.
Currently, a container is restricted to a single core - but the allocation of the container might allow for multiple cores to be used.
This has been done due to an MT bug in hadoop dfs code (HADOOP-8406) - which has been resolved. Unfortunately, not 2.0.x versions seem to support this (particularly cloudera versions which have cherry picked the patches to apply).
We can either wait for GA of hadoop 2.0 before modifying the code to add support for this.

Nothing else comes to mind off hand. No new changes have been introduced compared to yarn branch iirc - but the codebase there was significantly different : so a closer look as part of review would be great !

We have also NOT tested this for streaming (other than what is in maven tests) and mesos.
Any help in these would be greatly appreciated.

Thanks,
Mridul