mesos/spark — pull request #336 by mridulm (2012-12-18 00:40:26+00:00)
Title: Fix issues with the branch to allow it to work on non trivial YARN clusters

This is an export of our private svn to github which fixes and resolves issues we faced while running spark on a hadoop yarn cluster.
Lot of the changes are intertwined, and so not split into different patches.

1) Use host:port instead of host in the codebase everywhere (patch use of preferredLocations to revert to host)
Since multiple containers can be started on same node, using only "host" fails.

2) Handle container allocation and maintenance - re-request containers which were lost, release more containers than required, etc.

3) Handle data locality at allocation time (not task dispatch time) - that is, request for containers base of * : so that the subsequent launch of tasks (based on preferredLocations) can be dispatched to nodes which are data-local.

4) Add support for rack locality of task (at allocation and dispatch time).

5) Move to using host everywhere instead of ip. iirc hadoop expects this anyway - this is probably not required, but otherwise we end up with conflicts between IP vs host at various places where we compare.
So consistently using hostname solves that issue : though I did see a bugfix from you regarding picking right IP - so we might have issues to resolve in case of multi-honed machines, etc.
I dont have a good solution for this problem : atleast current solution works for our clusters fine !

Simple and/or Localized changes -

6) Run the yarn job as the user launching the job - acl's, hadoop base directory, etc depend on this.
Note that I am yet to support kerberos authorization/tokens, etc - this will need to be added later.

7) Support long runing tasks - essentially periodically report progress to yarn : else it kills job after idle timeout (10 mins usually).

8) There are classpath issues which needed to be fixed.

9) Fix file handle leak

10) Changes to arg parsing (so that multiple arg's can be passed to the job - via repeated --args : trying to tokenize and parse was causing issues.

11) Add support for Queue

12) Do a best case effort to stop the spark context and notify yarn about application exit.
TODO: Propagate actual application success/failure to yarn.

Please note that we are not very familiar with scala (or spark for that matter) - so any and all suggestions to make this patch better are welcome !