This is an export of our private branch to github to bootstrap the review process to spark.
Unfortunately, all the changes are merged into this single commit.
The fixes/changes are.

1) Use host:port instead of host in the codebase everywhere (patch use
of preferredLocations to revert to host)
Since multiple containers can be started on same node, using only "host" fails.

2) Handle container allocation and maintenance - re-request containers
which were lost, release more containers than required, etc.

3) Handle data locality at allocation time (not task dispatch time) -
that is, request for containers base of <input format, path>* : so
that the subsequent launch of tasks (based on preferredLocations) can
be dispatched to nodes which are data-local.

4) Add support for rack locality of task (at allocation and dispatch time).

5) Move to using host everywhere instead of ip.
iirc hadoop expects this anyway - this is probably not required, but
otherwise we end up with conflicts between IP vs host at various
places where we compare.
So consistently using hostname solves that issue : though I did see a
bugfix from you regarding picking right IP - so we might have issues
to resolve in case of multi-honed machines, etc.
I dont have a good solution for this problem : atleast current
solution works for our clusters fine :-)

6) Run the yarn job as the user launching the job - acl's, hadoop base
directory, etc depend on this.
Note that I am yet to support kerberos authorization/tokens, etc -
this will need to be added later.

7) Support long runing tasks - essentially periodically report
progress to yarn : else it kills job after idle timeout (10 mins
usually).

8) There are classpath issues which needed to be fixed.

9) Fix file handle leak - this is orthogonal to yarn port : just a
bugfix (so can be addressed independently).

10) Changes to arg parsing (so that multiple arg's can be passed to
the job - via repeated --args : trying to tokenize and parse was
causing issues.

11) Add support for Queue

12) finalize SparkContext tries to do a stop as best case effort (so
that 10 mins does not need to expire before job is killed in case of
user not stopping the context).
Cleanup some comments, split 'long' lines into multiple lines
Cleanup some comments
Change the more verbose info to debug - the logs are unmanageable otherwise
Revert log changes
log resource request in INFO
Cleanup some code, comments and add some explainations
remove context.finalize, rely on shutdown hook : more predictable and less error prone (not to mention, less interactions with other modes)
Add link to yarn mapred allocation
Tweak how long we wait for containers, also add finishApplicationMaster to shutdown hook
Allow some time for slaves to register with master after allocation - before jobs are launched
3 seconds to init yarn containers, wait for slaves to bootstrap before initializing
Introduce postStartHook to more cleanly initialize the scheduler
1) Fix issues with TaskSetManager introduced by previous commits (hyper local vs host local).
2) Ensure that task dispatch to nodes tries to prioritize host, rack and then any : as opposed to simply offering hosts in arbitrary order.
3) Move spark.master to host from ip - had missed this earlier.  fix bug with not propagating SPARK_YARN_MODE
4) Added more debug code - note that this will need to be removed once things are reasonably stable.
Do not propagate spark.hostname to clients (that is to be set by the client themselves : causes issues with cache status reporting
Cache computation of parseHost
tab to space
Changes to address review comments:

a) Modify changes to bring it in-line with spark/scala coding style. (Hopefully I did not miss anything !).
b) Remove explicit type information as suggested by Matei.
c) Also removed conditional check in SparkContext to set spark.master.host - always setting it to hostname (not ip).
Integrate review comments from Matei
Integrate review comments from Matei