mesos/spark — pull request #589 by mridulm (2013-05-01 15:04:23+00:00)
Title: Add support for hyper local scheduling

There are a couple of things which require review (among others).

a) Resolving executorId to executor host + port.
There is a change (hack ?) introduce to SparkEnv to achieve this due to lack of better way to do this.
This implicitly pulls ClusterScheduler as reference (cant be explicitly referenced though) - I am not very familiar with the design choice of splitting SparkEnv from SparkContext : so this change might be violating that ?

b) I am not familiar with ZippedRDD (or use of zip in general in scala) and what the expectation for preferredLocation for ZippedRDD is.
I suspect current code for preferredLocation is substandard for ZippedRDD : I have a TODO note there since I am not sure how to handle it properly.

This has been tested on a yarn cluster and on via maven test (and some tests modified to ensure they pass).

This had been initially removed from the yarn merge due to the disruptive nature of the change (more than yarn support change :) ). Since that patch is reasonably tested, we can take a shot at this now.

Any and all feedback more than welcome !

Thanks,
Mridul