mesos/spark — pull request #170 by dennybritz (2012-08-13 20:28:08+00:00)
Title: Launching Spark over YARN

**Usage Example:**
SPARK_JAR=./core/target/spark-core-assembly-0.6.0-SNAPSHOT.jar ./run spark.deploy.yarn.Client **--jar** examples/target/scala-2.9.1/spark-examples_2.9.1-0.6.0-SNAPSHOT.jar **--class** spark.examples.SparkPi **--args** standalone **--num-workers** 3 **--worker-memory** 512M

**Notes:**
- Need to run sbt-assembly to generate the core-assembly jar used above.
- YARN does not support requesting containers/resources by the number of cores. Using `Runtime.getRuntime.availableProcessors()` for each slave for now.
- The user must use a the String "standalone" as the master's URL. That starts the scheduler without trying to connect to a cluster.
- There are several TODOs. In particular 1. Checking for and handling container/app failure, 2. Letting the user set the context priority, and some code refactoring.
- Tested locally, but not on a real cluster.