mesos/spark — pull request #749 by benh (2013-07-30 06:50:04+00:00)
Title: Added property 'spark.executor.uri' for launching on Mesos.

This enables launching Spark without requiring it already be "installed". Example:

$ ./make_distribution.sh
$ mv dist spark-x.y.z
$ tar czf spark-x.y.z.tar.gz spark-x.y.z
$ hadoop fs -put spark-x.y.z.tar.gz /path/to/spark-x.y.z.tar.gz
$ SPARK_EXECUTOR_URI=hdfs://host:port/path/to/spark-x.y.z.tar.gz MESOS_NATIVE_LIBRARY=/path/to/libmesos.dylib|so MASTER=host:port ./spark-shell

A job can set the 'spark.executor.uri' directly prior to creating the context. Note that for any version of Mesos before 0.12.1 you must 'export MESOS_NATIVE_LIBRARY' in the conf/spark-env.sh _within_ the spark-x.y.z.tar.gz they upload (and it must point to the MESOS_NATIVE_LIBRARY on the machine, which may be different than the laptop running a job or REPL). This limitation will be removed in Mesos 0.12.1 and on.