apache/incubator-spark — pull request #11 by tgravescs (2013-09-23 14:13:20+00:00)
Title: Support putting files/archives in the hadoop distributed cache for spark on yarn

Add options for to put files/archives into the hadoop distributed cache. The options --files and --archives were added. They are very similar to the Hadoop options.

I also added an option to control the replication factor of the files uploaded into the staging directory. This includes the spark jar, app jar, and any distributed cache files.

we now attempt to cleanup the staging directory. There are some cases that it doesn't handle but it will work if the applications finish nicely, which is at least an improvement.

Question - Is there a reason the spark examples all do the System.exit() at the end?

Note this is the same as https://github.com/mesos/spark/pull/939 so we can probably close that one if we are using apache now.