Allow spark on yarn to be run from HDFS. Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs.
Fix access to Secure HDFS
Remove the runAsUser as it breaks secure hdfs access
Don't call the doAs if user is unknown or the same user that is already running
Use SPARK_HOME instead of user.dir in ExecutorRunnerTest
Propagate the SparkContext local property from the thread that calls the spark-repl to the actual execution thread.
Replaced the daemon thread started by DAGScheduler with an actor
Remove unnecessary null checking
replace the thread with a Akka scheduler
Merge pull request #152 from rxin/repl

Propagate SparkContext local properties from spark-repl caller thread to the repl execution thread.
Add spark-tools assembly to spark-class classpath.

This allows the JavaAPICompletenessChecker to be
run with Spark 0.8+.
Merge pull request #149 from tgravescs/fixSecureHdfsAccess

Fix secure hdfs access for spark on yarn

https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it works with secure hdfs on standalone. Fixing it at least for spark on yarn.

The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account things calling the getPartitions before sparkContext is initialized. The DAGScheduler does this as it tries to getShuffleMapStage.
Don't reset job group when a new job description is set.
Merge pull request #155 from rxin/jobgroup

Don't reset job group when a new job description is set.
Merge pull request #154 from soulmachine/ClusterScheduler

Replace the thread inside ClusterScheduler.start() with an Akka scheduler

Threads are precious resources so that we shouldn't abuse them
Merge pull request #147 from JoshRosen/fix-java-api-completeness-checker

Add spark-tools assembly to spark-class'ss classpath

This commit adds an assembly for `spark-tools` and adds it to `spark-class`'s classpath, allowing the JavaAPICompletenessChecker to be run against Spark 0.8+ with

    ./spark-class org.apache.spark.tools.JavaAPICompletenessChecker

Previously, this tool was run through the `run` script.  I chose to add this to `run-example` because I didn't want to duplicate code in a `run-tool` script.
Call Kryo setReferences before calling user specified Kryo registrator.
add tachyon module
Moved the Spark internal class registration for Kryo into an object, and added more classes (e.g. MapStatus, BlockManagerId) to the registration.
Merge pull request #157 from rxin/kryo

3 Kryo related changes.

1. Call Kryo setReferences before calling user specified Kryo registrator. This is done so the user specified registrator can override the default setting.

2. Register more internal classes (MapStatus, BlockManagerId).

3. Slightly refactored the internal class registration to allocate less memory.