Merge pull request #143 from rxin/scheduler-hang

Ignore a task update status if the executor doesn't exist anymore.

Otherwise if the scheduler receives a task update message when the executor's been removed, the scheduler would hang.

It is pretty hard to add unit tests for these right now because it is hard to mock the cluster scheduler. We should do that once @kayousterhout finishes merging the local scheduler and the cluster scheduler.
Removed unused return value in SparkContext.runJob
Attempt to fix SparkListenerSuite breakage

Could not reproduce locally, but this test could've been flaky if the
build machine was too fast.
Merge pull request #145 from aarondav/sls-fix

Attempt to fix SparkListenerSuite breakage

Could not reproduce locally, but this test could've been flaky if the build machine was too fast, due to typo. (index 0 is intentionally slowed down to ensure total time is >= 1 ms)

This should be merged into branch-0.8 as well.
Merge pull request #144 from liancheng/runjob-clean

Removed unused return value in SparkContext.runJob

Return type of this `runJob` version is `Unit`:

    def runJob[T, U: ClassManifest](
        rdd: RDD[T],
        func: (TaskContext, Iterator[T]) => U,
        partitions: Seq[Int],
        allowLocal: Boolean,
        resultHandler: (Int, U) => Unit) {
        ...
    }

It's obviously unnecessary to "return" `result`.
Add Spark multi-user support for standalone mode and Mesos
include the appid in the cmd line arguments to Executors
Merge pull request #23 from jerryshao/multi-user

Add Spark multi-user support for standalone mode and Mesos

This PR add multi-user support for Spark both standalone mode and Mesos (coarse and fine grained ) mode, user can specify the user name who submit app through environment variable `SPARK_USER` or use default one. Executor will communicate with Hadoop using  specified user name.

Also I fixed one bug in JobLogger when different user wrote job log to specified folder which has no right file  permission.

I separate previous [PR750](https://github.com/mesos/spark/pull/750) into two PRs, in this PR I only solve multi-user support problem. I will try to solve security auth problem in subsequent PR because security auth is a complicated problem especially for Shark Server like long-run app (both Kerberos TGT and HDFS delegation token should be renewed or re-created through app's run time).
very basic regression test to make sure appId doesnt get dropped in future
fix formatting
Merge pull request #148 from squito/include_appId

Include appId in executor cmd line args

add the appId back into the executor cmd line args.

I also made a pretty lame regression test, just to make sure it doesn't get dropped in the future.  not sure it will run on the build server, though, b/c `ExecutorRunner.buildCommandSeq()` expects to be abel to run the scripts in `bin`.
Use SPARK_HOME instead of user.dir in ExecutorRunnerTest
Propagate the SparkContext local property from the thread that calls the spark-repl to the actual execution thread.
Merge pull request #152 from rxin/repl

Propagate SparkContext local properties from spark-repl caller thread to the repl execution thread.
Add spark-tools assembly to spark-class classpath.

This allows the JavaAPICompletenessChecker to be
run with Spark 0.8+.