Merge pull request #145 from aarondav/sls-fix

Attempt to fix SparkListenerSuite breakage

Could not reproduce locally, but this test could've been flaky if the build machine was too fast, due to typo. (index 0 is intentionally slowed down to ensure total time is >= 1 ms)

This should be merged into branch-0.8 as well.

(cherry picked from commit 951024feeadcf73b50c3c80ec9e75c7e2214a7a4)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #144 from liancheng/runjob-clean

Removed unused return value in SparkContext.runJob

Return type of this `runJob` version is `Unit`:

    def runJob[T, U: ClassManifest](
        rdd: RDD[T],
        func: (TaskContext, Iterator[T]) => U,
        partitions: Seq[Int],
        allowLocal: Boolean,
        resultHandler: (Int, U) => Unit) {
        ...
    }

It's obviously unnecessary to "return" `result`.

(cherry picked from commit aadeda5e7697a433c82879033e758fbc403680dc)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #23 from jerryshao/multi-user

Add Spark multi-user support for standalone mode and Mesos

This PR add multi-user support for Spark both standalone mode and Mesos (coarse and fine grained ) mode, user can specify the user name who submit app through environment variable `SPARK_USER` or use default one. Executor will communicate with Hadoop using  specified user name.

Also I fixed one bug in JobLogger when different user wrote job log to specified folder which has no right file  permission.

I separate previous [PR750](https://github.com/mesos/spark/pull/750) into two PRs, in this PR I only solve multi-user support problem. I will try to solve security auth problem in subsequent PR because security auth is a complicated problem especially for Shark Server like long-run app (both Kerberos TGT and HDFS delegation token should be renewed or re-created through app's run time).

(cherry picked from commit be7e8da98ad04d66b61cd7fc8af7ae61a649d71c)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #148 from squito/include_appId

Include appId in executor cmd line args

add the appId back into the executor cmd line args.

I also made a pretty lame regression test, just to make sure it doesn't get dropped in the future.  not sure it will run on the build server, though, b/c `ExecutorRunner.buildCommandSeq()` expects to be abel to run the scripts in `bin`.

(cherry picked from commit 3d4ad84b63e440fd3f4b3edb1b120ff7c14a42d1)
Signed-off-by: Reynold Xin <rxin@apache.org>
Use SPARK_HOME instead of user.dir in ExecutorRunnerTest

(cherry picked from commit dd63c548c228d7775670e4664be18ebd1c62bed7)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #152 from rxin/repl

Propagate SparkContext local properties from spark-repl caller thread to the repl execution thread.

(cherry picked from commit 72a601ec318d017e5ec2b878abeac19e30ebb554)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #155 from rxin/jobgroup

Don't reset job group when a new job description is set.

(cherry picked from commit 83bf1920c85c33d8d725831b28bc5d11b53c27bd)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #149 from tgravescs/fixSecureHdfsAccess

Fix secure hdfs access for spark on yarn

https://github.com/apache/incubator-spark/pull/23 broke secure hdfs access. Not sure if it works with secure hdfs on standalone. Fixing it at least for spark on yarn.

The broadcasting of jobconf change also broke secure hdfs access as it didn't take into account things calling the getPartitions before sparkContext is initialized. The DAGScheduler does this as it tries to getShuffleMapStage.

(cherry picked from commit 8af99f2356ed19fe43d722ada02f8802cbd46d40)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #147 from JoshRosen/fix-java-api-completeness-checker

Add spark-tools assembly to spark-class'ss classpath

This commit adds an assembly for `spark-tools` and adds it to `spark-class`'s classpath, allowing the JavaAPICompletenessChecker to be run against Spark 0.8+ with

    ./spark-class org.apache.spark.tools.JavaAPICompletenessChecker

Previously, this tool was run through the `run` script.  I chose to add this to `run-example` because I didn't want to duplicate code in a `run-tool` script.

(cherry picked from commit 3efc0195625977335914f0a18cf32bd4e9b1d6d4)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #157 from rxin/kryo

3 Kryo related changes.

1. Call Kryo setReferences before calling user specified Kryo registrator. This is done so the user specified registrator can override the default setting.

2. Register more internal classes (MapStatus, BlockManagerId).

3. Slightly refactored the internal class registration to allocate less memory.

(cherry picked from commit 58d4f6c8a5d9739dc2a3f26f116528457336f0d3)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge branch 'branch-0.8' of https://github.com/apache/incubator-spark into master-csd
Merge pull request #164 from tdas/kafka-fix

Made block generator thread safe to fix Kafka bug.

This is a very important bug fix. Data can and was being lost in the kafka due to this.

(cherry picked from commit dfd1ebc2d1e5c34a5979648e571302ae81a178f5)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #171 from RIA-pierre-borckmans/master

Fixed typos in the CDH4 distributions version codes.

Nothing important, but annoying when doing a copy/paste...

(cherry picked from commit d76f5203af1f714efea30106bef9c2a3a80fd56e)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #170 from liancheng/hadooprdd-doc-typo

Fixed a scaladoc typo in HadoopRDD.scala

(cherry picked from commit 5a4f483652c3dc0730d1c908a6a46e2f77d270e1)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #169 from kayousterhout/mesos_fix

Don't ignore spark.cores.max when using Mesos Coarse mode

totalCoresAcquired is decremented but never incremented, causing Spark to effectively ignore spark.cores.max in coarse grained Mesos mode.

(cherry picked from commit 1a4cfbea334c7b0dae287eab4c3131c8f4b8a992)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #182 from rxin/vector

Slightly enhanced PrimitiveVector:

1. Added trim() method
2. Added size method.
3. Renamed getUnderlyingArray to array.
4. Minor documentation update.

(cherry picked from commit e2ebc3a9d8bca83bf842b134f2f056c1af0ad2be)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #173 from kayousterhout/scheduler_hang

Fix bug where scheduler could hang after task failure.

When a task fails, we need to call reviveOffers() so that the
task can be rescheduled on a different machine. In the current code,
the state in ClusterTaskSetManager indicating which tasks are
pending may be updated after revive offers is called (there's a
race condition here), so when revive offers is called, the task set
manager does not yet realize that there are failed tasks that need
to be relaunched.

This isn't currently unit tested but will be once my pull request for
merging the cluster and local schedulers goes in -- at which point
many more of the unit tests will exercise the code paths through
the cluster scheduler (currently the failure test suite uses the local
scheduler, which is why we didn't see this bug before).

(cherry picked from commit 96e0fb46309698b685c811a65bd8e1a691389994)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #175 from kayousterhout/no_retry_not_serializable

Don't retry tasks when they fail due to a NotSerializableException

As with my previous pull request, this will be unit tested once the Cluster and Local schedulers get merged.

(cherry picked from commit dfd40e9f6f87ff1f205944997cdbbb6bb7f0312c)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #160 from xiajunluan/JIRA-923

Fix bug JIRA-923

Fix column sort issue in UI for JIRA-923.
https://spark-project.atlassian.net/browse/SPARK-923

Signed-off-by: Reynold Xin <rxin@apache.org>

Conflicts:
	core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
	core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala

(cherry picked from commit b8bf04a085162478a64ca5d7be15d1af2f6a930e)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #153 from ankurdave/stop-spot-cluster

Enable stopping and starting a spot cluster

Clusters launched using `--spot-price` contain an on-demand master and spot slaves. Because EC2 does not support stopping spot instances, the spark-ec2 script previously could only destroy such clusters.

This pull request makes it possible to stop and restart a spot cluster.
* The `stop` command works as expected for a spot cluster: the master is stopped and the slaves are terminated.
* To start a stopped spot cluster, the user must invoke `launch --use-existing-master`. This launches fresh spot slaves but resumes the existing master.

(cherry picked from commit 87f2f4e5c2812351cdd1b2e35e2b12f62eeb3fdc)
Signed-off-by: Reynold Xin <rxin@apache.org>