Merge pull request #171 from RIA-pierre-borckmans/master

Fixed typos in the CDH4 distributions version codes.

Nothing important, but annoying when doing a copy/paste...

(cherry picked from commit d76f5203af1f714efea30106bef9c2a3a80fd56e)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #170 from liancheng/hadooprdd-doc-typo

Fixed a scaladoc typo in HadoopRDD.scala

(cherry picked from commit 5a4f483652c3dc0730d1c908a6a46e2f77d270e1)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #169 from kayousterhout/mesos_fix

Don't ignore spark.cores.max when using Mesos Coarse mode

totalCoresAcquired is decremented but never incremented, causing Spark to effectively ignore spark.cores.max in coarse grained Mesos mode.

(cherry picked from commit 1a4cfbea334c7b0dae287eab4c3131c8f4b8a992)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #182 from rxin/vector

Slightly enhanced PrimitiveVector:

1. Added trim() method
2. Added size method.
3. Renamed getUnderlyingArray to array.
4. Minor documentation update.

(cherry picked from commit e2ebc3a9d8bca83bf842b134f2f056c1af0ad2be)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #173 from kayousterhout/scheduler_hang

Fix bug where scheduler could hang after task failure.

When a task fails, we need to call reviveOffers() so that the
task can be rescheduled on a different machine. In the current code,
the state in ClusterTaskSetManager indicating which tasks are
pending may be updated after revive offers is called (there's a
race condition here), so when revive offers is called, the task set
manager does not yet realize that there are failed tasks that need
to be relaunched.

This isn't currently unit tested but will be once my pull request for
merging the cluster and local schedulers goes in -- at which point
many more of the unit tests will exercise the code paths through
the cluster scheduler (currently the failure test suite uses the local
scheduler, which is why we didn't see this bug before).

(cherry picked from commit 96e0fb46309698b685c811a65bd8e1a691389994)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #175 from kayousterhout/no_retry_not_serializable

Don't retry tasks when they fail due to a NotSerializableException

As with my previous pull request, this will be unit tested once the Cluster and Local schedulers get merged.

(cherry picked from commit dfd40e9f6f87ff1f205944997cdbbb6bb7f0312c)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #160 from xiajunluan/JIRA-923

Fix bug JIRA-923

Fix column sort issue in UI for JIRA-923.
https://spark-project.atlassian.net/browse/SPARK-923

Signed-off-by: Reynold Xin <rxin@apache.org>

Conflicts:
	core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala
	core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala

(cherry picked from commit b8bf04a085162478a64ca5d7be15d1af2f6a930e)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #153 from ankurdave/stop-spot-cluster

Enable stopping and starting a spot cluster

Clusters launched using `--spot-price` contain an on-demand master and spot slaves. Because EC2 does not support stopping spot instances, the spark-ec2 script previously could only destroy such clusters.

This pull request makes it possible to stop and restart a spot cluster.
* The `stop` command works as expected for a spot cluster: the master is stopped and the slaves are terminated.
* To start a stopped spot cluster, the user must invoke `launch --use-existing-master`. This launches fresh spot slaves but resumes the existing master.

(cherry picked from commit 87f2f4e5c2812351cdd1b2e35e2b12f62eeb3fdc)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #165 from NathanHowell/kerberos-master

spark-assembly.jar fails to authenticate with YARN ResourceManager

The META-INF/services/ sbt MergeStrategy was discarding support for Kerberos, among others. This pull request changes to a merge strategy similar to sbt-assembly's default. I've also included an update to sbt-assembly 0.9.2, a minor fix to it's zip file handling.

(cherry picked from commit 9290e5bcd2c8e4d8bbf1d0ce1ac09bbf62ece4e0)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #137 from tgravescs/sparkYarnJarsHdfsRebase

Allow spark on yarn to be run from HDFS.

Allows the spark.jar, app.jar, and log4j.properties to be put into hdfs.  Allows you to specify the files on a different hdfs cluster and it will copy them over. It makes sure permissions are correct and makes sure to put things into public distributed cache so they can be reused amongst users if their permissions are appropriate.  Also add a bit of error handling for missing arguments.

(cherry picked from commit f49ea28d25728e19e56b140a2f374631c94153bc)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #166 from ahirreddy/simr-spark-ui

SIMR Backend Scheduler will now write Spark UI URL to HDFS, which is to ...

...be retrieved by SIMR clients

(cherry picked from commit 39af914b273e35ff431844951ee8dfadcbc0c400)
Signed-off-by: Reynold Xin <rxin@apache.org>
Merge pull request #174 from ahirreddy/master

Write Spark UI url to driver file on HDFS

This makes the SIMR code path simpler

(cherry picked from commit ed25105fd9733acd631dab0993560ac66ffeae16)
Signed-off-by: Reynold Xin <rxin@apache.org>