Merge pull request #189 from tgravescs/sparkYarnErrorHandling

Impove Spark on Yarn Error handling

Improve cli error handling and only allow a certain number of worker failures before failing the application.  This will help prevent users from doing foolish things and their jobs running forever.  For instance using 32 bit java but trying to allocate 8G containers. This loops forever without this change, now it errors out after a certain number of retries.  The number of tries is configurable.  Also increase the frequency we ping the RM to increase speed at which we get containers if they die. The Yarn MR app defaults to pinging the RM every 1 seconds, so the default of 5 seconds here is fine. But that is configurable as well in case people want to change it.

I do want to make sure there aren't any cases that calling stopExecutors in CoarseGrainedSchedulerBackend would cause problems?  I couldn't think of any and testing on standalone cluster as well as yarn.
(cherry picked from commit aa638ed9c140174a47df082ed5631ffe8e624ee6)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #178 from hsaputra/simplecleanupcode

Simple cleanup on Spark's Scala code

Simple cleanup on Spark's Scala code while testing some modules:
-) Remove some of unused imports as I found them
-) Remove ";" in the imports statements
-) Remove () at the end of method calls like size that does not have size effect.
(cherry picked from commit 1b5b358309a5adfc12b75b0ebb4254ad8e69f5a0)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #191 from hsaputra/removesemicolonscala

Cleanup to remove semicolons (;) from Scala code

-) The main reason for this PR is to remove semicolons from single statements of Scala code.
-) Remove unused imports as I see them
-) Fix ASF comment header from some of files (bad copy paste I suppose)
(cherry picked from commit 4b895013cc965b37d44fd255656da470a3d2c222)

Conflicts:

	examples/src/main/scala/org/apache/spark/streaming/examples/MQTTWordCount.scala

Squash into 191
Merge pull request #101 from colorant/yarn-client-scheduler

For SPARK-527, Support spark-shell when running on YARN

sync to trunk and resubmit here

In current YARN mode approaching, the application is run in the Application Master as a user program thus the whole spark context is on remote.

This approaching won't support application that involve local interaction and need to be run on where it is launched.

So In this pull request I have a YarnClientClusterScheduler and backend added.

With this scheduler, the user application is launched locally,While the executor will be launched by YARN on remote nodes with a thin AM which only launch the executor and monitor the Driver Actor status, so that when client app is done, it can finish the YARN Application as well.

This enables spark-shell to run upon YARN.

This also enable other Spark applications to have the spark context to run locally with a master-url "yarn-client". Thus e.g. SparkPi could have the result output locally on console instead of output in the log of the remote machine where AM is running on.

Docs also updated to show how to use this yarn-client mode.
(cherry picked from commit eb4296c8f7561aaf8782479dd5cd7c9320b7fa6b)

Conflicts:

	core/src/main/scala/org/apache/spark/SparkContext.scala
Merge pull request #199 from harveyfeng/yarn-2.2

Hadoop 2.2 migration

Includes support for the YARN API stabilized in the Hadoop 2.2 release, and a few style patches.

Short description for each set of commits:

a98f5a0 - "Misc style changes in the 'yarn' package"
a67ebf4 - "A few more style fixes in the 'yarn' package"
Both of these are some minor style changes, such as fixing lines over 100 chars, to the existing YARN code.

ab8652f - "Add a 'new-yarn' directory ... "
Copies everything from `SPARK_HOME/yarn` to `SPARK_HOME/new-yarn`. No actual code changes here.

4f1c3fa - "Hadoop 2.2 YARN API migration ..."
API patches to code in the `SPARK_HOME/new-yarn` directory. There are a few more small style changes mixed in, too.
Based on @colorant's Hadoop 2.2 support for the scala-2.10 branch in #141.

a1a1c62 - "Add optional Hadoop 2.2 settings in sbt build ... "
If Spark should be built against Hadoop 2.2, then:
a) the `org.apache.spark.deploy.yarn` package will be compiled from the `new-yarn` directory.
b) Protobuf v2.5 will be used as a Spark dependency, since Hadoop 2.2 depends on it. Also, Spark will be built against a version of Akka v2.0.5 that's built against Protobuf 2.5, named `akka-2.0.5-protobuf-2.5`. The patched Akka is here: https://github.com/harveyfeng/akka/tree/2.0.5-protobuf-2.5, and was published to local Ivy during testing.

There's also a new boolean environment variable, `SPARK_IS_NEW_HADOOP`, that users can manually set if their `SPARK_HADOOP_VERSION` specification does not start with `2.2`, which is how the build file tries to detect a 2.2 version. Not sure if this is necessary or done in the best way, though...
(cherry picked from commit 72b696156c8662cae2cef4b943520b4be86148ea)

Conflicts:

	project/SparkBuild.scala
	streaming/pom.xml
Small fix for Harvey's patch
Merge pull request #234 from alig/master

Updated documentation about the YARN v2.2 build process
(cherry picked from commit 241336add5be07fca5ff6c17eed368df7d0c3e3c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #235 from pwendell/master

Minor doc fixes and updating README
(cherry picked from commit e5d5728b72e58046cc175ab06b5f1c7be4957711)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #237 from pwendell/formatting-fix

Formatting fix

This is a single-line change. The diff appears larger here due to github being out of sync.
(cherry picked from commit 10c3c0c6524d0cf6c59b6f2227bf316cdeb7d06c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #236 from pwendell/shuffle-docs

Adding disclaimer for shuffle file consolidation
(cherry picked from commit 1b38f5f2774982d524742e987b6cef26ccaae676)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
typo fix
Merge pull request #240 from pwendell/master

SPARK-917 Improve API links in nav bar
(cherry picked from commit 6494d62fe40ac408b14de3f0f3de8ec896a0ae6e)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Clean-up of changes file