Merge pull request #191 from hsaputra/removesemicolonscala

Cleanup to remove semicolons (;) from Scala code

-) The main reason for this PR is to remove semicolons from single statements of Scala code.
-) Remove unused imports as I see them
-) Fix ASF comment header from some of files (bad copy paste I suppose)
(cherry picked from commit 4b895013cc965b37d44fd255656da470a3d2c222)

Conflicts:

	examples/src/main/scala/org/apache/spark/streaming/examples/MQTTWordCount.scala

Squash into 191
Merge pull request #101 from colorant/yarn-client-scheduler

For SPARK-527, Support spark-shell when running on YARN

sync to trunk and resubmit here

In current YARN mode approaching, the application is run in the Application Master as a user program thus the whole spark context is on remote.

This approaching won't support application that involve local interaction and need to be run on where it is launched.

So In this pull request I have a YarnClientClusterScheduler and backend added.

With this scheduler, the user application is launched locally,While the executor will be launched by YARN on remote nodes with a thin AM which only launch the executor and monitor the Driver Actor status, so that when client app is done, it can finish the YARN Application as well.

This enables spark-shell to run upon YARN.

This also enable other Spark applications to have the spark context to run locally with a master-url "yarn-client". Thus e.g. SparkPi could have the result output locally on console instead of output in the log of the remote machine where AM is running on.

Docs also updated to show how to use this yarn-client mode.
(cherry picked from commit eb4296c8f7561aaf8782479dd5cd7c9320b7fa6b)

Conflicts:

	core/src/main/scala/org/apache/spark/SparkContext.scala
Merge pull request #199 from harveyfeng/yarn-2.2

Hadoop 2.2 migration

Includes support for the YARN API stabilized in the Hadoop 2.2 release, and a few style patches.

Short description for each set of commits:

a98f5a0 - "Misc style changes in the 'yarn' package"
a67ebf4 - "A few more style fixes in the 'yarn' package"
Both of these are some minor style changes, such as fixing lines over 100 chars, to the existing YARN code.

ab8652f - "Add a 'new-yarn' directory ... "
Copies everything from `SPARK_HOME/yarn` to `SPARK_HOME/new-yarn`. No actual code changes here.

4f1c3fa - "Hadoop 2.2 YARN API migration ..."
API patches to code in the `SPARK_HOME/new-yarn` directory. There are a few more small style changes mixed in, too.
Based on @colorant's Hadoop 2.2 support for the scala-2.10 branch in #141.

a1a1c62 - "Add optional Hadoop 2.2 settings in sbt build ... "
If Spark should be built against Hadoop 2.2, then:
a) the `org.apache.spark.deploy.yarn` package will be compiled from the `new-yarn` directory.
b) Protobuf v2.5 will be used as a Spark dependency, since Hadoop 2.2 depends on it. Also, Spark will be built against a version of Akka v2.0.5 that's built against Protobuf 2.5, named `akka-2.0.5-protobuf-2.5`. The patched Akka is here: https://github.com/harveyfeng/akka/tree/2.0.5-protobuf-2.5, and was published to local Ivy during testing.

There's also a new boolean environment variable, `SPARK_IS_NEW_HADOOP`, that users can manually set if their `SPARK_HADOOP_VERSION` specification does not start with `2.2`, which is how the build file tries to detect a 2.2 version. Not sure if this is necessary or done in the best way, though...
(cherry picked from commit 72b696156c8662cae2cef4b943520b4be86148ea)

Conflicts:

	project/SparkBuild.scala
	streaming/pom.xml
Small fix for Harvey's patch
Merge pull request #234 from alig/master

Updated documentation about the YARN v2.2 build process
(cherry picked from commit 241336add5be07fca5ff6c17eed368df7d0c3e3c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #235 from pwendell/master

Minor doc fixes and updating README
(cherry picked from commit e5d5728b72e58046cc175ab06b5f1c7be4957711)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #237 from pwendell/formatting-fix

Formatting fix

This is a single-line change. The diff appears larger here due to github being out of sync.
(cherry picked from commit 10c3c0c6524d0cf6c59b6f2227bf316cdeb7d06c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #236 from pwendell/shuffle-docs

Adding disclaimer for shuffle file consolidation
(cherry picked from commit 1b38f5f2774982d524742e987b6cef26ccaae676)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
typo fix