Merge pull request #199 from harveyfeng/yarn-2.2

Hadoop 2.2 migration

Includes support for the YARN API stabilized in the Hadoop 2.2 release, and a few style patches.

Short description for each set of commits:

a98f5a0 - "Misc style changes in the 'yarn' package"
a67ebf4 - "A few more style fixes in the 'yarn' package"
Both of these are some minor style changes, such as fixing lines over 100 chars, to the existing YARN code.

ab8652f - "Add a 'new-yarn' directory ... "
Copies everything from `SPARK_HOME/yarn` to `SPARK_HOME/new-yarn`. No actual code changes here.

4f1c3fa - "Hadoop 2.2 YARN API migration ..."
API patches to code in the `SPARK_HOME/new-yarn` directory. There are a few more small style changes mixed in, too.
Based on @colorant's Hadoop 2.2 support for the scala-2.10 branch in #141.

a1a1c62 - "Add optional Hadoop 2.2 settings in sbt build ... "
If Spark should be built against Hadoop 2.2, then:
a) the `org.apache.spark.deploy.yarn` package will be compiled from the `new-yarn` directory.
b) Protobuf v2.5 will be used as a Spark dependency, since Hadoop 2.2 depends on it. Also, Spark will be built against a version of Akka v2.0.5 that's built against Protobuf 2.5, named `akka-2.0.5-protobuf-2.5`. The patched Akka is here: https://github.com/harveyfeng/akka/tree/2.0.5-protobuf-2.5, and was published to local Ivy during testing.

There's also a new boolean environment variable, `SPARK_IS_NEW_HADOOP`, that users can manually set if their `SPARK_HADOOP_VERSION` specification does not start with `2.2`, which is how the build file tries to detect a 2.2 version. Not sure if this is necessary or done in the best way, though...
(cherry picked from commit 72b696156c8662cae2cef4b943520b4be86148ea)

Conflicts:

	project/SparkBuild.scala
	streaming/pom.xml
Small fix for Harvey's patch
Merge pull request #234 from alig/master

Updated documentation about the YARN v2.2 build process
(cherry picked from commit 241336add5be07fca5ff6c17eed368df7d0c3e3c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #235 from pwendell/master

Minor doc fixes and updating README
(cherry picked from commit e5d5728b72e58046cc175ab06b5f1c7be4957711)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #237 from pwendell/formatting-fix

Formatting fix

This is a single-line change. The diff appears larger here due to github being out of sync.
(cherry picked from commit 10c3c0c6524d0cf6c59b6f2227bf316cdeb7d06c)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
Merge pull request #236 from pwendell/shuffle-docs

Adding disclaimer for shuffle file consolidation
(cherry picked from commit 1b38f5f2774982d524742e987b6cef26ccaae676)

Signed-off-by: Patrick Wendell <pwendell@gmail.com>
typo fix