apache/incubator-spark — pull request #199 by harveyfeng (2013-11-24 01:19:47+00:00)
Title: Hadoop 2.2 migration

Includes support for the YARN API stabilized in the Hadoop 2.2 release, and a few style patches.

Short description for each set of commits:

a98f5a0 "Misc style changes in the 'yarn' package"
e158035 "A few more style fixes in the 'yarn' package"
Both of these are some minor style changes, such as fixing lines over 100 chars, to the existing YARN code.

1a7a7a9 "Add a 'new-yarn' directory ... "
Copies everything from `SPARK_HOME/yarn` to `SPARK_HOME/new-yarn`. No actual code changes here.

22caceb "Hadoop 2.2 YARN API migration"
API patches to code in the `new-yarn` directory. There are a few more small style changes mixed in, too.
Based on @colorant's work in Hadoop 2.2 support for the `scala-2.10` branch in #141.

9c45901 "Add optional Hadoop 2.2 settings in sbt build ... "
If the `hadoopVersion` specifies 2.2.x, then:
a) the `org.apache.spark.deploy.yarn` package will be compiled from the `new-yarn` directory.
b) Protobuf v2.5 will be used as a Spark dependency, since Hadoop 2.2 depends on it. Also, Spark will be built against a version of Akka v2.0.5 that's built against Protobuf 2.5, named `akka-2.0.5-protobuf-2.5`. The patched Akka is here: https://github.com/harveyfeng/akka/tree/2.0.5-protobuf-2.5, and was published to local Ivy during testing.

There's also a new boolean environment variable, `SPARK_IS_HADOOP_2`, that users can manually set if their `SPARK_HADOOP_VERSION` specification does not start with `2.2` (e.g., if the desired Hadoop 2.2 dependency is named "custom-hadoop-2.2"). Not too sure if this is necessary or done in the best way, though...