Microsoft-CISL/REEF — pull request #976 by markusweimer (2014-08-29 01:09:01+00:00)
Title: Updates to job submission and launch on YARN clusters

This PR mostly deals with an issue in `YarnConfigurationConstructor` where the configuration wouldn't be searched properly. However, it also contains the following updates:

* JobJarMaker no longer tries to overwrite files that exist twice on the classpath. Instead, it will print a message into FINEST and ignore the second file with the same name.
* LaunchClass now provides more elaborate error checking
* Updated the classpath assembly to allow for a prefix to the classpath to be provided by the runtime. This is helpful to make sure that e.g. the local YARN configuration is always first in the classpath
* Added new class `HadoopEnvironment` to provide constants for the names of the various Hadoop environment variables.
* Simplified `YarnConfigurationConstructor` to rely on the classpath to find the configuration
* `YarnClasspathProvider` now uses the `YarnConfiguration` to assemble the classpath instead of relying on environment variables
* `HelloReefYarn` now only adds its own JAR file to the Driver classpath, assuming that everything else is already on the Driver. This assumes that we run via `yarn jar reef-examples-?-shaded.jar com.microsoft.reef.examples.hello.HelloReefYarn`