Microsoft-CISL/REEF — pull request #878 by markusweimer (2014-06-25 01:49:33+00:00)
Title: HDInsight support and new file system layout.

This pull request addresses two separate issues:

1. HDInsight support: This adds support for job submission to YARN's REST API as used by pre-releases of HDInsight.
2. New file system layout: This redefines the file system structure of REEF's Driver and Evaluator working directories.

Details as follow below:

**HDInsight support**

YARN will get a REST API for job submission that expands on the current API for job management. The newly added module `reef-runtime-hdinsight` adds support for the current version of that under the assumption that data is also on Azure. Future work will move this to the final YARN REST API and expand this in scope beyond HDInsight.

The module `reef-examples-hdinsight` contains the example code for this new runtime. For now, that is to run `HelloREEF`. The rationale for yet-another examples submodule is that in the case of HDInsight, we can't actually exclude the Hadoop JARs from the client, as it isn't executed on the cluster (but on the submitting machine).

**New filesystem layout**

REEF used to put all files for the current Driver or Evaluator into its working directory. This is unfortunate, as it precluded us from packaging things together and have YARN expand it for us. With this PR, we instead move to a different file system structure. The authorative answer is in the code in `com.microsoft.reef.runtime.common.files.REEFFileNames`, but here is the high level view:

In each working directory (Driver or Evaluator) we create the following folders:

* `reef` serves as the root for all things REEF such that we don't clash with other libraries applications might be using.
* `reef/local` contains all files local to this container (e.g. the configuration, local libraries, ...)
* `reef/global` contains all files global to the application (e.g. libraries)

When we submit a job, we submit a JAR that contains all three folders above for the Driver. When we submit an Evaluator, we submit two JARs: One with the global stuff (that is also only copied to HDFS only once) and one with the local files for that Evaluators.

The upside of this approach is that the number of files an application consists of is immaterial for the number of filesystem operations we perform. Hence, it is now very much feasible to submit REEF jobs from an IDE without requiring the build of shaded jars or such upfront.

All three runtimes (YARN, HDInsight, Local) have been updated to adhere to this new layout. Also, the unit tests have been updated accordingly.