stratosphere/stratosphere — pull request #539 by rmetzger (2014-03-06 18:43:38+00:00)
Title: [yarn] Major YARN-Client improvements

- HDFS security token support (resolves https://github.com/stratosphere/stratosphere/issues/492)
- "ship/" directory to transfer files to all TaskManagers (user-files)
- Log4j-based logging (YARN now respects the logging configuration) (resolves https://github.com/stratosphere/stratosphere/issues/493)
- the YARN-client deletes all "temp" files from HDFS.
- The JVMs started by YARN now respect the configured JVM opts in the yaml-file (resolves https://github.com/stratosphere/stratosphere/issues/501)
- the JobManager webinterface shows the log file (e.g it is aware of the YARN-specific log-directory)
- The YARN-client now creates a hidden ".yarn-jobmanager" with the address of the JobManager in YARN. users do not have to specify the -m argument anymore.
- Fix a little bug with the JobManager's "cloud" model for taskManager with less that 1GB memory.
- TM ports are now assigned based on the ports availability on the machine. (Users can still "hardcode" a port)
- Tested on Cloudera Hadoop 5 Beta 2
- Tested on Cloduera Hadoop 5 Beta 2 WITH CDH5-B2 Maven Dependencies.
- Tested on Hadoop 2.2.0
- Tested on Hadoop 2.3.0 (resolves https://github.com/stratosphere/stratosphere/issues/500)
- Tested on Amazon EMR