[yarn] Major YARN-Client improvements
- HDFS security token support
- "ship/" directory to transfer files to all TaskManagers (user-files)
- Log4j-based logging (YARN now respects the logging configuration)
- the YARN-client deletes all "temp" files from HDFS.
- The JVMs started by YARN now respect the configured JVM opts in the yaml-file
- the JobManager webinterface shows the log file (e.g it is aware of the YARN-specific log-directory)
- The YARN-client now creates a hidden ".yarn-jobmanager" with the address of the JobManager in YARN. users do not have to specify the -m argument anymore.
- Fix a little bug with the JobManager's "cloud" model for taskManager with less that 1GB memory.
- Tested on Cloudera Hadoop 5 Beta 2
- Tested on Cloduera Hadoop 5 Beta 2 WITH CDH5-B2 Maven Dependencies.
- Tested on Hadoop 2.2.0
- Tested on Amazon EMR
Fix the external packaged program test after merge with the new java api
Relax generic bound in tuple type infor to work around bug in Java 6 compiler type inference.
Local file system is used by default when URI is missing
Removed final declaration from Tuple classes to allow inheritance (Fabian Hueske)
Fix in TupleType comparator generation (Fabian Hueske)
Cogroup Implementation (Markus Holzemer)
Added missing Apache license header (Fabian Hueske)
Initial commit for join operator for new Java API (Fabian Hueske)
Added cross operator to new Java API (Fabian Hueske)
First part of CoGroup (Markus Holzemer)
Added join example prog and fix in join operator (Fabian Hueske)
Unified operator translation (Fabian Hueske)
Add Cross Prog for API implementation (Fabian Hueske)
Implement CsvInputFormat for new Java API, including tests.
Fix logging messages in data source and sink tasks.
Fix error in data source task with new InputFormat siignatures
Add group reduce to new java api.
Finaliz Key Extractors for new Java API
Fix bug in intitialization of broadcast serializers.
Added first set of comparators.
Added generic type serialization via avro
Add first stab at KeyExtractor implementation (Aljoscha Krettek)
Add trivial parts of serialization (Aljoscha Krettek)
Connected serializers to the typeutils.
Fix tests to run with the NEW-API changes (Aljoscha Krettek)
Scala Post pass got confused because of ReduceNode renaming. (Aljoscha Krettek)
Some input format mishaps. (Aljoscha Krettek)
First set of new simple type serializers.
Connect first paths of new Java API to optimizer.
Add name() to Operator plus some other stuff (Aljoscha Krettek)
Clean up moved annotations
Add CoGroup, Cross, and Union operators (Aljoscha Krettek)
Make reduce operators children of UdfOperator (Aljoscha Krettek)
Adjust input formats to new signature
Adjusted, extended, and tested parsers for new java api compatibility.
First draft of new java api.
Removing dependency to Combinable annotation. (Jesus Camacho Rodriguez)
Moving all annotations to Java API. (Jesus Camacho Rodriguez)
Moving all annotations from the core to the Java API package. Created SemanticProperties in the core to store properties inferred currently from Java annotations. (Jesus Camacho Rodriguez)
Moving all annotations to Java API (Jesus Camacho Rodriguez)
Add RAT plugin in the MAVEN build. Also add licenses where-ever needed.
Fixes a bug when using Double types as sorting key
Fixes a NullPointerException in PlanUnwrappingReduceGroupOperator
included double quotes around scalars to handle paths with spaces
[yarn] set the same user for the services running in the YARN containers.
Fix #584
Fix #568
Adjustments for java8
Local environment has console logging deactivated by default.
LocalEnvironment used the number of processor cores as the default DOP.
Change MutableObjectIterator to allow immutable objects

This makes it possible to remove the Reference<T> hack.
Remove Reference<T> hack (plus some other clean-up)

Also remove Serializer<T> and only use TypeSerializer<T> which is now
Serializable.
Reset AsciiUtils to original behavior (do not implement MutableOjectIterator any more)
Minor cleanups.
Renamed AsciiUtils to SimpleStringUtils, because they actually work on more than ascii strings (all strings without surrogate chars)
Removed array datamodel and Scala Grabbag
Small step towards semi join. Added semi join functions.
TODO: Special exec strategy (runtime + optimizer) based on hash join required.
      Either no duplicates of filtering side in HT or filtered tuples removed from HT after first match.
Updated TypeExtractor and TupleTpyeInfo to work on subclasses of Tuple1, Tuple2, ...
Merge Fix
includeFields support for new Java API's CsvReader
Removed ReferenceWrappingReducer. Cleaned imports in PlanOperators. Fixes in ReduceFunction.