CsvInputFormat can optionally skip an initial header line.
Default mode for output formats is not to overwrite existing files.
Globally configurable default modes for overwrite / output directory behavior.
Configurable behavior for FileOutputFormat through regular parameters, rather than configuration.
OutputFormat unit test discard existing temp files by default.
Additional test case for broadcast variables in branching data flows.
Added options to configure the default behavior of the LocalExecutor (ports, overwrite, hdfs config)
Cleaned up BroadcastBranchingITCase (added Apache header)
Compiler deadlock detection
fix NullPointException in web interface when inspecting a job immediately after it finished
delay the display of job history with 8 sec instead of using a high network overhead method
Fixed file output configuration and logging for various test cases.
Added functionality to re-initialize file format defaults for use with local executor.
Issue #519 Add setValue methods to Value types
Termination Criterion for Bulk Iterations
Fix bug in data source cost accounting.
Cleanup of termination criterion for bulk iterations.
fix s3 deployment for stratosphere YARN version
[yarn] Major YARN-Client improvements
- HDFS security token support
- "ship/" directory to transfer files to all TaskManagers (user-files)
- Log4j-based logging (YARN now respects the logging configuration)
- the YARN-client deletes all "temp" files from HDFS.
- The JVMs started by YARN now respect the configured JVM opts in the yaml-file
- the JobManager webinterface shows the log file (e.g it is aware of the YARN-specific log-directory)
- The YARN-client now creates a hidden ".yarn-jobmanager" with the address of the JobManager in YARN. users do not have to specify the -m argument anymore.
- Fix a little bug with the JobManager's "cloud" model for taskManager with less that 1GB memory.
- Tested on Cloudera Hadoop 5 Beta 2
- Tested on Cloduera Hadoop 5 Beta 2 WITH CDH5-B2 Maven Dependencies.
- Tested on Hadoop 2.2.0
- Tested on Amazon EMR
Fix the external packaged program test after merge with the new java api
Relax generic bound in tuple type infor to work around bug in Java 6 compiler type inference.
Local file system is used by default when URI is missing
Removed final declaration from Tuple classes to allow inheritance (Fabian Hueske)
Fix in TupleType comparator generation (Fabian Hueske)
Cogroup Implementation (Markus Holzemer)
Added missing Apache license header (Fabian Hueske)
Initial commit for join operator for new Java API (Fabian Hueske)
Added cross operator to new Java API (Fabian Hueske)
First part of CoGroup (Markus Holzemer)
Added join example prog and fix in join operator (Fabian Hueske)
Unified operator translation (Fabian Hueske)
Add Cross Prog for API implementation (Fabian Hueske)
Implement CsvInputFormat for new Java API, including tests.
Fix logging messages in data source and sink tasks.
Fix error in data source task with new InputFormat siignatures
Add group reduce to new java api.
Finaliz Key Extractors for new Java API
Fix bug in intitialization of broadcast serializers.
Added first set of comparators.
Added generic type serialization via avro
Add first stab at KeyExtractor implementation (Aljoscha Krettek)
Add trivial parts of serialization (Aljoscha Krettek)
Connected serializers to the typeutils.
Fix tests to run with the NEW-API changes (Aljoscha Krettek)
Scala Post pass got confused because of ReduceNode renaming. (Aljoscha Krettek)
Some input format mishaps. (Aljoscha Krettek)
First set of new simple type serializers.
Connect first paths of new Java API to optimizer.
Add name() to Operator plus some other stuff (Aljoscha Krettek)
Clean up moved annotations
Add CoGroup, Cross, and Union operators (Aljoscha Krettek)
Make reduce operators children of UdfOperator (Aljoscha Krettek)
Adjust input formats to new signature
Adjusted, extended, and tested parsers for new java api compatibility.
First draft of new java api.
Removing dependency to Combinable annotation. (Jesus Camacho Rodriguez)
Moving all annotations to Java API. (Jesus Camacho Rodriguez)
Moving all annotations from the core to the Java API package. Created SemanticProperties in the core to store properties inferred currently from Java annotations. (Jesus Camacho Rodriguez)
Moving all annotations to Java API (Jesus Camacho Rodriguez)
Add RAT plugin in the MAVEN build. Also add licenses where-ever needed.
Fixes a bug when using Double types as sorting key