stratosphere/stratosphere — pull request #437 by rmetzger (2014-01-25 12:54:15+00:00)
Title: hadoop-compat addons project with HadoopDataSource

I picked up the code of https://github.com/stratosphere/stratosphere/pull/424 and enhanced it with the following:

* make hadoop-compat compatible to hadoop yarn and remove code in package `org.apache.hadoop`.
* Extended usercode object wrapper (@aljoscha, please validate my changes), fixed InputFormat serialization
* introduce pluggable type converter

Lets discuss if we want the project being called "hadoop-compat" or if you prefer "hadoop-compatability" ?

Open issues:
* If the Hadoop IF is a FileInputFormat, we should make the split assignment file locality aware
* Add junit tests for various formats such as ORC, Parquet and Avro.
* Write website documentation for this code.
* Add HadoopDataSource
* See if it is possible to map hadoop counters to our accumulators