dimalabs/ozone — pull request #68 by rmetzger (2013-08-23 10:24:23+00:00)
Title: AvroInputFormat for Ozone

This pull request contains an early prototype of an AvroInputFormat. Avro is a data serialization system that is widely used with Apache Hadoop. Many processing systems such as Hive or Pig support it.
I need to add support for Avro to Ozone because of customer requirements.

Please note that this pull request is not ready to merge: The test case is not done yet and the only a few primitive datatypes are supported (string, boolean, int).

I'd like to discuss the following questions (with you)
1) Do we want Avro Support in Ozone
2) Should we map the avro types to PactRecords or should we add the avro types to the system?

1) Pro: Widely used, easy to implement
Con: New dependencies; in the course of the implementation of Hive for ozone, we will most likely use HCatalog as a generic interface to many input file formats, including Avro. So one HCatalog is integrated, the AvroInputFormat would be redundant

2) Avro supports complex data types (http://avro.apache.org/docs/current/spec.html#schema_complex) which we can't map into our current data model (that's at least what I understand).
My current approach is to forbid nested or complex data structures with Avro and translate every AvroRecord into a PactRecord, preserving the order of the fields.

If we agree to add the InputFormat the ozone, I'll extend the test cases and type conversion to all primitive data types .