gerritjvv/elephant-bird — pull request #8 by ccattuto (2012-03-14 22:56:23+00:00)
Title: Preliminary support for nested JSON objects

This is a fork that implements very preliminary support for nested JSON objects:

- com.twitter.elephantbird.mapreduce.input.LzoJsonRecordReader now returns a nested MapWritable (with Writable values) that mirrors the parsed JSONObject.

- com.twitter.elephantbird.pig8.load.LzoJsonLoader now rewrites the MapWritable returned by the LzoJsonRecordReader into a Map that Pig can use.

This allows one to access deep fields in JSON records like this:

X = LOAD .... AS (json: map[]);
Y = FOREACH X GENERATE json#'xxx'#'yyy' as yyy;

TODO: clean up the type mapping stuff, make sure String values
are properly handled (on occasion they require an explicit cast to chararray in Pig),
and add support for bags/tuples.

The code is very rough and currently it only walks nested dictionaries,
filtering out bags/tuples.