apache/incubator-parquet-mr — pull request #14 by NathanHowell (2014-07-08 23:45:56+00:00)
Title: PARQUET-14: Support lifted coercions of file schemas into compatible read schemas.

**NOTE: This is not fully compatible with the old parquet-column/io layer, has some code duplication, is missing tests, etc, etc. I am submitting this request now to start a discussion on the new concepts it contains and to discuss how (or if) to introduce breaking API changes.**

-----------------

Pig and Hive assume certain values are nullable but should support
reading from Parquet files that do not have a representation of null
values. This patch introduces the notion type lifting. Lifted values
exist at runtime merely to provide a translation between the disk schema
and requested schema.

For example, a Parquet file with the following schema can not be read
from Pig or Hive:

```
message file_schema {
optional binary key (UTF8);
repeated group records {
optional binary name (UTF8);
optional binary type;
}
}
```

The requested schema that e.g. Pig requires wraps the `records` group
in an optional group in order to provide an explicit representation
of a null value.

```
message requested_schema {
optional binary key (UTF8);
optional group records (LIST) {
repeated group records {
optional binary name (UTF8);
optional binary type;
}
}
}
```

The file and requested schemas are not equal but are compatible for
reads with a simple transformation. When a nullable field present
in the requested schema but not on the file schema a GroupType wrapper
is inserted. This lifts non-nullable values into a nullable value.