apache/spark — pull request #2032 by ueshin (2014-08-19 08:48:37+00:00)
Title: [WIP][SPARK-3036][SPARK-3037][SQL] Add MapType/ArrayType containing null value support to Parquet.

JIRA:
- https://issues.apache.org/jira/browse/SPARK-3036
- https://issues.apache.org/jira/browse/SPARK-3037

Currently this uses the following Parquet schema for `MapType` when `valueContainsNull` is `true`:

```
message root {
optional group a (MAP) {
repeated group map (MAP_KEY_VALUE) {
required int32 key;
optional int32 value;
}
}
}
```

for `ArrayType` when `containsNull` is `true`:

```
message root {
optional group a (LIST) {
repeated group bag {
optional int32 array;
}
}
}
```

We have to think about compatibilities with older version of Spark or Hive or others I mentioned in the JIRA issues.

Notice:
This PR is based on #1963 and #1889.
Please check them first.

/cc @marmbrus, @yhuai