apache/spark — pull request #3406 by yhuai (2014-11-21 20:27:40+00:00)
Title: [WIP][SPARK-4476][SQL] Use MapType for dict in json which has unique keys in each row.

This PR introduces the following changes:
* Users can provide a schema with MapTypes to a JSON dataset.
* Users can use `spark.sql.schema.convertStructToMap` to control if we will try to convert a `StructType` with too many fields (the number of fields is greater than or equal to the threshold set by `park.sql.schema.convertStructToMapThreshold`) with same data type or a tightest common data type for these field data types exist (using `HiveTypeCoercion.findTightestCommonType`). Right now, this conversion will not be applied to the schema itself (the top level `StructType`). Only inner structs are candidate for this auto conversion.
* I am introducing one new `private[sql]` method to `DataType` (`transformUp`) and one new `private[sql]` method to `StructType` (`transformFieldTypeUp`). These two can be used to transform data types based the given partial function. My implementation of struct to map conversion is based on these two methods. Another example is `nullTypeToStringType` in `JsonRDD`, which transforms all `NullType`s to `StringType`s for a given `StructType`.

This is a WIP patch. I will add more unit tests and complete newly added ones later.

JIRA: https://issues.apache.org/jira/browse/SPARK-4476

@davies @marmbrus @willb