apache/spark — pull request #2919 by jkbradley (2014-10-24 03:42:52+00:00)
Title: [SPARK-3572] [sql] [mllib]  User-Defined Types and MLlib Datasets

This PR adds User-Defined Types (UDTs) to SQL. It is a precursor to using SchemaRDD as a Dataset for the new MLlib API.

## Main additions

Public API
* SQL
* Added annotation SQLUserDefinedType (DeveloperApi)
* Added UDTRegistry (global object)
* Added abstract class UserDefinedType
* MLlib
* Vector, DenseVector, SparseVector are annotated with SQLUserDefinedType

Internals
* Made MLlib depend on SparkSQL.
* SQL
* ScalaReflection
* Methods for converting between Scala and Catalyst types now take DataType.
* convertRowToScala added in several locations in SQL
* schemaFor checks for SQLUserDefinedType annotation and checks UDTRegistry
* MLlib
* Added VectorUDT, DenseVectorUDT, SparseVectorUDT (private[spark])

Examples
* /examples/mllib/DatasetExample.scala: Demonstrates implicit conversion of RDD[LabeledPoint] to SchemaRDD

Unit Tests
* mllib/rdd/DatasetSuite.scala: Tests *VectorUDT
* sql/UserDefinedTypeSuite.scala: Tests fake version of DenseVector

## Design decisions

* UDTs override types natively recognized by SQL.

* Question: Should users be able to override primitive or built-in types?

## Items left for future PRs

* Java and Python APIs
* Serialization (Parquet, etc.)

CC: @mengxr @marmbrus