apache/spark — pull request #3000 by jkbradley (2014-10-29 19:11:18+00:00)
Title: [SPARK-4081] [mllib]  DatasetIndexer

This introduces a DatasetIndexer class which does the following:
* fit(): collect statistics about how many values each feature in a dataset (RDD[Vector]) can take
* getCategoricalFeatureIndexes(): use the statistics to choose (a) which features should be treated as categorical vs. continuous and (b) 0-based indices for categorical feature values
* transform(): use the result from getCategoricalFeatureIndexes() to re-index categorical feature values

Currently, this kind of functionality is done on an ad-hoc basis (e.g., for labels in DecisionTreeRunner). This attempts to standardize it.

The basic usage pattern is:
```
val myData1: RDD[Vector] = ...
val myData2: RDD[Vector] = ...
val datasetIndexer = new DatasetIndexer(maxCategories)
datasetIndexer.fit(myData1)
val indexedData1: RDD[Vector] = datasetIndexer.transform(myData1)
datasetIndexer.fit(myData2)
val indexedData2: RDD[Vector] = datasetIndexer.transform(myData2)
val categoricalFeaturesInfo: Map[Double, Int] = datasetIndexer.getCategoricalFeatureIndexes()
```

Design notes:
* This maintains sparsity in vectors by ensuring that categorical feature value 0.0 gets index 0.
* This does not yet support transforming data with new (unknown) categorical feature values. That can be added later.
* This does not take advantage of sparsity in the input during fit(); it could be more efficient when given SparseVectors.

CC: @mengxr @manishamde @codedeft This should be helpful for DecisionTree and RandomForest.