apache/spark — pull request #3427 by jkbradley (2014-11-24 08:47:46+00:00)
Title: [MLLIB] [WIP] [SPARK-3702] Standardizing abstractions and developer API for prediction

This is WIP effort to standardize abstractions and developer API for prediction tasks (classification and regression) for the new ML api (org.apache.spark.ml).
* Please comment on:
* abstractions, class hierarchy
* functionality required by each abstraction
* naming of types and methods
* ease of use for developers
* ease of use for users migrating from org.apache.spark.mllib
* Please ignore for now:
* missing tests and examples
* private/public API (I will make more things private[ml] after writing tests and examples.)
* style and other details
* the many TODO items noted in the code

Please refer to [https://issues.apache.org/jira/browse/SPARK-3702] for some discussion on design, and [this design doc](https://docs.google.com/document/d/1I-8PD0DSLEZzzXURYZwmqAFn_OMBc08hgDL1FZnVBmw/) for major design decisions.

This is not intended to cover all algorithms; e.g., one big missing item is porting the GeneralizedLinearModel class to the new API. But it hopefully lays a fair amount of groundwork.

I have included a limited number of concrete classes in this WIP PR, for purposes of illustration:
* LogisticRegression (edited, to show effects of abstract classes)
* NaiveBayes (simple to show ease of use for developers)
* AdaBoost (demonstration of meta-algorithms taking advantage of abstractions)
* (Note discussion of strong vs. weak types for ensemble methods in design doc.)
* This implementation is very incomplete but illustrates using the abstractions.
* LinearRegression (example of Regressor, for completeness)
* evaluators (to provide default evaluators in the class hierarchy)
* IterativeSolver and IterativeEstimator (to expose iterative algorithms)
* LabeledPoint (Q: Should this include an instance weight?)

Items remaining:
- [ ] helper method for simulating a distribution over weighted instances by subsampling (for algorithms which do not support instance weights)
- [ ] several TODO items noted in the code
- [ ] add tests and examples
- [ ] general cleanup
- [ ] make more of hierarchy private[ml]
- [ ] split into several smaller PRs

Thanks to @epahomov and @BigCrunsh for input, including from [https://github.com/apache/spark/pull/2137] which improves upon the org.apache.spark.mllib APIs.

CC: @etrain @shivaram @mengxr