apache/spark — pull request #1361 by freeman-lab (2014-07-10 19:53:46+00:00)
Title: Streaming mllib

This PR implements a streaming linear regression analysis, in which a linear regression model is trained online as new data arrive. The design is based on discussions with @tdas and @mengxr, in which we determined how to add this functionality in a general way, with minimal changes to existing libraries.

__Summary of additions:__

_StreamingRegression_
- An abstract class for fitting regression analyses online on streaming data, including training on (and updating) a model, and making predictions

_StreamingLinearRegressionWithSGD_
- Class and companion object for running streaming linear regression

_MLStreamingUtils_
- Utility for loading and parsing streaming data from a text file stream, could be extended with functions for loading data from Kafka, Network, etc.

_StreamingLinearRegression_
- Example use case: fitting a model online to data from one stream, and making predictions on other data

__Notes__
- I will definitely add tests but I wasn't sure where it makes sense to put them: mllib or streaming?
- If this looks good, I can use the StreamingRegression class to do all other regression analyses (Ridge, Lasso, etc.), and a similar StreamingClassification class would give us logistic and SVM classification.