apache/incubator-spark — pull request #575 by mengxr (2014-02-10 17:46:16+00:00)
Title: [Proposal] Adding sparse data support and update KMeans

This is a proposal for sparse data support in mllib (https://spark-project.atlassian.net/browse/MLLIB-18).

The idea of the proposal is that we define simple data models and factory methods for user to provide sparse input. Then instead of writing a linear algebra library for mllib, we take leverage on an existing linear algebra package in implementing algorithms. So we can change the underlying implementation without breaking the interfaces in the future. We need the following:

* data models for sparse vectors. We need data models for dense vector, sequential access sparse vector (backed by two parallel arrays), and random access sparse vector (backed by a primitive-typed hash map, not in this pull request.). Those are defined in the Vec class in this PR.
* a linear algebra package. We are considering either breeze or mahout-math. Both have pros and cons, and we can discuss more in the JIRA. This PR uses mahout-math. Mahout vectors do not implement serializable, so we need a serializable wrapper class (MahoutVectorWrapper) to use in spark. As a result, we added not only mahout-math but also mathout-core into dependencies because we need VectorWritable defined mahout-core for the wrapper class. But we can certainly remove most transitive dependencies of mahout-core.
* lightweight converters. The conversion between our data models and Mahout vectors shouldn't involve data copying. However, Mahout vectors hide their members. In this PR, Java reflection is used to get the private fields out. This doesn't seem to be a good solution, but I didn't figure out a better one.