apache/spark — pull request #3720 by mengxr (2014-12-17 07:26:54+00:00)
Title: [WIP][SPARK-3541][MLLIB] New ALS implementation with improved storage

This is a WIP.

This PR adds a new ALS implementation to `spark.ml` using the pipeline API, which should be able to scale to billions of ratings. Compared with the ALS under `spark.mllib`, the new implementation

1. uses the same algorithm,
2. uses float type for ratings,
3. uses primitive arrays to avoid GC,
4. sorts and compresses ratings on each block so that only O(k^2) memory is required for solving all least square problems.

I keep the `spark.mllib`'s ALS untouched for easy comparison. If the new implementation works well, it will replace the old one.