apache/incubator-spark — pull request #674 by derrickburns (2014-03-30 20:55:10+00:00)
Title: Generalized distance measure and weights

This pull request modifies the ML Lib K-Means clustering algorithm to support user defined distance functions (instead of Euclidean distance) and weight functions (instead of uniform).

The distance function can be any Bregman divergence function (see http://en.wikipedia.org/wiki/Bregman_divergence). Bregman divergence functions have been used with great success in many clustering algorithms (http://www.cs.utexas.edu/users/inderjit/public_papers/sdm-breg.pdf).

The weight function allows points to affect cluster centers in a non-uniform manner. This is useful in certain information theoretic clustering contexts (http://machinelearning.wustl.edu/mlpapers/paper_files/DhillonMK03.pdf).

Note: No changes have been made to the public interfaces, so these modifications are 100% backward compatible. However, this also means that the new features are not accessible from Python or Java. I leave it to you to figure out how you want to expose the new features from these languages.