scikit-learn/scikit-learn — pull request #1765 by larsmans (2013-03-12 17:45:44+00:00)
Title: MRG constant-space cluster reassignment in minibatch k-means

I just ran into a `MemoryError` while clustering batches of 10k samples with >250k features (CSR matrices, of course) into 500 clusters with `MiniBatchKMeans`. It turns out that the line

distance_to_centers = np.asarray(centers[nearest_center] - X)

in the random reassignment densifies my minibatches, **twice**. I rewrote the squared-distance-to-centers code in Cython to take only constant space, and did the dense version too while I was at it.

There was already a test for this, which passes. I can now run `partial_fit` on 24 size-10k batches in less than 2 minutes, which includes I/O, feature extraction and two reassignments on a low-powered workstation without any trouble.

CC @pprett, @ogrisel.