scikit-learn/scikit-learn — pull request #2749 by alimuldal (2014-01-14 01:07:47+00:00)
Title: Tentative implementation of out-of-core SVD

Hi all,

I've had a crack at implementing a function for out-of-core SVD on large memory-mapped arrays (as per issue #2661). It's based on the algorithm in [Halko, et al., 2011](http://arxiv.org/pdf/1007.5510), which is essentially identical to the one currently used by `randomized_svd` (originally from [Halko, et al., 2009](http://arxiv.org/abs/arXiv:0909.4061)). The main difference is that I've implemented a function for computing dot products on pre-cached blocks of memory-mapped arrays. Doing manual buffering in this way is many times faster than calling `np.dot` directly on an `np.memmap` array (see my [SO question/answer here](http://stackoverflow.com/q/20983882/1461210)).

For the time being I've just added another user option to `randomized_svd` to force buffering, with an additional keyword argument to control the size of the buffer in bytes. If people think this would be a potentially useful avenue to go down, I could have a look at integrating it into the `RandomizedPCA` class. However, there are potentially issues arising from the fact that the PCA classes force a copy of the input array (in order to do centering etc), which would be problematic for very large memory-mapped arrays.

Let me know what you think.