scikit-learn/scikit-learn — pull request #865 by VirgileFritsch (2012-05-22 14:06:56+00:00)
Title: Ledoitwolf computation

LedoitWolf shrinkage estimation with large number of features does not create memory troubles again.

The construction of empirical covariance matrices is avoided by performing the computation with blocks.
As a consequence, the ``ledoit_wolf`` method will not always be able to return the shrunk covariance matrix (because it does not fit in memory) and will return None instead. The maximum size of a memory-allowed covariance matrix is controled by a new parameter of the ``ledoit-wolf`` method.

I tested up to n_features = 100,000 on my laptop, using a block size of 1000 and 1000 observations. No memory usage increase was observed while I could barely reach 10,000 features x 1,000 observations before on the same computer.