scikit-learn/scikit-learn — pull request #446 by larsmans (2011-11-17 18:48:18+00:00)
Title: ENH incrementally build arrays in SVMlight loader to reduce memory usage

Memory use on [covtype.binary](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary.html#covtype.binary) reduced by nearly 2/3, at the expense of of a 40% slowdown. Should fix issue #369.

Method: classical dynamic arrays with O(1) amortized append. The growth factor may be tweaked further for better performance.

Test script:

from bz2 import BZ2File
from sklearn.datasets import load_svmlight_file
load_svmlight_file(BZ2File('covtype.libsvm.binary.bz2'))