scikit-learn/scikit-learn — pull request #1713 by larsmans (2013-02-25 15:58:43+00:00)
Title: MRG CounterVectorizer using arrays instead of lists

This is a new take on @ephes's code in #1135, rebased on current `master`, cleaned up and with one additional optimization for memory use.

Current `TfidfVectorizer` uses 447MB to load all of 20news:

Extracting features from the training dataset using a sparse vectorizer
done in 12.139995s at 1.817MB/s

Extracting features from the test dataset using the same vectorizer
done in 4.654452s at 2.965MB/s

New vectorizer uses 290MB:

Extracting features from the training dataset using a sparse vectorizer
done in 13.596905s at 1.622MB/s

Extracting features from the test dataset using the same vectorizer
done in 4.770538s at 2.893MB/s

That's roughly the same memory usage as `HashingVectorizer`, which, however, is a lot faster:

Extracting features from the training dataset using a sparse vectorizer
done in 6.676072s at 3.304MB/s

Extracting features from the test dataset using the same vectorizer
done in 4.032882s at 3.422MB/s

Seems reasonable to me, knowing that `CountVectorizer` currently can't handle very large sets at all. Since I haven't changed much and the innards of `CountVectorizer` have not significantly changed over the past half a year, I assume the old measurements are still largely valid.