scikit-learn/scikit-learn — pull request #1122 by ephes (2012-09-06 20:47:13+00:00)
Title: CountVectorizerBuffered

Ran out of memory with CountVectorizer (having 3.4 million documents with 3.5Gb data and only 8Gb RAM) . The problem is that CountVectorizer stores term_counts_per_doc for all documents before creating the sparse coo_matrix.

So i wrote a new class CountVectorizerBuffered, which creates a partial sparse feature matrix for every 5K documents and merges the partial feature matrices in the end. This fixes the memory issue, but it also alters the behavior of the class, which makes it kind of incompatible with CountVectorizer. For example it's not possible to set max_df, min_df or max_features and then simply call fit_transform, because fit_transform doesn't have the required token statistics for that. Therefore you now have to call fit before transform if you want to use max_df, min_df or max_features.

I thought about replacing CountVectorizer with CountVectorizerBuffered and i have a branch (countvectorizer) in which i did, but this breaks a lot of test-cases... well, i'm not sure what to do here.