scikit-learn/scikit-learn — pull request #1135 by ephes (2012-09-10 18:11:13+00:00)
Title: new CountVectorizer using arrays instead of lists

This implementation doesn't change the interface of CountVectorizer and passes the tests. Here are some benchmark figures:

Time needed to turn 500K documents into count vectors:
* old list based CountVectorizer: 2m31.797s
* new array based CountVectorizer: 2m43.890s

Peak memory needed turning 500K documents into count vectors:
* old list based CountVectorizer: 6.3gb
* new array based CountVectorizer: 920mb

The array based CountVectorizer is a little bit slower, because it has to recreate the feature arrays after the terms are sorted and the dimensions are remapped. It also introduces a new dependency on the array module.

Previous attempt to reduce the memory footprint of CountVectorizer by stacking together sparse matrices:
https://github.com/scikit-learn/scikit-learn/pull/1122