scikit-learn/scikit-learn — pull request #2923 by justhalf (2014-03-03 09:38:00+00:00)
Title: ENH Parallelize CountVectorizer

Add parameter n_jobs to CountVectorizer (also TfidfVectrizer, which uses CountVectorizer), similar to one in KMeans
Include timing benchmark on Brown corpus (using nltk)
Include unit tests for the multiprocessing version of CountVectorizer
Fix a few typos

Benchmark result on Brown corpus (500 documents, amounting to 1,161,192 words) shows that the one-core version finishes in 38s, while the multi-core version finishes in 12s on a four-core (2.66 GHz Intel Core i5) iMac running OSX 10.8.5, a three-fold speedup. On another personal test on different machine, the one-core version finishes in 155s, while the multi-core version finishes in 38s, a four-fold speedup. So the parallelization clearly improves the speed, even for Brown corpus, which is not very large.
The parallelization is very useful for very large data (like the one I'm building this one for, around 115k documents), which can mean the difference between a few days and a couple of hours.