That is a possibility if you batch up the documents like you did before, so
that the analyzer is built only once per job.


On 4 March 2014 21:52, Aldrian Obaja <notifications@github.com> wrote:

>  In sklearn/feature_extraction/text.py:
>
> >          else:
> > -            j_indices = np.array([], dtype=np.int32)
> > -        indptr = np.frombuffer(indptr, dtype=np.intc)
> > -        values = np.ones(len(j_indices))
> > +            def _count_vocab_process(documents, result_pipe):
>
> How about passing the vectorizer object itself to Parallel? Then inside
> the method it can access the analyzer, I guess. I just worry about the
> memory usage.
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/scikit-learn/scikit-learn/pull/2923/files#r10250207>
> .
>