Interesting, although the benchmarks don't suggest this has been a big
problem with Brown. Does joblib help facilitate batching to sizes
appropriate for n_jobs.


On 4 March 2014 23:08, Olivier Grisel <notifications@github.com> wrote:

>  In sklearn/feature_extraction/text.py:
>
> >          else:
> > -            j_indices = np.array([], dtype=np.int32)
> > -        indptr = np.frombuffer(indptr, dtype=np.intc)
> > -        values = np.ones(len(j_indices))
> > +            def _count_vocab_process(documents, result_pipe):
>
> +1 for batching jobs as well (e.g. 10000 documents at once) to hide the
> per-job execution overhead. Batching a large number of documents will also
> reduce the number of vocabularies to pass between process (they can be very
> expensive to unpickle as the keys will be re-hashed on the un-pickling
> side).
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/scikit-learn/scikit-learn/pull/2923/files#r10252070>
> .
>