Turning a list of lists into a list is just sum(list_of_lists, [])


On 5 March 2014 13:49, Aldrian Obaja <notifications@github.com> wrote:

>  In sklearn/feature_extraction/text.py:
>
> >          else:
> > -            j_indices = np.array([], dtype=np.int32)
> > -        indptr = np.frombuffer(indptr, dtype=np.intc)
> > -        values = np.ones(len(j_indices))
> > +            def _count_vocab_process(documents, result_pipe):
>
> Ok, so I would change _count_vocab_process to accept list of documents
> and return the result as list. Then is there a conventional way to get the
> combined result as a flat list (i.e., not a list of lists)? Because the
> result of each _count_vocab_process will be a list of dictionaries, and
> the result of Parallel then would be list of lists of dictionaries.
> Should I use itertools.chain?
>
> And I don't understand about the "analyzer only built once", since I don't
> see any memoization in build_analyzer, build_tokenizer, or
> build_preprocessor. Shouldn't there be some variables that stores the
> lambda functions? Because currently it will just run through all process
> again.
>
> Take for example build_tokenizer:
>
>     def build_tokenizer(self):
>         """Return a function that split a string in sequence of tokens"""
>         if self.tokenizer is not None:
>             return self.tokenizer
>         token_pattern = re.compile(self.token_pattern)
>         return lambda doc: token_pattern.findall(doc)
>
> Each time build_tokenizer is called, it will do the regex compilation, am
> I right?
>
> --
> Reply to this email directly or view it on GitHub<https://github.com/scikit-learn/scikit-learn/pull/2923/files#r10286007>
> .
>