scikit-learn/scikit-learn — pull request #995 by kernc (2012-08-02 20:41:33+00:00)
Title: feature_extraction.text.CountVectorizer analyzer 'char_nospace'

A new in-built analyzer that creates character n-grams but only inside word boundaries, e.g.
`'string translation'` to 5-grams: `['strin', 'tring', 'trans', 'ransl', 'ansla', 'nslat', 'slati', 'latio', 'ation']`

This produces substantially smaller feature sets (10MB vs 60MB) while only slightly reducing the score with one of the default learners.
Produced n-grams still provide resilience to misspellings/cases/affixes/derivations, but they don't account for any word order (because they are bound to single words). Along with reduced dimensionality, this may be desired in some cases.

I have no references to attach, but the provided code works. :)

I'm very open to suggestions regarding the analyzer keyword (currently `'char_nospace'`).