johnmyleswhite/TextAnalysis.jl — pull request #11 by tanmaykm (2014-01-18 08:09:14+00:00)
Title: Optimizations to TextAnalysis

These are few proposed optimizations to speed up text analysis, based on observations while indexing common crawl repository.

- TokenDocument modified to use SubStrings instead of UTF8String
- Specialized stemmer for handling substrings
- Creating inverse index of a document other than NGramDocument just gets tokens instead of getting one-grams
- Specialized ngram creation for one gram

With the above changes I notice close to 50% speed up with the specific cases that I am trying out.