johnmyleswhite/TextAnalysis.jl — pull request #14 by tanmaykm (2014-02-11 07:08:52+00:00)
Title: Multiple changes for speed improvement

This includes the following changes:
- `TokenDocument` now uses `SubString`s instead of `ByteString`
- creating inverse index gets `tokens` instead of `ngrams`
- specialize ngram creation for 1grams
- `StringDocument` can work with `MutableString`. Makes it much faster during the preprocess phase.
- Optimized stemmer for handling substrings and words within a larger string.
- added a simple regex cache to avoid recompilations while processing large corpuses
- fixed snowball stemmer building through bindeps

Tests to create an inverse index of large number of documents from the common crawl database show significant speed improvements. Creating an inverse index of a common crawl archive (100MB compressed with around 8000 documents) now takes 80 seconds instead of 200 seconds. Relevant test code can be found at https://github.com/tanmaykm/CommonCrawl.jl and https://github.com/tanmaykm/julia_examples/tree/master/common_crawl_indexer.

This also includes the changes in PR #11, so that can be deleted if this is acceptable.