mortardata/bacon-bits — pull request #3 by thedatachef (2013-09-25 02:57:48+00:00)
Title: Tfidf - first step, lucene tokenizer

pulls out arbitrary n-grams from text (unigrams, bigrams, trigrams, etc). For just unigrams for example:

NGramTokenize("1", "1")

or for just bigrams

NGramTokenize("2", "2")

for both

NGramTokenize("1", "2")

etc