Tfidf - first step, lucene tokenizer

pulls out arbitrary n-grams from text (unigrams, bigrams, trigrams, etc). For just unigrams for example:

NGramTokenize("1", "1")

or for just bigrams

NGramTokenize("2", "2")

for both

NGramTokenize("1", "2")

etc