support output form Hive
tiny optimization
enable lookup of 3grams
use 2gram and 3gram corpus
implement cross validation