fixed vectorize concurrency bug
added n-gram support
implemented df and llr check for pruning, llr seems a little high (~500 average)