initial commit
Added tokenizer. Grouping does not work yet.
WIP: tokenizing and profanity filtering
dict tokenizer uses now data frames and plyr. Can tokenize and tag with dictionary.
Parse dict into one regular expression.
Fixed chunker
store results
Added tokenizer without dictionary and a range of dictionaries.
Loops over dictionaries until enough tokens tagged. Creates n-grams. Allows first predictions.
Added analysis and changed tokenizer to use top 999 words.
Finished tokenizing.