Adding final version of the tokenizer code.