Added the tokenizer and tests.