Add tokenizer unit tests.