Implemented further tests for the tokenizer.
Implemented further tests for the tokenizer