Added error checking to tokenizer and added test cases.