fixed error cases for tokenizer