Tokenizer - remove unnecessary tokenizer states.