Ensured Tokenizer failure atomicity.