recommit the tokenizer