make tokenizer reentrant