make tokenizer re-entrant