tokenizer interface changes