tokenizer api refactored