Yeah, the tokenizer API is weird. We might want to rewrite it to expose more functionality we want in a cleaner way.