rewrite tokenizer to be JIT friendly, to avoid NYI