A fast tokenizer/lexer core that is well tested.