raw tokenizer/statemachine implementation