documented and simplified tokenizer automaton