Basic lexer implementation details incl. regex->dfa.