Factor out complex Lexer::tokenize() to separate functions