Expose tokens in lexer module. Code cleanup.