Tokens are back and greatly improved helper functions for lexer.