Moved RegexTokenizer and Token classes to separate lexer package.
Token type described with inner enum of class Token.
Added silly memoization.