Implementing the table driven lexer.