Reimplemented lexer using a channel of tokens as output.