A simple streamable lexer.  It transforms text into token objects.