Add a new lexer based on a scanner that produces tokens one at a time.
Get rid of the old lexer.
Add lexer perf benchmark.
Update comment.
Add newline.