Line analysis added

The tokenizer can now work on lines, operators, whitespace, and other
tokens.

Also added a unit test to make sure the above work.