Improved on tokenizer test suite.

Increased cleanliness, and broke test cases into two separate tests.