Several minor fixes to tokenizer.

Includes catching any incomplete blocks or statements, which put the
tokenizer in a non-FREE state, but are not invalid token exceptions.
Added basic tests for the tokenizer.