Tokenizer rewritten to match Syntax spec. Needs extensive testing, but works with simple tests.
Sync the tokenizer with the Syntax spec.