Missing export
Some tests for the tokenizer
Fixed whitespace handling at eof
Changes to command defn syntax
Pass out whitespace tokens for now and ignore them where needed

This is mostly for tests but also the right thing to do for further
work.