TS-3171: minor style updates to Tokenizer

Make some minor style tweaks to Tokenizer so that it is a little
more conventional and easier to use. Rewrite the regression test
so it tests something.