Added and improved tests of alternate tokenizers, revealing unfinished work and bugs which are now fixed.