moved test to TokenizerSpecTest
changed to handle tokens with whitespace/control chars