ported the rest of the tokenizer tests