generalize tokenizer used in basic tokenizer tests