fixed tokenizer switch from falling through in a jacked up way