tokenizer now working as expected