update tokenizer according to specifications