Update: tokenizer.json | tokenizer_config.json | tokenizer.model Align tokenizer with mistral-common

#1
by Derpyhuwe - opened

Hello! Is it possible to update these files from the original?

tokenizer.json
tokenizer_config.json
tokenizer.model

It fixed my problem loading in vllm because of the tokenizer.

Thanks for the model! It's great.

Sign up or log in to comment