Updated tokenizer

#1
by Dampfinchen - opened

Hello,

the Mistral tokenizer has seen multiple changes over the last couple of days. An update would be nice!

https://huggingface.co/mistralai/Mistral-7B-v0.1/commits/main

Edit: looks like you've set the correct values already.

Dampfinchen changed discussion status to closed

Sign up or log in to comment