Why don't the models of Yi-*-Chat have the "tokenizer.json" file?

#21
by joeking11829 - opened

I want to perform TGI benchmark tests on the Yi model's token throughput, but I need the "tokenizer.json" file for the model.

Any Idea ?
Thanks.

You can load the tokenizer and save the information to a JSON file on your own

Sign up or log in to comment