Why don't the models of Yi-*-Chat have the "tokenizer.json" file?
#21
by
joeking11829
- opened
I want to perform TGI benchmark tests on the Yi model's token throughput, but I need the "tokenizer.json" file for the model.
Any Idea ?
Thanks.
You can load the tokenizer and save the information to a JSON file on your own