NOT Support of a fast tokenizer implementation for Yi-1.5-34B-Chat
#3
by
paulcx
- opened
Can you add support for fast tokenizer? I remember yi-34b-chat supports it
yi-34b-chat is not supported either.
I thought a tokenizer.json file from Yi-1.5-34B or Yi-34B-Chat can be used for Yi-1.5-34B-Chat? Chat and Non-Chat model share same tokenizer right?
yes
lorinma
changed discussion status to
closed