Tokenizer config uses the wrong type which causes breaks with txt-gen-inf
#3
by
samulrich1
- opened
Tokenizer type is PreTrainedTokenizerFast but should logically be Qwen2Tokenizer unless there is a reason it is not?
Also missing vocab.json file unless this truly is a completely different tokenizer than the original Qwen 1.5
The next version will fix the issue of inconsistent tokenizer.
huybery
changed discussion status to
closed
@huybery As in a new model or there will be an update to this model's repo?