This file should be removed - it's no longer in the upstream repo
#1
by
TheBloke
- opened
It was broken in the original release, and they've since removed it. They've specified the special_tokens as added_tokens, but they're already listed in the tokenizer and special_tokens_map.json.
This file being here breaks making GGUF quants, and might break some other things too.
In the upstream repo they've moved to the new style Transformers special_tokens config
ehartford
changed pull request status to
merged