TheBloke commited on
Commit
24e606e
1 Parent(s): 6c3c191

This file should be removed - it's no longer in the upstream repo

Browse files

It was broken in the original release, and they've since removed it. They've specified the special_tokens as added_tokens, but they're already listed in the tokenizer and special_tokens_map.json.

This file being here breaks making GGUF quants, and might break some other things too.

In the upstream repo they've moved to the new style Transformers special_tokens config

Files changed (1) hide show
  1. added_tokens.json +0 -5
added_tokens.json DELETED
@@ -1,5 +0,0 @@
1
- {
2
- "</s>": 2,
3
- "<s>": 1,
4
- "<unk>": 0
5
- }