tanaymehta commited on
Commit
26de81c
1 Parent(s): 7f22f0a

Add Fill-in-Middle specific tokens

Browse files

Request for adding FIM-specific tokens (prefix, middle and suffix) as discussed in the PR for adding FIM pipeline to transformers (https://github.com/huggingface/transformers/pull/27828#issuecomment-1970847921)

Files changed (1) hide show
  1. tokenizer_config.json +4 -1
tokenizer_config.json CHANGED
@@ -30,5 +30,8 @@
30
  "normalized": true,
31
  "rstrip": false,
32
  "single_word": false
33
- }
 
 
 
34
  }
 
30
  "normalized": true,
31
  "rstrip": false,
32
  "single_word": false
33
+ },
34
+ "fim_prefix": "▁<PRE>",
35
+ "fim_middle": "▁<MID>",
36
+ "fim_suffix": "▁<SUF>"
37
  }