unicrs-gen / tokenizer_config.json
ZhankuiHe's picture
Upload tokenizer
580a13c verified
raw
history blame contribute delete
No virus
172 Bytes
{
"tokenizers": {
"item": "EntityTokenizer",
"context": "GPT2TokenizerFast",
"prompt": "RobertaTokenizerFast"
},
"tokenizer_key_for_decoding": "context"
}