multi_tokenizer / tokenizer_config.json
ZhankuiHe's picture
Upload tokenizer
007c018 verified
{
"tokenizers": {
"nltk": "NLTKTokenizer",
"entity": "EntityTokenizer"
},
"tokenizer_key_for_decode": "nltk"
}