tokun / tokenizer /special_tokens_map.json
apehex's picture
Upload folder using huggingface_hub
c51be42 verified
raw
history blame
153 Bytes
{
"bos_token": "\u0002",
"cls_token": "\u0011",
"eos_token": "\u0003",
"pad_token": "\u0000",
"sep_token": "\u001d",
"unk_token": "\u001a"
}