frpron / special_tokens_map.json
Marxav's picture
add tokenizer
56a7094
raw
history blame
84 Bytes
{"bos_token": ">", "eos_token": "<", "unk_token": "<|endoftext|>", "pad_token": "#"}