dickens / special_tokens_map.json
alexorona's picture
add tokenizer
3c46bd7
raw
history blame contribute delete
No virus
84 Bytes
{"eos_token": "<|endoftext|>", "unk_token": "<|endoftext|>", "pad_token": "<|pad|>"}