ts-bpe-4k / special_tokens_map.json
cyrilzhang's picture
Upload tokenizer
0ec054a
{
"eos_token": "<|eos|>",
"pad_token": "<|pad|>"
}