ProteinSkier / tokenizer_config.json
MarvinCui's picture
Upload 5 files
91783b9 verified
raw
history blame contribute delete
138 Bytes
{
"vocab_size": 37,
"pad_token": "<|pad|>",
"eos_token": "<|endoftext|>",
"unk_token": "<|unk|>",
"bos_token": "<|endoftext|>"
}