[HELP!!] question about special_tokens_map.json

#20
by zzz96 - opened

hello, i found alpaca-native 's special_tokens_map.json is bellow:
{
"bos_token": "/s'",
"eos_token": "/s",
"pad_token": "[PAD]",
"unk_token": "/s"
}
decapoda-research/llama-7b-hf 's special_tokens_map.json is bellow(empty):
{}
but the code of stanford-alpaca should generate special_tokens_map.json like this:
{
"bos_token": "s",
"eos_token": "/s",
"pad_token": "[PAD]",
"unk_token": "unk"
}
Do you change the code of stanford-alpaca? Thank you ~
ps:《/s》can not show in huggingface ,so i delete the "《》"

Sign up or log in to comment