Updated special_tokens_map.json and tokenizer_config.json
#2
by
chameleon-lizard
- opened
In the standard LLaMA 3 special_tokens_map.json and tokenizer_config.json the eos token is set to <|end_of_text|>. This leads to model spamming assistant into chat, not stopping and chatting with itself.
This fix changes the eos token to <|eot_id|> in special_tokens_map.json and tokenizer_config.json, so the model can work with standard settings.