Infinity conversation, sould eos be <|eot_id|> ?
#1
by
Komposter43
- opened
Model doesn't stop in vllm chat conversation, may be eos should be '<|eot_id|>' ?
put both eos tokens as stopping criteria i`d suggest.
See related merged PR on the meta-llama repo https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/discussions/4. That change to the tokenizer basically just needs to be done here as well
I've made a PR for it https://huggingface.co/casperhansen/llama-3-70b-instruct-awq/discussions/2
Fixed!
casperhansen
changed discussion status to
closed