Problem with repeated generation of newline characters
#3
by
sszymczyk
- opened
When playing with TheProfessor on llama.cpp I noticed that it often enters a loop generating newline characters.
The output looks like:
<s><|im_start|>user
Given the family relationships:
* Ralph is Anthony's parent.
* Albert is Ralph's parent.
What is Anthony's relationship to Ralph?
Select the correct answer:
1. Anthony is Ralph's child.
2. Anthony is Ralph's parent.
Enclose the selected answer number in the <ANSWER> tag, for example: <ANSWER>1</ANSWER>.<|im_end|>
<|im_start|>assistant
<ANSWER>1</ANSWER>
(multiple newline characters to follow)
The model generated a correct answer and then entered a newline generation loop. It behaves this way even on f16 version (without quantization). Is the model broken?
That is true, and it is a limitation.
To fix this would require further finetuning, which I don't have sponsorship for.
The model is intended not for automated / agentic loads, but rather for speaking with a user ad-hoc. You can end a generation with your chat client, when you notice it generating these kinds of artifacts.
sszymczyk
changed discussion status to
closed