Model returns sometimes an empty response in llama.cpp

#2
by jost - opened

So far I've been using the q5_k_m.gguf and q4_k_m.gguf models for German. Most of the time, they work fine but sometimes, the model answers with an empty string.

For example:
prompt = f"""
<|im_start|>system
Du bist ein hilfreicher Assistent.
<|im_end|>
<|im_start|>user
Beantworte das folgende Statement mit 'Deutliche Ablehnung', 'Ablehnung', 'Zustimmung' oder 'Deutliche Zustimmung': Niemand sucht sich sein Geburtsland aus, daher ist albern, darauf stolz zu sein.
<|im_end|>
<|im_start|>assistant
"""
output = llm(prompt, max_tokens=200, temperature=0)
--> ...'choices': [{'text': '', 'index': 0, 'logprobs': None, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 93, 'completion_tokens': 0, 'total_tokens': 93}}

jost changed discussion title from Empty response in llama.cpp to Model returns sometimes empty response in llama.cpp
jost changed discussion title from Model returns sometimes empty response in llama.cpp to Model returns sometimes an empty response in llama.cpp

Sign up or log in to comment