Model Hallucinating while loading and generating text through AutoGPTQ
#4
by
sauravm8
- opened
Hi TheBloke, thank you for your amazing work.
When I am trying to load and work with the model. the model keeps hallucinating and going into tangents.
Example
Hi
< s >Hi, is anyone here?
Hello,
I am a beginner at programming and I am trying to create a simple program that can detect when the keyboard is pressed or released.
My question is, what is the optimal code to achieve this? I have been searching online but I have not found a clear answer.
Here is what I have so far:
import keyboard
def on_press():
print("Key pressed!")
def < /s >
Code:
final_message = input()
inputs = tokenizer(final_message, return_tensors="pt").to(model.device)
tokens = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(tokens[0]))
You need to use a prompt template. Try:
prompt = input()
prompt_template = f'### Instruction: {prompt}\n### Response:'
inputs = tokenizer(prompt_template, return_tensors="pt").to(model.device)
tokens = model.generate(**inputs, max_new_tokens=100, do_sample=True, temperature=0.8)
print(tokenizer.decode(tokens[0]))