How to set it up in a way that it just returns output without the system message or query, basically the information after [/INST].
#14
by
Pavan-growexx
- opened
Modifying responses
If you are using the TextIteratorStreamer class, then you can use skip_prompt=True
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
streamer = TextIteratorStreamer(tokenizer,skip_prompt=True)
if you are using a pipeline, I think there is a return_full_text=True/False parameter. not sure about this one.