How to set it up in a way that it just returns output without the system message or query, basically the information after [/INST].

by Pavan-growexx - opened

Modifying responses

If you are using the TextIteratorStreamer class, then you can use skip_prompt=True

tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
streamer = TextIteratorStreamer(tokenizer,skip_prompt=True)

if you are using a pipeline, I think there is a return_full_text=True/False parameter. not sure about this one.

Sign up or log in to comment