bigscience

/

bloom

like 4.57k

I want to stop text generation when a set of special characters are found, like ‘###’, but I can’t achieve it. I know that I can implement a piece of code to post-process the generated text and extract the expected result, but it would be interesting to stop text generation when a criteria is fulfilled to save some words/tokens in the task.

I’m using the parameter eos_token_id to that end, but it doesn’t work. I thought that I was doing something wrong, but I tried the model Incoder with the same eos_token_id parameter and it works!

Anyone know if this is a specific problem of BLOOM or am I doing something wrong?

tokenizer = AutoTokenizer.from_pretrained("bigscience/bloom")
end_sequence = '###'

payload = {
    "inputs": f"{context} \nHuman: {nl_query} \nAI: ",
    "parameters":
    {
        "top_p": 0.9,
        "temperature": 0.2,
        "max_new_tokens": 40,
        "eos_token_id": int(tokenizer.convert_tokens_to_ids(end_sequence)),
        "return_full_text": False
    }, 
    "options": 
    {  
          "use_cache": True,
          "wait_for_model": True
      }
}
response = requests.post(API_URL, headers=headers, json=payload)

Thanks!