I'm using BLOOMZ to generate text and intend or desire it to be "unending" in that I want to suppress the stop token. I can easily remove it from the result text but I'm also storing the output's tensor data. I was hoping there was a way to remove the stop token from the results without re-encoding the text.
- Encode input text into tensors
- Generate text (results_text)
- Remove input text from results_text
- Save results_text + input tensors (at this step)
In step #4 I want to remove the stop token from the input tensors I have collected in my variable. But my mind tells me I need to decode, remove it, re-encode it when I'm sure there is a way to represent the tensor or remove it from my variable another way?
Do you want the model to not stop?
In that case you can set
model.generate so it generates for your desired length. 👍
Or do you want to remove the stop token from the generation?
If so setting
skip_special_tokens=True is the way to go
I'm really looking for open-ended fragments, but when it decides to stop it stops. I'm using
max_length now, I've not tried
max_new_tokens on generate yet. Seems I might still accidentally reach the end if the new tokens/length is less than what the model considers 'done'.
Ah I meant
min_new_tokens, sorry - I.e. you can set
min_new_tokens to force a minimum number of generated tokens during which the eos token is ignored.