The model keeps generating until max_new_token limit is reached
#1
by
rjmehta
- opened
The model answers perfectly fine but doesn't honor EOS token. Using transformer 4.34.0 and latest exllama2. The model answers fine at start but generates until max new tokens and doest stop at EOS. Is the eos token id 2?
Not sure what the id is but in the config the eos token is /s in brackets <>
However i also see a lot of templates https://huggingface.co/TheBloke/Airoboros-L2-70b-2.2.1-GPTQ discussed here. Id go with the discussion format.