TheBloke

/

rocket-3B-GGUF

like 34

llama.cpp's ./main example uses the EOS token stored in the GGUF to figure out when to stop.

For context, convert-hf-to-gguf.py uses transformers.AutoTokenizer which uses settings from tokenizer_config.json.
The EOS token was wrong (but was recently fixed upstream), so the output never seemed to end when I first tried this model.

Consider re-converting this model so that the GGUF files contain the correct EOS token (which should be <|im_end|> (aka token id 50279) for this model).