This model requires a custom branch for GGUF
#3
by
Mihaiii
- opened
Were the GGUF created with the instructions from https://github.com/deepseek-ai/deepseek-LLM#ggufllamacpp ?
No, they were created using this other llama.cpp PR which enables making a GGUF using the Hugging Face AutoTokenizer, thus it uses tokenizer.json directly: https://github.com/ggerganov/llama.cpp/pull/3633
I wasn't aware of that other PR and I'm not sure what difference, if any, there would be in the output of the model from using that PR vs the one I did use.
Ah, I see. The sampling results are a little bit disappointing and I was hoping is a conversion issues, but all seems good on this side. :)
Closing as addressed. Thanks!
Mihaiii
changed discussion status to
closed