Gemma GGUF does not work!
#3
by
cohesionet
- opened
Hi all, Is there any one who is able to successfully run quantized gemma model? I tried a number of gguf models generated from different people including @mmnga , @mlabonne , mine @cohesionet . None of them works. I have not try @MaziyarPanahi yet.
Hi @cohesionet
So this issue I opened on Llama.cpp led into this PR: hopefully, once it's merged I can re-run the quant and update the weights: https://github.com/ggerganov/llama.cpp/pull/5650
any gguf work properly on llama.cpp？
@wukongai I am sorry, aren't these working properly? https://huggingface.co/MaziyarPanahi/GritLM-8x7B-GGUF/tree/main