Should we expect a version for GGUF?
This only works with GPTQ and EX.
For this reason, two and a half people from the entire community can take advantage of this, and most likely this is not a joke.
Is there a chance that you will make a version for GGUF? In this case, a lot of people will be able to use it, at least me and my friend who discussed this Lora.
Hi there, I don't know much about GGUF, but seeing on the quant method they use, they don't use a calibration dataset (It seems since their sizes are fixed it goes the same way for every model). So sadly I think it's not possible or I don't know how it could be done.
You can try this https://huggingface.co/tu9jn/Goliath-120b_SOTA_GGUF/tree/main
The uploader used ppipa_rp, so it is probably similar to this model, https://www.reddit.com/r/LocalLLaMA/comments/197mip0/comment/ki37saa/?utm_source=share&utm_medium=web2x&context=3