Possibility of new version with all K quants? V2?

#1
by Spacellary - opened

Greetings! πŸ‘‹ πŸ€—

New GGUF quants could be improved if we take what's being discussed here:

https://www.reddit.com/r/LocalLLaMA/comments/1993iro/comment/kie6w9l/

Possibility of new GGUF K quants version for this nice model?

Honestly it's my favorite 7B for roleplay, I prefer it over Kunoichi-7B/Maids even.

Maybe! I'm going to let the quant discussion settle down a bit, this still seems pretty investigatory.

@SanjiWatsuki - Alrighty. Thanks for your work so far! I’m also looking forward to any new developments from you. Stay awesome!

@SanjiWatsuki - Heya! Hope everything is as well as can be.

Using Kalo's general pseudo random text to create the Imatrix data from the F16 model, added additional quants:

https://huggingface.co/Lewdiculous/Loyal-Toppy-Bruins-Maid-7B-DARE-GGUF-Imatrix

Sign up or log in to comment