Koshkasa/DarkArtsForge_Zepar-24B-v1_IQ3_KT_attn

Model Card for Model ID

This is a mixed quantization of DarkArtsForge/Zepar-24B-v1 leveraging ik_llama.cpp trellis quantization as well as IQ_K quants, created to test various approaches to trellis quantization of medium weight models.
I also included the f16 GGUF for quicker quantization, as well as the imatrix I generated and used.

Quant Details

iq3_kt: ffn_gate, ffn_up (80)
iq4_kt: ffn_down, token_embd (41)
iq5_k: attn_q, attn_v, attn_k, attn_output (160)
iq6_k: output (1)
imatrix created based on calibration data by bartowski

quantized with ik_llama.cpp build: 9d07d868

incompatible with mainline llama.cpp
only tested with ik_llama.cpp (though I expect croco.cpp to work as well?)

Rationale

I wanted to run the model and there were no quants at the time I started baking, simple as. While we're at it, I tried testing a mixed trellis quantization of ffn, applying more precision to ffn_down. The massive byte budget savings allowed to quant attention up, and the resulting file is still small. KVO with 65k ctx in tolerable cache quantization fits even with system overhead. Why do that? Idk, but it's fun.

In conclusion

-Is it worth it?
No idea. I made it for personal use and as an experiment. It didn't loose too much brain based on quick benchmarks (winogrande = 78.5320, hellaswag(s=42) = 329/400). It loads (yay?), works with my preferred sampler settings (yay?), and outputs actual text (yay?). A quick 32k ctx run showed it remembers details from before and can write coherently about them. I'm yet to test it thoroughly in RP. WYSIWYG.

-Possible 12G VRAM full fit with KVO?
Probably. A mixed IQ3_KT ffn / IQ4_KT attn would likely fit with some context headroom. By my observations, it remains smart enough to be actually suprisingly usable, while keeping the bulk weight ffn tensors at 3.125 bpw. I have no use for such a quant, but I can bake one if someone needs it

Cheers

MistralAI - the beloved base model(s).
ikawrakow and contributors of ik_llama.cpp - I probably misused your wonderful creation.
DarkArtsForge/Naphula - for the effort of designing and producing the merge. And for reviving my interest in Mistral 24b.
+++ Everyone whose finetunes were included in the merge!
bartowski - for the calibration data + the myriad of quants we all benefit from.

Disclosure

imatrix generated on an intermediary q8_0 quant. I don't eat RAM for breakfast.
My only contribution is compute. This is neither my merge, nor my calibration data. Have fun.