Nexesenex/Mistral-Large-Instruct-2411-iMat-CQ-GGUF

E60/EQ60 means embeddings in ggml_type q6_0, thus only compatible with IK_Llama.CPP and Croco.CPP until this q6_0 quant eventually reaches mainline. E50/EQ50 or no mention means embeddings in mainline ggml_types (iq4_xs or Q5_0, usually)

For the IQ3 145L quantized model, the peformances close to IQ4_XS, with a few gigabites torn down. Best suited for 24-24-16GB GPU configs at 24-32k context with KV cache q6_0/q5_0 Or for more, of course.

For the IQ3_XSSL quantized model, the performances are probably akin to IQ3_XS.

These quants are made for my own use, and I decided to share them. Nothing special about them, except that they suit my needs.

Basically, my quant strategies obey to a few rules diverging from mainline.

• I often dump attn_q by one degree of quant, like mainline does for iq3_xxs in iq2_s, as well as attn_output.
• I often up attn_k and attn_v by one degree of quant, for example. Mainline usually neglects too much those tensors in the GQA era.
• I bump the embeddings, because they do not offload on the GPU (aside for Bitner and Maybe Gemma).
• I sometimes bump a whole FFN_down by one degree of quant, or down some layers of FFN_up and FFN_gate by one degree of quant.