Intel

/

gemma-4-31B-it-int4-AutoRound

like 50

Follow

Intel 4.07k

We'll see what we can do. It may be available next week, though it could take longer if we encounter additional issues.

We're a small team, and GGUF support is not our primary focus at the moment. Historically, each new model of gguf format has required some degree of manual integration and validation before it can be fully supported.

We'll see what we can do. It may be available next week, though it could take longer if we encounter additional issues.

We're a small team, and GGUF support is not our primary focus at the moment. Historically, each new model of gguf format has required some degree of manual integration and validation before it can be fully supported.

Believe me, this answer is the best I could even expect since now I know that it's a big deal. I threw my money to the wind on vast.ai in an attempt to solve this issue.

So, yes. I'll be waiting. I don't know how you did this miracle but Google doesn't want to hear my voice. Their QAT is inaccurate. They ruined the entire model knowledge base. And that's why I try to keep this exactly qaunt indifferent formats for various use cases. This AutoRound quant is the only working quant which keeps the original model's intellect and general knowledge base.

I reported this issue here:

https://huggingface.co/google/gemma-4-31B-it-qat-w4a16-ct/discussions/4

I cross my fingers in hope that you'll release these wights in different formats.

try this one Intel/gemma-4-31B-it-q4km-AutoRound-preview. But I don't think there is a big difference with the one provided by others. I am trying to tune another one.
BTW, you can tune other recipes via AutoRound as AutoRound has already supported it

try this one Intel/gemma-4-31B-it-q4km-AutoRound-preview. But I don't think there is a big difference with the one provided by others. I am trying to tune another one.
BTW, you can tune other recipes via AutoRound as AutoRound has already supported it

Thank you very much for your efforts!!! No doubt this quant is better than other GGUFs and significantly better than the original QAT from Google. But it still possesses lacks. Possible there is a difference in how vLLM and llama cpp deals with the weights.

I don't know how exactly vLLM works with the original AutoRound quant, but the original quant is astonishingly more accurate (especially in long tail... accurate dates, names, and other general knowledge).

Your original quant is almost identical to the original weights.

But anyway, even this GGUF is better than the original QAT from Google in W4A16.

P.s.: It differs, really differs from others. It keeps some crucial details. I'll undertake more tests. Now It's only my first impressions.

This model is not generated using the tuning mode that was applied to the INT4 model. I am running the tuning model for gguf

Differences between GGUF Q4_K_M and the INT4 model:
1 Q4_K_M uses mixed-bit quantization, which generally provides higher accuracy at the cost of a larger model size.
2 Q4_K_M uses smaller group sizes and double quantization, offering a better trade-off between accuracy and model size, although inference is typically slower.
3 The INT4 model does not quantize the embedding and LM head layers, which helps preserve accuracy at the cost of larger model size

In theory, the INT4 model could be converted to GGUF Q4_0 format, but such a conversion code is not currently supported.
Alternatively, you can use AutoRound to regenerate a similar GGUF Q40 model. However, this requires understanding the quantization details and tuning configuration. In addition, the tuning process is not fully deterministic due to several factors, so the regenerated model will not be identical to the original one, although it should achieve similar quality.