QAT versions please 🙏
Especially if you end up doing a v3 too... we NEED QAT versions for low-end hardware!!!
https://huggingface.co/collections/google/gemma-4-qat-q4-0
You should be able to just run the finetune the same way you did for the non-QAT, except you train on each qat-q4_0-unquantized version. Then community quantizations of the resulting qat-q4_0-unquantized finetune become extremely accurate, with little performance loss compared to the original FP16 weights. Gemma 4 26B-A4B Q4_0 at full 256K context fits into 24GB VRAM! It would mean so much for us to have that.
I love qat, i use UD unsloth XL (has fixes over 4.0). qat is essential so i always prefer qat over any FT
A very dubious take; QAT is almost like a fourth quantum, only lighter. There is no point in this.
A very dubious take; QAT is almost like a fourth quantum, only lighter. There is no point in this.
QAT is basically fixing what's wrong with normal quantization. QAT Q4_0 is less than 1% accuracy loss from non-QAT BF16 because they trained the model to simulate quantization errors. It learned from them in the qat-q4_0-unquantized model and you just need to finetune and quantize that one yourself.