axteam/gemma-4-31B-it-AWQ-8bit

gemma-4-31B-it · AWQ INT8 (W8A16)

8-bit quantization of google/gemma-4-31B-it (commit 842da37), produced for efficient vLLM serving. Quantized by axteam.

Base model	google/gemma-4-31B-it (commit 842da37)
Method	AWQ INT8 (W8A16)
Format	compressed-tensors pack-quantized
Weights / activations	W8A16 — INT8 symmetric weights, group size 32, BF16 activations
Architecture	Gemma 4 · Dense
Size on disk	~32.7 GB
Calibration	256 samples (Korean-heavy; -it includes 32 self-generated thinking-mode answers)
License	Gemma 4 / Apache-2.0

Quantization details

• Scheme — W8A16 — INT8 symmetric weights, group size 32, BF16 activations, stored as compressed-tensors pack-quantized.
• Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms. These are left unquantized to preserve quality (a small fraction of total parameters).
• Tokenizer / chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.

Quantization method — PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining, so it stays faithful to the original model's behaviour.

• RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Used only as an internal baseline here, not published.
• AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
• QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: AWQ — a calibration-based PTQ (256-sample calibration, no gradient).

Evaluation (measured)

Measured against the BF16 original on an internal mixed English/Korean set. Quantization variant: AWQ attention.

Metric	This model	BF16 original
Perplexity (lower is better)	1.1127	1.1120 (+0.1%)
Top-1 next-token agreement	99.2%	99.4%

These numbers measure fidelity to the base model, not absolute task scores.

Usage (vLLM)

vllm serve axteam/gemma-4-31B-it-AWQ-8bit

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/gemma-4-31B-it-AWQ-8bit")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.chat(
    [{"role": "user", "content": "Write a short joke about saving RAM."}],
    params,
)
print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64. Thinking mode: add the <|think|> token at the start of the system prompt to enable step-by-step reasoning.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

vllm serve axteam/gemma-4-31B-it-AWQ-8bit --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.

---

한국어 요약

google/gemma-4-31B-it 를 AWQ INT8 (W8A16) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.

• 방식 — INT8 대칭, 32개 묶음, 활성화는 BF16(W8A16). 같은 크기에서 원본에 가장 가까움. 형식은 compressed-tensors pack-quantized.
• BF16 유지 — 비전 타워·비전 투영·토큰 임베딩(lm_head와 묶임)·MoE 라우터·노름 (전체의 극히 일부).
• 토크나이저·chat_template·processor·generation_config 는 원본과 바이트까지 동일 (구글이 2026-07에 고친 chat_template 그대로). config.json 은 원본 + quantization_config.
• 양자화 방식(PTQ vs QAT) — 학습 후(PTQ) 방식이라 재학습 없이 원본 가중치를 저비트로 바꿉니다. RTN=보정 없이 반올림만(가장 빠르나 이상치 보상이 안 돼 4비트 품질 최저, 내부 기준용·비공개) · AWQ=소량 보정으로 큰 활성화와 곱해지는 중요 채널을 채널 스케일로 보호한 뒤 반올림(경사 없음, 4비트서 RTN보다 훨씬 좋음) · QAT=양자화를 흉내 내며 재학습해 가중치가 오차에 적응(저비트 최상이나 학습 스택·데이터·GPU 필요, 구글 Gemma QAT 체크포인트는 별도 — 본 레포는 QAT 아님). 저비트 품질 ≈ QAT ≥ AWQ > RTN. 본 빌드: AWQ — 보정 기반 PTQ(256문항 보정, 경사 없음).
• 품질(실측) — BF16 원본 대비 perplexity 1.1127(원본 1.1120, +0.1%), top-1 일치도 99.2%(원본 99.4%). 내부 영/한 평가셋에서 측정. 변형: AWQ attention.
• 사용 — vllm serve axteam/gemma-4-31B-it-AWQ-8bit. 권장 샘플링 temperature=1.0, top_p=0.95, top_k=64.
• KV 캐시 절감(선택, 서빙 시) — 기본은 BF16 KV(양자화 체크포인트 표준, RedHat·NVIDIA 동일). 가중치 양자화와 무관한 런타임 선택. 긴 문맥·대배치에선 vllm serve … --kv-cache-dtype fp8 로 KV 메모리 절반. (GGUF라면 llama.cpp --cache-type-k q8_0 --cache-type-v q8_0)
• 라이선스 — 구글 Gemma 4 파생물. Gemma 4 라이선스 적용. 원본 가중치·토크나이저·chat_template 권리는 Google DeepMind.