axteam/gemma-4-31B-FP8-dynamic

gemma-4-31B · FP8 (dynamic)

8-bit quantization of google/gemma-4-31B (commit 5bbc2fb), produced for efficient vLLM serving. Quantized by axteam.

Base model	google/gemma-4-31B (commit 5bbc2fb)
Method	FP8 (dynamic)
Format	compressed-tensors FP8_DYNAMIC
Weights / activations	W8A8 — FP8 E4M3 weights (per-output-channel scale), dynamic per-token FP8 activations
Architecture	Gemma 4 · Dense
Size on disk	~31.0 GB
Calibration	None (data-free static weight scales)
License	Gemma 4 / Apache-2.0

Quantization details

• Scheme — W8A8 — FP8 E4M3 weights (per-output-channel scale), dynamic per-token FP8 activations, stored as compressed-tensors FP8_DYNAMIC.
• Kept in BF16 — vision tower, vision projection, token embeddings (tied with lm_head), MoE router, and all norms. These are left unquantized to preserve quality (a small fraction of total parameters).
• Tokenizer / chat_template / processor_config / generation_config are byte-identical to the original, including Google's July-2026 corrected chat_template. config.json is the original plus a quantization_config block.

Quantization method — PTQ (AWQ / RTN) vs QAT

This model is a post-training quantization (PTQ): the released BF16 weights are quantized after training, with no retraining, so it stays faithful to the original model's behaviour.

• RTN (round-to-nearest) — weights are simply rounded to the quantization grid with per-group scales; no calibration data. Fastest, but outliers aren't compensated, so 4-bit fidelity is the lowest. Used only as an internal baseline here, not published.
• AWQ (activation-aware weight quantization) — a small calibration set finds the weight channels that multiply the largest activations; those salient channels are protected by per-channel scaling (the scale is folded into adjacent layers, e.g. LayerNorm → q/k/v) before rounding. Still PTQ (no gradient, seconds–minutes), and far better than RTN at 4-bit.
• QAT (quantization-aware training) — the model is fine-tuned with quantization simulated in the forward pass, so the weights adapt to quantization error via gradient descent. Best low-bit fidelity, but needs the full training stack, data and GPU hours. Google ships separate Gemma QAT checkpoints; this repo is not one of them.

Rough low-bit fidelity: QAT ≥ AWQ > RTN. Cost: QAT ≫ AWQ > RTN. This build: FP8 dynamic — a calibration-free PTQ (static per-channel weight scales + runtime per-token activation scales).

Evaluation

Quantization fidelity was checked against the BF16 original using perplexity and top-1 next-token agreement on an internal mixed English/Korean set. In our runs the 8-bit variants (AWQ INT8 / FP8) stay closest to the original, while the 4-bit variants (AWQ INT4 / NVFP4) trade a little quality for roughly half the footprint.

Usage (vLLM)

vllm serve axteam/gemma-4-31B-FP8-dynamic

from vllm import LLM, SamplingParams

llm = LLM(model="axteam/gemma-4-31B-FP8-dynamic")
params = SamplingParams(temperature=1.0, top_p=0.95, top_k=64, max_tokens=1024)
out = llm.chat(
    [{"role": "user", "content": "Write a short joke about saving RAM."}],
    params,
)
print(out[0].outputs[0].text)

Recommended sampling (per Google): temperature=1.0, top_p=0.95, top_k=64. Thinking mode: add the <|think|> token at the start of the system prompt to enable step-by-step reasoning.

Reducing KV-cache memory (optional, serve-time)

The KV cache is unquantized (BF16) here — the standard for quantized checkpoints (RedHatAI, NVIDIA, etc. ship the same way). KV-cache precision is a runtime choice, independent of the weight quantization above. For long context or large batches you can roughly halve KV-cache memory with FP8:

vllm serve axteam/gemma-4-31B-FP8-dynamic --kv-cache-dtype fp8

--kv-cache-dtype accepts auto (default, BF16), fp8 / fp8_e4m3, fp8_e5m2. For a GGUF build the llama.cpp equivalent is --cache-type-k q8_0 --cache-type-v q8_0.

License

This is a quantized derivative of Google's Gemma 4. Use is governed by the Gemma 4 license. All original weights, tokenizer, and chat template belong to Google DeepMind.

---

한국어 요약

google/gemma-4-31B 를 FP8 (dynamic) 로 양자화한 서빙용(8-bit) 모델입니다. 작성: axteam.

• 방식 — FP8 E4M3. 가중치는 출력 채널마다 스케일 하나(정적·보정 불필요), 활성화는 추론 시 토큰마다 동적 FP8. 형식은 compressed-tensors FP8_DYNAMIC.
• BF16 유지 — 비전 타워·비전 투영·토큰 임베딩(lm_head와 묶임)·MoE 라우터·노름 (전체의 극히 일부).
• 토크나이저·chat_template·processor·generation_config 는 원본과 바이트까지 동일 (구글이 2026-07에 고친 chat_template 그대로). config.json 은 원본 + quantization_config.
• 양자화 방식(PTQ vs QAT) — 학습 후(PTQ) 방식이라 재학습 없이 원본 가중치를 저비트로 바꿉니다. RTN=보정 없이 반올림만(가장 빠르나 이상치 보상이 안 돼 4비트 품질 최저, 내부 기준용·비공개) · AWQ=소량 보정으로 큰 활성화와 곱해지는 중요 채널을 채널 스케일로 보호한 뒤 반올림(경사 없음, 4비트서 RTN보다 훨씬 좋음) · QAT=양자화를 흉내 내며 재학습해 가중치가 오차에 적응(저비트 최상이나 학습 스택·데이터·GPU 필요, 구글 Gemma QAT 체크포인트는 별도 — 본 레포는 QAT 아님). 저비트 품질 ≈ QAT ≥ AWQ > RTN. 본 빌드: FP8 동적 — 보정 불필요 PTQ(가중치는 정적 채널 스케일, 활성화는 런타임 토큰별).
• 품질 — BF16 원본 대비 perplexity·top-1 일치도로 검증(영/한 혼합셋). 8비트(AWQ INT8·FP8)가 원본에 가장 가깝고, 4비트(AWQ INT4·NVFP4)는 용량을 절반으로 줄이는 대신 품질을 조금 양보.
• 사용 — vllm serve axteam/gemma-4-31B-FP8-dynamic. 권장 샘플링 temperature=1.0, top_p=0.95, top_k=64.
• KV 캐시 절감(선택, 서빙 시) — 기본은 BF16 KV(양자화 체크포인트 표준, RedHat·NVIDIA 동일). 가중치 양자화와 무관한 런타임 선택. 긴 문맥·대배치에선 vllm serve … --kv-cache-dtype fp8 로 KV 메모리 절반. (GGUF라면 llama.cpp --cache-type-k q8_0 --cache-type-v q8_0)
• 라이선스 — 구글 Gemma 4 파생물. Gemma 4 라이선스 적용. 원본 가중치·토크나이저·chat_template 권리는 Google DeepMind.