Gemma-4-12B-it-FP8

This model is an FP8 compressed checkpoint derived from google/gemma-4-12B-it.

It was quantized with llmcompressor using dynamic FP8 quantization over linear layers while preserving sensitive multimodal and output layers.

Model Details

Field	Value
Base model	google/gemma-4-12B-it
Quantization	FP8 dynamic
Quantization tool	llmcompressor
Serving tested with	vLLM
Approx. model folder size	~13 GB
Original BF16 size	~23.9 GB

Quantization Recipe

Targeted layers:

Linear

Ignored layers/patterns:

lm_head
re:.*embed.*
re:.*router
re:.*vision_tower.*
re:.*audio_tower.*

This was done to reduce the risk of corrupting embeddings, output projection, router behavior, or native multimodal projection paths.

Intended Use

This checkpoint is intended for:

• text generation
• question answering
• retrieval/context-grounded QA
• vLLM serving experiments
• FP8-vs-FP4 quantization comparison

Although the base architecture is multimodal, the reported benchmark here is text-only.

Usage With vLLM

vllm serve /path/to/model \
  --served-model-name fp8-gemma \
  --quantization compressed-tensors \
  --dtype bfloat16 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.70 \
  --enforce-eager

For text-only deployments, block multimodal inputs at serving time:

limit_mm_per_prompt={"image": 0, "audio": 0}

HotpotQA Benchmark

Benchmark:

Dataset: HotpotQA Distractor
Slice: validation[0:500]
Model name: fp8-gemma
Successful requests: 500/500

System prompt:

Answer questions using only the provided context. Some questions require combining facts from multiple passages. Return only the shortest correct answer. When possible, copy the exact name, date, place, number, or phrase from the context. Do not explain your reasoning or add extra text.

Metric	Value
Questions	500
Successful	500
Failures	0
Exact Correct	326/500
Exact Match	65.20%
Token F1	80.38%
Exact + Partial	442/500 = 88.40%
Avg Latency	0.53s
P50 Latency	0.48s
Max Latency	1.93s
Total Output Tokens	2474
Avg Output Tokens	4.95
Output Tokens/sec	9.35

Limitations

• Quantized checkpoint; behavior may differ from the original BF16 model.
• Reported benchmark is HotpotQA Distractor validation[0:500].

License

This checkpoint is derived from google/gemma-4-12B-it. Follow the applicable Gemma 4 terms and Apache 2.0 license information from the base model.