BCCard/MoAI-gemma-4-12B-it-mxfp4

Gemma 4 12B IT — MXFP4 Quantized

google/gemma-4-12B-it quantized to MXFP4 (W4A16) using GPTQ with special techniques that preserve Korean and English performance simultaneously.

Model Details

Property	Value
Base model	google/gemma-4-12B-it (BF16, ~24.4 GB)
Quantization format	MXFP4 W4A16 (compressed-tensors)
Disk size	7.9 GB (68% reduction from BF16)
VRAM (model weights)	~8 GB
Quantized scope	Decoder Linear layers (lm_head, embed_vision, embed_audio kept in BF16)
Calibration data	BCCard on-policy dataset (Korean/English split)

The 12B checkpoint is the gemma4_unified architecture variant — its vision stack (embed_vision.*) differs from the 31B's vision_tower, and is kept at original precision.

Serving

vLLM (recommended)

# Standard environment (CUDA 12.9+):
vllm serve gemma-4-12B-it-MXFP4 \
  --port 8000 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

# Blackwell with CUDA < 12.9:
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve gemma-4-12B-it-MXFP4 \
  --port 8000 \
  --linear-backend marlin \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Blackwell Compatibility

On Blackwell GPUs (SM 12.x) with containers running CUDA < 12.9, FlashInfer JIT compilation fails. The following workarounds are required:

Setting	Purpose
VLLM_USE_FLASHINFER_SAMPLER=0	Disables FlashInfer sampler (uses native PyTorch)
--linear-backend marlin	Uses pre-compiled MXFP4 GEMM kernels (no JIT)

Component matrix after workarounds:

Component	Backend	JIT Required
MXFP4 GEMM	Marlin (pre-compiled)	No
Attention	TRITON_ATTN	No (Triton JIT, not nvcc)
Sampling	Native PyTorch	No

Evaluation

Quality Benchmarks

Task	MXFP4 (this model)	BF16 baseline	Recovery
HellaSwag (English)	83.5%	81.0%	103.1%
KoBEST-HellaSwag	62.0%	64.0%	96.9%
KoBEST-COPA	98.5%	98.5%	100.0%
KoBEST-BoolQ	94.0%	93.0%	101.1%
KoBEST-SentiNeg	99.0%	99.5%	99.5%
KoBEST-WiC	82.0%	79.0%	103.8%
KMMLU (5 subjects)	55.5%	60.5%	91.7%
BC-Finance QA (char-F1)	0.350	0.344	101.7%
Average recovery			99.7%
Korean Generation	495 tok, rep 1.00, Hangul 58.8%	492 tok, rep 1.00, Hangul 58.3%	—
English Generation	430 tok, rep 0.99	432 tok, rep 0.99	—

Korean Knowledge (KMMLU) — Reference Points

Published scores from other Korean(-capable) models, for orientation. These are NOT apples-to-apples: vendors use the full 45-subject KMMLU (or the harder KMMLU-Redux subset) with their own prompting/shot settings, while this model's score is a 5-subject, 200-sample, 0-shot chat-MCQ sample from the harness above.

Model	Params	Disk	Score	Benchmark (setting/source)
A.X 4.0 (SKT)	72B	~144 GB	78.3	KMMLU, vendor
EXAONE 4.0 (LG)	32B	~64 GB	75.2	KMMLU, as reported in the HyperCLOVA X 32B Think report
GPT-4o (OpenAI)	—	— (API)	72.5	KMMLU, as reported by SKT A.X
HyperCLOVA X 32B Think (NAVER)	32B	~64 GB	71.3	KMMLU, vendor report
A.X 3.1 (SKT)	34B	~68 GB	69.2	KMMLU, vendor
A.X 3.1 Light (SKT)	7B	~14 GB	61.7	KMMLU, vendor
Gemma 4 12B IT (BF16 baseline)	12B	~24.4 GB	60.5	KMMLU 5-subject sample, 0-shot chat MCQ (this harness)
This model (Gemma 4 12B MXFP4)	12B	7.9 GB	55.5	KMMLU 5-subject sample, 0-shot chat MCQ (this harness)
Gemma 3 27B IT (Google)	27B	~54 GB	54.0	KMMLU-Redux, KMMLU-Redux/Pro paper
HyperCLOVA X SEED (NAVER)	3B	~6.5 GB	48.5	KMMLU, vendor
Gemma 3 12B IT (Google)	12B	~24 GB	46.7	KMMLU-Redux, KMMLU-Redux/Pro paper

Disk sizes for third-party models are approximate BF16 weight footprints (params × 2 bytes); this repo's model and its BF16 baseline are measured on-disk values.

Directional takeaway: at 12B/4-bit this model lands well above the previous Gemma generation's 12B (and its 27B on the harder Redux subset) on Korean knowledge, while purpose-built Korean 32B+/72B models remain ahead — as expected for their size. Against its own BF16 baseline under the identical harness, the model averages 99.7% recovery across all scored tasks, with KMMLU (91.7%) the only drop beyond sampling noise.

Compression Comparison

Format	Disk Size	vs BF16	Notes
BF16 (original)	~24.4 GB	100%	google/gemma-4-12B-it
FP8-dynamic	13 GB	~53%	community quantization with the same llm-compressor FP8_DYNAMIC recipe (no official RedHatAI 12B release)
This Model (MXFP4)	7.9 GB	32%	W4A16 GPTQ + MHR + PSA

Citation

@misc{gemma4-12b-mxfp4-mhr-psa,
  title={Gemma 4 12B IT MXFP4},
  author={Taeyoung Lee},
  year={2026},
  url={https://huggingface.co/BCCard/MoAI-gemma-4-12B-it-mxfp4}
}

License

This model is licensed under the Apache 2.0.