gemma-4-12B-it EAGLE-3 Draft

An EAGLE-3 draft (speculator) for accelerating gemma-4-12B-it via speculative decoding. Trained from scratch on on-policy responses regenerated by the verifier itself, over a general-purpose QA prompt set (~450k prompts: English Magpie + Korean/English sh2orc).

Overview

Method: EAGLE-3 (vllm-project/speculators)
Verifier (target): BCCard/gemma-4-12B-it-FP8-Dynamic (FP8; used for serving and hidden-state extraction)
Reused weights: BF16 embed/lm_head from google/gemma-4-12B-it (standard EAGLE-3)
Warm start: none — trained from scratch (no pretrained speculator initialization)
Sequence length: 4096

Training data

~450k prompts total, from two sources:

~150k from sh2orc/bccard-maywell-jojo0217-markai-lcw99-kendamarron-microsoft — Korean / English QA
~300k from Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered — English

Only the prompt (instruction) is used; original answers are discarded and regenerated on-policy by the verifier.

Serving (vLLM)

VLLM_USE_FLASHINFER_SAMPLER=0 vllm serve BCCard/gemma-4-12B-it-FP8-Dynamic -tp 1 \
  --max-model-len 8192 \
  --speculative-config '{
    "model": "BCCard/MoAI-gemma-4-12B-it-speculator.eagle3",
    "num_speculative_tokens": 4,
    "method": "eagle3",
    "draft_tensor_parallel_size": 1
  }'

Tune num_speculative_tokens in the 4–8 range based on measured acceptance / TPS. The draft uses the verifier's tokenizer.

Performance

Per-position acceptance at training time (validation split).

position	full_acc	cond_acc
0	0.665	0.665
1	0.412	0.620
2	0.264	0.640

Mean accepted length ≈ 2.3 tokens/step, roughly ~2.3x speedup (measure on your own traffic).

Train and validation metrics match closely, so there is no overfitting.

Limitations

Trained on general-purpose QA (mostly English Magpie + Korean/English sh2orc). Domain- or language-specific traffic (e.g. Korean finance) may benefit from another training cycle on matched data, raising acceptance.
Acceptance is measured against the verifier BCCard/gemma-4-12B-it-FP8-Dynamic. Pairing the draft with a different target will change results.

License

Apache 2.0. The base Gemma 4 (Apache 2.0 since 2026-04, the first Gemma family to adopt it) and the verifier BCCard/gemma-4-12B-it-FP8-Dynamic (Apache 2.0) are both Apache 2.0, and this draft was trained from scratch (no third-party speculator weights), so it is released under Apache 2.0 as well.

Apache 2.0 requires only attribution of the original copyright and disclosure of modifications, with no restrictions on commercial use, modification, or redistribution. (This is informational, not legal advice.)