Gemma 4 26B-A4B-it + DFlash, FP8 experts — radiance container
google/gemma-4-26B-A4B-it as a single .rad
container for the radiance inference engine (AMD RDNA4,
ROCm), with z-lab's DFlash block-diffusion
drafter merged in for speculative decoding. radiance's Gemma 4 26B-A4B support is by Perire.
| File | gemma4-26b-a4b-dflash-fp8.rad — 33.94 GiB |
| Weights | the routed experts' gate_up projections FP8 E4M3 per 128×128 block (bf16 scales); their down projections, the attention, the dense MLP, the router, the norms and the tied embedding bf16 as the checkpoint ships them |
| Speculator | DFlash, drafting 15 tokens a step. Its linears and its head are FP8 E4M3 per 128×128 block; the drafter only proposes, and the model verifies every token it emits |
| Context | 262,144 tokens (the checkpoint's trained length) |
| Modalities | text |
| Needs | radiance 1.5.0 or newer |
Serve
radiance --model gemma4-26b-a4b-dflash-fp8.rad --tp 2 --max-model-len 65536 --kv-cache-dtype fp8 \
--max-num-seqs 8 --max-num-batched-tokens 2048 --draft-window adaptive --host 0.0.0.0 --port 8000
The Docker image is stilldeadcode/radiance. The server speaks the OpenAI API (/v1/chat/completions,
/v1/completions), with tool calls and structured output. --draft-window adaptive lets the drafter's
window follow how much of it is accepted; without it every step drafts the container's 15 tokens.
Measured on two Radeon AI PRO R9700 (gfx1201), one sequence:
| prose | code | |
|---|---|---|
| fixed window (15) | 113.7 tok/s | 216.7 tok/s |
--draft-window adaptive |
159.0 tok/s | 249.3 tok/s |
An 11.5K-token prompt is prefilled in 1.0 s.
How this file was made
rad-convert google/gemma-4-26B-A4B-it --draft-model z-lab/gemma-4-26B-A4B-it-DFlash \
--recipe gemma4-26b-a4b-df.recipe -o gemma4-26b-a4b-dflash-fp8.rad
The recipe (gemma4-26b-a4b-df.recipe in this repository) names what is quantised; everything it does
not name is the checkpoint's bf16:
blk.*.ffn_gate_up_exps.*.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
dflash.fc.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
dflash.blk.*.attn_q.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
dflash.blk.*.attn_k.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
dflash.blk.*.attn_v.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
dflash.blk.*.attn_output.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
dflash.blk.*.ffn_gate_up.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
dflash.blk.*.ffn_down.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
dflash.draft_head.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16