Gemma-4-31B-IT — NVFP4 weights for the Nomos kernel
Kernel: github.com/lightofbaldr/nomos-nvfp4 (MIT) — a pure-Mojo inference kernel for Gemma-4-31B: 4-bit weights and activations, lossless speculative decoding via DFlash, tensor-core batch verify on NVIDIA Blackwell.
The exact weight set that kernel loads, converted from
google/gemma-4-31b-it into the kernel's native
NVFP4 layout. Published so that anyone running the kernel gets the same setup we run locally,
without having to reproduce the conversion.
This model is not fine-tuned. It is a format conversion of Gemma-4-31B-IT. Nothing has been trained, merged, or distilled into it.
What this is
| Source | google/gemma-4-31b-it (Google, QAT) |
| Format | NVFP4 (4-bit weights, W4A4-capable) — kernel-native, not HF safetensors |
| Files | 411 .nvfp4 tensors + 422 .bin (norms, scalars, embeddings) |
| Size | 21.3 GB |
| Layers | 60 · hidden 5376 · intermediate 21504 · vocab 262144 |
| Attention | 50 sliding layers (NH 32 / KV 16 / hd 256), 10 global (NH 32 / KV 4 / hd 512) |
Verify you have the right bytes:
layers_0_self_attn_q_proj_weight.nvfp4 sha256 starts b4908eb2
layers_30_mlp_down_proj_weight.nvfp4 sha256 starts 3fa309a8
embed_tokens_weight.nvfp4 sha256 starts 64dbb6e2
Performance — reported straight
Single-stream decode, one card. Two numbers, because only quoting the good one would be misleading:
| configuration | tok/s |
|---|---|
| our NVFP4 base decode alone | ~21.7 |
| llama.cpp Q4_0, same card | ~30 |
| our speculative-decode stack | ~50 |
| RTX 5090, lossless (measured separately) | 53.5 — +16.8% head-to-head vs llama.cpp |
Base decode alone is slower than llama.cpp's mature Q4_0 and we are not going to pretend
otherwise. The win comes from the speculative-decode stack, which reaches ~50 tok/s against
llama.cpp's realistic single-stream ceiling of ~30 (1.67×). That comparison is stack-vs-stack:
vanilla llama.cpp has no valid standalone Gemma-4-31B drafter, because the available drafters are
coupled next-n heads with no independent K/V and therefore cannot be passed as --model-draft.
These are our measurements on our hardware. Reproduce them before relying on them.
Using it
These weights are for the Nomos kernel, not transformers. They will not load with
AutoModelForCausalLM — the layout is the kernel's own flat NVFP4 format.
git clone https://github.com/lightofbaldr/nomos-nvfp4 && cd nomos-nvfp4
pixi install
bash refresh_build.sh
# this repo — target weights at the root, drafter in drafter/
hf download Adam1010/nomos-gemma-4-31b-nvfp4 --local-dir ~/nomos_data/gemma-4-31b-nvfp4
export WEIGHTS=~/nomos_data/gemma-4-31b-nvfp4/
export DFLASH_DIR=~/nomos_data/gemma-4-31b-nvfp4/drafter/ # TRAILING SLASH REQUIRED
./nomos bench # base decode
./nomos bench --spec # with speculative decoding
./nomos smoke --prompt "hello"
Serving:
export NOMOS_SERVE_SPEC=1
pyhost/serve_nomos.sh up
curl -s localhost:$NOMOS_PORT/health
Tokenizer is not included. Use the tokenizer from google/gemma-4-31b-it; the kernel loads it
via TOK_DIR. Gemma-4 frames turns with <|turn> / <turn|> (ids 105/106) and has no
<start_of_turn> — much published Gemma guidance describes the older scheme and does not apply to
this model. Always render prompts with the tokenizer's own apply_chat_template.
The drafter in drafter/ is DFlash by Z Lab (Chen, Liang, Liu), Apache-2.0 —
not our work; we converted the format and built the NVFP4-native lossless verify path. See
drafter/README.md for credit, citation and terms.
Thinking channel: Gemma-4 relies on its thought channel. Suppressing it, or truncating it with a
small max_tokens, produces output that looks like a numerics bug and is not one.
Contents
| path | what | licence |
|---|---|---|
*.nvfp4, *.bin |
Gemma-4-31B-IT target weights, NVFP4 | Gemma Terms of Use |
drafter/ |
DFlash drafter by Z Lab, converted | Apache-2.0 (+ Gemma terms in practice) |
Licence
The target weights are governed by the Gemma Terms of Use. Use is subject to Google's Gemma Prohibited Use Policy. By downloading you agree to those terms. This is a derivative of Gemma-4-31B-IT; all rights in the underlying model remain Google's. Redistribution must carry these terms forward.
Provenance
Converted from the Google QAT release. The source weight set was independently verified across two machines before conversion, and the spot hashes above are the identity check we use internally — we had a silent weight-corruption incident once and now treat byte-level verification as mandatory rather than optional.