Livesport/d3-flash-GGUF

d3-flash — GGUF

llama.cpp port of vllm-sr/d3-flash, the 9B model of Decision 3.0. It does not generate text: it answers typed questions about a shared state (noul / choice / score) and returns a probability for every option, in the same request shape as TypeSafe's Jev.

The upstream checkpoint is bf16 safetensors (~17 GB) that needs torch and flash-linear-attention. These files run under stock llama-server, which serves them on /v1/systemone natively — the decision metadata, the answer codes and the prompt template are inside the GGUF.

file	size	KL vs bf16	decision flips
d3-flash-Q8_0.gguf	8.87 GiB	0.000193	2 / 371
d3-flash-Q5_K_M-dyn.gguf	6.58 GiB	0.000866	5 / 371
d3-flash-Q4_K_M-dyn.gguf	5.96 GiB	0.001864	7 / 371
d3-flash-Q4_K_M.gguf	5.24 GiB	0.003530	6 / 371
d3-flash-imatrix.gguf	10 MB	—	—

-dyn keeps ssm_out and the attention projections (attn_qkv, attn_output, attn_gate) at q8_0. The 255-way decision head travels inside the GGUF as output.weight, expanded to one row per answer-code token; readout.safetensors and decision_config.json are shipped as well so any other runtime can load the head on its own.

Three things to know before deploying this

1. The attention is bidirectional, so there is no prefix to cache. attention_mode of the checkpoint is noncausal_full_attention: 8 of the 32 layers run softmax attention with the causal mask removed, the other 24 are linear (Gated DeltaNet) and keep theirs. The prompt is ordered State → Question → Options, so the state's representation depends on which question follows it. Each question is a full forward pass over the whole state.

This has a consequence for llama-server as of today's master: its decision path groups the questions of one request and lets them share the state's KV prefix, which is correct for a causal decision model and wrong for this one. Measured against the torch reference on 143 questions:

	mean |dp|	max |dp|	flips
stock server, multi-question request	0.00844	0.289	0 / 143
one question per request	0.00067	0.0069	0 / 143
patched server, multi-question request	0.00103	0.029	0 / 143

runtime/llama.cpp-noncausal-no-shared-prefix.patch makes can_share_prompt() return false when the context is non-causal. Without it, send one question per request.

2. Flash attention decides whether it fits a small card. The whole prompt goes through in one ubatch (chunking would cut the bidirectional window), so the attention scratch is quadratic in the prompt without FA. Measured on a GB10 with -np 1, Q4_K_M:

ctx	model	KV (8 layers)	recurrent	compute, -fa on	compute, -fa off	total, -fa on
2 048	4 812 MiB	64 MiB	50 MiB	424 MiB	480 MiB	5 350 MiB
4 096	4 812 MiB	128 MiB	50 MiB	864 MiB	1 504 MiB	5 854 MiB
8 192	4 812 MiB	256 MiB	50 MiB	1 792 MiB	5 184 MiB	6 910 MiB
16 384	4 812 MiB	512 MiB	50 MiB	3 841 MiB	19 072 MiB	9 215 MiB

So Q4_K_M fits an 8 GB card up to 8k tokens of context with flash attention; without it, only up to about 4k. The KV cache itself is cheap because only every fourth layer is attention.

3. It accepts any length and never truncates. Unlike Clef-Flash there is no silent cut-off, but a long state is expensive: each question re-reads it. Measured on a GB10 (torch runtime, bf16): 0.28 s for one question over 600 tokens, 1.63 s for five; 1.46 s / 6.43 s over 2.5k tokens. Pick this model for short states and for verdicts that matter, not for ten questions over a long document.

Quantization

llama-imatrix over 2 482 prompts (1.4 M tokens) rendered with the model's own d3_format.render and chat template from real Czech and English states, with a mix of noul / choice / score questions. Calibration in the production prompt shape, not plain prose.

The quality metric is KL divergence on the decision distribution, not token perplexity: the model has no language head, so llama-perplexity measures something it never does (the perplexity reported during imatrix collection is ~2·10⁵ and is meaningless). KL is measured against the bf16 GGUF on 371 held-out questions, a different seed from the calibration corpus.

A paired bootstrap (10 000 resamples over the same 371 questions):

pair	ΔKL	95 % CI	verdict
Q8_0 − Q5_K_M-dyn	−0.000673	[−0.001096, −0.000382]	real
Q5_K_M-dyn − Q4_K_M-dyn	−0.000998	[−0.001514, −0.000561]	real
Q4_K_M-dyn − Q4_K_M	−0.001666	[−0.002572, −0.000829]	real
Q4_K_M-dyn − Q5_K_M	−0.000637	[−0.001702, +0.000204]	not distinguishable

Plain Q5_K_M is therefore not published: at 6.02 GiB it is larger than Q4_K_M-dyn with no measurable advantage. Unlike the pplx-decider port, where a layout borrowed from another model of the family did not transfer, here the -dyn layout is worth its extra bytes at both Q4 and Q5.

Port fidelity

Before any quantization, the bf16 GGUF was checked against the original torch checkpoint on 143 Czech and English questions: mean |dp| 0.00067, max 0.0069, 0 flipped decisions (one question per request; see the table above for the shared-prefix caveat).

Running it

Stock llama-server from a build that knows the pplx-decider decision type:

llama-server -m d3-flash-Q4_K_M.gguf -c 8192 -b 8192 -ub 8192 -np 1 -fa on -ngl 99

-b/-ub must cover the whole prompt — chunked prefill would cut the bidirectional attention window at every chunk boundary.

curl -s localhost:8080/v1/systemone -H 'Content-Type: application/json' -d '{
  "state": "Spartu porazila Plzeň 2:1.",
  "questions": {
    "won":   {"type": "noul", "instructions": "Vyhrála Plzeň?"},
    "topic": {"type": "choice", "instructions": "Classify the state.",
              "criteria": {"sport": "sport", "politics": "politics", "other": "anything else"}}
  }}'

Converting the checkpoint yourself needs runtime/conversion_d3.py (drop it into conversion/ of llama.cpp as d3.py, register "D3Model": "d3" in conversion/__init__.py); it reuses the pplx-decider converter and adds d3's prompt, plus attention.causal = false from decision_config.json.

How it compares

Measured on identical sets against our own deployments (Rune 26B-A4B IQ3_S, Clef-Flash Q5, Decision 2.0 Lux-9B Q4) on a GB10:

	d3-flash	Rune	Clef-Flash	Lux-9B
7 580-question suite, 12 tasks	79.3 %	76.3 %	76.2 %	76.5 %
— English	86.6 %	82.4 %	82.6 %	83.6 %
— Czech	66.6 %	65.7 %	64.9 %	64.2 %
ECE	0.062	0.052	0.093	0.079
hard set (Czech sarcasm, threats)	84.8 %	93.5 %	71.7 %	73.9 %
BoolQ	92.9 %	81.8 %	89.8 %	89.4 %
shared prefix across questions	no	yes	yes	yes

It is the strongest model we have measured on the broad suite and on English, and the first 9B that holds its own against Rune in Czech. Rune stays ahead on Czech fact verification (CTKFacts 72.9 vs 68.6) and on the hard Czech set; d3-flash costs one full pass per question, which Rune and Clef do not.

Licence

Apache 2.0, inherited from the upstream checkpoint.