Livesport/clef-flash-GGUF

Clef-Flash · GGUF for llama.cpp hidden-state serving

This repository is a llama.cpp build of Cloudflare/clef-flash, a 9B decision model (Qwen3.5-9B backbone with a joint schema head). It holds everything needed to serve Clef's Jev/SystemOne-compatible /v1/systemone API on a single 8 GB GPU. Livesport runs it in production that way on a Quadro RTX 4000 (Turing, 8 GB).

Clef does not generate text. Its head reads the backbone's final hidden state of every token and scores every allowed option of every question in one forward pass. The GGUF here is therefore quantized for that job: its bits go into the backbone, not into the LM head.

Not for text generation. In these GGUFs output.weight (the LM head) is deliberately stored at Q2_K, because the Clef pipeline never uses its logits. The head takes the option tokens' rows from lm_head_f16.safetensors instead. The VRAM saved goes into a Q5_K_M backbone, which then needs 5.6 GB on the GPU: 0.3 GB more than a standard Q4_K_M and 0.2 GB less than a standard Q5_K_S, and closer to bf16 than either.

Files

file	size	contents
clef-flash-Q5_K_M-dyn.gguf	6.66 GB	backbone, text only (no vision tower, no MTP layers). Recipe below; 5.57 GB of it on the GPU, the 1.08 GB of Q8_0 token embeddings stay in host RAM
clef-flash-Q8_0-dyn.gguf	8.58 GB	the same recipe on a Q8_0 base, for GPUs with 12 GB or more: 7.49 GB on the GPU, practically lossless (KL 0.00017)
lm_head_f16.safetensors	2.03 GB	lm_head.weight of the original model in fp16, [248320, 4096]. Memory-map it, keep it off the GPU
joint_head.safetensors, joint_head_config.json	244 MB	the joint schema head, unchanged from Cloudflare
joint_schema_model.py		Cloudflare's head and record encoding, unchanged. Tokenization must match it exactly
tokenizer.json, tokenizer_config.json, config.json		tokenizer and config of the original model
clef-flash-imatrix.gguf	5 MB	the importance matrix used for the quantization
LICENSE		Apache-2.0
SHA256SUMS		sums of the files above

How it runs

request → encode_record()      (joint_schema_model.py: tokenization, question/option spans)
        → llama.cpp            per-token hidden states after the final norm, [n_tokens, 4096]
        → JointSchemaHead      (torch, fp16 on the GPU) → one logit per option → softmax

Hidden states from llama.cpp. Tested with llama.cpp 8212c78. Do not use embeddings = true: at this commit the output buffer then also reserves logits, n_vocab floats per token, which is about 1 MB of pinned host RAM per token. Use the nextn output instead (src/llama-ext.h). For Qwen3.5 it is output_norm(h) for every token, independent of batch.logits. Mark only the last token of each chunk as an output, so the LM head runs for one token per chunk:

cp.embeddings = false; cp.n_batch = cp.n_ubatch = 512;
llama_context * ctx = llama_init_from_model(model, cp);
llama_set_embeddings_nextn(ctx, true, /*masked=*/false);
// per request: llama_memory_clear(...); then decode the prompt in chunks of n_ubatch tokens (positions continue),
// logits flag only on each chunk's last token, and copy llama_get_embeddings_nextn(ctx) (m × 4096 floats) after each chunk

A request is computed from scratch every time (llama_memory_clear), so no state carries over between requests. Answers for the same request are bit-identical whatever ran before.

Head. Turing has no bf16, so run JointSchemaHead in fp16 (on our data, fp16 and fp32 give the same decisions as bf16). The head's output_embedding_weight argument only needs weight[token_ids]. Gather those rows from the memory-mapped lm_head_f16.safetensors and move only them to the GPU.

Native support in llama.cpp. The open llama.cpp PR #29831 adds Clef as a native architecture (clef) with a /v1/systemone endpoint in llama-server. Its GGUFs (ggml-org/Clef-Flash-GGUF) are a different format from the ones here. The files in this repository work with released llama.cpp.

Quantization

• Source: the original bf16 weights, converted to a bf16 GGUF with llama.cpp's convert_hf_to_gguf.py (--no-mtp, with Qwen3_5ForConditionalGeneration mapped to the text model).
• Base type: Q5_K_M, with these per-tensor overrides:
  • ssm_alpha, ssm_beta (gated delta net gates): F32, unquantized.
  • ssm_out and the full-attention attn_q/attn_k/attn_v/attn_output: Q8_0. The exception is ssm_out in the middle linear-attention layers, which is Q6_K.
  • Extra bits in the first and last layers (ffn_up 0–7, attn_gate, attn_qkv, ffn_down 0/27/31).
  • token_embd: Q8_0. llama.cpp keeps it in host RAM, so it costs no VRAM.
  • output: Q2_K. Unused, see above.
• Importance matrix: 1.5M tokens at context 2048, every one of them rendered in Clef's own input format (system prompt, STATE, SCHEMA FIELDS, options as JSON). Sources:
  • 5,018 decision records with 1–3 questions of all three types (noul, choice, score) in English and Czech. States come from the train splits of BoolQ, AG News and SST-2, from multi-article AG News digests, and from Czech news (CIIRC-NLP/czech_news_simple-cs, hynky/czech_news_dataset_v2).
  • bartowski's calibration_datav3, used as states.
• Disjoint from the evaluation: none of the evaluation items below are in the calibration data. The evaluation uses the validation and test splits.

Measured quality

How we measured. Each quantization ran through the full pipeline above, with the head in fp16 in every run, and we compared its option probabilities with the same pipeline on the bf16 GGUF. The items:

• 900 questions: 300 each from BoolQ (validation), AG News (test) and SST-2 (validation).
• 24 long synthetic states (2k–12k tokens) with 5 questions each.

KL is mean KL(bf16 ‖ quant) per question; agreement is the share of questions where both pick the same option. Measured on an NVIDIA GB10 with llama.cpp aa885f0:

GGUF	weights on GPU	BoolQ / AG / SST-2	agreement	mean |Δp|	|Δp| on long states	KL
bf16 (reference)	15.9 GB	90.0 / 92.3 / 96.0	—	—	—	—
Q8_0-dyn (this repo)	7.49 GB	90.0 / 92.3 / 96.0	99.9 %	0.0018	0.0029	0.00017
plain Q8_0	8.44 GB	90.0 / 92.3 / 96.0	100.0 %	0.0018	0.0027	0.00019
Q5_K_M-dyn (this repo)	5.57 GB	90.0 / 92.3 / 96.0	99.8 %	0.0036	0.0046	0.00063
same recipe, bartowski's imatrix	5.57 GB	90.0 / 92.3 / 96.0	99.6 %	0.0040	0.0042	0.00093
bartowski Q5_K_S	5.79 GB	89.7 / 92.3 / 96.0	99.2 %	0.0049	0.0058	0.00139
bartowski Q4_K_M	5.26 GB	89.7 / 92.3 / 95.7	99.2 %	0.0065	0.0085	0.00218
plain Q4_K_M without imatrix	5.05 GB	88.3 / 92.3 / 95.3	98.6 %	0.0126	0.0173	0.00629

For 0.3 GB more VRAM than a standard Q4_K_M, this build is 3.5× closer to bf16 by KL than the best public Q4_K_M we tested, and 2.2× closer than Q5_K_S, which needs more VRAM. Most of the gain comes from two things. The first is the per-tensor layout of current llama.cpp: F32 gated-delta-net gates, Q8_0 ssm_out and attention. The llama.cpp we started with applied none of it, and with it alone our Q4_K_M went from KL 0.0053 to 0.0026. The second is the Q5 backbone that the Q2_K LM head pays for. Using our decision-format imatrix instead of bartowski's general one lowers KL by a further third at the same recipe.

Running on an 8 GB GPU (measured in production)

clef-flash-Q5_K_M-dyn.gguf on one Quadro RTX 4000 8 GB (Turing, sm_75, CUDA 12.2). The serving stack:

• llama.cpp 8212c78 with the nextn bridge described above, ctx 16384, ubatch 512, flash attention, the GGUF read with O_DIRECT (load_mode = dio).
• The head in torch 2.5.1 fp16, with PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True.

quality vs a Q8_0 reference on the same GPU (same 924 items)	KL 0.00064, agreement 99.6 %, BoolQ / AG / SST-2 90.3 / 92.3 / 96.0 (Q8_0: 90.3 / 92.3 / 96.0)
VRAM after 16,384-token requests	7.33 GB of 8.19 GB (llama.cpp 6.51 GB fixed, torch 0.81 GB)
host RAM	2.4 GB non-reclaimable (1.1 GB of it pinned token embeddings), peak 3.2 GB
throughput	~1,100 tokens/s, the same as our earlier Q4_K_M build. llama-bench on that build: flat over ubatch 256–2048 and with or without flash attention, so compute-bound

Median latency of one request, one at a time. The backbone is 98 % of it; tokenizer and head stay under 45 ms even at 16k tokens:

state	1 question	5 questions
~600 tokens	0.57 s	0.86 s
~2,000 tokens	1.75 s	2.02 s
~7,400 tokens	6.58 s	6.97 s
16,384 tokens (maximum)	15.2 s	15.9 s

Requests are serialized on the one backbone. With the earlier Q4_K_M build, at the same speed, throughput stayed at ~55 requests/min for ~1k-token requests with 1, 2 or 4 parallel clients.

How it compares with the other decision models we serve

We run three decision models behind the same /v1/systemone API (Rune and Clef-Flash in production, Lux-9B in testing) and publish all three: Livesport/rune-26b-a4b-v3-GGUF, Livesport/clef-flash-GGUF and Livesport/Decision-2.0-Lux-9B-GGUF. This section is the same in all three cards.

How we measured. 7,580 questions from the validation and test splits of 12 public datasets, English and Czech, sampled with fixed seeds. Every item is one /v1/systemone request with one question; the state and the question text are the same for all three models. Each model ran from the exact file published in its repository, served as its card describes (Rune: llama-server with our adapter, request-scoped prefix reuse; Clef and Lux: the hidden-state bridge), one request at a time on an NVIDIA GB10. Accuracy is the share of questions where the most probable option is the gold label (for the star rating: the exact number of stars).

task	n	Rune IQ3_S-dyn	Clef-Flash Q5_K_M-dyn	Lux-9B Q4_K_M-dyn
SST-2 validation · sentiment, yes/no	872	93.5	94.5	93.6
AG News test · topic, 4 options	800	86.8	90.1	89.2
BoolQ validation · question over a passage, yes/no	800	81.8	89.8	89.4
ANLI test R1–R3 · adversarial NLI, 3 options	750	73.5	54.9	61.2
MNLI validation-matched · NLI, 3 options	500	83.0	84.8	90.8
Emotion test · 6 emotions	500	57.6	57.2	57.6
CLINC150 (plus) test · intent: the right one + 9 random + out-of-scope	600	92.8	99.7	97.3
CTKFacts NLI test (cs) · fact check, 3 options	558	72.9	75.1	67.6
CsFEVER NLI test (cs) · fact check, 3 options	500	72.2	71.2	69.8
ČSFD movie reviews test (cs) · star rating 0–5, score	500	53.0	55.4	57.0
Czech Facebook comments test (cs) · sentiment, 3 options	600	73.2	72.5	68.8
Mall.cz product reviews test (cs) · sentiment, 3 options	600	56.7	50.5	57.7
all English	4822	82.4	82.6	83.6
all Czech	2758	65.7	64.9	64.2
all 12 tasks	7580	76.3	76.2	76.5
calibration: ECE of the chosen option, all tasks (lower is better)	7580	0.052	0.093	0.079

Reading the table.

• Overall the three are within half a point of each other; they differ by task.
• Rune is best on adversarial reasoning (ANLI, by 12–19 points), on Czech fact checking (CsFEVER) and Czech Facebook sentiment, and it is the best calibrated: its probabilities match observed accuracy most closely.
• Clef-Flash is best on English topic, yes/no-over-a-passage and intent tasks, and on CTKFacts.
• Lux-9B is best on MNLI, the Czech star rating and Czech product reviews, and is close to Clef on the English classification tasks.
• We do not know the training data of the three models in full. Decision models are commonly trained on public NLI, QA and intent data, so results on MNLI, BoolQ or CLINC may be partly in-distribution for some of them.
• On our own (unpublished) English and Czech decision tasks the picture was the same: Rune leads on Czech faithfulness and fact checks, Clef and Lux on English routing and matching.

Speed. Median latency of one request on an NVIDIA GB10, one request at a time:

request	Rune IQ3_S-dyn	Clef-Flash Q5_K_M-dyn	Lux-9B Q4_K_M-dyn
~300 tokens, 1 question	0.17 s	0.22 s	0.17 s
~300 tokens, 5 questions	0.40 s	0.37 s	0.38 s
~1k tokens, 1 question	0.47 s	0.56 s	0.49 s
~2.5k tokens, 1 question	1.03 s	1.39 s	1.15 s
~2.5k tokens, 5 questions	1.33 s	1.44 s	1.42 s
~10k tokens, 1 question	4.84 s	5.25 s	4.66 s
~10k tokens, 10 questions	5.77 s	5.72 s	5.37 s
throughput, ~600-token requests with 1 question, 1–16 clients	4.3 req/s	3.1 req/s	3.9 req/s

Every server here runs one request at a time, so more parallel clients add queueing, not throughput.

On a Quadro RTX 4000 8 GB (Turing), measured in production:

• Rune IQ3_S-dyn on 2 GPUs: 0.49 s for ~730 tokens, 1.05 s for ~2.3k tokens, 4.12 s for ~8.9k tokens (1 question; 5 questions add 0.3–0.6 s).
• Clef-Flash Q5_K_M-dyn on 1 GPU: 0.57 s for ~600 tokens, 1.75 s for ~2k tokens, 6.58 s for ~7.4k tokens.
• Lux-9B Q4_K_M-dyn: not deployed on Turing yet. It has the same Qwen3.5-9B-class backbone and serving stack as Clef, so expect similar speed.
• The 9B dense backbones are about 1.6× slower than Rune's mixture of experts on Turing, but need one 8 GB GPU instead of two.

License and credits

• Clef-Flash by Cloudflare, Apache-2.0, post-trained from Qwen/Qwen3.5-9B. The joint schema head, its code, the tokenizer and the config are Cloudflare's files, unchanged.
• Quantization and serving notes: Livesport, Apache-2.0, provided as is.
• Thanks: bartowski for public quants and calibration data to compare against.