NilHRH/vulnerability-severity-clf-GGUF

vulnerability-severity-clf — GGUF quants

GGUF build of the severity classifier: every quantisation of the checkpoint in NilHRH/vulnerability-severity-clf. These are the same weights compressed differently, not separately trained models.

• Architecture: ModernBert encoder, 149.6M params + sequence-classification head
• Outputs, in order: Info, Low, Medium, High, Critical
• Pooling: CLS · multimodal projection: none
• Held-out test (full precision): accuracy 99.53%, macro-F1 0.9907

Every quant, measured

Accuracy is not guessed from the quant name. Each file was dequantised with gguf-py into the float32 values it stores, loaded into ModernBertForSequenceClassification, and scored on the same 3,598 held-out findings. weight error is the relative L2 distance of the dequantised tensors from the source weights. runs means llama.cpp built and executed the graph (llama-bench, build b11932).

quant	size	vs f16	test accuracy	vs f16	weight error (rel L2)	llama.cpp
F16	301.1 MB	100%	99.53%	+0.00 pp	0.0002	runs
Q8_0	160.8 MB	53%	99.53%	+0.00 pp	0.0063	runs
Q6_K	129.3 MB	43%	99.53%	+0.00 pp	0.0202	runs
Q5_K_M	116.2 MB	39%	99.53%	+0.00 pp	0.0340	runs
Q5_K_S	111.1 MB	37%	99.53%	+0.00 pp	0.0401	runs
Q5_0	109.9 MB	37%	99.53%	+0.00 pp	0.0502	runs
Q4_K_M	106.3 MB	35%	99.56%	+0.03 pp	0.0672	runs
Q3_K_L	100.9 MB	34%	99.56%	+0.03 pp	0.1426	runs
Q4_K_S	99.5 MB	33%	99.56%	+0.03 pp	0.0672	runs
Q4_0	96.0 MB	32%	99.50%	-0.03 pp	0.1007	runs
Q3_K_M	93.7 MB	31%	99.53%	+0.00 pp	0.1426	runs
Q3_K_S	83.9 MB	28%	99.47%	-0.06 pp	0.1678	runs
Q2_K	79.9 MB	27%	99.50%	-0.03 pp	0.2805	runs

The finding: on this task, quantisation is free

The full-precision file and the smallest one sit 1 finding apart on 3,598 rows. Every quant lands inside a ±1-finding band, so there is no accuracy reason to ship 300 MB:

• Q8_0 — indistinguishable from f16. The safe default if you don't want to think about it.
• Q4_K_M — same accuracy in this measurement, half the disk. What I would actually ship.
• Q2_K — 73% smaller than f16, still inside the band. Untested outside this data distribution, so treat it as a size experiment rather than a default.

That flatness is a property of this task, not of GGUF: five coarse classes over a policy that keys on a handful of signals is easy to preserve at low precision. A model whose output depended on fine-grained distinctions would not survive Q2_K.

Verified how

1. convert_hf_to_gguf.py (pinned to the same b11932 build) → f16
2. llama-quantize per quant
3. gguf-py re-read: general.architecture = modern-bert, five classifier.output_labels in the order above, tensor count identical to the f16 it came from
4. llama-bench pp32 on every file — all 13 load and execute at 149.6M params
5. dequantise → transformers → score the test split, per file (the table above)

Using it

llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M      # reproduce any row above

llama.cpp's modern-bert graph implements pool → dense → GELU → norm → classifier, which is why this checkpoint uses the canonical ModernBertForSequenceClassification head: the first model trained here had a 2-layer MLP head and llama.cpp has no second hidden layer for it, so it could not be quantised at all. For calibrated per-class probabilities or anything outside llama.cpp, use the Transformers weights in NilHRH/vulnerability-severity-clf.

What this is not

• A prioritiser, not a scanner — it ranks findings that already exist.
• Gold labels come from a deterministic policy over CVSS · EPSS · CISA KEV · exploit signals · asset exposure. No LLM wrote a label. A high score means the model reproduces that policy, not that it agrees with a human analyst.
• The asset context in the training data is synthetic; swap in real inventory and re-label.

Built from NVD, EPSS, CISA KEV, OSV and ExploitDB. Generated 2026-10-27 20:52:41 from run gguf1.