EryriLabs/Glimmer-Sentry-30B-GGUF

Glimmer-Sentry-30B GGUF

Quantised builds of Glimmer-Sentry-30B, a local detection-engineering copilot that writes, explains, tunes and translates detection rules (Sigma to KQL and SPL, YARA, Wazuh). Full eval numbers, training details, data licences and the responsible-use notes are on the main model card; this repo is just the llama.cpp builds.

Quant	Size	Fits	Validation result
IQ2_M	9.2GB	12GB GPU, low-RAM machines	Experimental. Coherent, well-formed KQL, clean termination — but see the low-bit caveat below
Q3_K_M	13.7GB	16GB GPU	Structurally correct holdout translation; one field-name substitution (same caveat)
Q4_K_M	16.9GB	24GB GPU, 32GB Mac	Recommended default. Byte-exact holdout Sigma→KQL vs sigma-cli
Q5_K_M	19.8GB	24GB GPU (tight), 32GB Mac	Byte-exact holdout Sigma→KQL vs sigma-cli
Q6_K	22.9GB	32GB+ GPU/Mac	Byte-exact holdout Sigma→KQL vs sigma-cli
Q8_0	29.6GB	48GB GPU, 64GB Mac, CPU with patience	Byte-exact holdout Sigma→KQL vs sigma-cli
imatrix	12.8MB	n/a	The importance matrix used for IQ2_M — published so you can make your own quants

Every quant ran the same test before upload: a held-out Sigma→KQL prompt through the real chat template, checked for termination, special-token leakage, and fidelity against the official sigma-cli converter output. Q4_K_M through Q8_0 all came back byte-exact, which honestly surprised me — quantisation down to 4-bit cost nothing measurable on that test.

The low-bit caveat: below Q4, a specific failure pattern appears — the query structure stays perfect but a single field name drifts to something plausible and wrong (both Q3_K_M and IQ2_M did it, same pattern). A query that references a field that doesn't exist fails loudly in Sentinel, so you'll catch it — but it means Q3_K_M and especially IQ2_M need their output checked field-by-field, not just eyeballed. If your hardware runs Q4_K_M, run Q4_K_M.

Every quant is made directly from the F16 GGUF, never requantised from another quant. The IQ2_M used an importance matrix computed over a 50/50 mix of general text and real detection-rule content (Sigma, KQL, SPL, YARA from the training pools) — the corpus recipe is in GGUF_NOTES.md.

Running it

Built against llama.cpp master, commit e79e4bf (2026-08-13) — you need a build from that date or newer, since support for this architecture is recent.

llama-cli -m Glimmer-Sentry-30B-Q4_K_M.gguf --jinja -c 8192 --temp 1.0 --top-p 0.95 --top-k 64 \
  -p "Convert this Sigma rule to Microsoft 365 Defender Advanced Hunting KQL: ..."

--jinja matters — the chat template is bespoke (<|start|>/<|message|>/<|eot|> turns with a reasoning channel) and the template baked into the GGUF handles it. Sampling defaults above are the base model's recommended settings. For YARA authoring give it at least 1536 tokens of headroom (-n 1536 or more); rules are long and a cut-off rule won't compile — this exact effect shows up in the eval numbers on the main card.

Text only. The base model has a vision tower, but it's frozen, untouched by the fine-tune, and not included in these files.

A note on how these were made

If you've tried to convert this base model yourself you'll have hit a wall: its tokenizer_config.json targets a transformers version that isn't on PyPI yet, so AutoTokenizer refuses to load it and llama.cpp's converter dies at the vocab step. The fix here was to load the tokenizer through the one code path that has always worked (the same loader our training and eval pipeline used), have it re-serialise itself in the current transformers schema, and then machine-verify the re-export against the original — identical vocab, identical IDs for all 2,048 added tokens, identical encodings across a battery of awkward test strings, byte-identical chat-template output. Nine checks, all passed, before conversion was attempted. No config fields were hand-edited or guessed. The conversion script and the verification results are in the main repo if you want to reproduce it.

Same Apache 2.0 licence as the main model. Same rule applies too: generated rules get reviewed and tested by a human before they go anywhere near production.