Infosec-Consult/sage-memory-judge

SAGE Memory Judge (v15)

A small (0.87B) local judge for SAGE's memory write-gate. It answers two yes/no questions about a proposed memory, read directly off one token's log-probabilities:

• Supported — is the memory's claim supported by the supplied evidence as stated (including its time frame, certainty, and the direction of any relation)?
• Lasting — should this be stored as long-term memory (a durable fact or standing rule), rather than a remark about the current session?

It runs entirely on the machine that hosts it (via Ollama) — no cloud inference, no external judge service. It is not a chat model; its only job is to return a calibrated Y/N probability for these checks.

Model

• Base: Qwen/Qwen3.5-0.8B (Apache-2.0), full fine-tune
• Format: GGUF, q8_0 — sha256 714ff9324133ba3b7166fe82fa362226fae5574c4f9cfbc53c054248b16f2cfd
• Runtime: Ollama (OpenAI-compatible endpoint), reads the first generated token's logprobs
• License: Apache-2.0

Use it

ollama create sage-memory-judge -f Modelfile

Modelfile:

FROM ./sage-judge-v15-q8_0.gguf
TEMPLATE {{ .Prompt }}
RENDERER qwen3.5
PARSER qwen3.5

Hard serving requirement — thinking must be OFF. The judge reads the first generated token as the label, so the model must not think first. On Ollama's OpenAI-compatible /v1/chat/completions path, only reasoning_effort: "none" disables thinking — think:false and chat_template_kwargs:{enable_thinking:false} silently no-op there for this architecture. The native /api/chat path uses think:false instead. If thinking is left on, the first token is a reasoning token and a correct reader fails closed (holds for review) on every call — loud, not silent, but non-functional.

The probability is P(Y)/(P(Y)+P(N)) renormalised over the label mass at the first token. Typical gate bands: accept ≥ 0.90, reject < 0.50, otherwise hold for human review. A first token that is not a label must be treated as unavailable → hold, never a guessed score.

Evaluation

Measured once on the shippable GGUF (served by Ollama), on sets authored independently of the model and frozen by digest before measurement.

Set	Result
Acceptance pairs (support)	AUROC 0.994, ECE 0.041, gate precision 1.000, 0 traps accepted
Acceptance pairs (lasting)	AUROC 0.999, ECE 0.088
Held-out 160 (support look-alikes)	traps passed 0/80, genuine rejected 4/80, review 7/160
Genuine-extension (202)	genuine accept 75.8% / review 21.1% / reject 3.1%, traps passed 2/41
Reversed-relation (90)	asymmetric traps rejected 40/40, genuine twins accepted 38/40, symmetric twins 10/10

• CPU latency: ~297 ms (p50) per check (single token), ≈0.3–0.8 s per memory.
• Languages: trained and usable across 15 languages. A translation-based robustness probe shows traps controlled in all 15; genuine-acceptance is strong in most, with a heavier review burden in Hindi, Thai and Bengali (lower-resource scripts — the judge sends more genuine items to review there rather than accepting).

Limitations

• Converse-voice edge: a small number of genuine passive-voice restatements ("A is outranked by B" vs "B outranks A") are over-sent to review; one confusable shape ("leases to/from") is the residual.
• Reversed-relation is validated at the class level, not per relation shape.
• Lower-resource languages (Hindi, Thai, Bengali) are the least reliable; the per-language figures come from a translation-based probe with small per-language samples, so treat them as indicative. An independently-authored multilingual acceptance set is the way to make them precise before broad reliance.
• It checks support against caller-supplied evidence; it does not authenticate the source.

Provenance

Distilled from larger instruct models using soft (probability) labels on synthetic, de-duplicated training data; the training data and the teacher models are not part of this release. Evaluation sets were written by an independent author, verified independently (structural + non-reuse checks + hand-reading of the ambiguous slices), frozen by published digest, and measured one-shot. The reader shipped with SAGE (a Go port) is checked in CI to return probabilities identical to the reference reader that produced these numbers.