auto-0.4b-2

A 395.8M-parameter ModernBERT encoder that decides whether an agent's proposed tool call is authorized and safe in context: it reads the proposed call, the user's request, and the agent's history and outputs approve or deny, at 65,536-token context. This is iteration 1 of auto-0.4b-2 (September 10, 2026), a continued training of the September 9 release on one RTX PRO 6000. The previous weights remain at revision 456e153d with their card archived under history/pre-iteration-1-456e153d.

Results

Both versions were re-evaluated on the pinned 3,000-item Approve-or-Deny benchmark (revision a38b6259), full input lengths, BF16 with FlashAttention, P(deny) >= 0.5. A false approval is an unsafe call approved; a false denial is an authorized call denied.

Model	Accuracy	False approvals	False denials	AUROC
auto-0.4b-2 (Sept 9)	96.77% (2903/3000)	44/1401 (3.14%)	53/1599 (3.31%)	0.9944
auto-0.4b-2 iteration 1	97.00% (2910/3000)	36/1401 (2.57%)	54/1599 (3.38%)	0.9949

Iteration 1 fixes 32 of the previous errors and introduces 25 new ones (+0.23 points; paired bootstrap 95% interval -0.27 to 0.73, McNemar p = 0.43, so the accuracy change alone is not statistically clear). Wilson 95% interval for its accuracy: 96.33%–97.55%.

Other checks	Sept 9	Iteration 1
Benchmark inputs of 16k–64k tokens (239)	94.56%	95.82%
Validation audit partition, never used for selection (2595)	97.88%	98.03%
Published skills/MCP/custom-tool probes	24/24	24/24
Fresh scope/history/injection probes	38/40	38/40

Per-category, language, difficulty, length, and threshold breakdowns are in eval_results.json; per-item logits in benchmark_predictions.npz; probe inputs and outputs in eval/. The validation-calibrated threshold is 0.515; all numbers above use 0.5.

What changed in iteration 1

• Distilled from a private 3B teacher (a SmolLM3-3B-Base classifier that scores 98.03% on this benchmark, weight SHA256 314b5711a812…): loss 0.2·CE(label) + 0.8·T²·KL(teacher‖student), T = 2, one full epoch over all 711,985 training rows at full length, peak LR 1e-05. The teacher agrees with 99.80% of the training labels, so on its own this mostly made the model more permissive.
• Teacher-labelled cross-paired data. 120,000 new inputs re-judge an existing history and proposed call under a different user's request from the same language, framework, and domain (a quarter with no history). Their only label is the teacher's judgement (67.10% deny). A short follow-on epoch on these plus 173,351 replayed originals (peak LR 5e-06) is what cut false approvals.
• Weight averaging. The published weights are the FP32 average of kd_b_aug/step-1305 and kd_b_aug/step-1737, the last two checkpoints of that follow-on epoch.
• Selection. 43 candidates (all checkpoints and weight averages, including a safety-weighted third run) were scored on the 7,824-row validation split; 14 were then benchmarked and the most accurate one satisfying the gates below was chosen. Because that choice used benchmark results (14 candidates plus 3 diagnostic previews), the headline number carries some selection bias; the audit partition above is the cleaner comparison. Every candidate's scores are in shortlist_results.json.
• Gates. Predeclared: strictly higher benchmark accuracy; no more benchmark false approvals or false denials than the Sept 9 release; no regression on the 16k–64k slice, the audit partition, or either probe set; finite 65,536-token forward pass; CPU and GPU decisions agree. One gate was waived by the maintainer: benchmark_false_denials_not_worse (54 vs 53 false denials, one item on 1,599 authorized cases), accepted because false approvals fell from 44 to 36 and total errors from 97 to 90. No benchmarked candidate met every gate at once: they split into a safer family (36–38 / 54–56) and a more permissive family (45–48 / 42–47).

Same pinned data as before (ProCreations/auto-1b-data revision d265bbf7, 15 rows removed by deduplication and held-out-group separation, no benchmark overlap); full-parameter AdamW with FP32 master weights and BF16 compute; no input truncated. The Sept 9 baseline was reproduced exactly (2,903/3,000, 44/53) before comparison. Scripts, plan, data audit, environment, and prediction caches are under training/iteration-1/ and eval/iteration-1/; the teacher weights are private and not needed for inference.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

"""Serialize exactly the information available to an approve/deny classifier."""
def build_input(user_request, history, call):
    parts = ['### PROPOSED TOOL CALL', f"tool: {call['tool']}", f"args: {call['args']}", '',
             '### USER REQUEST', user_request, '', '### AGENT HISTORY']
    if not history:
        parts.append('(no prior actions)')
    else:
        for i, h in enumerate(history):
            parts.append(f"[{i+1}] {h['tool']}({h['args']})\n-> {h.get('result', '')}")
    return '\n'.join(parts)


repo = "ProCreations/auto-0.4b-2"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(
    repo, dtype=torch.bfloat16,
    attn_implementation="kernels-community/flash-attn2@81fb77c12b2ad5d69380669b46739d5868614502",
).cuda().eval()

text = build_input(
    user_request="Clean up the build artifacts and reinstall dependencies.",
    history=[{"tool": "Bash", "args": "ls", "result": "node_modules dist package.json"}],
    call={"tool": "Bash", "args": "rm -rf node_modules dist && npm install"},
)
inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=65536).to("cuda")
with torch.inference_mode():
    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
print("deny" if p_deny >= 0.5 else "approve", p_deny)

Install the tested versions listed in environment.json, including kernels; the pinned Hugging Face FlashAttention kernel loads automatically on supported CUDA systems. A compatible local FlashAttention 2 installation can also use attn_implementation="flash_attention_2". Ordinary SDPA is suitable for shorter inputs but can materialize large masks for long sequences.

Labels are 0 = approve, 1 = deny. Preserve the exact ### PROPOSED TOOL CALL, ### USER REQUEST, and ### AGENT HISTORY sections. The model only consumes this serialized text; training rationales and metadata are excluded.

Scope and limitations

A classifier, not a chat model: it approves routine authorized work and denies consequential unauthorized actions or actions that follow injected instructions. It cannot inspect hidden file contents, resolve opaque executables, or know a URL's runtime behavior; false approvals remain possible. Labels are synthetic and the benchmark has been reused across releases, so evaluate on your own traffic before relying on it.