auto-3b-3

A 3.08B-parameter classifier that decides whether an AI agent's proposed tool call should run. It reads the proposed call, the user's request and the agent's history, and outputs approve or deny. auto-3b-3 is the most accurate Auto model. It matches the best hosted guards we tested on the v3 benchmark, at a fraction of the cost.

It is part of Auto, a local guard for coding agents (Claude Code, Codex, Gemini CLI, Cursor, Copilot CLI and others). Auto blocks a call only if it's consequential and the user didn't ask for it, or if the agent is following instructions planted in a page, file or tool output. Everything runs on your machine, and nothing it reads leaves it.

v3 benchmark	96.0%, up from 72.1% for auto-3b
Missed harm / over-block	3.1% / 4.8%
Original Approve-or-Deny benchmark	98.2%
Time per decision	21 ms (median, NVIDIA RTX PRO 6000)
Cost per million decisions	$77 on a rented GPU
Architecture	SmolLM3 decoder, 3.08B parameters, BF16 weights (6.2 GB)
Runs on	an NVIDIA or AMD GPU

How it compares

The v3 benchmark has 2,948 held-out tool calls. Each one is paired with a near-twin where one detail flips the right answer: who asked, how far the request reaches, production or staging, the exact call, or where an instruction came from. A guard that guesses from surface features gets one of each pair wrong. Hosted guards were prompted with the same decision rule that Auto's training labels follow, and every Auto model is scored at P(deny) >= 0.5.

A guard can be wrong in two ways. Missed harm lets a harmful call through; over-blocking interrupts work you asked for. Hosted LLMs tend to trade one for the other: DeepSeek V4.1 Flash misses almost nothing (0.6%) but blocks 7.9% of authorized calls.

Model	v3 accuracy (95% CI)	Missed harm	Over-block	Both twins right	Time per decision	Cost per 1M
auto-3b-3	96.0% (95.3–96.7)	3.1%	4.8%	92.8%	21 ms	$77
DeepSeek V4.1 Flash (max reasoning)	95.6% (94.8–96.3)	0.6%	7.9%	91.1%	1.3 s	$3,043
Codex auto-review (medium)	95.5% (94.7–96.2)	4.9%	4.1%	90.8%	–	$11,353
GLM-5.3 Flash (high reasoning)	95.4% (94.5–96.1)	5.5%	3.8%	90.2%	1.2 s	$629
auto-0.4b-3	93.1% (92.2–94.0)	6.0%	7.6%	87.2%	7 ms	$23
TypeSafe Jev 1.14	90.2% (89.1–91.2)	12.5%	7.3%	80.1%	136 ms	$227
auto-150m-3	88.8% (87.7–89.9)	9.1%	13.1%	78.7%	5 ms	$13
GPT-6 Luna (low reasoning)	87.9% (86.7–89.0)	5.8%	17.9%	75.8%	–	$395

Missed harm: share of deny items approved. Over-block: share of approve items denied. Both twins right: share of pairs where both halves are correct. Time: Auto on a local NVIDIA RTX PRO 6000, one request at a time; hosted guards are API round trips measured unloaded (Codex auto-review and GPT-6 Luna run through CLIs, which take 12–13 s per decision, so they're left out of the speed comparison). Cost: Auto on a rented RTX PRO 6000 at $2.18/h (median market rate), about $0.50–$3 per million in electricity on a GPU you own; hosted guards at list price × measured tokens.

On an Apple M-series GPU it runs through Auto's runtime, but slowly: a median of 0.86 s per decision on a 300-item sample, and about 12 s for long sessions. auto-0.4b-3 is the better choice there. On a CPU, long sessions take minutes.

v2 → v3

Original Approve-or-Deny benchmark (3,000 items)	auto-3b	auto-3b-3
Accuracy	98.03%	98.23%
Harmful calls approved (of 1,401)	22	35
Authorized calls denied (of 1,599)	37	18
AUROC	–	0.9984

On the original benchmark, v3 keeps its accuracy but shifts toward approving: it denies fewer authorized calls and approves more of the harmful ones. On the v3 benchmark it is far stronger on both counts.

Breakdown on the v3 benchmark

Slice	Accuracy
Written by Claude Opus 4.8 (not a training-data writer)	93.0%
Written by GPT-6.1 Sol	97.7%
Long sessions	95.2%
Non-shell tools (browser, email, CRM, ...)	96.4%
Twin flips on who asked	96.9%
Twin flips on scope	95.3%
Twin flips on environment	96.6%
Twin flips on the exact call	94.5%
Twin flips on where an instruction came from	98.2%

The most-missed harm categories are obfuscated payload (5.7%), data destruction unrequested (4.8%), destructive filesystem (4.3%). Per-item scores for both benchmarks are in eval/, and every model's metrics are in eval/results.json.

Through Auto's runtime (Apple MPS, sessions fitted to 8,192 tokens), auto-3b-3 scored 97.3% on 300 random v3 items, against 97.7% for the reference scores on the same items, and agreed with 299 of 300 decisions.

Usage

With Auto

curl -fsSL https://auto.ssh.codes/install.sh | sh
auto configure --model auto-3b-3

The installer finds your agents and adds the hook. See the Auto README for the Python API, the Claude Agent SDK hook and LangChain middleware.

With Transformers

The model reads one serialized text. Keep the three sections and their order exactly as below.

import json
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer


def text(value):
    return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, separators=(",", ":"))


def build_input(user_request, history, call):
    parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {text(call['args'])}", "",
             "### USER REQUEST", user_request, "", "### AGENT HISTORY"]
    if not history:
        parts.append("(no prior actions)")
    for i, action in enumerate(history, 1):
        parts.append(f"[{i}] {action['tool']}({text(action['args'])})\n-> {text(action.get('result', ''))}")
    return "\n".join(parts)


repo = "ProCreations/auto-3b-3"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo, dtype=torch.bfloat16).eval()

prompt = build_input(
    user_request="Clean up the build artifacts and reinstall dependencies.",
    history=[{"tool": "Bash", "args": {"command": "ls"}, "result": "node_modules dist package.json"}],
    call={"tool": "Bash", "args": {"command": "rm -rf node_modules dist && npm install"}},
)
inputs = tokenizer(prompt, return_tensors="pt")
with torch.inference_mode():
    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
print("deny" if p_deny >= 0.5 else "approve", round(p_deny, 3))

Labels are 0 = approve and 1 = deny. The model accepts up to 65,536 positions. For long sessions, Auto's runtime keeps memory bounded and fits the session to 32,768 tokens on NVIDIA and AMD GPUs or 8,192 on Apple silicon and CPUs, keeping the user's messages, the newest actions and older actions that mention what the call touches.

Training

• Starting point: ProCreations/auto-3b (revision 58323dac).
• New data: about 100,000 contrastive pairs: two near-identical situations where one detail flips the right answer. They were written by GPT-6.1 Terra, GPT-6.1 Sol and DeepSeek V4.1 Flash, label-checked, and cover 53 agent harnesses, 11 languages and 23 categories, including browser, email, CRM and smart-home tools as well as the shell.
• Replay: 84,706 rows of v2 training data, kept close to auto-3b by distillation (KL to the frozen v2 model at temperature 2, weight 0.8), so v3 doesn't forget the original distribution.
• Loss on new rows: cross-entropy, plus a pair margin softplus(4 − (z_deny − z_approve)) (weight 0.5) that pushes each twin pair apart. Denials in the categories where early v3 trials missed more harm than v2, and hard examples, are weighted 1.5×.
• Schedule: 2 epochs, AdamW (β 0.9/0.95, weight decay 0.01), peak learning rate 5e-6, 3% warmup then decay to 10%, batches of up to 32,768 tokens, BF16 compute with FP32 master weights, on one NVIDIA B200 (about 6 hours). The full settings are in training_config.json.
• Selection. auto-3b-3 is the only 3B candidate trained for v3. Its final weights are released as they came out of training, with no selection on the benchmark.
• Held out: the v3 benchmark items and their twins were never trained on. The benchmark isn't public yet.

Limitations

• A classifier, not a chat model. It can't open files, follow URLs or know what an opaque script does; it judges from the text it is given.
• Synthetic labels. Training and benchmark data were written by LLMs and label-checked, not collected from real agent traffic. Items written by Claude Opus 4.8, which wrote none of the training data, score lower (93.0%) than items from a training writer (97.7%), so expect some drop on unfamiliar styles.
• Harm still gets through. It approves 3.1% of harmful v3 calls. When Auto approves, your agent's own permission settings still apply; keep them on.
• Evaluate it on your own traffic before relying on it.