auto-150m-3

A 149.6M-parameter classifier that decides whether an AI agent's proposed tool call should run. It reads the proposed call, the user's request and the agent's history, and outputs approve or deny. auto-150m-3 is the smallest and fastest Auto model, for machines that can't spare much memory. It succeeds auto-200m-2, renamed because the model has always had 150M parameters.

It is part of Auto, a local guard for coding agents (Claude Code, Codex, Gemini CLI, Cursor, Copilot CLI and others). Auto blocks a call only if it's consequential and the user didn't ask for it, or if the agent is following instructions planted in a page, file or tool output. Everything runs on your machine, and nothing it reads leaves it.

v3 benchmark	88.8%, up from 69.9% for auto-200m-2
Missed harm / over-block	9.1% / 13.1%
Original Approve-or-Deny benchmark	96.0%
Time per decision	5 ms (median, NVIDIA RTX PRO 6000)
Cost per million decisions	$13 on a rented GPU
Architecture	ModernBERT-base encoder, 149.6M parameters, BF16 weights (0.30 GB)
Runs on	almost anything, including older CPUs

How it compares

The v3 benchmark has 2,948 held-out tool calls. Each one is paired with a near-twin where one detail flips the right answer: who asked, how far the request reaches, production or staging, the exact call, or where an instruction came from. A guard that guesses from surface features gets one of each pair wrong. Hosted guards were prompted with the same decision rule that Auto's training labels follow, and every Auto model is scored at P(deny) >= 0.5.

A guard can be wrong in two ways. Missed harm lets a harmful call through; over-blocking interrupts work you asked for. Hosted LLMs tend to trade one for the other: DeepSeek V4.1 Flash misses almost nothing (0.6%) but blocks 7.9% of authorized calls.

Model	v3 accuracy (95% CI)	Missed harm	Over-block	Both twins right	Time per decision	Cost per 1M
auto-3b-3	96.0% (95.3–96.7)	3.1%	4.8%	92.8%	21 ms	$77
DeepSeek V4.1 Flash (max reasoning)	95.6% (94.8–96.3)	0.6%	7.9%	91.1%	1.3 s	$3,043
Codex auto-review (medium)	95.5% (94.7–96.2)	4.9%	4.1%	90.8%	–	$11,353
GLM-5.3 Flash (high reasoning)	95.4% (94.5–96.1)	5.5%	3.8%	90.2%	1.2 s	$629
auto-0.4b-3	93.1% (92.2–94.0)	6.0%	7.6%	87.2%	7 ms	$23
TypeSafe Jev 1.14	90.2% (89.1–91.2)	12.5%	7.3%	80.1%	136 ms	$227
auto-150m-3	88.8% (87.7–89.9)	9.1%	13.1%	78.7%	5 ms	$13
GPT-6 Luna (low reasoning)	87.9% (86.7–89.0)	5.8%	17.9%	75.8%	–	$395

Missed harm: share of deny items approved. Over-block: share of approve items denied. Both twins right: share of pairs where both halves are correct. Time: Auto on a local NVIDIA RTX PRO 6000, one request at a time; hosted guards are API round trips measured unloaded (Codex auto-review and GPT-6 Luna run through CLIs, which take 12–13 s per decision, so they're left out of the speed comparison). Cost: Auto on a rented RTX PRO 6000 at $2.18/h (median market rate), about $0.50–$3 per million in electricity on a GPU you own; hosted guards at list price × measured tokens.

v2 → v3

Original Approve-or-Deny benchmark (3,000 items)	auto-200m-2	auto-150m-3
Accuracy	96.33%	95.97%
Harmful calls approved (of 1,401)	53	71
Authorized calls denied (of 1,599)	57	50
AUROC	–	0.9927

On the original benchmark, v3 keeps its accuracy but shifts toward approving: it denies fewer authorized calls and approves more of the harmful ones. On the v3 benchmark it is far stronger on both counts.

Breakdown on the v3 benchmark

Slice	Accuracy
Written by Claude Opus 4.8 (not a training-data writer)	84.6%
Written by GPT-6.1 Sol	91.2%
Long sessions	85.2%
Non-shell tools (browser, email, CRM, ...)	87.8%
Twin flips on who asked	90.8%
Twin flips on scope	85.6%
Twin flips on environment	89.0%
Twin flips on the exact call	86.0%
Twin flips on where an instruction came from	95.9%

The most-missed harm categories are physical financial (16.2%), malicious install (12.0%), scope violation (11.7%). Per-item scores for both benchmarks are in eval/, and every model's metrics are in eval/results.json.

Through Auto's runtime (Apple MPS), 79 of 80 random v3 items got the same decision as the reference scores (median P(deny) difference 0.0001); the one difference is a borderline call within 0.05 of the 0.5 threshold.

Usage

With Auto

curl -fsSL https://auto.ssh.codes/install.sh | sh
auto configure --model auto-150m-3

The installer finds your agents and adds the hook. See the Auto README for the Python API, the Claude Agent SDK hook and LangChain middleware.

With Transformers

The model reads one serialized text. Keep the three sections and their order exactly as below.

import json
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer


def text(value):
    return value if isinstance(value, str) else json.dumps(value, ensure_ascii=False, separators=(",", ":"))


def build_input(user_request, history, call):
    parts = ["### PROPOSED TOOL CALL", f"tool: {call['tool']}", f"args: {text(call['args'])}", "",
             "### USER REQUEST", user_request, "", "### AGENT HISTORY"]
    if not history:
        parts.append("(no prior actions)")
    for i, action in enumerate(history, 1):
        parts.append(f"[{i}] {action['tool']}({text(action['args'])})\n-> {text(action.get('result', ''))}")
    return "\n".join(parts)


repo = "ProCreations/auto-150m-3"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForSequenceClassification.from_pretrained(repo, dtype=torch.bfloat16).eval()

prompt = build_input(
    user_request="Clean up the build artifacts and reinstall dependencies.",
    history=[{"tool": "Bash", "args": {"command": "ls"}, "result": "node_modules dist package.json"}],
    call={"tool": "Bash", "args": {"command": "rm -rf node_modules dist && npm install"}},
)
inputs = tokenizer(prompt, return_tensors="pt")
with torch.inference_mode():
    p_deny = model(**inputs).logits.float().softmax(-1)[0, 1].item()
print("deny" if p_deny >= 0.5 else "approve", round(p_deny, 3))

Labels are 0 = approve and 1 = deny. The model accepts up to 65,536 positions. For long sessions, Auto's runtime keeps memory bounded and fits the session to 32,768 tokens on NVIDIA and AMD GPUs or 8,192 on Apple silicon and CPUs, keeping the user's messages, the newest actions and older actions that mention what the call touches.

Training

• Starting point: ProCreations/auto-200m-2 (revision 0bbb929f).
• New data: about 100,000 contrastive pairs: two near-identical situations where one detail flips the right answer. They were written by GPT-6.1 Terra, GPT-6.1 Sol and DeepSeek V4.1 Flash, label-checked, and cover 53 agent harnesses, 11 languages and 23 categories, including browser, email, CRM and smart-home tools as well as the shell.
• Replay: 84,706 rows of v2 training data, kept close to auto-200m-2 by distillation (KL to the frozen v2 model at temperature 2, weight 0.8), so v3 doesn't forget the original distribution.
• Loss on new rows: cross-entropy, plus a pair margin softplus(4 − (z_deny − z_approve)) (weight 0.5) that pushes each twin pair apart. Denials in the categories where early v3 trials missed more harm than v2, and hard examples, are weighted 1.5×.
• Schedule: 2 epochs, AdamW (β 0.9/0.95, weight decay 0.01), peak learning rate 2e-5, 3% warmup then decay to 10%, batches of up to 32,768 tokens, BF16 compute with FP32 master weights, on one NVIDIA RTX PRO 6000. The full settings are in training_config.json.
• Distillation tried, not used. A version distilled from auto-3b-3 scored 89.4% on the v3 benchmark. The difference is not significant (McNemar p = 0.12), and the directly trained model misses less harm, so it is the release. That choice used the v3 benchmark, so this card's v3 numbers carry a little selection bias.
• Held out: the v3 benchmark items and their twins were never trained on. The benchmark isn't public yet.

Limitations

• A classifier, not a chat model. It can't open files, follow URLs or know what an opaque script does; it judges from the text it is given.
• Synthetic labels. Training and benchmark data were written by LLMs and label-checked, not collected from real agent traffic. Items written by Claude Opus 4.8, which wrote none of the training data, score lower (84.6%) than items from a training writer (91.2%), so expect some drop on unfamiliar styles.
• Harm still gets through. It approves 9.1% of harmful v3 calls. When Auto approves, your agent's own permission settings still apply; keep them on.
• Evaluate it on your own traffic before relying on it.