Valkea/System1Router-effort

System1Router effort classifier

Predicts the reasoning effort a coding-assistant turn needs (low, medium, high, xhigh, max) from the prompt and its recent conversation. It is the local classifier of System1Router, which sets the effort of each Claude Code or OpenCode prompt automatically while keeping the model and its prompt cache.

Files

File	What
ettin-32m-effort-v1.onnx	ONNX graph with embedded weights. Inputs input_ids, attention_mask (int64, batch × ≤256); output: 5 effort logits
ettin-32m-effort-v1.tokenizer.json	Hugging Face tokenizer
ettin-32m-effort-v1.json	Manifest: effort order, input layout, file sizes and SHA-256

System1Router downloads these on system1router install. A new version gets a new name (-v2); old files stay.

Input

[CLS] prompt [SEP] context [SEP], at most 256 tokens. The prompt keeps up to a third; the context is cut from its oldest end. The context is the last 4,000 characters of:

Files in play: <paths>
Tools used: <tool names>
Recent conversation:
<the newest 3,000 characters of the conversation>

So a short follow-up ("ok, do it") after a long debugging exchange is judged by what it refers to.

Training

• Base: jhu-clsp/ettin-encoder-32m, fully fine-tuned (embeddings included), 4 epochs, lr 8e-5.
• Data: 2,535 prompts from real Claude Code and OpenCode sessions, each labelled in hindsight by Claude Opus with the effort the turn needed, given the prompt, context and outcome. The data is not published.
• Loss: class-weighted cross-entropy plus an expected-distance cost that weighs a too-low effort twice a too-high one.

Evaluation

509 prompts from 96 held-out sessions (split by session):

	Exact	Within one level	Mean abs. error
random	23–28%	65–74%	1.04–1.21
always medium	33%	91%	0.76
this model	60%	94%	0.47

Too low 25%, too high 15%. Labels are one judge's opinion, so part of the remaining error is label noise. About 10 ms per prompt on 4 CPU threads after a 0.5 s load; 130 MB.

Limits

• Trained on one developer's sessions: mostly Rust, TypeScript and Python projects.
• The token embeddings were fine-tuned on those sessions, so they carry traces of the vocabulary that appeared in them.
• An effort predictor, not a quality judge: it does not know which model runs the turn.

Retrain on your own sessions: experiments/TRAINING.md.