typical-small

Typical is an open System-One decision model family, built by OzLabs.

Give it a state — a support ticket, a document, a form, a log line, any text — a question with its rubric, and a label set defined at runtime, and it returns a calibrated probability over those labels plus an explicit abstain option. One forward pass, no generated text to parse, about 45 ms per decision on an H100. Every decision is one of three typed primitives: Choice (categorical), Score (ordinal), Noul (yes/no, exactly order-invariant).

typical-small is Release 1 (checkpoint ts1b): a clean 1.7B retrain with dedicated Score and Noul heads, a harder training curriculum, and 1,024-token states. It supersedes typical-small-preview as the reference 1.7B model. For more capability at 1.25x the latency, see typical-medium.

Why it's different

• The decision is read directly from the model's own candidate-conditioned computation. The state and candidates are encoded once; the answer is read off that computation instead of being sampled one token at a time.
• The state is KV-cached, so asking many questions about one state is cheap — each additional question is a short suffix against a cache that's already built.
• Latency is flat up to ~64 options (45 ms at K = 2, 46 ms at K = 32); 106 ms at K = 256 — the state dominates until the rendered option list does.
• Abstention is a first-class output, not a parsed refusal: every decision carries a calibrated P(∅) alongside the label probabilities.
• Probabilities, not verbalized confidence. No "I'm 80% sure" string to parse — an actual calibrated number.

Model family

	typical-small-preview	typical-small	typical-medium
Backbone	Qwen3-1.7B-Base	Qwen3-1.7B-Base	Qwen3-4B-Base
Tap layer	20 / 28	20 / 28	26 / 36
Per-decision latency, K=2 (one H100)	45 ms	45 ms	57 ms
Peak memory, K=2	6.4 GB	6.4 GB	14.6 GB
JevBench standard / easy / hard	.750 / 1.00 / .387	.694 / 1.00 / .432	.806 / 1.00 / .423
MMLU-Pro among-K	.330	.343	.458
Held-out workflow noul / score	.699 / .498	.715 / .520	.811 / .528
Calibration (typed-decisions NLL)	2.06	1.25	1.18

Full ablations and matched-baseline tables: REPORT.md §3ae (this release) and §3af (typical-medium).

Results

	typical-small (ts1b)
SNLI / MNLI / BoolQ / ANLI	.894 / .859 / .824 / .497
CLINC-150 / TREC-fine / HWU64 / 20NG	.804 / .508 / .761 / .540
Held-out workflow: noul / score / style	.715 / .520 / .859
External floors: PagerDuty (floor .792) / jevlogs (floor .697)	.817 / .710
JevBench standard / easy / hard (Brier std / hard)	.694 / 1.00 / .432 (.40 / .79)

JevBench standard, same 231 public ids, next to the open field:

jeff (GLiFormer 400M)	Laya (ModernBERT)	typical-small	open-alternative-jev (Qwen3.5-4B)	system-one-open (Gemma E2B LoRA)
.750	.694	.694	.833	.931

Public-subset per-item comparison on the 231 public ids; not a ranked leaderboard entry.

Quickstart

huggingface-cli download OzLabs/typical-small --include "inference/*" --local-dir typical-small
pip install -r typical-small/inference/requirements.txt

import sys; sys.path.insert(0, "typical-small/inference")
from typical import Typical

m = Typical.from_pretrained("OzLabs/typical-small", device="auto")

ticket = ("Customer reports repeated 500 errors on checkout after the last deploy; "
          "asking for a refund.")
m.choice(ticket, "Which team should handle this?", ["billing", "infra", "support", "security"])
m.score(ticket, "How severe is this ticket?", ["1", "2", "3", "4"])
m.noul(ticket, "Does this ticket violate the refund-abuse policy (3+ refunds in 30 days)?")

Verified: this exact sequence (download, install, run) works on CPU.

choice/score return {label: probability, ...} plus p_null; noul returns P(yes). decide mirrors the internal PCDMDecider.decide exactly, including the runtime block (latency, p_null, truncation flags) — see inference/README.md for the full API and inference/example.py for a runnable script.

Training recipe

• Data mix E .40 / K .15 / W .35 / U .10 of every batch (evidence, knowledge-MCQ, rubric- conditioned workflow, soft-target uncertainty), plus 15–20% null augmentation on W rows (--null_aug W:0.20) so the model learns to abstain, not just choose.
• 1,024-token decision states (vs. 256 in the preview), so policy documents and multi-hop states are in-distribution.
• Typed heads: Score is K-way Choice trained with ordinal-smoothed targets (τ = 0.7, zero head code); Noul is a dedicated Bernoulli head, routed per row — only rows whose candidates are exactly {yes, no} use it, every other 2-way set scores as ordinary Choice.
• 12,000 steps, effective batch 64 (bs 64), LoRA rank 16 on the top 8 layers of a 20-of-28-layer truncated backbone.
• Held-out by construction: one rubric style and three whole domains of the hard curriculum (data_wh), plus temporal/numeric/probability/trade-off composition (level 7), are eval-only — never trained on.

Data sources and licenses:

Source	Role	License
SNLI, MultiNLI, ANLI, BoolQ, MMLU-aux, ARC, OpenBookQA, CommonsenseQA, SciQ, QASC, LogiQA2, AQuA-RAT, MedMCQA	evidence + knowledge-MCQ	permissive research licenses (CC-BY-SA / MIT / Apache-2.0 family)
cua-ai/cua-s1-forms	workflow (GUI-form synth)	MIT
dwidlee/systemone-lite-general	workflow (rule-labeled typed decisions)	MIT
MagaBitmex/jev-4b-distill-data	workflow (programmatic gold)	Apache-2.0
in-repo rubric-conditioned generator (data_wf, data_wf_long)	workflow	generated, no external license
in-repo rule-engine generator (data_wh, DecisionMix v2)	hard curriculum	generated, no external license
Zhengping/UNLI (validation split)	soft-target uncertainty	MIT
metaeval/ambient (ambiguous rows, trained on)	soft-target uncertainty	license undeclared on its HF card — flagged, not asserted
metaeval/chaos-mnli-ambiguity (eval-only, never trained on)	soft-target uncertainty	license undeclared on its HF card — flagged, not asserted

Evaluation-only, never trained on: JevBench (public subset), typed-decisions, PagerDuty, tree-choice, jevlogs (Loghub research license, caveated), Mind2Web.

Limitations

• Hard tier is .432 (chance .336, SE ≈ .045 at n = 111) — the least reliable tier; the open field's best is .73.
• Held-out rubric styles regress 4 points vs. the preview, traded for the first 1.7B model to clear the PagerDuty and jevlogs constant-prediction floors.
• Long policy documents (over ~1,024 tokens) remain the weakest hard-tier slice.
• Mind2Web stays below its constant-prediction floor — general web-agent candidate sets are outside this training distribution.
• Inference code is included in this repo; training code release to follow.

License

Apache-2.0 (backbone Qwen/Qwen3-1.7B-Base is Apache-2.0; this release's weights are a LoRA adapter plus small Choice/Score/Noul heads over that backbone). Training-data licenses above; JevBench numbers are a public-subset run against fstandhartinger/jevbench v1.2.1, not a submitted or ranked leaderboard entry.

Files in this repository

• best.pt — this checkpoint (ts1b).
• results.json / eval_wf_full.json / probe_results.json / jevbench_summary.json — full eval suite, post-hoc held-out/external eval, probe results, JevBench run summary.
• inference/ — the self-contained Typical inference package (see Quickstart above).
• MANIFEST.json — checksum + provenance record for this release.

Lineage

typical-small-preview (Release 0, frozen reference) → typical-small (Release 1, this repo) → typical-medium (Release 1 at 4B, more capability at 1.25x the latency).

Coming: typical-large — a 14B ladder checkpoint scores JevBench .875/.468 (standard/hard), but probability calibration at that size is not yet fixed (§3ab). Trained, not released.

Typical is built by OzLabs.