typical-small-preview

Typical is an open System-One decision model family, built by OzLabs.

Give it a state — a support ticket, a document, a form, a log line, any text — a question with its rubric, and a label set defined at runtime, and it returns a calibrated probability over those labels plus an explicit abstain option. One forward pass, no generated text to parse, about 45 ms per decision on an H100. Every decision is one of three typed primitives: Choice (categorical), Score (ordinal), Noul (yes/no).

typical-small-preview is Release 0: a frozen checkpoint (nc_v3_tap20_wf) that establishes the architecture — native contextual readout, factored abstention, rubric-conditioned workflow training. It is the reference point for typical-small (Release 1, dedicated Score/Noul heads, 1,024-token states) and typical-medium (Release 1 at 4B). In this preview, Score and Noul are trained as plain Choice, not as dedicated ordinal/Bernoulli heads.

Why it's different

• The decision is read directly from the model's own candidate-conditioned computation. The state and candidates are encoded once; the answer is read off that computation instead of being sampled one token at a time.
• The state is KV-cached, so asking many questions about one state is cheap — each additional question is a short suffix against a cache that's already built.
• Latency is flat up to ~64 options (45 ms at K = 2, 46 ms at K = 32); 106 ms at K = 256 — the state dominates until the rendered option list does.
• Abstention is a first-class output, not a parsed refusal: every decision carries a calibrated P(∅) alongside the label probabilities.
• Probabilities, not verbalized confidence. No "I'm 80% sure" string to parse — an actual calibrated number.

Model family

	typical-small-preview	typical-small	typical-medium
Backbone	Qwen3-1.7B-Base	Qwen3-1.7B-Base	Qwen3-4B-Base
Tap layer	20 / 28	20 / 28	26 / 36
Per-decision latency, K=2 (one H100)	45 ms	45 ms	57 ms
Peak memory, K=2	6.4 GB	6.4 GB	14.6 GB
JevBench standard / easy / hard	.750 / 1.00 / .387	.694 / 1.00 / .432	.806 / 1.00 / .423
MMLU-Pro among-K	.330	.343	.458
Held-out workflow noul / score	.699 / .498	.715 / .520	.811 / .528
Calibration (typed-decisions NLL)	2.06	1.25	1.18

Full ablations and matched-baseline tables: REPORT.md §3t / §3w (this release).

Results

	typical-small-preview (nc_v3_tap20_wf)
SNLI / MNLI / BoolQ / ANLI	.894 / .849 / .831 / .481
CLINC-150 / TREC-fine / HWU64 / 20NG	.734 / .372 / .735 / .522
Held-out workflow: noul / score / style	.699 / .498 / .901
JevBench standard / easy / hard (Brier std / hard)	.750 / 1.00 / .387 (.40 / .88)

JevBench standard, same 231 public ids, next to the open field:

typical-small-preview	jeff (GLiFormer 400M)	Laya (ModernBERT)	open-alternative-jev (Qwen3.5-4B)	system-one-open (Gemma E2B LoRA)
.750	.750	.694	.833	.931

Public-subset per-item comparison on the 231 public ids; not a ranked leaderboard entry.

Quickstart

huggingface-cli download OzLabs/typical-small-preview --include "inference/*" --local-dir typical-small-preview
pip install -r typical-small-preview/inference/requirements.txt

import sys; sys.path.insert(0, "typical-small-preview/inference")
from typical import Typical

m = Typical.from_pretrained("OzLabs/typical-small-preview", device="auto")

ticket = ("Customer reports repeated 500 errors on checkout after the last deploy; "
          "asking for a refund.")
m.choice(ticket, "Which team should handle this?", ["billing", "infra", "support", "security"])
m.score(ticket, "How severe is this ticket?", ["1", "2", "3", "4"])
m.noul(ticket, "Does this ticket violate the refund-abuse policy (3+ refunds in 30 days)?")

Verified: this exact sequence (download, install, run) works on CPU (checked against typical-small's identical inference package).

choice/score return {label: probability, ...} plus p_null; noul returns P(yes) (routed through the K-way path in this preview — see Limitations). decide mirrors the internal PCDMDecider.decide exactly, including the runtime block (latency, p_null, truncation flags) — see inference/README.md for the full API and inference/example.py for a runnable script.

Training recipe

• Data mix E .35 / K .25 / W .40 of every batch (evidence, knowledge-MCQ, rubric-conditioned workflow) — no soft-target uncertainty corpus and no null augmentation in this release.
• 256-token decision states.
• Unified Choice head: Score is trained as a K-way categorical Choice; Noul is a 2-way Choice over ["no","yes"] — neither has a dedicated typed head yet (see typical-small).
• 12,000 steps, effective batch 64 (bs 64), LoRA rank 16 on the top 8 layers of a 20-of-28-layer truncated backbone.
• Held-out by construction: one rubric style per workflow family is held out of train/val entirely, to measure rubric dependence rather than pattern-matching a fixed decision.

Data sources and licenses:

Source	Role	License
SNLI, MultiNLI, ANLI, BoolQ, MMLU-aux, ARC, OpenBookQA, CommonsenseQA, SciQ, QASC, LogiQA2, AQuA-RAT, MedMCQA	evidence + knowledge-MCQ	permissive research licenses (CC-BY-SA / MIT / Apache-2.0 family)
cua-ai/cua-s1-forms	workflow (GUI-form synth)	MIT
dwidlee/systemone-lite-general	workflow (rule-labeled typed decisions)	MIT
MagaBitmex/jev-4b-distill-data	workflow (programmatic gold)	Apache-2.0
in-repo rubric-conditioned generator (data_wf)	workflow	generated, no external license

Evaluation-only, never trained on: JevBench (public subset), typed-decisions, PagerDuty, tree-choice, jevlogs (Loghub research license, caveated), Mind2Web.

Limitations

• Hard tier is .387 (chance .336, SE ≈ .045 at n = 111) — the least reliable tier; the open field's best is .73. Long policy states are the weakest slice (≤ 256 tokens .64 vs. > 1,024 tokens .23).
• Score and Noul are trained as plain Choice, not dedicated ordinal/Bernoulli heads — order sensitivity on Noul reaches Δp ≈ .12–.55 under label reversal (fixed in typical-small).
• Workflow-mix training induces over-abstention on CLINC-150 and TREC-fine, a diagnosed null-weighting/sampling interaction, not a discrimination loss.
• Very large candidate sets (K in the hundreds to thousands) need the energy → top-r → native path, not this native head directly.
• Inference code is included in this repo; training code release to follow.

License

Apache-2.0 (backbone Qwen/Qwen3-1.7B-Base is Apache-2.0; this release's weights are a LoRA adapter plus a small decision head over that backbone). Training-data licenses above; JevBench numbers are a public-subset run against fstandhartinger/jevbench v1.2.1, not a submitted or ranked leaderboard entry.

Files in this repository

• best.pt — this checkpoint (nc_v3_tap20_wf).
• results.json / eval_wf.json / probe_results.json / jevbench_summary.json — full eval suite, post-hoc held-out/external eval, probe results, JevBench run summary.
• baseline_nc_v3_tap20/ — the matched pre-workflow baseline, same five files.
• inference/ — the self-contained Typical inference package (see Quickstart above).
• MANIFEST.json — checksum + provenance record for this release.

Lineage

typical-small-preview (Release 0, this repo) → typical-small (Release 1, dedicated typed heads) → typical-medium (Release 1 at 4B).

Coming: typical-large — a 14B ladder checkpoint scores JevBench .875/.468 (standard/hard), but probability calibration at that size is not yet fixed (§3ab). Trained, not released.

Typical is built by OzLabs.