assay-0.6b

Calibrated typed decisions from one forward pass. Send a state and named typed questions (bool yes/no, choice over 2..255 described options, score over 2..10 ordered levels); get a probability distribution per question, a confidence and an evidence score. No text is generated, so nothing can come back off-schema.

Code, server and training recipe: https://github.com/bgokden/assay

How it is built

• Backbone Qwen/Qwen3-0.6B-Base with a LoRA adapter (r=16, alpha=32, lr=5e-05, 1.0 epoch, batch 8 x 1 accumulation); merged weights are in this repository, the adapter is in adapter/. The merged weights load with transformers like any Qwen checkpoint.
• The answer is read from the model's own next-token logits over option label tokens at a single decision position, so the base model's zero-shot competence is the starting point.
• Questions are isolated branches over a shared state (block attention mask, restarted positions): packed and separate requests agree exactly.
• Trained with cross-entropy against soft targets: human label distributions where the source has them, SORD-smoothed levels for ordinal questions, one-hot otherwise. Choice options are shuffled per example.
• An evidence head (linear on the decision token, assay_head.safetensors) predicts whether the state supports the question, trained on passage-swapped negatives.
• Global temperature 1.199 fitted on the calibration split of the training tasks and applied unchanged everywhere else.

Evaluation

split	n	accuracy	Brier	NLL	ECE	confident errors
seen tasks (dev), raw	6113	0.705	0.395	0.761	0.057	0.030
seen tasks (dev), scaled	6113	0.705	0.391	0.744	0.030	0.019
unseen tasks (holdout), raw	2020	0.704	0.402	0.732	0.052	0.031
unseen tasks (holdout), scaled	2020	0.704	0.397	0.705	0.037	0.019
kev transfer-v4 dev, raw	764	0.636	0.517	0.912	0.157	0.077
kev transfer-v4 dev, scaled	764	0.636	0.499	0.851	0.124	0.045

"Unseen tasks" are eleven datasets never used in training (bbc_news, app_reviews, scitail, medical_questions_pairs, tweet_irony, ethos, stance_climate, dream, copa, truthful_qa, hh_rlhf). "kev transfer-v4 dev" is the public suite from jaredpalmer/kev-suites (mmlu, emotion, sciq, tweet_offensive, qnli, paws and synthetic rule holdouts); none of its sources are in the training data. Brier is the multi-class sum of squared errors (0..2), ECE uses 15 bins, confident errors are answers with p >= 0.9 that are wrong.

transfer-v4 source	n	accuracy	Brier	ECE
composition_held_and_or	32	0.500	0.667	0.331
composition_held_conditional	32	0.625	0.616	0.348
composition_held_or_not	32	0.562	0.470	0.273
contrastive_authorization	40	0.500	0.837	0.450
contrastive_deadline	40	0.925	0.214	0.269
emotion	116	0.457	0.708	0.235
mmlu	116	0.457	0.636	0.168
paws	80	0.588	0.548	0.266
qnli	80	0.775	0.313	0.107
sciq	116	0.888	0.193	0.068
tweet_offensive	80	0.713	0.445	0.177

Latency on one RTX 5090 (bf16, transformers, packed questions over one state versus separate requests):

questions  packed_ms  separate_ms
        1       17.4         17.2
        3       18.1         52.2
        6       20.9        104.1
       12       18.8        207.6
       24       21.8        416.7

Abstention

conformal.json holds per-question-type thresholds fitted on the seen-task calibration split (alpha 0.1, delta 0.05): a prediction-set threshold with coverage at least 1 - alpha and an act threshold on the top probability whose acted-on error rate is at most alpha at confidence 1 - delta, both on inputs distributed like the calibration split. assay.server returns them as act and set on every answer. How they carry over to other tasks:

split	type	coverage	set size	act rate / error among acted
unseen tasks	bool	0.88	1.28	49% / 14.8%
unseen tasks	choice	0.92	1.80	39% / 5.6%
unseen tasks	score	0.77	2.37	no threshold
transfer suite	bool	0.74	1.17	68% / 31.2%
transfer suite	choice	0.87	1.91	36% / 15.8%
transfer suite	score	1.00	1.75	no threshold

Score questions get prediction sets but usually no act threshold, because exact-level accuracy is the wrong error notion for ordinal answers. Refit on your own labelled data with python -m assay.conformal for a guarantee about your distribution.

The family

model	size	unseen tasks	transfer-v4
assay-0.6b	0.6B	0.704 / 0.397	0.636 / 0.499
assay-1.7b	1.7B	0.752 / 0.334	0.670 / 0.436
assay-4b	4B	0.803 / 0.271	0.784 / 0.302
assay-27b	27B	0.842 / 0.221	0.842 / 0.229
assay-compiled-base	149M	0.606 / 0.494	0.542 / 0.572

Accuracy / Brier after temperature scaling. Same recipe, same splits, different backbones; per-tier abstention and latency are in docs/models.md.

Serving

python -m assay.server --model Berk/assay-0.6b --port 8000

POST /v1/decide is the native shape; POST /v1/systemone and /v1/systemone/batch accept the shape other open decision models use (criteria options, noul booleans); POST /v1/decide_graph walks a decision tree in one forward pass; POST /v1/agents registers an agent -- a graph plus the actions its outcomes stand for -- and /v1/agents/<name>/run decides a case. Requests arriving together share a pass, and /health and /metrics are for operations. assay.backends.sglang runs the same model on an SGLang deployment.

Guides: deployment and the full API, agents, runnable examples.

Train one on your own data

python -m assay.pipeline --config <your>.json runs training, temperature calibration, evaluation and the conformal thresholds over your own records, and writes a directory this same server and publisher accept. The repository's examples/ has a configuration per tier and a dataset in the record format; records written for other decision models (criteria options, noul booleans) load unchanged.

Usage

from assay import load_model
from assay.schema import Question

model = load_model("Berk/assay-0.6b")
answers = model.answer(
    state="My card was charged twice for order A-104.",
    questions={
        "refund": Question(type="bool", instructions="Does the customer ask for money back?"),
        "team": Question(type="choice", instructions="Which team should handle this?",
                         options={"billing": "Charges and refunds", "technical": "Bugs"}),
    },
)
print(answers["team"].probabilities, answers["refund"].p_true, answers["refund"].evidence)

Limitations

Text only, English training data. No arithmetic, counting, date comparison or multi-hop reasoning in one pass; keep those in code. Accuracy drops with unrelated state. The evidence head is trained on coarse swapped-passage negatives. Probabilities are calibrated in aggregate on the evaluated distributions, which is not a guarantee about any single answer or about your data; check calibration on your own labels before acting on thresholds.

Training data

Fifty-five public classification, inference, reading-comprehension and preference datasets rendered as typed questions with described options, plus a synthetic policy-application generator (see assay/data/tasks.py in the repository for the rubrics). Each dataset keeps its own licence; the per-dataset list is in docs/datasets.md. Several sources carry non-commercial or research-only terms; check them before commercial use.

Relationship to other work

Assay is an independent project. Jev and System One are names of TypeSafe AI's products and are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data. Assay is not affiliated with or endorsed by either.