assay-27b

Calibrated typed decisions from one forward pass. Send a state and named typed questions (bool yes/no, choice over 2..255 described options, score over 2..10 ordered levels); get a probability distribution per question, a confidence and an evidence score. No text is generated, so nothing can come back off-schema.

Code, server and training recipe: https://github.com/bgokden/assay

How it is built

• Backbone Qwen/Qwen3.8-27B with a LoRA adapter (r=16, alpha=32, lr=5e-05, 1.0 epoch, batch 4 x 2 accumulation, 4-bit base (QLoRA)); this repository holds the adapter (adapter/) and the evidence head; the base is loaded from Qwen/Qwen3.8-27B in 4bit (bitsandbytes) at load time. Loading downloads the base model separately; the 4-bit base needs about 15 GB of GPU memory.
• The answer is read from the model's own next-token logits over option label tokens at a single decision position, so the base model's zero-shot competence is the starting point.
• Questions are isolated branches over a shared state (block attention mask, restarted positions): packed and separate requests agree exactly.
• Trained with cross-entropy against soft targets: human label distributions where the source has them, SORD-smoothed levels for ordinal questions, one-hot otherwise. Choice options are shuffled per example.
• An evidence head (linear on the decision token, assay_head.safetensors) predicts whether the state supports the question, trained on passage-swapped negatives.
• Global temperature 1.235 fitted on the calibration split of the training tasks and applied unchanged everywhere else.

Evaluation

split	n	accuracy	Brier	NLL	ECE	confident errors
seen tasks (dev), raw	5513	0.834	0.245	0.466	0.048	0.036
seen tasks (dev), scaled	5513	0.834	0.243	0.451	0.040	0.021
unseen tasks (holdout), raw	2020	0.842	0.220	0.392	0.031	0.015
unseen tasks (holdout), scaled	2020	0.842	0.221	0.390	0.040	0.005
kev transfer-v4 dev, raw	764	0.842	0.234	0.460	0.065	0.047
kev transfer-v4 dev, scaled	764	0.842	0.229	0.430	0.041	0.038

"Unseen tasks" are eleven datasets never used in training (bbc_news, app_reviews, scitail, medical_questions_pairs, tweet_irony, ethos, stance_climate, dream, copa, truthful_qa, hh_rlhf). "kev transfer-v4 dev" is the public suite from jaredpalmer/kev-suites (mmlu, emotion, sciq, tweet_offensive, qnli, paws and synthetic rule holdouts); none of its sources are in the training data. Brier is the multi-class sum of squared errors (0..2), ECE uses 15 bins, confident errors are answers with p >= 0.9 that are wrong.

transfer-v4 source	n	accuracy	Brier	ECE
composition_held_and_or	32	1.000	0.002	0.012
composition_held_conditional	32	0.812	0.347	0.177
composition_held_or_not	32	0.906	0.168	0.087
contrastive_authorization	40	1.000	0.000	0.003
contrastive_deadline	40	0.950	0.047	0.084
emotion	116	0.647	0.478	0.077
mmlu	116	0.784	0.333	0.119
paws	80	0.775	0.331	0.151
qnli	80	0.963	0.062	0.055
sciq	116	0.983	0.038	0.034
tweet_offensive	80	0.738	0.331	0.151

Latency on one RTX 5090 (bf16, transformers, packed questions over one state versus separate requests):

questions  packed_ms  separate_ms
        1      109.5        109.6
        3      194.0        333.4
        6      250.4        664.7
       12      412.0       1330.3
       24      750.8       2666.8

Usage

from assay.model import AssayModel
from assay.schema import Question

model = AssayModel.from_pretrained("Berk/assay-27b")
answers = model.answer(
    state="My card was charged twice for order A-104.",
    questions={
        "refund": Question(type="bool", instructions="Does the customer ask for money back?"),
        "team": Question(type="choice", instructions="Which team should handle this?",
                         options={"billing": "Charges and refunds", "technical": "Bugs"}),
    },
)
print(answers["team"].probabilities, answers["refund"].p_true, answers["refund"].evidence)

Limitations

Text only, English training data. No arithmetic, counting, date comparison or multi-hop reasoning in one pass; keep those in code. Accuracy drops with unrelated state. The evidence head is trained on coarse swapped-passage negatives. Probabilities are calibrated in aggregate on the evaluated distributions, which is not a guarantee about any single answer or about your data; check calibration on your own labels before acting on thresholds.

Training data

Fifty-five public classification, inference, reading-comprehension and preference datasets rendered as typed questions with described options, plus a synthetic policy-application generator (see assay/data/tasks.py in the repository for the rubrics). Each dataset keeps its own licence; the per-dataset list is in docs/datasets.md. Several sources carry non-commercial or research-only terms; check them before commercial use.

Relationship to other work

Assay is an independent project. Jev and System One are names of TypeSafe AI's products and are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data. Assay is not affiliated with or endorsed by either.