assay-1.7b

Calibrated typed decisions from one forward pass. Send a state and named typed questions (noul yes/no, choice over 2..255 described options, score over 2..10 ordered levels); get a probability distribution per question, a confidence and an evidence score. No text is generated, so nothing can come back off-schema.

Code, server and training recipe: https://github.com/bgokden/assay

How it is built

• Backbone Qwen/Qwen3-1.7B-Base with a LoRA adapter (r=16, alpha=32, lr=5e-05, 1.0 epoch, batch 8); merged weights are in this repository, the adapter is in adapter/.
• The answer is read from the model's own next-token logits over option label tokens at a single decision position, so the base model's zero-shot competence is the starting point.
• Questions are isolated branches over a shared state (block attention mask, restarted positions): packed and separate requests agree exactly.
• Trained with cross-entropy against soft targets: human label distributions where the source has them, SORD-smoothed levels for ordinal questions, one-hot otherwise. Choice options are shuffled per example.
• An evidence head (linear on the decision token, assay_head.safetensors) predicts whether the state supports the question, trained on passage-swapped negatives.
• Global temperature 1.211 fitted on the calibration split of the training tasks and applied unchanged everywhere else.

Evaluation

split	n	accuracy	Brier	NLL	ECE	confident errors
seen tasks (dev), raw	5513	0.740	0.359	0.700	0.055	0.041
seen tasks (dev), scaled	5513	0.740	0.355	0.680	0.035	0.029
unseen tasks (holdout), raw	2020	0.752	0.337	0.626	0.047	0.032
unseen tasks (holdout), scaled	2020	0.752	0.334	0.601	0.024	0.022
kev transfer-v4 dev, raw	764	0.670	0.453	0.794	0.147	0.065
kev transfer-v4 dev, scaled	764	0.670	0.436	0.743	0.115	0.039

"Unseen tasks" are eleven datasets never used in training (bbc_news, app_reviews, scitail, medical_questions_pairs, tweet_irony, ethos, stance_climate, dream, copa, truthful_qa, hh_rlhf). "kev transfer-v4 dev" is the public suite from jaredpalmer/kev-suites (mmlu, emotion, sciq, tweet_offensive, qnli, paws and synthetic rule holdouts); none of its sources are in the training data. Brier is the multi-class sum of squared errors (0..2), ECE uses 15 bins, confident errors are answers with p >= 0.9 that are wrong.

transfer-v4 source	n	accuracy	Brier	ECE
composition_held_and_or	32	0.688	0.430	0.177
composition_held_conditional	32	0.469	0.501	0.258
composition_held_or_not	32	0.719	0.361	0.188
contrastive_authorization	40	0.525	0.703	0.391
contrastive_deadline	40	0.300	0.897	0.443
emotion	116	0.595	0.550	0.149
mmlu	116	0.569	0.534	0.109
paws	80	0.662	0.439	0.161
qnli	80	0.875	0.232	0.087
sciq	116	0.888	0.161	0.070
tweet_offensive	80	0.725	0.376	0.127

Latency on one RTX 5090 (bf16, transformers, packed questions over one state versus separate requests):

questions  packed_ms  separate_ms
        1       17.8         17.9
        3       17.9         53.6
        6       19.7        106.6
       12       21.1        212.2
       24       32.1        428.7

Usage

from assay.model import AssayModel
from assay.schema import Question

model = AssayModel.from_pretrained("Berk/assay-1.7b")
answers = model.answer(
    state="My card was charged twice for order A-104.",
    questions={
        "refund": Question(type="noul", instructions="Does the customer ask for money back?"),
        "team": Question(type="choice", instructions="Which team should handle this?",
                         options={"billing": "Charges and refunds", "technical": "Bugs"}),
    },
)
print(answers["team"].probabilities, answers["refund"].noul, answers["refund"].evidence)

Limitations

Text only, English training data. No arithmetic, counting, date comparison or multi-hop reasoning in one pass; keep those in code. Accuracy drops with unrelated state. The evidence head is trained on coarse swapped-passage negatives. Probabilities are calibrated in aggregate on the evaluated distributions, which is not a guarantee about any single answer or about your data; check calibration on your own labels before acting on thresholds.

Training data

Fifty-five public classification, inference, reading-comprehension and preference datasets rendered as typed questions with described options (see assay/data/tasks.py in the repository for the full list and rubrics). Each dataset keeps its own license.