assay-compiled-base

An encoder-tier decision model from the Assay project: typed questions (bool, choice, score) over a state produce calibrated probability distributions with an evidence signal, with no text generation and no language model. It is the small, CPU-friendly tier; the decoder models (Berk/assay-4b, Berk/assay-27b) are far more accurate.

Architecture compiled with late interaction on Alibaba-NLP/gte-modernbert-base (8 query slots). The state is encoded once into token embeddings. Each question is compiled once: the instruction becomes a set of query vectors (learned slots plus a projection of its pooled encoding), each option becomes a vector (its pooled encoding) and its token encodings. A decision is cross-attention from the queries over the state tokens, a small reader MLP, a bilinear score against each option vector plus a per-option bias, and a late-interaction term: the mean over option tokens of the best cosine match among state tokens, times a learned scale. Compiled questions can be cached and reused across states; after the state encode, a decision is a few small matrix products.

Evaluation

split	n	accuracy	Brier	NLL	ECE	confident errors
seen tasks (dev), scaled	6113	0.668	0.442	0.806	0.037	0.019
unseen tasks (holdout), raw	2020	0.606	0.522	0.895	0.133	0.035
unseen tasks (holdout), scaled	2020	0.606	0.494	0.818	0.061	0.011
kev transfer-v4 dev, raw	764	0.542	0.613	1.048	0.174	0.064
kev transfer-v4 dev, scaled	764	0.542	0.572	0.952	0.095	0.012

"Scaled" applies the temperature 1.485 fitted on the seen-task calibration split. Unseen tasks are eleven datasets never trained on; the transfer suite is jaredpalmer/kev-suites transfer-v4 dev, whose sources are excluded from training. Single-text classification (topic, sentiment, spam) is strong; questions that need knowledge (MMLU) or multi-step reasoning are near chance. See the repository's docs/roadmap.md for the full comparison against the cross-encoder and the decoders.

CPU latency (milliseconds; the state is encoded once, questions are compiled once and cached, decide runs per state x question set):

runs/compiled-late-gte-base on cpu, 8 threads
questions  encode_state_ms  compile_ms  decide_ms  end_to_end_ms
        1            28.95       29.63      1.381          62.25
        3            30.67       61.52      1.753          94.48
        6            30.64       88.46      2.844         122.80
       12            30.51       88.45      4.867         124.75
       24            30.64       88.42     10.694         131.05

Usage

from assay.compiled import load_any
from assay.schema import Question

model = load_any("Berk/assay-compiled-base", device="cpu")
answers = model.answer(
    "My card was charged twice for order A-104.",
    {
        "refund": Question(type="bool", instructions="Does the customer ask for money back?"),
        "team": Question(type="choice", instructions="Which team should handle this?",
                         options={"billing": "Charges and refunds", "technical": "Bugs"}),
    },
)
print(answers["team"].probabilities, answers["refund"].p_true)

Trained with assay.train_compiled (3.0 epochs, lr 5e-05, head lr 0.0005, batch 32) on the Assay data (55 public datasets rendered as typed questions, synthetic policy and date cases, and 40k generic questions labelled by assay-27b). Each dataset keeps its own licence; the list is in docs/datasets.md.

Limitations

English only. No knowledge beyond what the encoder carries, no arithmetic, no multi-hop reasoning. Calibrated in aggregate on the evaluated distributions, not per answer; check on your own labels before acting on thresholds.

Relationship to other work

Assay is an independent project. Jev and System One are names of TypeSafe AI's products and are mentioned only to describe and compare; kev-suites is Jared Palmer's evaluation data. Assay is not affiliated with or endorsed by either.