gavel-base

A gavel ends a hearing with a verdict. This model does the same for software: the state and the allowed answers go in, one typed answer and a calibrated probability come out. Nothing is written, thus there is no generation loop and no parsing afterwards.

150 million parameters, an encoder, no language model inside.

How it decides

Each option becomes a hypothesis about the state, a three-way entailment head scores it, and a softmax over the options is the decision.

premise     = the state (or the question, when there is no state)
hypothesis  = "<question> The answer is <option>."
score       = logits[entailment]
decision    = argmax over the options

Two properties follow from this shape:

• The answer space is defined at request time. The option text is part of the input, therefore the model has no fixed classes and a new category needs no retraining.
• The order of the options cannot change the result, because every option is scored in its own sequence.

Three earlier designs were tried and thrown away: option markers inside one sequence, span pooling over the option text, and the standard multiple-choice head. All three stayed at cross entropy ln(number of options) for thousands of steps — an untrained head cannot read an abstract option string, so no gradient separates the options. The entailment form above trains from the first hundred steps.

Use it

import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

name = "chukfinley/gavel-base"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForSequenceClassification.from_pretrained(name).eval()

state = "The customer was charged twice for the same subscription period."
question = "Which queue should handle this request?"
options = ["Billing support", "Account access support", "Technical fault"]

pairs = [(state, f"{question} The answer is {o}.") for o in options]
batch = tok([p for p, _ in pairs], [h for _, h in pairs],
            padding=True, truncation=True, max_length=512, return_tensors="pt")
with torch.no_grad():
    entail = model(**batch).logits[:, 0] / 2.2677      # fitted temperature
print(dict(zip(options, torch.softmax(entail, dim=-1).tolist())))

Measured

Every number below was measured on one RTX 3060, on the rows named.

Task	gavel-base 150M	reference
Tool selection (function-calling corpora, 400 held-out rows)	0.993	0.758 before this sector was trained
SciQ with passage	0.980	—
CLINC intent routing	0.947	Qwen3.5-4B letter logits 0.968
BoolQ	0.785	Qwen3.5-4B 0.863
ANLI-R1	0.565	Qwen3.5-4B 0.503
WANLI (OpenJev fixture, best checkpoint)	0.714	OpenJev frozen Qwen3.5-4B 0.637
Browser actions (Mind2Web, element choice)	0.655	—
Belebele German	0.555	0.305 before the language data
Belebele Italian, never trained	0.550	transfer to an unseen language
MMLU	0.303	Qwen3.5-4B letter logits 0.688
TypeSafe business fixture (equal-case agreement)	0.409	published Jev 0.883

Expected calibration error after temperature fitting: 0.024 on the development strata. The temperature is in the config and in the snippet above; without it the model is overconfident (0.098).

Where it is good and where it is not

It is strong when the answer is in the text it was given: routing, intent, tool choice, yes/no over a passage, entailment. It is weak when the answer has to be recalled: MMLU and medical questions sit near chance, because 150 million parameters do not store what a 4-billion-parameter decoder read during pretraining. Long multi-step reasoning over structured documents (the TypeSafe invoice packets) is the weakest case of all.

A larger model is still training

This checkpoint is the first one worth publishing, not the last. A larger run is on the card right now: 542000 training rows against the 380000 behind this one, 40000 steps instead of 8000, and after it a 395-million-parameter encoder and specialised branches for routing, documents and agents. The numbers here will be replaced when those finish.

Training data

All public and human-labelled, converted into one record format: MNLI, WANLI, ANLI, SNLI, BoolQ, ARC, CommonsenseQA, QASC, WinoGrande, HellaSwag, SWAG, MMLU auxiliary, SciQ, MedMCQA, AQuA, RACE, QuALITY, Banking77, AG News, DBpedia, tweet_eval, SMS spam, Yelp, XNLI, PAWS-X, MASSIVE, Belebele, RouterBench, Mind2Web, and two function-calling corpora.

Generated in addition, with labels that follow from the construction and not from a teacher model: invoice and incident packets, routing and severity cases, and abstain rows built by removing the evidence that decides the answer.

Code, data builders and the full result files: https://github.com/chukfinley/gavel

MIT licence.