MLukauskas/amber-0.4b

Amber 0.4B

Amber 0.4B is a typed decision engine: given a state and typed questions (multiple-choice choice or yes/no noul, each option described by its criteria), it returns a probability for every supplied option. It is ModernBERT-large (395M parameters, frozen) with a LoRA adapter (r=8, alpha 16) and a learned option scorer (width 512): the context and each option are encoded separately and the scorer turns the pair of vectors into a logit; probabilities are the softmax over a question's options.

Decision Index 0.3

Public index 49.74 (raw 62.82): rank 64 of 168 by public index on the board as of 2026-10-15, and rank 1 of 39 among models under 1.5B parameters (served, as the board counts them); the next is OneJev1.5-0.8B at 42.71. The board ranks on the Full score, of which the public index is 20%; the maintainers run the other 80% (private tests of the same skills, 50%, and decision tasks from new domains, 30%) on submitted models, so Amber's Full score is not out yet.

Complete run, 140,178 scored requests, 0 unsupported, nothing truncated. The run is a complete 0.2.1 run of the same engine, checkpoint and runner (150,317 requests) resumed under 0.3, which ran only the 2,638 rebuilt GSM8K requests. Median 48.5 ms, p95 115.2 ms per request on one NVIDIA GeForce RTX 3080 Ti (bf16 encoder, fp32 option scorer; torch 2.14.0+cu126).

Model	Public index	Full score	Median ms
Sieve 4B	50.06	46.11	93.4
Jevfire	49.80	45.55	89.6
Amber 0.4B	49.74	-	48.5
decisio 0.4.0 (Gemma 4 12B)	49.72	47.47	37.8
LiquidAI d1-3B	48.99	38.90	16.1

Area	Skill	Raw
Knowledge & Reasoning	18.9	38.1
Language Understanding	48.7	64.6
Retrieval & Classification	64.3	71.9
Tools & Automation	78.9	82.0
Arts & Human Taste	49.5	68.8

Best on that board: POP909.

Every benchmark in the index

Area	Benchmark	Metric	Raw	Skill
Knowledge & Reasoning	GPQA Diamond	accuracy	0.260	0.014
Knowledge & Reasoning	GSM8K	accuracy	0.173	0.000
Knowledge & Reasoning	ChessBench	accuracy	0.093	0.012
Knowledge & Reasoning	MuSR	accuracy	0.396	0.040
Knowledge & Reasoning	SATA-Bench	case exact accuracy	0.376	0.367
Knowledge & Reasoning	CRUXEval	accuracy	0.456	0.137
Knowledge & Reasoning	CLadder	accuracy	0.830	0.660
Knowledge & Reasoning	HLE	accuracy	0.174	0.011
Knowledge & Reasoning	MMLU-Pro	accuracy	0.132	0.024
Knowledge & Reasoning	BBH	accuracy	0.615	0.442
Knowledge & Reasoning	WinoGrande	accuracy	0.693	0.386
Language Understanding	ContractNLI	macro-F1	0.792	0.700
Language Understanding	ANLI	macro-F1	0.500	0.250
Language Understanding	HellaSwag	accuracy	0.717	0.623
Language Understanding	ACOS	per-review F1	0.419	0.401
Language Understanding	FinEntity	macro-F1	0.914	0.874
Language Understanding	iSarcasmEval	Sarcasm F1 · track A, English	0.352	0.166
Language Understanding	VAST	macro-F1	0.772	0.657
Language Understanding	NLI4CT	macro-F1	0.636	0.291
Language Understanding	RAGTruth	F1 on hallucinated class	0.729	0.439
Retrieval & Classification	BANKING77	macro-F1	0.902	0.900
Retrieval & Classification	CLINC150	macro-F1	0.915	0.915
Retrieval & Classification	BRIGHT	nDCG@10	0.249	0.151
Retrieval & Classification	Amazon ESCI	macro-F1	0.518	0.395
Retrieval & Classification	PhishNChips	accuracy	0.884	0.768
Retrieval & Classification	HoVer	accuracy	0.863	0.725
Tools & Automation	BFCL	case exact accuracy	0.939	0.918
Tools & Automation	ToolRet	nDCG@10	0.649	0.595
Tools & Automation	API-Bank	accuracy	0.789	0.785
Tools & Automation	Home appliances	case exact accuracy	0.955	0.955
Tools & Automation	When2Call	accuracy	0.752	0.669
Arts & Human Taste	BPoMP	accuracy	0.935	0.869
Arts & Human Taste	Humicroedit	accuracy	0.612	0.224
Arts & Human Taste	POP909	accuracy	0.816	0.815
Arts & Human Taste	cfcolor	accuracy	0.593	0.186
Arts & Human Taste	Habermas	accuracy	0.581	0.391
Arts & Human Taste	New Yorker	accuracy	0.591	0.489

Policy, the same for every benchmark: One fixed rendering for every benchmark (amber.eval.decision_index): string state as is, object state as key: value lines, option = its description or key. Softmax over the option scorer's logits; noul = P(true) over its true / false criteria. No abstention, no truncation, no option filtering. A context over the limit is read in consecutive windows of its longest field (state or question), each with the other fields, and scored on the token-weighted mean of the window vectors; an over-length option is unsupported.

Use

With the decision-index kit installed (pip install -e decision-index, commit f5a6e4fd85fed05e77878dfc4966d0568bfb45ec) and this repository as the working directory:

pip install -r requirements.txt
PYTHONPATH=. python -m decision_index run --engine amber_engine:AmberEngine \
    --option weights=amber.safetensors --option dtype=bf16 \
    --suite-dir <suite-0.3> --out runs/amber-0.4b --compact

The engine loads answerdotai/ModernBERT-large at revision 45bb4654a4d5aaff24dd11d4781fa46d39bf8c13 and applies the adapter and scorer in amber.safetensors.

• dtype is the encoder's (bf16, the default, or fp32). The option scorer stays in fp32, since option logits sit far from 0 and bf16 would tie close options.
• batch_tokens (default 8192) caps the padded tokens per encoder batch.
• Long contexts: a context over 8,192 tokens is read in consecutive windows of its longest field, each with the other field, and scored on the token-weighted mean of the window vectors.
• Long options: an option over 8,192 tokens is refused as unsupported.

Files:

• amber.safetensors: the LoRA adapter (r=8, alpha 16, on Wqkv, Wo, Wi) and the option scorer. fp32, 228 tensors, sha256 c02f5a4ef32da2931413e59b6a92aa01d1d4602912dc83027b4397137fe24346.
• amber_engine.py: the engine, one file. sha256 ffaade8f6ce39520785588c6817a6e60fb2ad4ff2b54e650f05ce64168a3a2c5.

The scored run used the training code's engine on the full training checkpoint (sha256 72c1095921df2a1974eee4d86ef39bc5bae73c33889abedba8dc271c3e01200e). That checkpoint's backbone equals the base model at the revision above, tensor for tensor. amber_engine.py was checked against the run on the shortest answered request of 44 benchmarks (bf16, cuda): 156 of 156 answers with the same choice, probabilities within 7.3e-03.

Training

Amber was trained in rounds. Each round starts from the previous round's checkpoint, trains on that round's mix, and in the later rounds is averaged 50/50 with the checkpoint it started from; the weights here are such an average. The last rounds ran 90,000 steps on one GPU: one record per step, gradient accumulation 4, learning rate 1e-4 decaying linearly to 0 over the last quarter, cross-entropy plus Brier score over each record's options, and auxiliary heads (abstention, answer selection, fact reading) that this engine does not use. The backbone stays frozen; the LoRA adapters, the option scorer and the heads train.

Data

The last round's mix has 588,486 records from 125 sources. No Decision Index row is in it. Where an upstream is also a Decision Index source, the training side uses only parts the index does not score (other splits, songs, palettes, events), and every builder drops a training text that equals a Decision Index row's or shares a 13-word span with one (lower-cased, word characters only), checked against the selected and added rows of the 0.2 suite; the table names the exceptions.

Sources	Records	Upstream, part used	Decision Index benchmark; how its items stay out	Licence
tools.home_gen, tools.home_ha	88,382	Homes and requests generated in this repo; acon96/Home-Assistant-Requests @75f5abe train, mapped to the kit's device format	Home appliances: none of the kit's homes, device names or rooms; requests equal to a scored one dropped	MIT (requests)
tools.toolret_*	39,770	mangopy/ToolRet-Training-20w training queries and tools (revision at download)	ToolRet, BFCL: text overlap on the query	dataset terms
tools.when2call*	23,699	nvidia/When2Call training preference pairs (revision at download)	When2Call: text overlap on the question	CC BY 4.0
tools.apibank_*	20,400	API-Bank lv1/lv2 training dialogues	API-Bank: text overlap; the kit's tool names left out	dataset terms
pop909.pop909	40,000	POP909-CL @be90943: the 115 songs the kit does not sample, every target beat, 12 transpositions	POP909: songs left out by song_id, so no scored song appears	research use
kn.bbh*	37,408	Generated in this repo in the formats of 14 BBH tasks (no BBH text)	BBH: an input equal to a scored one dropped	generated
kn.cladder	25,000	Generated in this repo: causal questions over pseudo-words	CLadder: none of the kit's prompts or pseudo-words	generated
arts.bpomp	24,000	Yorth/limerick-topic @9527c47 limericks with BPoMP-style perturbations	BPoMP: limericks sharing an 8-word span with any BPoMP original dropped	dataset terms
lu.winogrande, lu.hellaswag	47,948	allenai/winogrande @01e7417 and Rowan/hellaswag @218ec52, train	WinoGrande, HellaSwag (validation scored): text overlap	dataset terms
lu.anli_r*	16,400	facebook/anli @8e4813d, train_r1 to train_r3	ANLI: text overlap	CC BY-NC 4.0
lu.vast, lu.contractnli, lu.nli4ct, lu.acos, lu.isarcasm_*, lu.ragtruth	43,199	The kit's pinned sources, non-test splits: VAST train; ContractNLI, NLI4CT, ACOS train and dev; iSarcasm train; RAGTruth train	VAST, ContractNLI (exact contract match), NLI4CT, ACOS, iSarcasmEval, RAGTruth: text overlap	dataset terms; RAGTruth contexts: MS MARCO non-commercial, Yelp no redistribution
ra.clinc*	9,784	CLINC150 train and validation, out-of-scope included	CLINC150: text overlap	CC BY 3.0
ra.banking77	9,934	mteb/banking77 @18072d2 train	BANKING77 (test scored): text overlap	CC BY 4.0
ra.esci	15,991	Amazon ESCI training pairs	Amazon ESCI: overlap on the query	Apache-2.0
ra.hover	8,890	HoVer training claims with the kit's Wikipedia database	HoVer: overlap on the claim	CC BY-SA 4.0
ra.humicroedit	9,000	SemEval-2020 Task 7 train, FunLines and dev	Humicroedit: text overlap	task terms
ra.newyorker	10,311	jmhessel/newyorker_caption_contest @d81cbab matching, train and validation	New Yorker: overlap on the descriptions	CC BY 4.0
ra.habermas	1,918	Habermas Machine training panels	Habermas: overlap on the question	dataset terms
ra.cfcolor	8,000	cfcolor training ratings	cfcolor: no test palette used (split, not text overlap)	dataset terms
ra.forecast	10,000	ForecastBench questions resolved before 2025-08-01	ForecastBench (retired in 0.3): the events 0.2.1 asks about left out	dataset terms
ra.sentfin	9,360	pyRis/SEntFiN @eba4361, a different financial-entity corpus	FinEntity: overlap with FinEntity's texts	dataset terms
ret.phish	12,000	flwrlabs/fed-phishing-urls @941d453 training URLs inside generated emails	PhishNChips: the index's URLs and hosts left out (not text overlap)	dataset terms
sata.multirc, sata.reuters	14,165	SuperGLUE MultiRC train; Reuters-21578 ModApte train	SATA-Bench: overlap on paragraphs and sentences	dataset terms
sata.raw_*	32,927	sata-bench/sata-bench-raw @727cc25, the unlabelled pool SATA-Bench was drawn from	SATA-Bench: its questions carved out, plus overlap on the first 40 words	dataset terms
decisionmix.*	7,090	AG News, BoolQ, SST-5, TREC and BANKING77 (intent transfer), train splits	not filtered; see below	AG News, SST, TREC: research use; BoolQ CC BY-SA 3.0; BANKING77 CC BY 4.0
kev.*	6,033	jaredpalmer/kev-suites @a88f56d decision-v7 training partitions and generated variants	not filtered; see below	Apache-2.0 (suite); sources' own terms
rulemix.*, workflowmix.*	16,877	Generated in this repo: policy rules and multi-step workflows	none	generated

decisionmix.* and kev.* were built without the Decision Index filter. An exact-text scan of their 13,123 records against every suite row finds no BANKING77 or other scored item; at most 20 of HoVer's 4,000 rows share a Wikipedia paragraph with a BoolQ training passage (a different question about the same paragraph).

Earlier rounds also used sources the last mix no longer has, under the same filters: tools.toolret (E6, 23,735 records, an earlier ToolRet cut); in E9, kn.crux (8,000 functions generated in this repo, any function equal to a CRUXEval row's dropped) and 26,980 multiple-choice questions at pinned revisions, from the train and validation splits of ARC, OpenBookQA, SciQ, QASC and CommonsenseQA, MMLU dev and validation, and MedMCQA validation, with a question sharing text with any MMLU, MMLU-Pro, GPQA, ARC, SATA-Bench or HLE row dropped; in E15, sata.toxicity (9,504, RealToxicityPrompts with the SATA-Bench filter) and generated BBH tracking tasks (9,000). No round used GSM8K.

Limitations

• Knowledge & Reasoning is its weakest area: the backbone is a 395M encoder that does not generate, so questions that need world knowledge or multi-step arithmetic stay near chance.
• Mostly English: Arabic appears only in the iSarcasm training rows.
• The probabilities are calibrated on the training mix, not per deployment; re-check thresholds on your own data.