MLukauskas/amber-0.4b-v2

Amber 0.4B v2

Amber 0.4B v2 is a 397.9M-parameter typed decision engine. Given a state and typed questions (multiple-choice choice or yes/no noul, each option described by its criteria), it returns a probability for every supplied option. It is cross-encoder/ettin-reranker-400m-v1 (Apache-2.0) with every weight fine-tuned and a small option head. Each question is one encoder pass over [CLS] instructions [SEP] state [SEP] [MASK] option 1 … [MASK] option K [SEP], so every option sees the question, the state and the other options. An option's logit is an MLP over its [MASK] token, the mean of its tokens and [CLS]; probabilities are the softmax over the question's options, and a noul answer is P(true).

🇱🇹 Built in Lithuania. Amber is named after Baltic amber, gintaras in Lithuanian, which the sea washes up on the Lithuanian coast.

Decision Index 0.3

Public index 47.42 (raw 60.00). Complete run: 140,178 requests, 0 unsupported, nothing truncated. Median 19.2 ms, mean 41.0 ms, p95 115.6 ms per request on one NVIDIA GeForce RTX 3080 Ti (bf16, torch 2.14.0+cu126).

The board (edition 0.4) ranks on the Full score. That score is 60% private tests of the same skills and 40% private decision tasks from new domains, which the maintainers run on submitted models, so it is not out yet. Board models up to 1.25 times this size (board snapshot 2026-11-04):

Model	Parameters	Public index	Full score
vLLM-SR vela3-fast (Vela 3.0, 0.4B)	380.4M	50.36	29.27
Amber 0.4B	395.9M	49.74	26.00
GLiClass v4 edge (0.4B)	439.8M	47.49	31.21
Amber 0.4B v2	397.9M	47.42	pending
Crivo-400M	395.8M	39.79	23.74
vLLM-SR vela3-fast-vision (Vela 3.0, 0.25B)	254.1M	37.50	22.11

Area	Skill	Raw
Knowledge & Reasoning	17.8	37.5
Language Understanding	53.3	68.2
Retrieval & Classification	68.0	75.0
Tools & Automation	62.5	64.9
Arts & Human Taste	40.1	57.9

Every benchmark in the index

Area	Benchmark	Metric	Raw	Skill
Knowledge & Reasoning	GPQA Diamond	accuracy	0.245	0.000
Knowledge & Reasoning	GSM8K	accuracy	0.234	0.074
Knowledge & Reasoning	ChessBench	accuracy	0.085	0.003
Knowledge & Reasoning	MuSR	accuracy	0.586	0.342
Knowledge & Reasoning	SATA-Bench	case exact accuracy	0.256	0.246
Knowledge & Reasoning	CRUXEval	accuracy	0.416	0.073
Knowledge & Reasoning	CLadder	accuracy	0.587	0.174
Knowledge & Reasoning	HLE	accuracy	0.142	0.000
Knowledge & Reasoning	MMLU-Pro	accuracy	0.232	0.137
Knowledge & Reasoning	BBH	accuracy	0.597	0.416
Knowledge & Reasoning	WinoGrande	accuracy	0.731	0.462
Language Understanding	ContractNLI	macro-F1	0.798	0.709
Language Understanding	ANLI	macro-F1	0.570	0.356
Language Understanding	HellaSwag	accuracy	0.844	0.792
Language Understanding	ACOS	per-review F1	0.474	0.457
Language Understanding	FinEntity	macro-F1	0.805	0.714
Language Understanding	iSarcasmEval	Sarcasm F1 · track A, English	0.475	0.325
Language Understanding	VAST	macro-F1	0.784	0.675
Language Understanding	NLI4CT	macro-F1	0.682	0.382
Language Understanding	RAGTruth	F1 on hallucinated class	0.696	0.370
Retrieval & Classification	BANKING77	macro-F1	0.936	0.935
Retrieval & Classification	CLINC150	macro-F1	0.934	0.934
Retrieval & Classification	BRIGHT	nDCG@10	0.344	0.258
Retrieval & Classification	Amazon ESCI	macro-F1	0.532	0.413
Retrieval & Classification	PhishNChips	accuracy	0.892	0.785
Retrieval & Classification	HoVer	accuracy	0.871	0.742
Tools & Automation	BFCL	case exact accuracy	0.943	0.923
Tools & Automation	ToolRet	nDCG@10	0.673	0.622
Tools & Automation	API-Bank	accuracy	0.681	0.675
Tools & Automation	Home appliances	case exact accuracy	0.023	0.023
Tools & Automation	When2Call	accuracy	0.859	0.812
Arts & Human Taste	BPoMP	accuracy	0.960	0.920
Arts & Human Taste	Humicroedit	accuracy	0.630	0.260
Arts & Human Taste	POP909	accuracy	0.016	0.009
Arts & Human Taste	cfcolor	accuracy	0.618	0.235
Arts & Human Taste	Habermas	accuracy	0.589	0.403
Arts & Human Taste	New Yorker	accuracy	0.661	0.576

Policy, the same for every benchmark: one rendering for every benchmark. A string state is used as is and an object state becomes key: value lines; an option is its criteria text, or its key. No abstention, no truncation, no option filtering. A question longer than 32,768 tokens is unsupported.

Use

With the decision-index kit installed and this repository as the working directory:

pip install -r requirements.txt
PYTHONPATH=. python -m decision_index run --engine amber_uni_engine:UniEngine \
    --suite-dir <suite-0.3> --out runs/amber-0.4b-v2 --compact

The scored run used no options. The engine reads the weights, uni_config.json and encoder/ from its own directory, and nothing is downloaded. Options:

• dtype: bf16 (default) or fp32.
• token_budget (default 65536): the most padded tokens per forward pass. A request's questions are sorted by length and packed.
• max_len (default 32768): a question needing more tokens is unsupported.
• orders (default 1, from uni_config.json): averages the probabilities over up to 4 fixed option orders (as given, reversed, rotated by half, rotated by half and reversed).
• Long questions: a question over 2,048 tokens runs alone and unpadded. The encoder's sliding-window layers then attend in blocks to the keys within their window, not through a dense L × L mask, which would take about 14 GiB at 30k tokens. The attention is the same: on a 15,200-token question, Amber 17M's probabilities were bit-identical to the dense path.

Files:

File	sha256 (first 16)
amber_banner.jpg	0b165507de8387d0
amber_uni_core.py	17a4270d6e02cc97
amber_uni_engine.py	541314792b915357
encoder/config.json	4cb5132219b46f8a
encoder/tokenizer.json	6c8aaa9a542084f2
encoder/tokenizer_config.json	09baa14905255c98
model.safetensors	92fb4bc5a5bcfee3
requirements.txt	13efb642d4e4bd72
uni_config.json	4d624933048d418b

model.safetensors holds the encoder and the option head (174 tensors, bf16). It is the uniform average of the three EMA checkpoints listed under Training (each an exponential moving average, decay 0.999, of the training weights). encoder/ is the base model's config and tokenizer. Requirements:

safetensors==0.8.0
torch==2.14.0
transformers==5.17.0

Training

The uniform average of three EMA checkpoints: the end of g400-kd18k, g400-c36k at step 12,000 and the end of g400-c36k. They come from two training cycles from cross-encoder/ettin-reranker-400m-v1 at revision 5dca36282a5d85f368d2544002513a29159b4c9e, 31.2 h in all on one RTX 3080 Ti:

1. g400-kd18k: 18,000 steps from the base model, seed 1, learning rate 3e-5 for the encoder and 5e-4 for the head, 4 micro-batches of up to 8,192 padded tokens per step, teacher f400-v2 (15.4 h).
2. g400-c36k: 18,000 more steps from g400-kd18k's EMA weights, with a new warm-up and cosine decay, seed 2, learning rate 3e-5 for the encoder and 5e-4 for the head, 4 micro-batches of up to 8,192 padded tokens per step, teacher g400-kd18k (15.8 h).

• Batches. Each step has up to 32,768 padded tokens of whole questions (about 85 questions). Training sequences are up to 4,096 tokens. A longer one is cut to the largest common cap on the instructions, the state and each option that fits; a cut state keeps its head and tail. Inference never cuts.
• Optimizer. AdamW, 5% warm-up, then cosine decay to 0 in each cycle, weight decay 0.01, dropout 0.1. The weights kept are the exponential moving average (decay 0.999) of the training weights.
• Augmentation. With probability 0.5 a question's options are shuffled. With probability 0.5 its state moves into the instructions, as in many Decision Index rows.
• Loss. Cross-entropy against soft targets: 0.5 gold and 0.5 the option probabilities of a larger or earlier development model, the teacher. f400-v2 is an Ettin reranker 400M trained for 6,000 steps on the same mix with gold targets only; g400-kd18k is the first cycle above. A task family where the teacher's top option matches the gold on less than 60% of its rows keeps the gold alone.

Data

Questions are drawn from four stores, with weights 0.45 / 0.10 / 0.30 / 0.15. Every store drops a row whose text (state, question or options) shares a 13-word span or a short sentence with a Decision Index 0.3 suite row, and rows from sources derived from a suite benchmark.

Store	Rows	What it is	Licence
tsj-c3k-clean	390,285	tasksource/tasksource-jev-typed-decisions @6f92aad, train split: at most 3,000 rows per source (358 sources, 215 families). Sources that are a Decision Index benchmark's own dataset are left out, as are 145 families held out for validation and the multilingual set. 1,129 rows were dropped for suite overlap and 6,331 for derived sources.	per row (the dataset's license column)
tsj-bench	37,887	The same dataset's rows from the 21 sources that are Decision Index suite datasets, train splits only: ARC, ANLI, BANKING77, CLINC150, CommonsenseQA, ContractNLI, ESCI, HellaSwag, HoVer, Humicroedit, iSarcasm, NLI4CT, OpenBookQA, RAGTruth, WinoGrande. At most 3,000 per source; 5,887 rows dropped for suite overlap.	per row; ANLI CC BY-NC 4.0
amber-di	323,836	The Decision Index-format records of Amber 0.4B's training mix, whose card lists sources, splits and filters, plus the MRPC / QQP paraphrase records of its later rounds. Generators built to imitate one benchmark (home appliances, POP909, CLadder, SATA-Bench raw pool) are left out, as are multi-answer and abstaining records. At most 15,000 per source; 61,328 rows dropped for suite overlap.	as listed on the Amber 0.4B card (RAGTruth contexts: MS MARCO non-commercial)
fc	134,125	Tool-use decisions parsed from glaiveai/glaive-function-calling-v2, Team-ACE/ToolACE and NousResearch/hermes-function-calling-v1: which tool to call, whether to call one now, ask or decline, which tool description fits, and which reply comes next. 865 rows dropped for suite overlap.	Apache-2.0

The weights are released under CC BY-NC 4.0 because the training data includes non-commercial sources (ANLI under CC BY-NC 4.0, RAGTruth's MS MARCO contexts, QQP under Quora's terms). The base model is Apache-2.0.

Limitations

• Knowledge & Reasoning is its weakest area: a small encoder that does not generate stays near chance on questions that need world knowledge or multi-step arithmetic.
• Home appliances and POP909 are near zero: the generators built for them were left out of training, because their gains on the public benchmarks did not carry over to the private tests.
• Mostly English.
• The probabilities are calibrated on the training mix, not per deployment; re-check thresholds on your own data.