AnamK/jdi-ptr14-0.8b

bigind_ptr14nosyn_soup3

A 0.8B text-only decision model for typed multiple-choice / yes-no decisions over a structured state (the Jev / Decision Index request format). Built on Qwen3.5-0.8B. At inference it runs only the first 14 of 24 decoder layers. A 2 MB bilinear pointer head then picks the option. There is no text generation and no LM head.

Results (Decision Index 0.3, public suite, kit 9eb2dbe)

	Value
Public index (balanced skill, coverage-adjusted)	35.79 (complete run: all 140,178 scoreable requests)
Areas (skill ×100)	Knowledge 13.5 · Language 40.3 · Retrieval 52.6 · Tools 53.0 · Arts 16.8
Display-only (not counted; native metric)	MMLU 49.5% acc · ARC-Easy 87.7% · ARC-Challenge 73.3% · SimpleBench 2/10 · RouterBench quality 0.610 · SGD macro-F1 0.385 on the frozen rows (0.937 on builder-fixed rows)
Full score (public 20% + private 80%)	not measured (private parts are run by the maintainers)

The public index is partly in-distribution: train splits of 15+ suite benchmarks are in the training data (below). Gains on the public index are larger than we expect on the private parts of the Full score. In our out-of-suite checks (48 held-out tasksource-format tasks and an 18-task leaderboard-format set) this model was not better than the model trained without the benchmark train splits (−0.85 [−2.9, +1.2] and −0.9 [−6.3, +3.0] points).

How to run

Repo: AnamK/jdi-ptr14-0.8b. Put engine/ on PYTHONPATH and run the Decision Index kit with --engine jdi_ptr14.engine:Ptr14Engine --option model=AnamK/jdi-ptr14-0.8b.

Use the engine in engine/jdi_ptr14 (see engine/README.md): bf16, 14 layers, ptr_head_L14.pt, fixed prompt, shared-prefix batching (batch_tokens=8192), and no truncation (longer prompts are refused).

Method

• Base: Qwen/Qwen3.5-0.8B (hybrid Gated DeltaNet + full attention, 24 layers).
• Readout: The prompt numbers the options ([i] key: description). A pointer head compares the hidden state of the last prompt token (after the assistant prefill Best answer: [) with the hidden state at the end of each option line, both taken at the output of layer 14. A softmax over options gives the probabilities. This works for any number of options (up to 160 in training, 151+ in the suite), unlike a digit/letter readout.
• Training: LoRA (rank 16) on decoder layers 0–13, with the pointer head trained jointly at layer 14.
  • Loss: cross-entropy + 0.5 × Brier against soft or one-hot targets.
  • Setup: lr 1e-4, 1 epoch (5,587 steps, token-budget batches of about 6k tokens), max length 32,768. Option order is shuffled per row. Base weights in bf16, LoRA in fp32.
• Soup: a uniform weight average of 3 runs with the same initialisation and different data-order/option-shuffle seeds, merged into the dense checkpoint. The pointer head is averaged too.
• Compute: 3 runs of about 3 GPU-hours each on RTX 3090s.

Training data (178,783 rows; all text; no images)

1. General decision data (41,354 rows): about 640 sources from tasksource/tasksource-jev-typed-decisions (≤60 rows per source, K≤9), plus 5,845 rows of the synthetic business-workflow set LocalLLaMA/typed-decisions, which is generic and not shaped after any suite benchmark.
   • Sources whose names match a suite benchmark were removed by a name blocklist: ANLI, ARC, BANKING77, CLINC, ContractNLI, ESCI, HellaSwag, HoVer, Humicroedit, NLI4CT, WinoGrande, and, by accident, WANLI.
2. Train splits of datasets behind Decision Index benchmarks ("DI-source train splits"), rendered in the suite's request format with the kit's own adapter logic. The suite's evaluation split is never used.
   • BANKING77, CLINC150+OOS, ANLI r1–r3, ARC-Easy/Challenge, WinoGrande, HellaSwag, ContractNLI, NLI4CT and Humicroedit.
   • Amazon ESCI (train split, label-balanced), HoVer (train, label-balanced, oracle evidence), GSM8K (train, 4- and 10-way answer-group options) and API-Bank (training-data lv1–lv3).
   • SGD (dstc8 train dialogues), When2Call (train_pref), RAGTruth (train), iSarcasmEval (train), VAST (train) and New Yorker caption matching (train contests, none shared with the suite).
   • MMLU auxiliary_train (also rendered in MMLU-Pro style), MASSIVE-en and MTOP-en intents, ToolACE rendered as BFCL-style per-tool yes/no (BFCL has no train split), and 6,989 tasksource rows with K>9.
3. Not included: the generator-made CLadder-style (5,186 rows) and BPoMP-style (7,895 rows) data used in our earlier 38.08 model. This model contains no synthetic data generated to match a benchmark's format.
   • The format-matched rows that remain are real train-split examples (or ToolACE / MMLU-aux examples) re-rendered in the suite's request layout.

Decontamination against the full public suite (selected + added + GSM8K rows, about 153–158k requests):

• Round 1 and round 2 data: a row is dropped if any normalised segment of ≥30 characters (state, instructions, any option) equals a suite segment, or if it shares any word 13-gram with the suite.
  • Before the 13-gram check, template tokens are masked: tokens covered by an 8-gram that occurs in ≥50 suite rows.
  • Removed: 8,593 of 59,293 rows (round 1) and 5,783 of 80,298 (round 2). Most were ToolACE/When2Call tool schemas shared with BFCL/ToolRet, HoVer dev-evidence overlap, API-Bank tool schemas, GSM8K and ESCI.
• The earliest in-domain set (25,295 rows): a per-family exact-state check and a state-only 13-gram check.
  • Limitation: drop counts were not saved, and there is no cross-family check.
• The general tasksource mix (41k rows) was filtered by source name only.
  • Small residual risks: swag vs HellaSwag contexts, FEVER-family vs HoVer, corr2cause vs CLadder.

Licence notes on the training data: RAGTruth contexts and RACE (inside MMLU-aux) are non-commercial, VAST has no stated licence, and ANLI is CC BY-NC. Review these before choosing a model licence.

Limitations

• Benchmark-specific gains. Much of the public-index improvement comes from benchmark train splits. It did not carry over to held-out decision tasks. Expect a lower Full score than the public index suggests.
• Known weak spots.
  • PhishNChips is near chance: no training data, and predictions collapse to "legitimate".
  • ChessBench, CRUXEval, HLE and POP909 are at or near zero skill.
  • FinEntity regressed (Negative recall), and MuSR/SATA are noisy.
• Language and modality. Mostly English text; small Arabic share (iSarcasmEval). No image input.
• Numerics. bf16. Shared-prefix batching matches per-question scoring to bf16 noise; about 1% of near-tie answers can flip between bf16 code paths.
• Not a chat model. The 14-layer trunk plus head cannot generate text. The remaining 10 layers of the checkpoint are unused at inference.