mini-chessformer-v1

A 7.57M-parameter Chessformer-lite transformer for chess: 64 square tokens + Geometric Attention Bias (GAB) + from–to policy + contempt-conditioned WDL value, trained from scratch on data/train/data.parquet (Jan–Mar 2026 Lichess).

Goal: match burrowdweller/minichess-gpt-v1-final under matched MCTS at ≤50% train FLOPs. Composite play + cost gate: PASS.

Results (c=0, search-contempt off)

Gate	Value	Threshold
Paired match score vs v1-final	0.700 (W11/D6/L3)	≥0.40	✅
Holdout MCTS mean CPL	306.7	382.0 (parent+25)	✅ (lower=better)
Train FLOPs	4.665e17	4.665e17 (0.50×v1)	✅
Wall (RTX 4070)	9.80 h	10.5 h	✅
Contempt conditioning shift (c=0→0.5)	Δdraw=−0.10, Δdecisive=+0.10	ε=0.05	✅

T3 contempt conditioning is alive: raising contempt from 0 to 0.5 measurably reduces draws and increases decisive outcomes in the predicted direction, without collapse.

Architecture

ChessformerLiteConfig(
    d_model=256, n_layers=6, n_heads=8, d_ff=384,  # narrow FFN (~1.5x)
    use_gab=True,
    gab_compress_dim=32, gab_code_dim=256, gab_n_templates=32,
    contempt_hidden=64,
)
# 7,566,471 params

• Encoder: 64 square tokens (piece id embedding + learned square PE) + 1 CLS token (state projection + contempt embedding). 6 pre-norm transformer blocks.
• GAB: Geometric Attention Bias — per-layer attention bias generated from square features via a templates-and-coefficients mechanism (Smolgen-style), added to attention logits.
• Policy head: factored from–to (64×64 = 4096) + 176 promotion slots → dense MOVE_SPACE = 4272 logits. Order matches engine.interfaces.
• Value head: FiLM-conditioned WDL (3-class: win/draw/loss, mover POV). Contempt embedding modulates the CLS via gamma * cls + beta.

Inputs (ONNX)

Name	dtype	Shape	Notes
square_ids	int64	[B, 64]	piece ids 0–12 (empty=0, white P..K=1..6, black P..K=7..12)
state_features	float32	[B, 8]	[stm, WK, WQ, BK, BQ, ep_file/7 or -1, halfmove_bucket, repetition]
contempt	float32	[B]	scalar per batch; 0.0 for tournament/production play

Outputs

Name	dtype	Shape	Notes
policy	float32	[B, 4272]	raw logits (MOVE_SPACE); apply legal_mask + softmax
wdl	float32	[B, 3]	raw logits (win/draw/loss, mover POV); softmax to get probs

ONNX parity

Validated against the PyTorch forward at both c=0.0 and c=0.5:

| Sweep | max |policy err| | max |wdl err| | |---|---:|---:| | c=0.0 | 1.18e-5 | 2.86e-6 | | c=0.5 | 1.26e-5 | 2.44e-6 |

Both under 1e-3 — ONNX graph preserves T3 conditioning exactly.

Training

• Data: /mnt/c/Users/jun/chessdb/data/train/data.parquet (streaming, multi-epoch)
• Steps: 617,523 · Batch: 256 · Seq len: 65 (1 CLS + 64 squares)
• Warmup: 61,752 (10%) · LR: 3e-4 → cosine → 0 · WD: 0.01
• Seed: 0 · Device: CUDA (RTX 4070)
• Final loss: ≈1.98 (policy ≈1.17, value ≈0.81)

FLOP/cost ledger: runs/chessformer_lite/s2/flop_ledger.jsonl (separate from production lineage).

Files

File	Size	Description
mini-chessformer-v1.onnx	30.4 MB	ONNX artifact (opset 17, dynamic batch)
mini-chessformer-v1.pt	91.0 MB	PyTorch checkpoint (trainer state: model + optimizer + scheduler + step)
mini-chessformer-v1.json	1.1 KB	Export metadata (hashes, config, I/O shapes, parity)
REPORT.md	—	S2 scale + play eval report (this card's source)
inference.py	—	Standalone ONNX inference + board encoding + 4272-move tables + legal mask. No repo imports.
browser/manifest.json	445 B	Exact chess-gpt-package-v1 package manifest
browser/entry.js	133 KB	Self-contained browser arena entrypoint: chess.js + MCTS + Chessformer-lite adapter
browser/model_final.onnx	30.4 MB	Browser package copy of the ONNX artifact

Usage (standalone)

This repository ships inference.py — a self-contained ONNX inference module that carries its own copy of the board encoding, the 4272-move table, and the legal-move mask. It depends only on chess, numpy, and onnxruntime. You do not need to clone the chessdb repo or import engine.interfaces / experiments.chessformer_lite.encode.

CLI smoke (startpos / midgame / promotion, dynamic batch)

python inference.py --model mini-chessformer-v1.onnx
# pass --contempt 0.5 to exercise T3 conditioning
# pass --fens "<fen1>" "<fen2>" ... for custom positions

Python API

import chess
from inference import ChessformerLiteONNX

eng = ChessformerLiteONNX("mini-chessformer-v1.onnx")

# single-board (returns raw logits + best legal move)
policy_logits, wdl_logits, best_move = eng.evaluate(board, contempt=0.0)

# batched (returns raw logits, Nx4272 and Nx3)
policy, wdl = eng.evaluate_batch([board1, board2, board3], contempt=0.0)

Inputs:

• square_ids int64 [B, 64] — piece ids 0–12 (empty=0, white P..K=1..6, black P..K=7..12)
• state_features float32 [B, 8] — [stm, WK, WQ, BK, BQ, ep_file/7 or -1, halfmove_bucket, repetition]
• contempt float32 [B] — scalar per batch; 0.0 for tournament/production

Outputs:

• policy float32 [B, 4272] — raw logits; apply legal_mask + softmax before MCTS
• wdl float32 [B, 3] — raw logits (win/draw/loss, mover POV); softmax to get probs

Citation / Provenance

Trained on the feature/chessformer-lite branch of junisbuilding/chessdb (commit 3ac1ed4). Parent baseline: burrowdweller/minichess-gpt-v1-final.

Caveats / Known follow-ups

• Match sample is 20 games (point estimate +147 Elo, 95% CI spans 0). Directional, not tight.
• Contempt on the CLS token leaks into policy via attention — slight confound with the "policy independent of c in v1" design. Worth detaching c from the policy path in a follow-up.
• FiLM γ initializes near 0, slowing early value conditioning. γ←1 identity init recommended.