Chess transformer 270M (squares64 recurrent)
~269M parameter recurrent chess policy/value net. Width-only scale of the 99M incumbent (1216d / 16H). Attention is only over the 64 board squares. Turn, castling, and en passant are FiLM on the square stream, not extra tokens.
This file is latest.pt at exp271 distill step 45000 (2026-09-11 13:09 UTC).
Warm start from the public pretrain snapshot (step 43000), then 2000 Polar-NorMuon
steps of online KD from the frozen 99M teacher.
Architecture
| Params | 268.55M |
| Hidden / heads | 1216d / 16 |
| Encoder dim | 256 |
| Trunk | prefix 4 + bank 7×3 unrolls + suffix 4 |
| Effective depth | 29 (15 unique layer modules) |
| Attention | 64×64 squares only |
| Embeds | fused piece×color (13) |
| Side info | FiLM (turn / castling / EP), zero-init |
| Heads | spatial policy (vocab 1968 compact) + 3-way WDL |
| Recurrent grads | bank grads divided by 3 after backward |
Config is in model_config.json (Squares64RecurrentConfig).
Training
Boards from avewright/chess-soft-sf19.
Policy soft target is Hinton KD (T=2, teacher_kd_frac=1.0) on logits from
avewright/chess-transformer-100m-squares64.
Hard CE stays on the Stockfish 19 best move. WDL stays on the SF19 labels
(soft_alpha=0.85, value_weight=0.15).
| Optimizer | Polar-NorMuon (re-init; public 43000 ckpt was weights-only) |
| Batch | 64 |
| Train loss @ 45000 | ~0.96 |
| Val SF19 @ 45000 | hard_ce 1.83 / teacher_kl 0.73 / wdl_ce 0.89 |
Pretrain mix (through step 43000) was Lichess MultiPV + Syzygy + a small disagreement sprinkle. That recipe is in older commits.
Strength (search-free greedy)
No book, no Syzygy, no MCTS. Legal-masked policy argmax vs Stockfish 19
UCI_LimitStrength / UCI_Elo. 8 openings × 2 colors × 2 = 32 games.
Not FIDE / Lichess Elo.
| Opponent | W-D-L | Score | Estimate |
|---|---|---|---|
| SF19 2050 | 4–18–10 | 0.406 | ~1984 |
| SF19 2150 | 7–11–14 | 0.391 | ~2073 |
Same 2150 protocol, 99M teacher: 3–16–13 (0.344, ~2038). 32-game CIs overlap.
Inference
Requires this repo (chess_inference.py, chess_squares64.py) and compact vocab 1968.
pip install torch huggingface_hub python-chess
export MOVE_VOCAB_VERSION=compact
import os
os.environ["MOVE_VOCAB_VERSION"] = "compact"
import chess
from huggingface_hub import hf_hub_download
from chess_inference import load_checkpoint, get_model_move
path = hf_hub_download("avewright/chess-transformer-270m-squares64", "latest.pt")
device = "cpu" # or "cuda"
model = load_checkpoint(path, device=device)
board = chess.Board()
move, info = get_model_move(model, board, device)
print(move, info["top_moves"], info["wdl"])
Files
latest.pt— inference checkpoint (model_state_dict,config,arch,steps)model_config.json— architectureelo_eval_sf2050.json/elo_eval_sf2150.json— 32-game screens at step 45000