avewright/chess-transformer-437m-ft3h

Chess Transformer 437M — FT3h (best Elo)

Greedy policy checkpoint from the exp191 meta-attention line. Best measured Elo in this project: ~1950 (Stockfish LimitStrength gauntlet).

File	Description
best_model.pt	Inference weights (~1.75 GB). Keys: model_state_dict, config, vocab_version, training meta.
config.json	Architecture + FT3h training hyperparameters
elo_eval.json	Full greedy gauntlet dump
README.md	This card

Play checkpoint: use greedy argmax over legal moves (temperature 0). Weak MCTS settings previously hurt Elo vs greedy.

Architecture

• 437M params (hidden_dim=1280, num_layers=18, num_heads=20)
• Strengthened board encoder + meta-factored attention + Shaw-on-pos
• Compact move vocab (1968), STM-normalized fused board tokens
• Policy + 3-class value head
• Trained with NorMuon + PolarExpress, torch.compile, cautious WD

Elo (greedy, movetime 0.05s SF side)

SF Elo	Score	W-D-L (16 games)
1450	0.750	10-4-2
1600	0.562	6-6-4
1750	0.562	7-4-5
1900	0.594	6-7-3
2050	0.312	2-6-8

Estimate ~1950 (bracketed by SF1900 / SF2050). Black is weaker at high SF; French / English / Réti openings are relative weak spots. Losses are mostly tactical checkmates.

Training lineage

exp191 pretrain / early soft FTs
  → exp191_soft_ft3_lichess
  → exp191_soft_ft3b_unseen          (~1832 Elo)
  → exp191_soft_ft3h_edge_end  ★     (~1950 Elo)   ← this upload

Later soft / loss / tactics corrective FTs (ft3e/f/g/i, loss_fix, blunder_hard, tactics_hard) regressed Elo and were discarded. Do not prefer them over this file.

FT3h recipe (this checkpoint)

Setting	Value
Init	exp191_soft_ft3b_unseen/best.pt
Steps	6000 (best @ step 5000 by top1)
Batch	256
Soft frac	0.85
Soft α (vs hard CE)	0.38
Deep mix	0.42
Muon / Adam LR	0.0009 / 4.5e-5
Select metric	top1 (not soft_loss)
HFlip	0.5

Data mixture used at FT3h

1. Shallow soft MultiPV (~2.0M train) — outputs/exp186_sf_multipv/soft_cache.pt
Stockfish MultiPV=8 soft targets on general positions (exp186 harvest from Lichess/SF-labeled position corpora such as avewright/chess-positions-lichess-sf / avewright/chess-positions-sf-labeled). Also mirrored on HF as dataset work under avewright/chess-soft-multipv-lichess.

2. Deep soft mix (~3.24M train) — outputs/unseen_quality_mix_ft3h/soft_cache.pt
Built to avoid overlap with prior FT caches (~33.9M excluded keys):

Component	Role
Pool: soft_cache_virgin_b_12m.pt (Lichess cloud-eval soft)	Unseen deep positions, phase-balanced → ~3.1M
Phase / edge / endgame / puzzle / syzygy spice	Attempted; most spice was already in the exclude set so little new volume landed
Mean label depth (mix report)	~71 (cloud-eval style depths; quality varies by source)
Phase mix	~21% opening / 46% middlegame / 33% endgame

3. Hard ballast (~1.0M / step mix) — hard_ballast_d15_n2000000_s42.pt
Hard CE on Stockfish best-move labels, min depth ≥15, for stability when soft frac < 1.

Prior stage data (inherited via FT3b init)

FT3b was fine-tuned from ft3_lichess on:

• Same shallow exp186 soft cache
• Deep: soft_cache_unseen_ft3.pt (~3.2M Lichess-eval soft, held out from earlier FT3)
• Soft α 0.55, deep mix 0.55, hard ballast

Weights therefore already encode earlier Lichess MultiPV soft pretraining before the FT3h edge/end-lean mix.

Related public datasets

• avewright/chess-soft-multipv-lichess — soft MultiPV caches
• avewright/chess-soft-syzygy — syzygy soft (~498k)
• Lichess/chess-puzzles — puzzles (used in later experiments; not a major FT3h spice contributor)

Load

import torch
from huggingface_hub import hf_hub_download

path = hf_hub_download("avewright/chess-transformer-437m-ft3h", "best_model.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
state = {k.replace("_orig_mod.", ""): v for k, v in ckpt["model_state_dict"].items()}
config = ckpt["config"]  # dict → ChessTransformerConfig(**config)
# build_model(config); load_state_dict(state); vocab_version = ckpt.get("vocab_version", "compact")

Requires this repo’s ChessTransformer / compact move vocab (MOVE_VOCAB_VERSION=compact).

License

Weights: MIT (same as project). Upstream Lichess eval / puzzle data retain their respective terms (puzzles: CC0).