Chessformer: A Unified Architecture for Chess Modeling
Authors: Daniel Monroe
George Eilender
Philip Chalmers
Zhenwei Tang
Ashton Anderson
Venue: ICLR 2026 Poster (2026)
Keywords: Transformers
Interpretability
Human-Aligned AI
Chess
Action Prediction
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We present a transformer architecture for chess that advances the state of the art across several chess modeling tasks.

Abstract: Chess has played a uniquely important role as a testbed domain for artificial intelligence. Applying new architectures to improve absolute chess performance, and more recently to predict human moves at specified skill levels, has therefore garnered attention in the machine learning literature. Current approaches to these problems employ transformer models with widely varying architectural designs, and use unintuitive tokenization schemes that are not amenable to interpretability techniques, which hinders their applicability for teaching and human-AI interaction. We introduce Chessformer, a novel chess transformer model design that consists of an encoder-only model which processes chessboard squares as input tokens, instead of moves or the entire position, a dynamic positional encoding scheme that allows the model to flexibly adapt to the unique geometries present in chess, and an attention-based policy output design. We show that Chessformer advances the state of the art in all three major chess modeling goals: it significantly improves the chess-playing performance of a state-of-the-art chess engine, it surpasses the previous best human move-matching prediction performance with a much smaller model, and it enables substantial interpretability benefits. Our unified approach constitutes a broad advance across several important tasks in chess AI, and also demonstrates the benefits of carefully adapting transformers' tokenization systems, output systems, and positional encodings to reflect the structure of a domain of interest.

Under review as a conference paper at ICLR 2026
000
CHESSFORMER: CHESS MODELING
A UNIFIED ARCHITECTURE FOR
001
002
003
004
005
Anonymous authors
Paper under double-blind review
006
007
008
009
ABSTRACT
010
011
Chess has played a uniquely important role as a testhed domain for artificial in- telligence. Applying new architectures to improve absolute chess performance, and more recently to predict human moves at specified skill levels, has therefore garnered attention in the machine learning literature. Current approaches to these problems employ transformer models with widely varying architectural designs, and use unintuitive tokenization schemes that are not amenable to interpretability techniques, which hinders their applicability for teaching and human-Al interac- tion. We introduce Chessformer, a novel chess transformer model design that consists of an encoder-only model which processes chesshoard squares as input tokens, instead of moves or the entire position; a dynamic positional encoding scheme that allows the model to flexibly adapt to the unique geometries present in chess; and an attention-basee advances the state of the art in all three major chess modeling goals: it signifi- cantly improves the chess-playing performance of a state-of-the-art chess engine, it surpasses the previous best human move-matching prediction performance with a much smaller model, and it enables substantial interpretability benefits. Our uni- fied approach constitutes a broad advance across several important tasks in chess AI, and also demonstrates the benefits of carefully adapting transformers' tok- enization systems, output systems, and positional encodings to reflect the structure of a domain of interest.
012
013
014
015
016
017
018
019
020
021
policy output design. We show that Chessformer
022
023
024
025
026
027
028
029
030
031
032
1 INTRODUCTION
033
034
A central goal of artificial intelligence (AI) is to build systems that are simultancously high- performing and human-compatible. Capable yet intelligible models not only solve tasks but can also collaborate with and teach human users by understanding their strengths, weaknesses, and learning trajectories. Chess provides a particularly well-suited model system for this dual goal: while mod- em engines are decisively superhuman, their behavior is often opaque even to experts, leaving a persistent gap on the human side of the problem.
035
036
037
038
039
040
The current literature on developing strong chess engines and accurate human move prediction is dis- jointed and times unprincipled. Prior systems for human emulation span convolutional stacks over board images (Mellroy-Your))
041
042
et al. 2020). skill-aware attention atop convolutional features (Tang al. 2024), and language modeling over move histories (Zhang al., 2025); other efforts distill strong oracles (Ruoss et al. 2024) with one-dimensional position representations that are potentially misaligned with the underlying action space, complicating both efficiency and analysis.
043
044
045
046
We introduce Chessformer, a unified architecture that advances the state of the art on three fronts at once: produces substantial gains over prior methods in raw chess engine ability, achieves state of the art human move-matching performance, and it admits downstream interpretability much more naturally than previous models. Concretely, Chessformer is an encoder-only transformer that treats the 64 board squares as tokens, pairs this square-token body with an attention-based "source- destination" policy head, and equips the trunk with Geometric Attention Bias (GAB), a novel dy- namic positional-be bias generator that adapts attention to the geometry of position. It has also been adopted in the leading open-source Leela Chess Zero chess engine in configurations that defeated Stockfish, perennial world champion, in computer-chess competitions.
047
048
049
050
051
052
053

---

Under review as a conference paper at ICLR 2026
054
Empirically, we find that GAB is key driver of these gains. Ablations show consistent improve- ments over absolute and relative position encodings across Elo, puzzle accuracy, and policy and value accuracy, with sizable performance-per compuuuu advantages. These results reinforce broader lesson: adapting tokenization, output heads, and positional encodings to the domain's 's structure al- lows transformer models to more flexibly adapt to both task mastery and human-compatibiityy In chess, this yields a single model family that pushes engine strength, advances human move match- ing, and makes square-level attributions and attention pattern interpretability straightforward, all while being over an order of magnitude more parameter and compute efficient than prior art. That Chessformer achieves these simultaneously surprising, both because chess very well-studied problem which is considered gold standard for AI, and because past approaches have focused on only one of these at a time. In short, Chessformer demonstrates that principled, architectural choices can deliver state-of-the-arr chess performance and human emulation, while also enabling interpretability analyses that can power downstream homan-compatible applications, offering template for other structured decision-making domains. We open-source our full pipeline, thub. including all code and training data, at htttps:: http:///gggtuub...ccssss.saaa com/CsSaabmmalaa.
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
RELATED WORK
070
071
072
Traditionally, computational approaches to chess have focused on maximizing absolute playing
strength by developing hand-crafted scarch heuristics and position evaluations. This approach gave rise to Deep Blue (Campbell et al.; 2002), which in 1997 became the first computer to defeat a reigning human World Chess Champion under official tournament conditions, and Stockfish (Rom- stad et al. 2023), which is generally considered the strongest chess engine available today. Alp- haZero (Silver et al. (0188) introduced a different recipe based on Monte Carlo Tree Search (MCTS) and reinforcement learning, training neural networks through self- play to predict state values and policy distributions over subsequent actions. Its open-source re-implementation Leela Chess Zero Pascutto & Linscott (019) refined this approach with new neural network architectures and search strategies and often ranks as a close runner-up to Stockfish computer chess competitions. Even without search, transformer-based agents can achieve grandmaster-level strength when strong ora- cles are distilled into them (Ruoss et al. 2024).
073
074
075
076
077
078
079
080
081
082
083 A more recent line of work aims to develop chess systems that are not only strong but also human- compatible, in the sense that they understand and are attuned to the behavior of humans, by modeling human play across skill levels. This was first explored by MAIA (Mellroy- Young et al. 2020), which employed a set of convolutional neural networks, each trained to model players at a specific rating range. iaaobb et al. (2022) augmented these with search to better replicate strong play. MATA-2 [Tng et al. 2024) simplified MATA'S 's approach with a unified model, introducing a skill-aware self- attention layer that tokenized the channels of the output of stack of convolution layers. ALLIE (Zhng et al.) 2025) viewed this behavior replication task through the lens of language modeling, training a decoder-only transformer model on move-based representation of the game trajectory to achieve state-of -ftiee-ff-hhetrrtre the-art human move matching. These human emulation methods were exploited by Hamade et al. (2024) to investigate human-Al cooperation in chess. Modeling individual behavior, Mcllroy- Young et al. (202T) demonstrated that human players can be reliably identified from just a few of their games, while Mcllroy- Young et al. (2022) improved move-matching performance on individual players by finetuning on their games.
084
085
086
087
088
089
090
091
092
093
094
095
096
Chess has also served as a model system for broader problems. [arebrother et al. (2024) used chess to demonstrate that classifying rather than regressing improves scalability in deep reinforcement learning, while Feng et al. (2025) investigated creative generative Al in the testbed of chess puzzles. In mechanistic interpretability. which studies the mechanisms underlying the behavior of Al models, McGrath et al.] (2022) used linear probes to identify concepts learned by AlphaZero, while Jenner et al. (2024) found evidence of learned planning in transformer model trained by the Leela Chess Zero project.
097
098
099
100
101
102
103
104
METHODOLOGY
105
106
Here we describe our Chessformer architecture and general training setup. Fundamentally, Chess- former is encoder-only transformer that processes the 64 chessboard squares tokens and aug-
107
2

---

Under review as a conference paper at ICLR 2026
108
109
Mallieiodd Atlentio Mechansom
110
Atandon Incul
faed
111
+
112
Q
Teaatppe
113
Hesse
Speet
OK
114
115
Reatape
116
118
119
Cocmaitt
BEM
120
Mullppection
121
Additionn
122
123
124
Figure 1: Chessformer attention mechanism. Chessformer adopts the natural visual representation of the chessboard, processing the 64 board squares as tokens. It augments the dot-product attention mechanism with Geometric Attention Bias (GAB), a novel position encoding that generates biases for attention logits from compressed representation of the board state. A consistent theme of attention in Chessformer models harmonious collaboration between the semantic dot-product at- tention component and the positional GAB component. In the example shown, the querying square white's s rook on cl, highlighted in red. Higher attention scores are colored yellow, while lower attention scores are colored purple. The dot-product logits highlight important pieces, while the
125
126
127
128
129
130
131
GAB biases focus on squares that are rook's move away. Together, they point white's rook to the pinned black knight c5.
132
133
134
135
ments self-attention layers with novel dynamic position encoding called Geometric Attention Bias (GAB). We evaluate this architecture on two chess modeling tasks: emulating human play and max- imizing playing strength as the distillation target of strong engine. More concretely, for both tasks, we assemble a dataset chess games and train Chessformer models in a supervised fash- ion to take in board states from those games and predict both game outcomes and played moves (or for engine distillation, distributions over moves corresponding to playouts chosen during engine self-play).
136
137
138
139
140
141
142
143
3.1
BOARD REPRESENTATION SQUARES AS TOKENS
144
145
Existing transformer-baase methods in chess use variety of tokenization schemes: ALLIE repre- sents position through the trajectory past moves, MAIA-2 tokenizes the channels of the output of stack of convolution blocks, and Ruoss al. (024)) applies rotary position embeddings (Su lt al. 2024) on top of a tokenized representation of the 64 board squares and additional information contained in the Forsyth-Ed Edwards Notation (FEN) (Edwards 1994) representation of the position. We argue in Appendix D that these formulations are misaligned with the domain of chess and may therefore hinder performance. Our proposed Chessformer instead adopts the natural visual repre- sentation that treats individual chessboard squares as tokens. This allows model representations to correspond to more uniform units and fixes the relative positions of tokens in two dimensions, en- abling more effective position encodings. It also allows tokens to specialize to their corresponding squares rather than represent the entire board state, which greatly reduces the load on parameters.
146
147
148
149
150
151
152
154
155
156
Concretely, positions are represented each indicating which of the 12 pieces are present on the corresponding square, and the board is flipped with the side to move. To obtain the input for given board state, concatenate represen- tations of the current and past n positions, where n is non-negative integer controlling the amount of history information conditioned upon, and repeat the earliest position if some or all of these past positions are not available. Unless otherwise stated, n 7. For the engine distillation setup. we also concatenate auxiliary information that necessary for compatibility with the Leela Chess
sequence of 64 one-hot or zero vectors of dimension 12,
157
158
159
160
161
3

---

Under review as a conference paper at ICLR 2026
162
Zero infrastructure, though this did not noticeably affect performance in initial experiments; Appendix A.2 The final board representation consists of 64 tokens, one for each square.
163
164
Our setup is most similar to that of Ruoss et al. (2024), which tokenized the 64 board squares in addition to other information about the position. for a total of 77 tokens. However, that work adopted rotary position embeddings {Su et al.
165
166
a linearized representation of the board squares, enforcing a one-dimensional structure on the tokens that is not grounded in the domain. This is especially deleterious given the central role that position plays in chess. For example, among relationships between squares, their architecture maximally decays the attention strength between opposite corners, even though those comners lie on a main diagonal that critical for checkmate patterns, We propose a novel position encoding called the Geometric Attention Bias (GAB) that rectifies this by allowing self-attention to dynamically and flexibly model piece movement in two dimensions.
2024
167
168
169
170
171
172
173
174
175
3.2
GEOMETRIC ATTENTION BIAS
176
177
178
Self-attention in Transformer architectures is permutation-invarian so positional information must be introduced to the model through some kind of position encoding. In language and vision settings, simple Euclidean distance is arguably the predominant notion of position, and thus static position encoding schemes, like rotary position embeddings and absolute and relative biases (refer to Ap- pendix C for details), power state-of-the-art art models in these domains. However, chess follows its own special geometry. in which the six piece types move in particular ways. In addition, positional relationships can vary heavily with the board state. As simple example, relationships correspond- ing to the movement of a particular piece are only sensible if that piece is present on the board. But chess is rife with more complex interactions; for example, connections between distant squares are weaker in locked positions. A more versatile positional encoding is necessary.
179
180
181
182
183
184
185
186
187 To capture the variable geometry of chess, we propose adaptive position encoding called Geo- metric Attention Bias (GAB). GAB uses a compressed representation of the board state to generate biases for each attention head from set of "templates". To compress the board state, tokens first undergo a linear projection of dimension d, and are flattened, followed by a linear projection of dimension d2 with GELU activation and layer normalization. To generate the attention biases, apply another linear projection of depth hdg followed by activation and normalization, and reshape to h dg. We apply final linear projection, shared by the whole model to reduce parameter count and accelerate learning, to form biases h x 4096 which are reshaped to fa x 64 x 64 and added to the dot-product logits before softmax. This final projection can be viewed as dynamically mixing a set of d3 attention bias templates. Pseudocode for generating GAB biases can be found in Figure
188
189
190
191
192
193
194
195
196
197
Our approach has a number of benefits. First, models positional information globally rather than through individual tokens. This aligns well with our later finding that Chessformer models mainly adapt the GAB component of the self-attention computation to global positional features like the game stage (opening, middlegame, endgame), rather than local features like the locations of indi- vidual pieces. Second, representing attention logits as the sum of a semantic component generated by dot-product attention and positional component generated through GAB allows self-attention layers to assign relevant aspects of the attention computation to each part. The choice of a dynamic position encoding enables attention heads to be repurposed based on the board state, behavior we explore in Section ( Finally, formulating the interaction additively allows us to use existing memory-efficied attention kernels (Dao et al. 2022) during training and inference.
198
199
200
201
202
203
204
205
206
207
208
3.3
OUTPUT HEADS
209
210
All models we train have two output heads: value head that predicts the game outcome (win, draw, and loss), and a policy head that predicts the move (or in the case of engine oracle self-play games, the distribution of moves corresponding to playouts) chosen during the game. To generate the game outcome prediction, we apply mean pooling to the encoder body's 's output, followed by a layer normalization layer. We then apply linear projection to dimension 128. followed by a ReLU nonlinearity, followed by a linear projection of dimension 3 whose output is taken as the logits for the game outcome prediction target.
211
212
213
214
215
4

---

Under review as a conference paper at ICLR 2026
216
Table 1: Main results for human modeling. The MAIA-3 family of models achieves new state-of- the-art performance with a fraction of the parameter count.
217
218
219
Accuracy (%)
#Params
History
Search
220
MATA-3-79M
57.1 + 0.1
79M
X
221
MAIA-3-23M
56.6 + 0.1
23M
X
222
MAIA-3-5M
55.4 + 0.1
5M
V
X
223
55.9 + 0.1
355M
J
/
224
ALLIE-POLICY
55.7 + 0.1
355M
V
X
225
MATA-2
52.0 + 0.1
23M
X
X
226
MAIA
51.6 + 0.1
92M
X
X
227
GPT-3.5
53.7 + 0.1
175B
X
228
229
230
231
Prior work has modeled move distributions variety of ways. ALLIE autoregressively predicted tokens corresponding to each of the 1968 possible moves in Universal Chess Interface (UCI) nota- tion, while MAIA-2 used an MLP layer. We propose a policy head based on self-attention that re- flects the "from-to" structure of the underlying action space, encoding moves by the starting square and destination square of the moved piece. Given 64 tokens returned by the encoder body, we generate via linear projection set of query vectors corresponding the starting square and a set of key vectors corresponding to the destination square, both with depth equal to the depth of the encoder body. Logits for moves calculated via scaled dot-product, resulting in 64x64 matrix representing all possible traversals from one chessboard square to another. This is sufficient to rep- resent all moves except castling and promotions, implementation details for which are reported in Appendix A.3
232
233
234
235
236
237
238
239
240
241
Though we did not observe large performance benefit from this policy head formulation, found it to substantially improve the interpretability of MLP activations, which we attribute to its alignment with the domain: see Section 6 We hope that this design choice will help position our proposed architecture as useful test case for future mechanistic interpretability research.
242
243
244
245
246
247
4
PREDICTING HUMAN PLAY
248
249
4.1
DATASET
250
251
We construct training dataset consisting of blitz games played on the online chess platform Lichess from January 2023 to July 2025. As the bulk of these games are from the middle of the skill distribution, we re-sample the games during training so that all skill levels are equally represented. The standard evaluation metric for human emulation is move-mat accuracy, which is the rate at which, given a board position encountered in a real game, a model correctly predicts the move adopt the dataset of 884,049 positions curated by Zhang et al. (2025), which we call the ALLIE test set. These were formed by sampling Lichess blitz games from 2022 and removing the first 10 moves from each game, as these can be easily memorized, as well as positions which occur after the first time a player has less than 30 seconds on the clock, which tend be more noisy due time pressure. During training, retain the first 10 moves but discard moves made under time pressure in the same way. We describe the processing of training data more detail in Appendix A.
252
253
254
255
played by human player. For main analysis,
256
257
258
259
260
261
262
263
4.2
TRAINING METHODOLOGY
264
265
We perform three main training runs at scales of million, 23 million, and 79 million parameters, calling the largest of these MAIA-3. We ablate the position encoding at the smallest scale, comparing to the absolute and relative biases described in Appendix C We also train smaller 3-million- parameter ablation show that GAB enables significantly smaller models to outperform these. For
266
267
268
269
GAB models at the 5M and 3M scale, we replace the first linear projection and flattening layer of GAB with average pooling these parameter-intensivee at small scales. Initial experiments
5

---

Under review as a conference paper at ICLR 2026
270
Table 2: Position encoding ablations for human emulation.
271
272
Loss
Accuracy (%)
FLOPS
#Params
273
Policy
Value
Policy
Value
274
275
Absolute
1.418
0.754
54.7 0.1
62.6 + 0.1
268M
4.58M
276
Relative bias
1.420
0.754
54.6 0.1
62.6
268M
4.58M
MAIA-3 MAIA-3-3M
1.419
0.739
0.1
62.6 0.1
164M
2.98M
277
MAIA-3-5M 1.387
0.736
5.14
62.8 + 0.1
276M
4.91M
278
279
280
281
showed this to have only a minor effect on performance, decreasing move-matching accuracy by approximately 0.2%.
282
283
Skill in chess is modeled with the Elo system, where Elo ratings vary roughly from 0 for weak players to 3000 for the strongest human players. We condition human-emulating models on the skill levels of both players by prepending two "soft embeddings" of dimension 128, corresponding to the Elo ratings of the players, to each of the 64 tokens. Following ALLIE, we compute an embedding ek for Elo rating k linear interpolation between two learnable embeddings: a weak embedding (Eweak) corresponding to Elo and a strong engine-level embedding (Carong) corresponding to 5000
284
285
286
287
288
289
50y0 Representing the Elo ratings scalar inputs would achieve the same representational capacity, but this design enables the flexibility to, for example, model individual behavior stylometry. There are several Elo rating systems currently in use that are generally incomparable. Ratings on Lichess, from which we source our training data, tend to be slightly inflated compared to the more standard FIDE ratings
Elo. Formally, we set = 7Eweek + (1 es where 7 = 5000-k
290
291
292
293
294
The final input for our human emulation models consists of 64 tokens, a concatenation of represen- tations of the current and n past board states and two strength embeddings of dimension 128. This comes out to a depth of 12 x (1 + n) + 2 x 128, which is 352 for the n hyperparameter choice used in our main training and ablations runs. Despite the dimensions of this input being dominated by these embedding vectors, we did not find the choice of embedding dimension 128 to impact per- formance. Detailed information our training setup and model hyperparameters can be found in
295
296
297
298
299
Appendix[A. A.I
300
301
302
4.3
BOARD HISTORY
303
304
History-less representations of chess have a "Markov Property" in that future board states are in- dependent of previous states given the current state, with the (typically negligible) caveat of the threefold repetition nule that a game shall end in a draw when a position repeats three times. Though optimal play in given position is almost always independent of previous moves, it is not clear that this should extend to human play. Humans often form long-term plans or exhibit consistent weak- nesses that may be discernible from their previous actions, which may make this information useful when replicating their play. Prior human emulation work varies in its use of history information, with move-token methods like ALLIE conditioning on the full game history, and equare-based meth- ods like MAIA and MAIA-2 omitting history information entirely. To determine the usefulness of history information in modeling human chess play, condition models the current and past board states for varying values of n. Ablation details can be found in Appendix
305
306
307
308
309
310
311
312
313
314
315
4.4
RESULTS
316
317
Table rrpports overall move- matchin accuracies of MATA-3 models on the ALLIE test set, demon- strating a marked improvement in move-matching performance and parameter efficiency. Figure S shows that these gains hold across a wide range of skill levels, with the improvement increasing in the game rating. Impressively, our 79M-paramete and 23M-parameter models achieve respec- tive mote-natching accuracies of 57.1% % and 56.6%, outperforming the offte-ofthhe-ee 355-milli 355-million- parameter searchless (55.7%) and search-enables (55.9%) ALLIE methods at significantly smaller scales. Our 5M- -arrmmemte model reaches a move-matching accuracy of 55.4%, achieving results comparable to the state of the art 70 times fewer parameters.
318
319
320
321
322
323

---

Under review as a conference paper at ICLR 2026
324
The MAIA-2 paper trained two models, one rapid games and one blitz games, and we evaluate the latter for a fair comparison. Results for MAIA and GPT-3.5 are taken from Zhang et al. (2025). MAIA* was formed by choosing the MAIA model with Elo rating closest to the active player Elo rating, and GPT-3.5, which outperformed more recent models like GPT-4, operates on a move-based board representation.
325
326
327
328
329
We plot the move-matching accuracies of MAIA-3 by the Elo ratings of the active and opponent players in Figure Results for ablating the position encoding are presented in Table while results for ablating the number of history positions n are reported in Table X The baseline MAIA-3-5M model significantly outperforms ablations equipped with absolute and relative biases, while MAIA- 3-3M matches their performance at 30% fewer parameters and 40% fewer FLOPS. In other words,
330
331
332
333
334
GAB enables a Pareto improvement across model scale, computation, and policy and value metrics.
335
336
337
5
OPTIMIZING PLAYING STRENGTH
338
339
340
We now test our Chessformer architecture-a 64-token encoder transformer with Geometric At- tention Bias (GAB) and attention policy-on the task of optimizing raw searchless chess playing strength. To do so, we distill the Leela Chess Zero engine into Chessformer models, and perform ablations on the position representations to assess their impact. We then analyze a 191-million parameter Chessformer model trained by that project, which we call Leela-CF, comparing agents constructed from that model to prior work on searchless playing strength. Finally, we demonstrate that these gains in searchless playing strength can be extended to engine strength- first by perform- ing tournament between configurations of Leela Chess Zero equipped with either the Leela-CH Chessformer or the Lecla-CNN convolutional model previously used in tournaments- and then by describing several prominent computer chess tournaments in which Chessformer-equipped configu- rations of Leela Chess Zero defeated pools of engines that included Stockfish, a frequent champion in these kinds of events.
341
342
343
344
345
346
347
348
349
350
351
352
353
5.1
TRAINING METHODOLOGY
354
355
356
Leela Chess Zero is an open-source recreation of AlphaZero, which iteratively teaches a randomly initialized neural network to interact with a do- main by generating self-play games between MCTS- augmented versions of that neural network. This neural network outputs a policy distribution that pre- dicts the distribution of moves chosen by the MCTS algorithm and a value that predicts the outcome of the game. In effect, a model continually generates strong search-enan -eablee oracles whose play is then dis- tilled back into that model.
Table 3: Results for raw playing strength. Chessformer results in significant gains in both Elo and puzzle solving rate, using fewer FLOPS than competing models.
357
358
359
360
Flo
Puzzles (%)
FLOPS
361
Leela-C Prela-CF-pollccy (ours)
2374 37
93.5 + 0.5
7.6B
362
Leela-CI CF-value (ours)
2466 1: 36
07.2 1 0.3
152B
363
AC-9M
2144 1: 42
86.2 1 0.7
14.2B
AC-136M
2257 = 36
92.7 + 0.5
215B
364
AC-270M
2299
36
94.2 0.5
427B
365
-Nolocc Lodla-CNN-vatue
2096 40
82.1 1: 0.8
12.5B
366
The most expensive component of the AlphaZero process by far the generation of training games, which typically requires hundreds of model evalua- tions per position. We skip this step by fixing dataset of self-play games from an April 2024 reinforcement learning run of Leela Chess Zero, keeping only those games that occurred once the model's strength had leveled off. That run used a transformer model of roughly 100 million pa- rameters at 600 nodes per move with a square-based token representation and our GAB biases and attention policy, In this way, we move to the supervised setting, distilling a search-augmented ver- sion of one model into another.
2168
36
92.5 : 0.5
249B
367
368
369
370
371
372
373
374
375
We motivate our design choices under this setup, ablating the position encoding of a 4-million- parameter model with those described in Appendix C We also train a Chessformer model with 2.5 million parameters to determine the extent to which our techniques can replace the need for model size. Our oracle distillation setup is described in more detail in Appendix A.2
376
377
7

---

Under review as a conference paper at ICLR 2026
378
5.2
EVALUATION METHODOLOGY
379
380
We include in our analysis two types of agents: those that function based on policy information by picking the highest-ranked move in the policy distribution predicted by the model, and those that function based on value information, emulating a depth-1 search by evaluating the model for each legal move and selecting the move that maximizes the resulting evaluation. The policy strategy requires a single model evaluation, while the value maximization strategy requires an evaluation for each legal move. To estimate the floating point operations (FLOPS) per evaluation used by an agent of the value maximization type, we multiply the model FLOPS by 20, which is a conservative estimate of the average number of legal moves available in a position.
381
382
383
384
385
386
387
388 We construct agents from the D0l-million-parmmeee by its model name followed by the strategy it uses (e.g.. Leela-CF-policy and Leela-CF-value) We also construct agents from both strategies using a -95-million-parameter convolutional neural network trained by the Leela Chess Zero project, which we call Leela-CNN. Our analysis also includes models from Ruoss et al. (2024) that use the value maximization approach. We use the final checkpoints of their main runs having parameter counts of 9 million, 136 million, and 270 million, referring to these AC-9M, AC-136M, and AC-270M, respectively.
model using both strategies, denoting an agent
389
390
391
392
393
394
395
Our evaluation setup is adapted almost exactly from Ruoss et al. (2024). We compare agents on both tournament strength and puzzle solving ability. To measure the former, we play 200 games between each pair of agents and calculate relative Elo ratings using BayesElo (Coulom 2008). We perform separate tournaments for the main and ablation runs, anchoring the Elo value of the absolute position encoding to 0 and the Elo value of AC-270M to the value reported by Ruoss et al. To measure puzzle solving ability, we report the accuracy of these models on a test set of 10 thousand puzzles curated by that work. For our ablation runs, we also report accuracy and loss metrics on the value and policy heads. These were calculated on 1,403,999 test positions, sourced from the same reinforcement learning run, that did not intersect the training set.
396
397
398
399
400
401
402
403
404
5.3
RESULTS
405
406
As shown in Table the Chessformer policy agent Leela- Le-la-CF-policy matches or outperforms all other agents we consider in both tournament strength and puzzle-solving ability despite having the lowest computational cost. We also see performance on the puzzle set approaching saturation, suggesting that new metries for evaluation might be needed.
407
408
409
410
Our ablation results, reported in Table 6 show a monotonic improvement in performance from ab- solute to relative bias to GAB encodings. GAB outperforms the baseline absolute position encoding by 1.96% on move-matching accuracy, 0.34% on game outcome prediction accuracy, 3.37% on puzzle-solving accuracy, and 80 Elo rating points in tournament strength. We note that the max- imum possible policy and value accuracies are well under 100%, both because Leela's self-play methodology is inherently nondeterminstic and because there are positions with variety of best moves. Impressively, GAB enables a model to perform on-par with the absolute position ablation at nearly half the parameter count and computation.
411
412
413
414
415
416
417
418
5.4
ENGINE STRENGTH
419
420
To demonstrate that our Chessformer models have the capacity to push engine strength, we perform a tournament between configurations of the Leela Chess Zero engine paired with either the Leela-CNN model, a 195-million- p5-million-ppramette tournaments, or Leela-CF, a 191-miliion parameter Chessformer. We play 2000 games between these configurations at three time controls, described in further detail in Appendix B As shown in Table 7 the Leela-CF Chessformer model consistently increases the playing strength of Leela Chess Zero by over 100 Elo. To contextualize these gains, the difference in playing strength between Stockfish versions 16 and 17, corresponding to 14 months of continuous development progress, was measured at around 46 Elo under a similar testing setup (Stockfish Team 2024). This is especially notable due to the difficulty of improving top engines; the Stockfish project continually employs thousands of CPU cores to test over 10,000 potential improvements per year (Stockfish Team 2022).
421
422
parameter convolution-base
model previously used by Leela Chess Zero at
423
424
425
426
427
428
429
430
431
Configurations of Leela Chess Zero equipped with Chessformer models defeated Stockfish, a peren- nial champion, at several online computer chess tournaments hosted by the Top Chess Engine Cham-
8

---

Under review as a conference paper at ICLR 2026
432
pionship (TCEC). These victories included the single-elimination TCEC Cup 11 tournamen where 32 engines faced off in brackets, and two Swiss-system tournament the TCEC Swiss 6 and events, each of which had around 40 contestants. In the cup event, Leela Chess Zero advanced through every round, defeating Stockfish in the finals. In the Swiss events, Leela Chess Zero scored first place both times, once one point ahead of the field and once tiebreak.
433
434
435
436
437
438
6
INTERPRETABILITY
439
440
recent line of work has used chess as testbed for mechanistic interpretability, which aims to identify the mechanisms by which an AI model computes its outputs. Our Nomain-grounder archi- tectural choice eases this process by allowing squar-speccife attribution of activations and attention patterns. As a preliminary investigation of the interpretability benefits of our approach, we present an overview the mechanisms by which Geometric Attention Bias (GAB) and dot-product atten- tion cooperate.
441
442
443
444
445
446
447
6.1
GAB MAPS
448
449
We inspect the attention heads of both MATA-3 and Leela-CF, finding that they tend to represent semantic information with the dot-product attention logits and positional information with the GAB biases. In general, GAB appears to represent different movement types and measures of proximity on the chessboard. In contrast, the dot-product attention (DPA) maps appear focus more on global semantic information such as important enemy pieces.
450
451
452
453
454
To quantitatively evaluate this observation, we measure how much the rows of the GAB and DPA attention maps vary both between positions and within a single position. For between- position variability, we randomly sample 30,000 positions from Lichess blitz games played in June 2019. For each query square, we extract the corresponding row-centered, 1.2-normalized GAB attention rows across all positions and compute the average cosine distance between them (i.e., GAB-GAB similarity for that square). We repeat the same procedure for the DPA maps (DPA-DPA similarity), averaging over all query squares, heads, and layers.
455
456
457
458
459
460
461
For within- variability, we fix a position and compute the average cosine distance among all pairs of row cew-centered L2- norrmallz GAB maps across different query squares for a given head and layer in that position (i.e.. RAB-GAB similarity within the same position). We then average over all heads and layers. We do the same for DPA. In both cases, we compare GAB-GAL and DPA-DPA statisties; we do not measure the similarity between GAB and DPA directly.
462
463
464
465
466
As shown in Table the query square's GAB bias exhibits substantial variation across positions, unlike fixed positional encoding. This suggests that GAB adapts meaningfully the position context. However, this variation is much smaller than that of dot-product attention, indicating that GAB is still relatively stable between positions. At the same time, GAB is more variable across query squares than dot- -ro-product attention, so it is more -quare-specffic within positions. Taken to- gether, these findings suggest that GAB primarily captures specific information relevant the query square, whereas dot-product attention is less tied to specific squares and instead reflects semantic information about the position as whole.
467
468
469
470
471
472
473
474
Following this intuition, there are several examples in our models of GAB biases adapting to global features of the board state, like the game stage. Figure 2 shows an example of this where the GAB component of the attention computation transitions from modeling general piece movement to modeling king movement. We list more examples of GAB and DPA maps in Appendix F.2
475
476
477
478
479
6.2
IDENTIFYING INTERPRETABLE FEATURES
480
481
A prominent approach to transformer interpretabilitt consists of training sparse autoencoders [Brckknn et al. 2023) or transcoders (Dunefsky et al. (004) on a model's MLP activations, then interpreting their basis vectors by examining the inputs which activate them the most to identify in-
482
483
484
485
Sec http:///s//aaaaaaaaaaaaaaaaaaaaa "See
https://en.... wiki ot htg.:///011.aaaaoooaoooaaoaaooor

---

Under review as a conference paper at ICLR 2026
486
terpretable features and circuits within the model Lindsey et al. (2024). Much previous work applies mechanistic interpretability to the testbed of chess Karvonen et al. (2024).
487
488
Our architecture enables much finer interpretation based transcoder features. Instead of merely examining the positions which maximally activate an MLP feature, one can identify the squares at which feature is maximally activated. To explore this useful property of our architecture, a cross-layer transcoder on MAIA-3 (training details Section A.4). We find that our resulting features are interpretable at high and rich in the space of concepts they cover. For example, many features correspond to core tactical concepts like forks and pins, whose top activated squares are those involved in the tactical dynamics. More importantly, we find features that would be other- wise uninterpretable from only their top activated positions, but whose top activated squares exhibit a clear pattern. Space constraints preclude us from including all 8192 transcoder features, but we visualize and annotate the top activated positions of the first 20 features of both transcoder layers in Figure (since feature order arbitrary, these are essentially randomly selected). The abun- dance of interpretable features in MAIA-3 and the usefulness of square-level activation attribution in parsing their meanings establish our architecture as a significant step forward towards the goal of interpretable chess modeling.
489
490
train
491
492
493
494
495
496
497
498
499
500
501
502
Table 4: Cosine distance of attention rows within and between positions for Geometric At- tention Bias (GAB) and dot- dro-product attention (DPA). GAB exhibits higher variability within positions than DPA, but lower variability be- tween positions.
503
504
505
A
506
507
è
508
509
Cosine distance
GAB
DPA
510
Between positions
0.230
0.770
511
Figure 2: GAB bias maps in L14H11 of Leela- CF in the early and late game. The GAB bias for this head transitions from modeling a wide range of movement the early game (left) king movement in the late game (right).
Within positions
0.995
0.184
512
513
514
515
516
517
DISCUSSION
518
519
520
Our central contribution is Chessformer, an architecture for chess modeling that jointly advances the state of the art for engine strength and human-move prediction, while doing in a natu- rally interpretable way. A Chessformer variant was integrated into a strong open-source engine and contributed to match wins over Stockfish in multiple computer-chess tournaments.
521
522
523
524
For human emulation, MAIA-3 reaches 57. 1% move-matching accuracy and surpasses prior search- less (55.7%) and search-enabled (55.9%) ALLIE methods with fewer than one-fourth the parameters, indicating a significant improvement in architecture design. Ablations show monotonic improve- ment from absolute relative to GAB position representation, with GAB delivering +1.96% policy accuracy, +0.34% value accuracy, +3.37% puzzle-solving accuracy, and +80 Elo, and matching an absolute-encoding baseline at roughly half the compute. Together, these results establish that our methodology is doing substantive work, yielding performance-per-come
525
526
527
528
529
advantages and better
530
Pareto points across Elo, puzzles, and policy and value accuracies.
531
532
Beyond chess, the general lesson is that it may be possible to make progress on both raw task performance and compaiiiiiaiibilitty goals by allowing models to better conform to the domain at hand. In our case, this alignment closes the gap between modeling form and action space, enabling simultaneous gains in mastery (engine Elo; oracle/puzzle metrics) and coman-compatibiliyy (move- matching across skill levels) while allowing for interpretability testing that could enable progress further downstream tasks.
533
534
535
536
537
538
Our methods have limitations that suggest concrete next steps. GAB is currently chess- specific, and its benefits may rely on domains where geometric relations are central. Extending this tem- plate other structured decision problems invites exploration of further geometric modeling.
539
10

---

Under review as a conference paper at ICLR 2026
540
The interpretability benefits square-level maps, complementary roles of GAB versus dot- product attention are promising but require deeper exploration.
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
11

---

Under review as a conference paper at ICLR 2026
594
REFERENCES
595
596
Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jenmyn, Tom Con- erly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatieeld-oods, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https//transformer- Secuutt.pub20023/mooeemmaa /nnoooosmmmtt----ttttssseeeeeeee
597
598
599
600
601
602
603
Murray Campbell, A. Joseph Hoane, and Feng hsiung Hsu. Deep blue. Artificial Intelligence, 134 (1):57-83, January 2002. ISSN 0004-3702.
604
605
Rémi Coulom. Whole-history rating: A bayesian rating system for players of time-varying strength.
606 In Computers and Games, 2008.
607
608 Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: fast and memory-efficient exact attention with io-awareness. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
609
610
611
612
Jacob Dunefsky, Philippe Chlenski, and Neel Nanda. Transcoders find interpretable I1m feature circuits In A. Globerson, L. Mackey. D. Belgrave, A. Fan, U. Pa- quet, J. Tomezak, and C. Zhang (eds.), Advances Neural Information Processing Sys- tems, volume 37, pp. 24375-24410. Curran Associates, Inc., 2024. doi: 10.52202/ 079017-0768. URL http://0000000cccccccceeeeeeeea0
613
614
615
616
ngs neurips wwtp:ipppppppppppppppppepe hcpp:
httpppapeerrreeeeeeaaeeeeeeeeeee files/ pd
617
618
619
Steven Edwards.
Standard:
Portable game notation specification and implemen-
tation guide, 1994.
URL https:://aa002:000 us. archiverivee.... ortpe///rrg///////ss//
620
621
622
Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Ir- pan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal. Stop regressing: Training value functions via classification for scalable deep RL. In Forty-first International Conference on Machine Learning, 2024. URL httptss://ppeeeeeiieeeee net./
623
624
625
626
627
Xidong Feng, Vivek Veeriah, Marcus Chiam, Michael D Dennis, Federico Barbero, Johan Obando- Ceron, Jiaxin Shi, Satinder Singh, Shaobo Hou, Nenad Tomasev, and Tom Zahavy. Generating creative chess puzzles. In The Thirty ninth Annual Conference on Neural Information Processing Systems, 2025. URL
628
629
630
631
632
Andrew Grant. Openbench. htttppss/ggtaaeccccm/////aaatttp 2125-11-22
Accessed:
633
634
Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bert- simas. Finding neurons in a haystack: Case studies with sparse probing. arXiv preprint arXiv:2305.016/1 2023.
635
636
637
638
Karim Hamade, Reid Mcllroy- Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. De- signing skill-compatible Al: Methodologies and frameworks in chess. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL htttps://ppeerevieeee htteeeiiewwwwwe/ee//
639
640
641
Forum?id=79rfqy33
642
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
643
644
doi: 10. 0.1109//CPR.20116999
645
646
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitatio networks. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7132-7141, 2018, doi: 10.1109/CVPR.2018. 00745.
647
12

---

Under review as a conference paper at ICLR 2026
648
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. In Ricardo Silva, Amir Globerson, and Amir Globerson (eds.), 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, 34th Conference on Uncertainty in Artificial Intelligence 2018, UAI 2018, pp- 876 885. Association For Uncertainty in Artificial Intelligence (AUAT), 2018
649
650
651
652
653
654
Athul Paul Jacob, David J Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Ja- cob Andreas, and Noam Brown. Modeling strong and human-like gameplay with kl-regularized search. In Proceedings of the 39th International Conference Machine Learning, volume 162, pp. 9695-9728, PMLR, July 2022.
655
656
657
658 Erik Jenner, Shreyas Kapur, Vasil Georgiev, Cameron Allen, Scott Emmons, and Stuart Russell. Evidence of learned look-ahead in chess chess-playing neural network. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 31410-311733777 Curran Associates, Inc., 2024. doi: 10.52202/ 079017-0987. httpatppp/ppppeeeessseeeeeeeepaa
659
660
661
662
ottp://ceeeeoaeeeeeeereoeoocepas cwppppprrrpppppppppppppppp.. fi wwtpee/p_eeeeeepppeeeeeeea
663
oel.:faeec7n111100Deeeeeeeeeaae2 Papeeadaa-a--pppp-aa---eee Poter pdf
664
Adam Karvonen. Emergent world models and latent variable estimation in chess- chess-playing language models. In First Conference on Language Modeling, August 2024.
665
666
667
Adam Karvonen, Benjamin Wright, Can Rager, Rico Angell, Jannik Brinkmann, Logan Smith, Claudio Mayrink Verdun, David Bau, and Samuel Marks. Measuring progress in dictio- nary learning for language model interpretability with board game models. arXiv preprins arXiv:2408. 00/13, 2024. doi: 10.48550/arXiv.44000 200.00000..0000.131.. URL httphhtt///aatt ptttps://arxiv.org)
668
669
670
671
00113
672
673
Jack Lindsey, Adly Templeton, Jonathan Marcus, Thomas Conerly, Joshua Batson, and Christopher Olah. Sparse crosscoders for cross-layer features and model diffing. Transformer Circuits Thread, 2024. URL /aattps:/ttrant ooreorrrrr---eeee ceaerreireee www.itsss..t..ieeieeeeerrii www.rruuttsspuupe/oeeeee
674
675
html
Preprint technical note.
676
677
Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, UIrich Paquet, and Vladimir Kramnik. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47))e220662511) 2022.
678
679
680
681
Reid Mcllroy- Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning superhuman ai with human behavior: Chess as a model system. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1677-1687, August 2020.
682
683
ISBN 978-1-4503-7998
684
685
Reid Mcllroy- Young, Yu Wang. Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Detecting in- dividual decision-making style: Exploring behavioral stylometry in chess. In Advances in Neural Information Processing Systems, volume 34, pp. 24497, 2021.
686
687
688
689
Reid Mcllroy- Young, Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Learn- ing models of individual behavior in chess. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1253-1263, 2022.
690
691
692
Aaron Mei. Understanding how chess-playing language models compute linear board represen- tations. In ICML 2025 Workshop on Methods and Opportunities at Small Scale, 2025. URL https::/cpeereviiee net.eewwweeeeeeee fwr.mmrummmmmmmmmmoooomooo (ttooummmnd-22ooyyyye.
693
694
695
696
Pascutto and Gary Linscott. Leela chess zero, March 2019.
697
698 Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
699
700
701
Tord
Romstad, Marco Costalba, Joona Kiiski, and et al.
Stockfish.
https://
stockfi
shchess. iscchessshhhess. org 2023. Accessed: 2025-11-29.
13

---

Under review as a conference paper at ICLR 2026
702
Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, El- liot Catt, John Reid, Cannada A. Lewis, Joel Veness, and Tim Genewein. Amortized planning with large-scale transformers: A case study on chess. In A. Globerson, L. Mackey, D. Bel- grave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Pro- cessing Systems, volume 37, pp. 63765 65790. Curran Associates, Inc., 2024. doi: 10.52202/ sttprerprrieeeeeppprereep o022///11e/E00033333/0/6/7626622 ottp:://paooa.aaoooooooaaoore-er ort.aaaeeecoccoooo.ooooooooooonn
703
704
705
706
707
079017-2102. URL httpss//pprooeeiinsnneerss..pp.p
708
pdf
709
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362
710
711
712
1144, 2018.
713
Stockfish Team. Stockfish testing framework. http:://eeats..tocfffahhciise
714
tests Accessed: 2025-11-22.
715
716
Stockfish Team.
Stockfish 15.
717
stockfish- 15/
September 2022.
Accessed: 2025-11-19.
718
Stockfish Team.
Stockfish September 2024.
719
stockfish 17/
Accessed: 2025-11-19.
720
721
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomput, 568(C), mar 2024. ISSN 0925-2312. doi: 10.101 //016//..encomm2023..... 127063. URL https::/doi. org/ 101.///............ 1016/j.
722
723
724
net.om 2023. 127063
725
Zhenwei Tang, Difan Jiao, Reid Mcllroy- Young, Jon Kleinberg, Siddhartha Sen, and Ashton An- derson. Maia-2: unified model for human-ai alignment in chess. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomezak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 20919-20944. Curran Associates, Inc., 2024. doi: 10.52202/ 079017-0659. URL https::/procediii.e
727
728
729
neur wttp:/pppppppteec http://ppprr_flleepppprrrrr www.arpreeraereeeeeeeeeeae pdf
730
731
Yiming Zhang. Athul Jacob, Vivian Lai, Daniel Fried, and Daphne Ippolito. Human- aligned chess with bit of search. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (eds.), International Conference on Representation Learning, volume 2025, pp. 4815-4836,
732
733
734
2025. URL /ttp/ttts///rrroetiiinossss iclr co/paper hra.aaaaaecccccccceeeellaaeeeeer
http:/pppprr.llspppaaaeee/eeeeee files/paper/225//1122
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
14

---

Under review as a conference paper at ICLR 2026
756
A IMPLEMENTATION DETAILS
757
758
A.1
HUMAN EMULATION
759
760
All human-mimickinn models were trained with the AdamW optimizer the dataset described in Section 4 During training, we sample 32 positions per game at random, or take all of them if 32 positions are not available. All runs have 8 layers, head dimension 32, and MLP expansion factor of 2. Our 3M, 5M, 23M, and 79M models have embedding dimensions of 192, 256, 512, and 1024, respectively. GAB is configured with di = 32 and d2 = d3 128 for the 23M and 79M models and average pooling and d2 = d3 = 64 for the 5M model and all its ablations. All runs were trained for 1 million steps with a cyclic cosine annealed learning rate schedule. The 23M and 79M main runs were performed on 8 A100 GPUs, taking a few days and week respectively, and all other runs were trained on 2 A 100 GPUs for around a week.
761
762
763
764
765
766
767
768
Because low-rated games are overrepresented in our training dataset, we downsample this dataset during training to ensure that skill levels are equally represented. In particular, the rating spectrum is divided into 22 bins: 20 bins uniformly spanning 600 to 2600 Elo with 100-point intervals, plus two bins for players rated below 600 or above 2600. For each game, we compute the average Elo of the two players and assign the game to the corresponding bin. We organize the raw game data into chunks of 20,000 games. For each chunk, we iterate through games sequentially and distribute them into bins until each bin accumulates 10 games. The process terminates when either all games in the chunk are consumed or all bins reach 10 games. This encourages equal representation across skill levels, removing the bias toward low-rated games. We then uniformly at random select 32 positions per game, or take every position if that many are not available.
769
770
771
772
773
774
775
776
777
778
The ALLIE test set is suitable to measure overall performance, and all overall performance metrics we report on the human emulation task employ it. However, it lacks positions at very high and low skill levels therefore cannot be used form statistically significant conclusions these skill levels. For example, it has only 205 positions rated above 2850, which would result in a margin of error of over 7%. To rectify this, we augment the ALLIE test set with very highly and lowly rated Lichess blitz games from August and September 2025 to form the ALLIB-AUGMER
779
780
781
782
783
dataset, which we use to compare modeling performance across skill levels- it is used in Figure Figure [ and Figure and nowhere else.
784
785
786 To obtain the -ULIE-AUGGETEED test set from the ALLIE test set, we first take rating intervals of length 100 between 550 and 2950 in which ALLIE contains less than 20 thousand positions. These intervals are so chosen so that they correspond to the bins in Figure 5 and Figure 6 which round the game rating. We initialize empty buckets for each such interval and iterate through September and August 2025 Lichess blitz games, adding the game to the corresponding bucket so long as it has less than thousand games. These games are added on top of the ALLIE test set to form the final ALLIE-AUGMENTE set of 1,087,778 positions.
787
788
789
790
791
792
793
794
A.2
PLAYING STRENGTH
795
As described in Section 5 we train in the supervised setting on games generated from past rein- forcement learning run of the Leela Chess Zero project. This strategy works well in practice and was used train Leela-CF, which surpassed the model that generated its training data, as well as each of the models used in the victories against Stockfish described in Section 5) Initial experiments showed that Chessformer models trained on self-play games produced by convolutional neural networks and other Chessformer models reach virtually identical playing strengths, suggesting that the strength of the oracle is far more important than the model underlying its search process.
796
797
798
799
800
801
802 To maximize compatibility with the Leela Chess Zero infrastructure, we follow their input scheme as closely as possible We form 64 tokens of depth 96 for the current and past board states, following Section 3.1 We also concatenate indicators of whether each of the current and past 7 board states was a repetition, whether each of the 4 castling options are available, whether black is to move, and the number of plies since the last capture pawn move, divided by 100. Finally, we concatenate a O and 1, which are relics that were originally intended to allow convolutional models to detect edges. This information is concatenated to each token, giving 64 pre-embedding tokens of depth (96 + + 4 + 1 1+ 2) = 112. Initial experiments did not show this additional information to alter modeling performance.
803
804
805
806
807
808
809
15

---

Under review as a conference paper at ICLR 2026
810
Table 5: Human Move-Matching Training Configuration for Reproducibility
811
812
Parameter
Value
813
Training Setup
814
batch_size_tral
128
815
batch_size__
816
inddiettaccummlation
4
817
num_workers
818
Optimization
819
Ir min_ wd (weight decay)
0.00005
820
0.00001
821
0.000001
822
grad clip nomm warmup steps cosine_cycles refresh.Ir.scheduuee
3.5
823
1,000
824
50,000
825
true
Mixed Precision
827
use amp amp_init_scale amp_max_seaa
true
828
256
829
8,192
830
amp-growth_facte ffcctrr
1.5
831
amp growth interval amp backoff factor
2,000
832
0.5
833
Loss Weights coefficiet
0.1
835
836
837
838 All Chessformer models trained for this task used the Nadam optimizer with 81 0.9, 8 0.98, e = 10-7 7 and gradient clipping 10. Following Leela Chess Zero's setup, checkpoints were cal- culated using Stochastic Weight Averaging (Izmailov et al. 2018). which incrementally boosted performance.
839
840
841
842
The 191-million parameter Leela-CF model has hidden dimension 1024, MLP dimension 1536, head size of 32, and 15 layers, and GAB configured d2 = dg = 256, and the first linear project and flattening layers are replaced with average pooling, which initial experiments showed greatly improves parameter efficiency at small scales at the cost of a slight performance degradation. It was trained for million steps, with the learning rate initialized 2 10-3 and dropped to 3 10-4 4.49 million steps and 3 x 10-5 at 5.47 million steps. The 195-million parameter Leela-CNN is squeeze-excitation excitation ResNet (Hu al. 2018) (He al. 2016) with 512 filters and depth 40.
843
844
845
846
847
849
850
Table Ablation results for raw playing strength. Accuracies and Elo values are reported with 95% confidence intervals
851
852
853
Loss
Accuracy (%)
Puzzles (%)
Elo
FLOPS
#Params
854
Policy
Value
Policy
Value
Absolute
0.363
0.567
56.6
+ 0.1
88.7 0.1
61.0 1.0
0
210M
3.67M
856
Relative bias
0.346
0.565
57.5 0.1
88.8 0.1
63.2 # 1.0
+ 18
210M
3.67M
GAB-2 -2.5M
0.360
0.567
57.0 77.010.1
88.7 0.710.11
61.5 1 1.0
4 41
131M
2.51M
857
GAB-4M 0.330
0.562
58.5 + 0.1
89.0 + 0.1
64.2 + 1.0
83 +
18
228M
4.01M
858
859
860
We train ablations with the three position representations described in Section 3 with 8 layers, em- bedding dimension 256, head dimension 32, and MLP dimension 256. GAB configured with d1 8 and d2 dg 32. The smaller model has embedding dimension and MLP dimension 192, with all else held constant. Each was trained for 1.4 million steps on a single A100 GPU with batch size of 2048 in approximately four days. The learning rate held constant at 5 10-1
861
862
16

---

Under review as a conference paper at ICLR 2026
864
A.3
SPECIAL MOVES
865
866 A source and destination square are sufficient to represent all moves that can occur within the rules of chess, with some exceptions. When pawn advances to the last rank of the board, it must be promoted to a knight, bishop, rook, or queen. To represent these special moves, we apply a linear projection to the key vectors for squares in the last rank, generating an additive bias for each possible promotion piece. This bias is then applied to the logits representing all possible traversals between the penultimate rank and the promotion rank to generate additional logits for each possible promotion. Following the standard in computer chess, en passant captures are encoded as diagonal moves, and castling is encoded as the king moving two spaces horizontally.
867
868
869
870
871
872
873
874
def sm bias (x: torch. Tensor)
terch. Tensor:
875
x. x.shape [0]
876
y sm1 (x) (B, 64, d_1)
y y. reshape (B, -1) $ (8, 6ád_1)
877
y = sm act (sm2 (y)) # (B, 2)
878
y Inl (y)
879
Y = sm act (y)) # (B, Hxd_3)
880
y 1n2 (y) wwe.vviewww (B, num_heade gen_size) & (B, H, gen_size)
torch. (i Y, poif.
881
return b.view self.num_heads, 64, 64)
882
883
Figure 3: Torch-like pseudocode for GAB.
884
885
886
A.4 TRANSCODER TRAINING
887
888
For interpretability purposes, we train cross-layer transcoder on MLP activations collected from layers 3 and 4 (in other words, the 4th and 5th layers) of an earlier checkpoint of MAIA-3. The transcoder consists of encoders for each layer and decoders going between the two layers (including between each layer and itself), trained on reconstruction and sparsity loss. We train only on layers 3 and because of a) compute constraints and b) our preliminary investigations finding that these layers tended to contain the most interpretable representations. It is common practice to train sparse autoencoders and transcoders on medium deeth layers of models, as these layers are often where the most interpretable representations are found Gurnee et al. (2023).
889
890
891
892
893
894
895
896 Let the base transformer have Luse layers, and let S = {40 < & .... < (x-1} be a subset of K layers which we train the transcoder (for example, S [3.4] when training only between layers 3 and 4). For each Eg E S we denote by R.,.e E R Bxxxx the pre-MLP residual stream and by Mi, IR BxTXD the corresponding MLP output. After per-layer standardization over batch and tokens, we write
897
898
899
900
901
X E BXTXXD YA E R
h k= = O=....,K- 1-
902
for the normalized inputs and targets. The transcoder operates on the index set {0...., K 1}, with indices i.j referring to layers L, lg E S.
903
904
905
906
Encoder:
(e) Z: = X; ( +b(e)
(e) E BBXTXM
(1)
907
A; = ReLU(Z, T,) E BXTXM
(2)
908
909
where w/(0) E DXM b(o) E RM and T; E R a learned threshold broadcast over batch and tokens.
910
911
912
913
Decoders (for all 0 O Y = A, W! 1-j
+
+b/0,
E BBXTXD
(3)
914
i Y,
915
=
Y ドーク
(4)
916
i=0
917
where w:d)) E CRDx and b (d) b, E RD.
17

---

Under review as a conference paper at ICLR 2026
918
919
Y - y 12
920
Reconstruction loss:
MSE
BTD
(5)
921
922
Liccon
CAISE.
(6)
K j=ū
923
924
925
K
K-1
926
Decoder-weighted sparsity penalty:
f
( L All2:
f f= ,,..,, M, (7)
K
927
928
T
M
1
929
81
tanh CTT. (Ai)bass)
(8)
BTM
930
b=
f=1
931
K-1
932
Caparse
8
K
933
934
where À O and c >> 0 are scalar hyperparameters.
935
936
937
Total loss: Liotal Lrecon
Eparre.
(10)
938
We use batch size of 22 games, learning rate of 5e-5, and expansion factor of 8. For training data, we use blitz games from lichess played during July 2019, filtered in the exact same way as in base model training pipeline. We use the same data to sample the top activating tokens for each feature. At the end of training, our transcoder achieves a reconstruction MSE of 1.6% and a sparsity (po) of .90
939
940
941
943
944
B
IMPLEMENTATION DETAILS FOR TOURNAMENT
945
946
Table 7: Tournament performance of Leela-CF Chessformer against Leela-CNN convolution model. The time control is chosen so that N is roughly the number of playouts the Leela-CNN model performs during each game. Elo values are reported with 95% confidence intervals.
947
948
949
950
N
Elo Gain
Wins
Losses
Draws
951
160k
112 + 7
846
223
931
952
320k
111 + 7
806
186
1008
953
640k
105
+
779
190
1031
954
955
To gauge the impact of Chessformer raw engine strength, perform a tournament between versions of the Lecla Chess Zero engine configured with either the Leela-CNN model, a 195-miliion- parameter convolution-basee million parameter Chessformer. Because the high computational load of long engine analyses, use the distributed testing framework OpenBench (Grant) run 2000 games between these configurations, so that each of 8 RTX 4090 and 4 A100 GPUs runs single game at a time with engines alternating use of the hardware. Games are played in pairs starting from positions sampled from UHO Lichess_4852.11 4852.v1..pd, a book of 2.6 million unbalanced human openings curated by the Stockfish project.
956
957
model previously used by Leela tournaments, or Leela-CF, 191-
958
959
960
961
962
963
964
To calculate the base time control 7 for each GPU, we benchmark the speed of Leela-CNN and set to an estimate of the amount of time the GPU would take to perform N MCTS playouts, varying the value of N to determine the effect of thinking time on the performance difference. The increment set to T/100, and Leela Chess Zero was free to use its time according to its time management algorithm. In other words, N is an estimate of the total amount of playouts the Leela- CNN configuration performs over the game. On average, the speed Leela-CNN on these GPUs is approximately 20,000 playouts per second, so T x N/20000 seconds. Calculating the time control dynamically for each worker to adjust for variations in processing power is standard in modern distributed engine testing frameworks like Stockfish's Fishtest (Stockfish Team).
965
966
967
968
969
970
971
18

---

Under review as a conference paper at ICLR 2026
972
C POSITIONAL ENCODING BASELINES
973
974
Absolute Position Embeddings Perhaps the simplest choice of position encoding is the absolute position embedding, which consists of adding a learned embedding to each token and was notably used by GPT-2 (Radford et al. 2019). Formally, given sequence of token embeddings X1,... Xn. one applies
975
976
977
978
979
X; + X; +
(1I)
980
981
prior the transformer sublayers.
982
983
Relative Position Biases Unlike absolute position emheddings, relative position encodings model positional information based on the relative displacement between tokens. One simple variant introduces relative biases f& which are added to the attention logits:
984
985
986
987
Eij (x,W9)(X, +Si-j
(12)
988
Vd
989
We consider the two-dimensional analog of this technique, where square the chessboard is assigned coordinates (i,j), with i and ranging from 0 to 7. The bias for querying square (11,51) and key square (iz, j22 is thus jz where fa,b is defined for -7 S a, b S 7. This adds
990
991
992 15 X 15 parameters per attention head.
993
994
D TOKENIZATION
995
996
number of tokenization schemes have been proposed for chess. We review some of these and attempt to give insight into why our recipe, square-based representation with a strong position encoding, significantly outperforms them.
997
998
999
1000
The MAIA-2 architecture consists of series of convolution blocks operating on square-based vi- sual board representation, followed by self-attention layers that process the depth-64 output channels as tokens. Though interesting design choice, this does not align with the standard, well-tested recipe of transformers in domains like vision and language, which tokenizes inputs by partitioning them in space rather than through internal model representations.
1001
1002
1003
1004
1005
Move-token formulations like ALLIE, on the other hand, rely on the established methodology of language modeling but lack another vital property: specialization. Processing inputs in parallel should allow a model to "divide and conquer", so that the overall computation is split into units that are processed with the same parameters. However, there is emerging evidence that move-token models do not specialize effectively and instead simply reconstruct the board state at each token
1006
1007
1008
1009
(Mei 2025), (Karvonen] 2024). It is also not intuitively clear why trajectory-based representation should be natural in the Markovian domain of chess.
1010
1011
1012
1013
E ADDITIONAL ANALYSIS FOR HUMAN EMULATION
1014
1015
Here provide additional analysis on our human emulation results. We first ablate n, the number of past positions concatenated to the current one to form the input, at the 5M scale. Interestingly, as shown in Table g there large increase in performance between n 0 and significant difference between = 7 and = 31. The improvement is concentrated at low game ratings and decreases steadily up to high ratings. This contrasts with the effect of model size on move-matching performance, shown in Figure 5 and Figure 6 The impact of scale modeling performance several times higher for strong play as it is for weak play.
1016
1017
7, but no
1018
1019
1020
1021
1022
1023
To ensure that positions in which no or little history information is available remain in-distribution, mask out during training uniformly random amount of history information with probability 5%. In initial experiments, this was found to negligibly affect performance (< 0.05% reduction in move-matching accuracy) relative to always retaining all past board states.
1024
1025
19

---

Under review as a conference paper at ICLR 2026
1026
Despite marked improvements in parameter and compute efficiency, MAAIA-379MM improves on the state-of-the-ar move-natching accuracy by only 1.2%. We postulate that despite Chessformer's
1027
1028 modeling capability, it runs into performance ceiling at low and intermediate skill levels. Play at
1029
these skill levels is unsophisticated and easy to model, but inconsistent and stochastic enough that
1030
the accuracy appears saturate at around 50%. MAIA-3-79M shines however at modeling highly
1031
skilled play, advancing the searchless state of the art by up to 5% for very strong play. Prior work has
1032
struggled emulate strong play, often relying on search to shore up weak human-aligned models.
That MAIA-3 not only outperforms even search-enables methods but achieves its largest gains at
1033
these very high skill levels suggests that our methodology jointly enables both human alignment and
1034
mastery.
1035
1036
1037
Table 8: Move-matching accuracy by history length for human emulation. The value reported is n, the number of past positions excluding the current one fed inputted into the model.
1038
1039
History
Accuracy (%)
1040
1041
0
54.0 + 0.1
1042
7
55.4 0.1
1043
31
55.4 + 0.1
1044
1045
To understand why this is the case, we decompose the error for human emulation into aleatoric uncertainty, the amount of uncertainty that is inherent to the task, and epistemic uncertainty, the amount of uncertainty that can be reduced through stronger models. Low-rated play tends to be unsophisticated. reducing the amount of improvement available from better modeling approaches, but stochastic and inconsistent, reducing the ceiling on predictability. In other words, it has high aleatoric uncertainty, explaining the low accuracies for these players but low epistemic uncertainty, explaining the minor gains provided by scale. In contrast, highly skilled play more accurate and thus more consistent, but more difficult to predict due its sophistication; it has low aleatoric uncer- tainty but high epistemic uncertainty, giving plenty of room for improvement. History information likely improves move- move-matching matching performance by providing information about the player and reduc- ing the alcatoric uncertainty of the task, particularly for low-rated play where is higher.
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
2820
1058
7620
4144
1059
0.
2400
1060
1061
1062
3800
153
1063
Oxpontnt
3600
112
1064
1400
1065
-3.4
-200
1066
1067
1068
600
1069
0.
1070
Aetho Payark Sell
1071
1072
Figure 4: Move-matching accuracies of MAIA-3 for pairs of skill levels. Interestingly, players are more predictable when they are paired against weaker opponents. This figure uses the ALLIE- AUGMENTED test set, described in Appendix A.1
1073
1074
1075
1076
With this in mind, we believe that future human emulation work should focus highly skilled play, where the performance ceiling appears to be much higher. One interesting question is maximum accuracy on this task increases monotonically with the game rating, aligning with our intuition about the consistency of strong players, or drop off for ratings above 2600 in line with our observed performance.
1077
1078
1079
20

---

Under review as a conference paper at ICLR 2026
1080
1081
1082
030
625
535
n: 0
20.0
62.5
57.5
Reterie
SN Dbaset
Abectutte
hove-matchiingg (%)
00.0
92.5
1083
1084
525
1085
530
10.0
20.0
1086
675
67.
67.
1087
650
45.0
45.0
1088
180
260
1009
2300
20o0
2l10
2000
25000
1089
Game rating
Game rating
Game rating
1090
1091
Figure 5: Human move-matching accuracies the ALLIB-AUGMENTED test set by number of
1092
history positions n (left), position encoding (middle), and scale (right). History information helps
1093
most for weaker play, while scale and effective position encodings have large effect for stronger play. We omit results for n 31 history positions as they are virtually identical to those for = 7, and also omit ALLIE-ADAPHR
1094
SEARCH due to compute constraints. For all game ratings other than 2900, the margin of error is less than a percent.
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
(bsent
65
85
6.5
1110
Selartvv
Abso
1111
Perplexity
1112
5.0
5.0
6.
1113
1114
4.0
4.0
4.0
1115
3.3
1116
Rede
24100
800
1050
1400
pert
6000
1010
1117
rating
Game rating
1118
1119
Figure 6: Human move-matching perplexity on the ALLIE- AUGEETED test set by number of
1120
history positions (left), position encoding (middle), and scale (right). We omit results for 31 history positions as they are virtually identical to those for n =
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
21

---

Under review as a conference paper at ICLR 2026
1134
F ADDITIONAL RESULTS
1135
1136
F.1
TOP ACTIVATED TOKENS FOR TRANSCODER
1137
1138
L3F000O
LSP0001
1139
1140
1141
AMB
1142
1143
1144
1145
1146
1147
L3F0002
L3F0003
1148
1149
MoAL
1150
1151
1152
1153
8
10
319
1154
1155
1156
1157
1158
1159
1160
1161
N
1162
1163
L3F0006
L3F0007
1164
1165
1166
1167
1168
1169
1170
80
1171
L3F0008
LSFDODD
1172
1173
1174
010
BLL
A1
mm
1000
1175
1176
1177
1178
1179
1180
Figure 7: Annotations for features 0-9 of layer 3. L3F0000: Square that the active player advance a pawn to in order to attack an enemy bishop. L3F0001: Active player's knight, usually under attack. L3300002 Square the side of the board that is controlled by the active player's rook or queen. L.300033 Vacant square adjacent to a rook the corner. L330000: An enemy pawn in the corner, in front of the active player's 's pawn, sheltering the opponent's king. L3F0005: Either a rook in the corner or center pawn movement option. L3F0006: Not interpretable. L3F0007: square diagonally adjacent the opponent's king that is controlled by the active player's pawn. L3F0008: A square contested by both friendly and enemy pawns. L3F0009: Enemy minor piece on the edge the board pinned an enemy rook.
1181
1182
1183
1184
1185
1186
1187
22

---

Under review as a conference paper at ICLR 2026
1188
L2FO010
L3H0011
1189
1190
1191
1192
1193
1194
1195
8
12 98
1196
1197
13F0012
1198
1199
1200
1201
1202
1203
1204
1205
L3F0014
L3F0015
1206
1207
1208
1209
1210
1211
M
1212
1213
1214
1215
1216
DR
1217
1218
02
1219
8
1220
81
10
1221
1222
L3F0018
LSF0019
1223
1224
1225
4-808
1226
1227
1228
1229
120 911
1100 60
1230
1231
Figure 7: Annotations for features 10-19 of layer 3 of MAIA-3: L3F0010: Queenside activation
1232
square for active player's knight in Queen's 's gambit structures. L3F0011: Square on b3, b6, f3, or f6 in the opening that have been weakened by the lack of supporting pawn. L3F0012: Square that the active player's knight can move to to give check. L330013: Enemy rook checking the active player's king. L.3F0014: Active player's s king on the starting position, or an adjacent square the king has castled. L3F0015: Enemy king in danger of being checkmated the back rank. L3F0016 Square controlled by both friendly and enemy pawns. L3F0017: Enemy knight developed on the side of the board. L300188 8: Enemy knight attacked by active player's bishop and defended by enemy bishop. L30019: Not interpretable.
1233
1234
1235
1236
1237
1238
1239
1240
1241
23

---

Under review as a conference paper at ICLR 2026
1242
L4F0000
L4F0001
1243
1244
1245
1246
1247
1248
1249
01
1250
1251
L4F00G2
L4F0063
1252
1253
1254
100 1al-taaea
1255
1256
1257
1258
98
1259
L4F0004
L4F0006
1260
MAM
1261
1262
1263
1264
11
1265
00
1266
B
AURE
1267
1268
L450008
L4F0007
1269
1270
80
EALRURG
1271
1272
1273
1274
2
1275
1276
L4F0008
1277
1278
1279
1280
1281
1282
1283
1284
1285
Figure Annotations for features 0-9 layer of MAIA-3: L4F0000: Not interpretable L4F0001: Active player's bishop on strong diagonal, often paired up with queen. L4F0002: Enemy center pawn targeted for capture in the opening. L4F0003: Square deep in opponent's terri- tory attacked either by two rooks or rook and queen. 14F0004: Either long castling or tension between active player's f6 pawn and opponent's g7 pawn. L4F0005: Enemy knight pinned by the ac- 1289
1286
1287
1288
tive player's bishop. L4F0006: Usually square controlled by the active player's bishop. L4F0007: Not interpretable.
1290
1291
1292
1293
1294
1295
24

---

Under review as a conference paper at ICLR 2026
1296
L4FO0T10
L4HD011
1297
1298
1299
1300
1301
1302
1303
24
1304
1305
1400013
1306
1307
1308
1309
1310
1311
1312
1313
L4F0074
L4F0015
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
ORL
1325
BLB
1326
1327
1328
1329
1330
L4F0011
L4F0019
1331
1332
3.0 m
1333
1334
1335
1336
1337
1338
1339
Figure 8: Annotations for features 10-19 of layer 4 MAIA-3: L4F0010: Not interpretable.
1340
L4F0011: Enemy pawn attacking or threatening to attack an active player's 's minor piece L4F0012:
1341 Not fully interpretable; miscellaneous key squares in endgames. L4F0013: Active player's vulner- able king in the corner. 14F001: Square that is will be controlled by enemy pawn, especially if is close to promotion. L440015: Not interpretable. I4F0016: Square deep in opponent's territory controlled by active player's bishop. L4F0017: Active player's centralized piece in the middlegame. L4F0018: Key target square for enemy pawn push. L4F0019: Blockading square for enemy pawn.
1342
1343
1344
1345
1346
1347
1348
1349
25

---

Under review as a conference paper at ICLR 2026
1350
F.2
ADDITIONAL GAB AND ATTENTION HEAD HEAT MAPS
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
M
1370
1371
1372
1373
row-center
SCali:: +2.77010 [row
1374
1375
Figure 9: Layer head MAIA-3 GAB and DPA maps, left and right respectively
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
M
W
1397
1398
1399
1400
1401
1402
#2.4396
center
(row)
1403
Figure 10: Layer 4 head MAIA-3 GAB and DPA maps, left and right respectively
26

---

Under review as a conference paper at ICLR 2026
1404
1405
1406
1407
1408
2
1409
1410
1411
1412
1413
1414
-2
1415
1416
1417
1418
1419
1420
1421
1
1422
1423
1424
1425
1426
1427
25
1428
1429
1430
1431
1432
1433
4
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
F3
1455
1456
1457
Figure 11: Additional Lecla-CF GAB maps from layer 3
27

---

Under review as a conference paper at ICLR 2026
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
2
1474
3
1475
1476
1477
1478
1479
2
1480
1481
2
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
2
52
1493
2
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
A
1507
1508
1509
Figure 12: Additional Leela-CF DPA maps from layer 3
1510
1511
28


Reviews:
[{"review_id": "CMw1EQIzaE", "reviewer": "Reviewer_kykM", "summary": "The paper introduces Chessformer, a novel transformer architecture specifically designed for the domain of chess, which significantly improves move prediction and playing strength over prior approaches. Chessformer makes multiple domain-specific architectural improvements: encoding the 64 board squares as tokens, using an attention-based “source-destination” policy head instead of naively one-hot encoding all possible legal moves, and using a Geoemtric Attention Bias (GAB) to convey positional information. Evaluations on move prediction and game-playing show that the Chessformer matches or outperforms prior approaches at a fraction of the cost.", "questions": "* How did the paper arrive at concatenating the current and the 7 past positions? It would be interesting to ablate this particular architectural choice.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The proposed approach is sensible and appears to work well in practice. The empirical validation is comprehensive and shows that Chessformers match or outperform prior work at a fraction of the scale and cost. The paper conducts ablations to show that the Geometric Attention Bias outperforms traditional positional encoding schemes. The Geometric Attention Bias and the “source-destination” attention head are novel and well-suited to chess. The proposed architecture modifications promise to facilitate domain-specific interpretability research by being more suited to the geometry of chess. The paper is well-written and easy to follow.", "weaknesses": "The main weakness of this work is that it is restricted to chess and, therefore, likely to be of marginal importance beyond the chess-ML community. Chess has served as an important testbed for many ideas in AI; however, this paper’s contribution is to make chess-specific adaptations to Ruoss et al. (2024) to obtain better performance. While the paper does a fine job at that, it is not quite clear to me what anyone outside of the chess community can learn from this work.\n\nGiven the above, the primary contribution of this paper should be to advance the state-of-the-art in the narrow subdomain of searchless chess modeling in a _reproducible_ manner, i.e., by releasing the code, model parameters, and/or the dataset. To the best of my knowledge, the paper does not address any of these aspects, unlike prior work (Ruoss et al., 2024).\n\nThe paper makes the unsubstantiated claim that “Chessformer significantly improves the chess-playing performance of a state-of-the-art chess engine” and “contributed to match wins over Stockfish in multiple computer-chess tournaments”. However, there is no empirical evidence to back up this claim.\n\nThe paper claims to use a novel encoding scheme; however, it is quite similar to the one proposed by Ruoss et al. (2024), who utilize an expanded FEN notation, i.e., 64 board states (and some additional information, which this paper also encodes). The main difference between the two approaches is that Ruoss et al. (2024) only feed the current board state to the transformer, while this paper proposes to concatenate it with the previous 7 board states.\n\nThere are a few typos:\n* L269 “name changed”\n* L352 “absolute position”\n* L355 “Table 3”\n* L376 “A recent line”\n* L429 “much finer interpretation”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Zyl6ikaZ1F", "reviewer": "Reviewer_XrJD", "summary": "This paper presents Chessformer, a transformer-based model for chess that unifies engine play, human move prediction, and interpretability on board attention. The model represents chess positions using 64 square tokens and introduces a Geometric Attention Bias (GAB), which is a dynamic positional encoding that adapts to board geometry. It also uses an attention-based policy head aligned with chess’s “from–to” move structure. Experiments show Chessformer outperforms prior models like Allie in both playing strength and human move prediction while being more efficient and interpretable.", "questions": "- How was the Lichess 2023 dataset processed in practice? How many samples per Elo range were used, and what criteria guided the downsampling to balance skill levels?\n- You mentioned \"Chessformer models mainly adapt the GAB biases to global positional features like the game stage (opening, middlegame, endgame), rather than the locations of individual pieces.\" Why is GAB able to recognize different game stages?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The introduction of GAB is a creative and well-motivated innovation that aligns the model’s attention with the spatial and semantic structure of the chessboard.\n- The experiments are very comprehensive, covering both engine and human benchmarks. The ablation studies also show consistent performance gains.", "weaknesses": "- While the Geometric Attention Bias is central to the paper’s contribution, it is only presented in pseudocode within the appendix. A main-text figure illustrating its structure, input–output flow, and how it modulates attention across the board would make the concept far more intuitive and strengthen readers’ understanding.\n- The description of how Lichess data were sampled and balanced across Elo levels is brief and lacks specific counts or sampling ratios, which may limit reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVwKNfUx7p", "reviewer": "Reviewer_FWRW", "summary": "The paper introduces a novel transformer architecture specialized to playing chess (e.g. approximating optimal play, or for imitating human play). A key novel component is GAP (\"geometric attention bias\"), which essentially adds an additional component computed by an MLP to the pre-softmax attention scores, as a more dynamic alternative to traditional positional encodings. The new \"Chessformer\" architecture also uses a new type of policy head. Experiments for both optimal and human-imitating play show clear performance gains over other architectures, even at lower parameter counts and inference costs. The new architecture also has properties that make it easier to interpret, and the paper takes initial steps towards understanding the functions of different model components.", "questions": "1. Does GAP fully replace any positional encoding? If so that seems interesting/surprising, since if I understand correctly, GAP only directly affects attention scores, so MLPs and attention value vectors would not directly receive any positional information. Did you experiment with e.g. both GAP and an absolute positional encoding? And do you have guesses for why putting positional encodings directly into the residual stream of the transformer (rather than only attention patterns) isn't important for performance?\n2. How cherry-picked are the two SAE features shown in the appendix? E.g. did you look at 100 features and these were the only ones this interpretable, or are half the features roughly this clean?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "- The architecture contains clever ideas and is well-motivated for the domain of chess.\n- The empirical results are very impressive, demonstrating clear gains over previous work and ablations, even when using over an order of magnitude less inference compute than prior work. The evaluation methodology is convincing.\n- I feel that there are lessons to be learned beyond only chess. The paper is a great example showing that a simple off-the-shelf transformer baseline can be decisively beaten in non-language domains with a more specialized architecture. And the motivation for GAP could apply to other domains as well where positions/distances aren't well-described by a static approach.\n- I found the SAE interpretability results (in appendix B.1) highly intriguing.", "weaknesses": "- While I think some of the lessons from this paper could generalize to other domains, the target audience may still be a bit narrow.\n- The description of the architecture (in particular GAP and the policy head) could likely be made easier to follow with some figures showing those novel components", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gYVBEczy7Y", "reviewer": "Reviewer_6GBt", "summary": "This work introduces a transformer architecture meant to be trained over next-move prediction in chess. The two main changes are (1) GAB and (2) formulating the policy prediction over moves more simply (from start to end square). \n\nThe authors take care to provide pseudonyms for models/works that are presumably related to their work. (Sometimes it makes it harder to understand the paper and the relative position of what they are trying contribute.)", "questions": "# Main Questions\n* What is the primary goal of this work? It seems that the architecture is being sold as separate from the \"state-of-the-art chess engine.\" When this is stated is this in reference to Table 3 (Table 3: Main results for raw playing strength.). \n* The second listed contribution, the human-emulation matching, again, seems better, but marginally so, to the point it is unclear (or not sufficiently). The third, again, is the interpretability, and it seems again likely/possibly interesting, but just not shown.\n* Do you report or comment on the stat. range or std of the scores reported in your tables? Anything to help contextualize the results would be helpful. The accuracy deltas being so small make it hard to appreciate the results. Are we near a ceiling of performance? Is the task hard or is the data stochastic? \n* Q: \"Empirically, we find that GAB is a key driver of these gains.\" Note that in Table 1, GAB also requires more FLOPS and param. Where is this clearly demonstrated?\n\n\n# Minor Questions\n* Q: In section 3.2, \"we concatenate representations of the past 7...\" How are the concat? I am reading this as \"stacking\" the information onto each of the 64 tokens. What are the final dimensions?\n* Q: Are the embeddings for the weak/strong players learned/updated? (Any comments on the relatively large dimension for this 128dim; the rest of the embeddings is 12 for the pieces + some other auxiliary information?) It is unclear what the actual/final architecture and dimensions of the model are.\n\n# Minor Notes\n* Table 2. The parameter sizes are very different from those before. I think grouping together the different models and sizes actually used into a clear section would help.\n* As shown in Table 5.3 --> As shown in Table 3,\n* L429 \"finder\"\n* L247 \"achieves its largest gains at the highest calibers of strength *shows that* Chessformer\" --> \"suggests that\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The writing itself is clear\n* Figure 2 was interesting ; The tables 1,2 are clear\n* Introduces a new intuitive way of framing how to format moves (an attention-based “source-destination” policy head)", "weaknesses": "* The first point raised in the abstract is only stated: \"it significantly improves the chess-playing performance of a state-of-the-art chess engine.\" However, it does not describe this or give any information on this claim. While I am sympathetic to the challenge of anonymity issues, we cannot really take these 1-line statements into consideration. \n* The first (and primary?) set of results looks at the loss and accuracy of the models vs. another model (Figure 1) vs. ablations on human scores (Table 1). The differences while real and positive are not really shown to be material. GAB exceeds the Absolute approach by 0.16 percent. Even if this were statistically significant it is not obvious it is meaningful. Likewise, the Figure 1 results are similar. \n* Overall, it seems that the result is a strong one--a good architectural improvement--but that the demonstration of this based off of trust, not clear scientific ablations or clear documentation of what was done. This makes it hard to understand/evaluate the position of the paper.\n* I did not find the interp. section compelling. There is a single figure comparison (cherry-picked? randomly-picked? representative?) in the first subsection. Likewise, in the next subsection they make similar one line note about SAE-like results. Again, it is not that I totally doubt the results are real. Rather, the results are insufficiently demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)