Human-Aligned Chess With a Bit of Search
Authors: Yiming Zhang
Athul Paul Jacob
Vivian Lai
Daniel Fried
Daphne Ippolito
Venue: ICLR 2025 Poster (2025)
Keywords: chess
alignment
adaptive MCTS
inference-time scaling
Primary area: generative models
TL;DR: Learning to play chess (piece movement, pondering and resignation) from humans, and adapting to skill levels from beginner (1000 Elo) to expert (2500 Elo) with time-adaptive MCTS.

Abstract: Chess has long been a testbed for AI's quest to match human intelligence, and in recent years, chess AI systems have surpassed the strongest humans at the game.
However, these systems are *not human-aligned*; they are unable to match the skill levels of all human partners or model human-like behaviors beyond piece movement.
In this paper, we introduce Allie, a chess-playing AI designed to bridge the gap between artificial and human intelligence in this classic game.
Allie is trained on log sequences of real chess games to model the behaviors of human chess players across the skill spectrum, including non-move behaviors such as pondering times and resignations
In offline evaluations, we find that Allie exhibits humanlike behavior: it outperforms the existing state-of-the-art in human chess move prediction and ``ponders'' at critical positions.
The model learns to reliably assign reward at each game state, which can be used at inference as a reward function in a novel *time-adaptive* Monte-Carlo tree search (MCTS) procedure, where the amount of search depends on how long humans would think in the same positions.
Adaptive search enables remarkable *skill calibration*; in a large-scale online evaluation against players with ratings from 1000 to 2600 Elo, our adaptive search method leads to a skill gap of only 49 Elo on average, substantially outperforming search-free and standard MCTS baselines.
Against grandmaster-level (2500 Elo) opponents, Allie with adaptive search exhibits the strength of a fellow grandmaster, all while learning *exclusively from humans*.

Published as a conference paper at ICLR 2025
HUMAN-ALIGNER CHESS WITH A BIT OF SEARCH
Yiming Zhang' Athul Paul Jacob² 2MIT
Vivian Lais 3 Visa Research
Daniel Fried
Daphne Ippolito
CCanegie Mellon University
ABSTRACT
Chess has long been a testbed for Al's quest to match human intelligence, and in recent years, chess AI systems have surpassed the strongest humans the game. However, these strong Al systems are not human-aligned;; they are unable match the skill levels of all human partners model human-like behaviors beyond piece movement. In this paper, we introduce ALLIE, a chess-playing Al designed bridge the gap between artificial and human intelligence in this classic game. ALLIE is trained on log sequences of real chess games to model the behaviors of human chess players across the skill spectrum, including non-m move behaviors such as pondering times and resignations In offline evaluations, we find that ALLIE exhibits humanlike hehavior: it outperforms the existing state-of-the-ard in human chess move prediction and "ponders" at critical positions. The model learns to reliably assign reward at each game state, which can be used at inference reward function in a novel time-adaptive Monte-Carle Carlo tree search (MCTS) procedure, where the amount of search depends on how long humans would think in the same positions. Adaptive search enables remarkable skill calibration; in a large-scale online evaluation against players with ratings from 1000 to 2600 Elo, our adaptive search method leads to skill gap of only 49 Elo average, substantially outperforming search-free and standard MCTS baselines. Against grandmaster-levee (2500 Elo) opponents, ALLIE with adaptive search exhibits the strength of fellow grandmaster, all while learning exclusively from humans.'
INTRODUCTION
Computer chess is among the most studied problems in Artificial Intelligence. In the pursuit of stronger chess engines, decades of hardware and algorithmic improvements since the first computer chess programs (Turing, 1948; Shannon, 1950) have led to the development of increasingly strong chess engines (Campbell al., 2002). Current wate-of-the-ar systems, such as Stockfish (Romstad al., 2008) and AlphaZero (Silver et al., 2017) 7) have reached strength far beyond even the best human players. However, these systems are not calibrated to play at levels matching human strength, and they make moves that are inserutable even to human experts.
In this work, we revisit the classic challenge of computer chess, but with a different objective: developing skill-calibrated and human-alie alignndd chess AI. By skill-calibrated, we mean an system that is evenly matched (i.e., winning approximately 50% of games) against players across the skill spectrum, while maintaining humanlike play. Skill calibration of Al systems is open research challenge, and could prove valuable for domains requiring superhuman AI systems to collaborate with and be overseen by imperfect human partners. Similar Mcllroy- Mcllroy-Young define human- mamman-liggeddd as whether the model behaves indistinguishably from a human player. Our definition extends beyond just move selection: other key aspects, such as time spent pondering a move and the decision to resign in losing positions, are fundamental to how humans play chess. By incorporating these humanlike behaviors, our chess engine ALLIE aims to serve as an engaging and realistic sparring partner for human players.
al. (2020), we
Our approach draws upon recent success in language modeling. Large decoder-only Transformer models, when trained on vast text corpora (Radford et al., 2019; Touvron et al., 2023), learn to
Code, data and model weights are available on GitHub. Ponddrring in chess means spending time to make move humans usually spend more time critical positions. Resignation is the act of conceding game out of respect for the other player a losing position.

---

Published as a conference paper at ICLR 2025
generate text that is sometimes indistinguishable to human- human-written written content (Dugan et al., 2023). Similar to language, chess has a natural sequential representation with moves taking the place of tokens. It is therefore natural to model chess like language: we train decoder-only Transformer model (Vaswani et al., 2017) on a large dataset of human chess game trajectories to model how humans play chess. Our resulting model predicts human moves at state-of-the--rt level (competitive with GPT-3.5 despite many fewer parameters), and demonstrates humanlike behavior in other aspects of chess play, such as pondering and resigning, which previous systems are incapable of modeling. The model also demonstrates a remarkable ability to predict game outcomes at intermediate board positions, achieved solely through supervision on human game outcomes.
Using ALLIE'S next move distribution and value estimation learned exclusively from humans, we add a bit of search at inference time. Specifically, our time-adaptive Monte-Carde tree search (MCTS) method allocates limited inference budget proportional to the predicted human ponder time, enabling more intensive search at critical positions. In a large-scale human study of 7,483 games with 2,412 human players, we find that our adaptive search method enables skill calibration to strengths ranging from beginner to expert levels with a skill gap of only 49 Elo points on average across the skill spectrum. Against 2500 Elo opponents, our adaptive search method enables ALLIE to achieve near-perfect skill calibration, substantially outperforming both search-free free baselines and a traditional MCTS approach with equal computational budget.
RELATED WORK
Most existing approaches in chess engine development have focused on creating the best possible system. Early successful engines like Deep Blue relied hand-coded rules and extensive search algorithms (Campbell et al., 2002). In contrast, AlphaZero (Silver et al., 2017) used self- and Monte-Carlo tree search (MCTS) to learn probability distribution over actions (policy) and estimate game outcomes with a value function. AlphaZero also employed MCTS at inference time to select winning moves. We explore variation of this MCTS algorithm in Section 3.3, using policy and value functions learned directly from human games, and inference time search budget allocated proportional human ponder time.
More recently, Mcllroy-Yound et al. (2020) introduced 'MALA', a neural network trained on human chess games rather than through self-play, proposing a new goal of creating a human- allinned chess AI and achieved remarkable accuracy in modeling how humans play chess. Following the success of MAIA, has been shown chess players can be reliably identified using a small number of games through their playing style (Mellroy-Your substantially boosts the model's capability of predicting the individual's moves (Mellroy- Young et al., 2022). Recently, Maia-2 (Tang et al., 2024) further unifies the Maia models at different skill levels into a single model. Jacob et al. (2022) showed that policy and value functions learned from humans can be combined with MCTS to improve policy strength, and we extend upon their work and demonstrate that adaptive search enables ALLIE to almost perfectly match the strengths of human players up the grandmaster level. By learning value estimates generated by an oracle search engine, Ruoss et al. (2024) showed that neural networks can achieve grandmaster-levee performance without inference-time search. Our approach differs in that our networks are supervised on human data alone.
et al., 2021), and fine-tuning on individual gameplay
Our proposed method is inspired by Toshniwal et al. (2022)'s idea of treating chess like a language modeling task. Feng et al. (2023) fine-tuned language model on chess games, books and commentary and demonstrated that the model can track pieces throughout games and solve chess puzzles, and Karvonen (2024) demonstrated that a language model trained to predict chess moves exhibits emergent understanding of chess concepts. Zhang et al. (2024) similarly showed that Transformer model trained on human games can be made to play at a higher skill level than the games its training data by using a low sampling temperature.
BUILDING ALLIE, A HUMAN CHESS MODEL
Here, we describe how we represent chess game, and our training and inference methods.
Fll is standard measure of strength in two-player games (higher is stronger). A 2500 Elo level corresponds 99.6% percentile of players the popular chess website Lichess.
2

---

Published as a conference paper at ICLR 2025
(b)
Current gume
Game history
1731
Next move:
<7a6a>
Time Contadd
Decoder only Transforaser
Pondering time:
<3'2>>
secenda
Wi
Monn Hatreey
X
K
A
<6206>
Value:
<gir3>
chite u111 win
Natu
Figure 1: (a) The current game state can be represented as the sequence of moves that produced it. This sequence, which also includes metadata on the players' skill and the time setting (c.g. a blitz game), is inputted to a Transformer, which predicts the next move, pondering time for this move, and a value assessment of the move. (b) At inference time, we employee Monte-Cane Tree Search with the value predictions from the model. The number of rollouts Nsim is chosen dynamically based on the predicted pondering time.
3.1 REPRESENTING CHESS GAME SEQUENTIALLY
Vocabulary To apply language modeling techniques to chess, we need a sequential representation of a chess game. To this end, view chess game as sequence moves. We encode moves using Universal Chess Interface (UCT) notation, which specifies every chess move its starting and ending square (see example in Figure 1). We initialize the language model's vocabulary E as the set of possible moves under UCI notation (1968 in total). A board state is implied by the sequence of moves that led to that board state. Game metadata, including the two players' skill levels, time control (how much time each player is allowed to take over all the moves in game), and termination condition (e.g.. whether the game ends with a resignation or checkmate) are added to the vocabulary special tokens. 4 This representation is compact for training: contextualized by the previous tokens in sequence, each token in the dataset implicitly maps to a single board state, making training on a dataset with billions of chess positions feasible and efficient.
Strength conditioning Player skill in chess is computed using the Elo rating system (Elo, 1967). Elo scores normally fall in the range of 500 (beginner players) 3000 (top chess professionals). To calibrate the playing strength of ALLII to different levels of players, we model gameplay under conditional generation framework (Keskar al., 2019), where encodings of the Elo ratings of both players prepended to the game sequence.
The obvious way to encode Elo ratings as tokens would be to add items to our vocabulary representing each Elo score between 500 and 3000. However, this approach runs into data sparsity issues (a small number of games for each individual Elo rating), and this discrete representation fails to encode the fact that scalar distances between Elo scores meaningful (a difference of 5 between two players' Elo ratings indicates they are much closer in ability than a difference of 500). To address these issues, introduce soft control tokens, which interpolate between weak token, representing 500 Elo, and a strong token, representing 3000 Elo. For a player with Elo rating k; we compute a soft token ek by linearly interpolating between the weak and strong tokens; Ek 7' + (1 7) where 3000-k During training, we prefix each game with two soft tokens corresponding the two y 2500
players strengths.
3.2 TRAINING ALLIE TO MOVE, PONDER AND EVALUATE
Using sequential representation of a chess game, we can naturally apply standard sequence modeling techniques to model how human players make moves and when they decide to resign (we treat "resignation" as just another move token the model can assign probability to). ALLIE is built using a decoder-only Transformer model (architecture details in Section 4.2) which inputs the game history as sequence and has three output heads: (I) a policy head that outputs probability distribution over possible next moves, (2) pondering-time head to that outputs the number of seconds a human player would take to come up with this move, and (3) a value assessment head to that outputs scalar
"Time control and skill level are prepended to the start of the game sequence, and termination condition tokens appended to the end of the game sequence.
3

---

Published as a conference paper at ICLR 2025
value representing who is expected to win the game. The pondering- time and value assessment heads are erucial for the human-aligg aligned chess play that we aim to capture. The former allows ALLIE to behave like human, taking more time to make decisions in complex game states than simple ones, and the latter allows the model to discriminate between good moves and blunders. All three heads combined enable the adaptive MCTS procedure, detailed in Section 3.3.
All three prediction heads are individually parameterized as linear layers applied to the outputs of the final decoder layer. Given a dataset D = { (m, t, v)} of chess games, each represented sequence of moves m E ΣA, human think time before each move E IR and the ultimate game outcome v E { [black wins], O[draw], 1 [white wins]}. we train the model to minimize the log likelihood of the next move and mean squared errors of time and value predictions:
(()
Σ
Σ ISISN
(- (-log po (mxl mcs)) (Lo(m)))) 42)² + (vo w)²)
(m.D,))ED
A similar objective of jointly learning policy and value can be found in MCTS-based reinforcement learning algorithms (Silver et al., 2017; Schrittwieser et al., 2020).
3.3 POLICY IMPROVEMENT UNDER TIME CONSTRAINTS WITH ADAPTIVE SEARCH
Virtually all strong chess engines (Romstad et al., 2008; Pascutto & Linscott, 2019) rely on search, a process of exploring possible future moves to pick the best move. Past work has shown that search is crucial for achieving strong gameplay (Silver et al., 2017: Jones, 2021). Since ALLIE produces both policy and value estimators, planning algorithms such as Monte-Carlo tree search (MCTS) (Coulom, 2007) can be applied off-the-shelf for policy improvement. As shown in Figure 1b, MCTS works by rolling out multiple moves into the future, selecting paths that are most likely to lead to a win.
State-of-the-art search-based chess engines such as AlphaZero use a constant number of rollout steps for each move, leading to them assessing tens of thousands to millions of positions before playing a in blitz
move. Such large amounts of search are incompatible with our goal of numan-alignmest games, humans frequently makes moves with <1 second of time usage, and it is practically infeasible to search through such large number of rollouts on consumer hardware in this timeframe. On the other hand, in critical game states where the model predicts a human would spend more time to ponder, it is plausible that running deeper simulations would allow for better modeling of the elevated depth of human reasoning in such positions and improve policy strength.
To this end, we propose time-adaptivy MCTS procedure that aligns MCTS with human reasoning: at each position m, we dynamically set the number of rollouts Nai = [c to (m)]. where to (m) the position m and c constant." Another alternative implementation of time-adaptive MCTS would be to keep searching until a timeout is reached, but we opted against doing this in order to make our implementation independent of hardware efficiency.
the predicted human pondering-tine
4 EXPERIMENTAL SETUP
4.1 DATASET
We constructed a raw dataset of chess games using all blitz" games played in 2022 on Lichess, popular online chess platform.? ? To address the data's skew toward low-skill-level games, downsampled the dataset to have roughly equal numbers of games in bins in increments of 100 Elo. From this downsampled dataset, use 18 thousand games for testing, and the remaining games for training and validation. In total, the training set contains 91 million games and 6.6 billion tokens.
Our primary automatic evaluation metric is move-matching accuracy-how often does the model correctly predict the next move in the game. Following Mcllroy- Young et al. (2020), when eval- uating accuracy, we discard the first moves of each game, which reduces the impact of opening
*The value of is set that Nam 50 for the average position. Our MCTS implementation and hyperparameters follow AlphaZero (Silver al., 2017). See Appendix E.2.
'A blitz, game is one where each player usually can take 3-5 minutes across all their moves. "htpps: /atts:///aaaaaaeeeee 1 tiches.orr//
4

---

Published as a conference paper at ICLR 2025
memorization (there are only many ways to begin chess game). We further omit from evaluation any moves made under time pressure (when there is less than 30 seconds on the clock) to avoid the influence of random moves made due to being low on time. This leaves us with 884,049 positions from an evaluation test set. To further evaluate the abilities of ALLIE to produce valid chess moves under out-of di-trribbbtttoinnnnn game states, we also constructed dataset of random chess games, where each game contains moves that are randomly sampled among legal moves in each position.
4.2 MODEL ARCHITECTURE
Our model uses a standard decoder-only Transformer architecture (Vaswani et al., 2017) with 355M parameters. We initialize model parameters (excluding embeddings) using weights from the pre- trained GPT-2 medium model (Radford et al., 2019), and embeddings are trained from scratch since the vocabulary is not shared with natural language. It may seem surprising that that learned model weights for language modeling are useful for linguuisi task like chess, but this transfer technique is shown effective in other domains (Papadimitriou & Jurafsky, 2020; Shen et al., 2023). The value prediction head is followed by tanh activation layer that squeezes the value prediction to the range [-1,1], 1]. with the extreme values corresponding to wins for each of the two players. Time prediction labels are normalized to have variance 1, and all three loss terms are weighted equally. The model is trained for 2M steps with a global batch size of 131,072 tokens on our training set. This corresponds to roughly 40 epochs over the training data. Additional training details and hyperparameters are provided in Appendix E. In Appendix F, we explore the effect of both dataset size and parameter count on model capability. We find that our setting is mostly data-constrained model performance is limited by the number of human chess games available on the Internet-and doubling model size has only small effect the model's ability of predicting human moves.
4.3 BASELINES
We compare our ALLIE'S learned policy against MAIA (Mellroy- Young al., 2020), which, like ALLIE, is trained on guman-games to make next-move predictions. MAIA is a family of nine individual models, each trained Lichess games from players with Elo ratings in a given range. We refer to these as MAJA-(1100 1200,... 1900}. The MALA network architecture is residual CNN, and their move prediction objective used during training is similar to our approach, but the input representation is board state without full move history information. To unify the different Maia models into single strong baseline, we define a MAIA" model by adaptively choosing the Maia model with the closest Elo rating to the players' ratings. For example, a 1480-rated game would be evaluated using the Maia-1500 model. We note that publicly available MATA models are much smaller than ALLIE, and this has an effect on the relative performance of the models. We explore a variant of ALLIE with half the parameters in our ablation study (Section F).
The primary comparative metric we use for automatic evaluation is move-matching accuracy: what fraction of the time does the system correctly predict the move a human would have made. Other aspects of human-aligned chess play (e.g., modeling human moves vs. time usage) require different evaluation metrics, which we detail in Section 5. To the best of our knowledge, there are no existing chess engines that model how humans play chess in terms of pondering and resigning, we do not have a direct comparison with a baseline system for these behaviors.
Though large language models (LLMMs) such OpenAl's GPT--3.5-turbo-nnsttee our knowledge) been explicitly trained to play chess, they have been shown to reliably produce humanlike next moves." This is accomplished by prompting the LLM with textual representation of the game state using PGN notation." Due to dependency on the textual PGN notation, this approach not compatible with OpenAl's 's latest chat-based LLMs (e.g., GPT-4), and we report prompts and implementation details in Appendix B. It is difficult to make a fair comparison between ALLIE and GPT-3.5 because on the one hand, GPT-3.5 has many more parameters and potentially observed much more chess data during pre-training. On the other hand, GPT-3.5 was never intended to play chess, and the fact that it can play chess is somewhat remarkable. We report GPT-3.5 results just to provide context on performance achievable by a frontier large language model.
have not (to
*https: / i chal as carl ini ccw/wwwiiii wttp:rr/in////////////e chess-heesseeeeeeeeee 1 html
9The Portable Game Notation (PGN) is popular human-readable and human- writalee textual notation for chess games.
5

---

Published as a conference paper at ICLR 2025
Table All configurations of our chess-engine, ALLIE.
Config-
Description
ALLIE-POLICE
Softmax sampling according to with unit temperature. 2,500 Elo level.
ALLIE-GREEDY
Greedy decoding according pe conditioned with non adaptive MCTS (50 rollouts). LOLIC with adaptive MCTS (c set such that MCTS performs 50 rollouts on average across all positions).
ALLIE-SEARCH
ALLIE- SEEEE-AAAAPIV--AAAE SEARCH
Table 2: ALLIE learns to play valid chess moves. 95% confidence inter- vals are shown.
Table 3: ALLIE- ALLIE-POLICY outperforms state-of-the-ari meth- ods in human move prediction. Move prediction accuracy with 95% confidence intervals reported in the table.
Evaluation set
Top-1 1 move is valid (%)
Human plays...
ALLIE (%)
Maia" (%)
GPT-3.5 (%)
All moves
55.7 + 0.1
51.6 + 0.1
53.7 + 0.1
Lichess
100.0 + 0.0
Castling
74.3 + 0.5
73.3 0.6
72.4 0.6
Lichess (under check)
100.0 + 0.0
En passant
70.4 4.1
4.2
71.4 4.0
Random
99.9 0.0
Pawn promotion
86.9 + 1.7
8..1
86.0 1.7
Random (under check)
96.6 + 0.0
Threefold repetition
92.0
8..0
92.8 4.4
4.4 LARGE-SCALE HUMAN STUDY
In addition to conducting offline evaluation, we deployed the four configurations of ALLIE described in Table as well as MATA", play blitz games on the website Lichess. ALL.IE-POLIC' was conditioned to play adaptively at the opponent's s strength, and moves were sampled from the model distribution pa. BREIEY was conditioned to play at a 2,500 skill level, and top moves under the model distribution are played. This setting allowed us measure the upper bound of the policy strength." 10 ALLIE-SEARGE and SELLI--ADAPIVEE -SARCH employ inference-tine MCTS to improve move selection, with the latter using adaptive number of rollouts. Overall, collected 7, 7,483 blitz games with 2,412 human players over a multi-week period. After each game, players were invited to fill out survey about their experience. Survey results can be found in Appendix D.2.
RESULTS
To apply inference-time search to ALLIE, we first need to understand chess is at all learnable from -eneerree data (Section 5.1), and if so, how well ALLIE models human gameplay (Section 5.2). We discuss our main results on adaptive MCTS and skill calibration in a large-scale study against human players in Section 5.3.
5.1 DOES ALLIE LEARN THE RULES OF CHESS?
First, we ask whether the rules of chess are learnable from human-generated chess data. The model produces a softmax distribution over roughly two thousand possible chess moves, and we can test the model has indeed learned the rules of chess by checking if model assigns high probability to valid moves, and low probability to invalid moves. While we evaluate the model's behavior on actual human games, it also important to test if the model can generalize to out-of- diiitiibiitiiioi positions that are rare in human games but are nevertheless valid: model that has learned the rules of chess should play legal moves in randomly generated games as well. Beyond testing the model behavior in the aggregate, we further examine the model's behavior when special chess rules restricting valid moves (e.g., check) are in effect. !!
In Table 2, report how often the top move from the model distribution is valid. On both the human and random evaluation sets, we find that the top move is almost always valid: 100% of the time on human games, and 99.9% of the time on random games. Softmax distributions by definition assign
10 rating of 2500 is typically considered as the threshold for grandmaster level play.
See Appendix A for a glossary of chess terms.

---

Published as a conference paper at ICLR 2025
System
607
Maia*
Maia-1100
Maia-1200
Move-matching accuricy
55%
Maia-1300
Maia-1400
Maia-1500
50%
Maia-1600
Mata-1700
Maia-1800
49%
Maia-1900
GPT-35
Allie Polley
40%
Sdaptiveeee
600
200
800
500
1000
1100
1200
1900
100
1000
1002
2200
2200
2400
2000
200
1002
Game rating
Figure 2: Adaptive search enables matching human moves expert levels. Move-matching accuracy of ALLIE-POLIEE
ADAATIVE-SEAAE MAIA and GPT-3.5 are reported across skill levels. SEARCH has virtually the same move matching accuracy as ALLIE- AAALII-AAAAIIIESAAAAA and is omitted from the figure.
non-zero probabilities to all (including invalid) moves, but this probability is vanishingly small: 0.2% in both human and random games (see Table in Appendix C.I). In positions where the king under check," 12 the model still only assigns 0.2% of probability to all invalid moves. Our results suggest that the model has indeed learned the rules of chess from observing human chess games, and generalizes reasonably well to out-of out-of-distribe of-distribution positions.
5.2 HOw WELL DOES ALLIE MODEL HUMAN GAMEPLAY?
The ideal human-aligned chess bot should behave indistinguishably from a human chess player. A major aspect of humanlikeness is in the moves played: for given game state, a humanlike chess bot should play the same move as a human would in the same position. Beyond moves played, we argue that it important match the time humans ponder their moves before taking them, and resign when appropriate-ther these are also essential components of how humans play chess.
Moves. the Lichess evaluation set, we compare how often ALLIE, GPT-3.5, and the MAIA models play the same moves as humans. Following sellroy-Young et al. (2020), we consider the move-matching accuracy metric, defined as the fraction top-1 moves under the model distribution that matches human moves at the same positions. Over the entire test set, the top move produced by ALLIE matches human moves 55.7% of the time, compared to MATA 's 51.6% and GPT-3.5's 53.7% (Table 3).Shown in Figure 2, find that ALLIE matches human moves more accurately than MAIA and GPT-3.5 models across almost the entire skill spectrum. Notably, outperforms ALLIE-POLICY 2300 Elo and above, providing evidence that search is crucial for modeling the behavior of expert-level human players (Jacob et al., 2022).
We further report move-matching accuracy of special moves such castling, en passant, pawn promotion, and threefold repetition in Table 3. ALLIE reaches higher move-matching accuracy than MAIA* for all four types of special moves, and is competitive with GPT-3.5 overall.
Pondering time and resignation. Additional dimensions of human behavior, including pondering time and resignation, are also key aspects humanlike gameplay. We find a strong correlation between the model's predicted think time and human think time, with Pearson's suggests that ALLIE successfully learns to predict when humans do and do not ponder a position. Figure shows the distribution of ALLIE'S predicted think time for different amounts of time spent by humans. There is clear monotonic relationship. but interestingly ALLIE tends to predict lower
0.697. This
12This is game state where the set of valid moves is more restricted than usual: the player must make move that prevents the opponent from capturing their king piece.

---

Published as a conference paper at ICLR 2025
Positions
1.0
System
250,000
Allie
Stockfish
0.5
Model think (s)
200,000
Correlation
150,000
0.4
100,000
2-
S0,000
0.0
10
25
20
225
30
60
50
30
217
Human think time (6)
Moves from terminal
Figure 3: ALLIE'S 's time predictions are strongly correlated with ground-truth human time usage. In the figure. we show median and interquar- tile range of ALLIE'S predicted think time for different amount of time spent by humans and observe a clear monotonic relationship.
Figure ALLIE learns to assign reliable value estimates to board states by observing game outcomes alone. We report Pearson's correla- tion of value estimates by ALLIE and Stockfish with game outcomes. Game outcomes are in- creasingly predictable as the game progresses.
pondering times than humans do. This is probably because of the skew in pondering time distribution: the majority of moves in blitz games is played under 5 seconds, and the model is incentivized to "hedge" its prediction and output shorter pondering times.
We further evaluate whether ALLIE resign in losing positions like humans. We define resignation as when a special resignation token <resign> is assigned higher likelihood than all valid moves on the board, and the predicted board value is below -0.9 from the perspective of ALLIE. We focus our analysis on both the true positive rate (TPR), i.e., the number of positions where the model resigns when humans resign, and false positive rate (FPR), i.e., the number of positions where the model resigns when humans do not resign. Over the evaluation set, we observe a TPR of 86.4% indicating ALLIE usually resigns when a human would. ALLIE almost never resigns when a human wouldn't. 't, with FPR of 0. %. Our results highlight that ALLIE models human chess play holistically, not only in terms of moves played, but also in pondering time and resignation when approriate.
Reliable board value estimate. Before applying search algorithm such as Monte- Carlo tree search (MCTS), we need value function that guides exploration of promising game states. Recall that ALLIE is trained predict the outcome of games at each eesition-whice can be conveniently interpreted as board value function. In Figure 4, we show how well ALLIE'S value function and an oracle value function correlate with game outcomes." By observing only outcomes of games without additional supervision, we find that AL.III learns to assign surprisingly reliable value estimates to chess board states: ALLIE'S value estimates closely match that of the oracle, and predicts game outcomes just as well. Notably, ALLIE has access game metadata (in particular, player skill levels) that Stockfish does not, which may explain why it even outperforms Stockfish sometimes. Our results suggest that ALLIE learns credit assignment in chess by observing game outcomes alone, and provides the foundation for applying salue- search methods such as MCTS.
5.3 EVALUATING SKILL CALIBRATION VIA GAMES WITH HUMANS
Our offline evaluations suggest that ALLIE predicts human behavior well, but study whether ALLIE could calibrate to strength of human players, we had ALLIE play against real humans at a variety of skill levels. chess engine that is perfectly skill-calibrated should win 50% of games against players regardless of their skill level. Inspired by the expected calibration error metric (Naeini et al., 2015; Guo et al., 2017), we define a skill calibration error (SCE) metric. Games between the chess engine and humans are first partitioned into equally spaced bins based on skill level (player Elo). For a bin of games between the evaluated system and human players, we take the absolute difference between the system's 's estimated performance the set of games, and the average Elo of the human
We use evaluations of Stockfish (Romstad al., 2008) after 10 nodes searched.
8

---

Published as a conference paper at ICLR 2025
0.65
System
Mala
System peformance möng
Mais-110 1100
Mave-matching acturacy
0.60
Maia-1300
04
Mada-1100
Mais-1600
-200
System
Mass-1700 1700
Maiz-1500 1500
Alle-Pdkcy
0.50
400
Alll-Gredy
GPT-35
Allie-Policy
Alle- Seach
AlleeAdaative-Seechh
1,000
1,30
1,50
1,90
1,900
2,00
2.360
250
0%
20%
407
60%
90%
1007
A Average oppoment rating
Game progress (%)
Figure Search-free methods fail to match skill level of strong players. We estimate difference strength of various systems online players. Values close to 0 indicate good skill calibration.
Figure 6: Human move prediction accuracy in- creases consistently across models as the game progresses and jumps up sharply in the last por- tion of the game.
Table 4: Adaptive search enables remarkable skill calibration. Mean and maximum skill calibration
erros are computed by binning human players into 200-Elo groups. We also report systems' estimated performance against players at the lower and upper Elo ends the skill spectrum.
Skill Calibration Error Mean +
Online performance vs. 1100-rated
System
Max 4
2500-rated
Search-free
MAIA
146
336
1251
2138
ALLIE-POLICE
127
351
1134
2136
ALLIE-GREED
328
677
1799
2260
Search-based
ALLIE-SEARCH
80
166
1180
2318
ALLIE-ADEE ADARTIIN----ACCH
49
95
1196
2528
players as the calibration error: 14
SCE(B) = |SystemElo(B) HummnnEo(B)).
Search-free methods do not match the strength of experts. Figure 5, we show estimated ratings of the systems against human players across different strength levels, and well-calibrated systems should have a rating difference close to O. Mean and maximum skill calibration errors are reported in Table 4. We find that ALLIE-POLICY, ALLIE-GREEDS and MAIA* are not calibrated to opponent strength. ALLIE-POLICY and MAIA are more or less evenly matched against players below 2100 Elo, but against players above 2400 Elo, both models perform poorly, with ALLIE-POLICY scoring 1 11.1% % and MAIA scoring 12.5% on average. than weak players (< 2100 Elo), and yet still loses 75% of games to players above 2400 Elo. All search-free systems perform progressively worse against stronger players, suggesting that strength conditioning, sampling temperature
is considerably stronger
-LLIE- or multiple expert models for different skill levels (MAIA*) may not be sufficient to match the strength of strong human players.
Skill-calirrated chess play with adaptive search. Despite being a strong human move prediction model, search-free ALLIE configurations do not match the level of gameplay of strong (2 2000 Elo) players. Qualitatively, models blunder pieces and make suboptimal moves in ways that strong players do not (see online players' feedback in Figure 11, Section D.2). In this section, we discuss how we can improve the skill calibration of ALLIE-in particular its performance against strong players-and maintain humanlike play by incorporating an adaptive search method. Recall that
"We follow rules of the International Chess Federation for computing performance Elo and report evaluation details in the appendix D.I.

---

Published as a conference paper at ICLR 2025
Table 5: Some examples of the qualitative feedback we received in our post-game survey,
System
Feedback
ALLIE-
liked the fact Allie plays like human, and makes human mistakes. She's not like, let's say, Stockfish level 1 making absurd mistakes, nor an inhuman AI with perfect play, but humanlike player that fights for a win and makes human-rcasonable moves. Honestly, I'm not top player, but like to play with similar opponents and I'm also programmer with interest in AI, and feel satisfied with Allic's 's behaviour. Great job I really felt like was playing against a human, but have some opinions on this robot: Firstly, I noticed that he plays the opening well, which is a very good thing Secondly, I also noticed that in the middle of the game his accuracy decreases somewhat, he makes mistakes and inaccurate moves, and this is just like a human.
ADAPTIVE-
SEARCH
ALLIE- POLICY
ALIIE-ADAPIIVESSEAARCC uses an adaptive search budget allocated linearly according to predicted human pondering time at each position, and we compare it with equal-compute MCTS baseline, ALLJE-SFARCH.
We find that ALLIE- ADARTIVE-SEARCH improves skill calibration remarkably, achieving an average skill calibration eггоr of 49 Elo, and a maximum skill calibration error of 95 Elo. Figure 5 helps contextualize this finding, where we see the performance ratings of ALLIB-ADAM DALLI---DAAIIV-SSAARAA exhibit a near-linear relationship with opponent ratings. This is a substantial improvement over all cearch-free systems, all of which underperform 2 2400 Elo players by at least 200 Elo points.
More surprisingly, ALLIE- ADAPTIVE-SEARCE outperforms standard AlphaZero-like MCTS (ALLIE- SEARCH), in both overall skill calibration and performance against 2500 Elo human players. Our findings suggest that humanlike reasoning at "critical" positions is useful for reaching expert-level chess. Crucially, ALLIE-SEARCI SEARCH and ALLIE-ADAPTIVE-SAARAE achieving move-matching accuracy of 55.9% compared to 55.7% for ALLIE-POLICY and 51.6% for MAIA".
maintain humanlike play, both
6 DISCUSSION
In this work, we demonstrate a method for training a state-of-the-ar chess Al that models how humans play chess: our system ALLIF exhibits remarkable precision in playing humanlike moves, as well as pondering and resigning like humans. Through a time-adaptive Monte-Carlo tree search algorithm, ALLIE can be evenly matched with players from beginner (1100 Elo) to expert level (2500 Elo) with almost no skill gap, by learning chess exclusively from humans without the need of distilling from strong chess engine. We believe the techniques developed in this paper have broad applicability for other settings where aligning Al models with imperfect human reasoning is crucial, and we look forward to future explorations in other complex settings, such as the alignment and oversight of superhuman Al systems.
While offline evaluation metrics and quantitative analysis of games with real human players reveal ALLIE'S strengths, especially relative to prior approaches, more progress is still necessary to fully realize our goal of a nhuaan-aligned chess engine. In qualitative feedback, many players were positive about ALLIE (see Table 5), but several shortcomings were also repeatedly emphasized. Players especially noted ALLIE'S ' propensity toward late-game blunders and that its pondering times were sometimes long in positions where there is only one reasonable move. However, since players all knew they were playing against a bot, it hard to disentangle their perspectives from this knowledge. For example, contrasting with the qualitative feedback, we empirically observed that move prediction accuracy actually improves as games progress, especially in the last few turns (see Figure 6). For future work, it would be interesting to conduct proper Turing test, where players do not know whether they are playing against an AI or a human--payer of a similar Elo level.
Our approach relies on pre-training, which is limited by available data: the vast majority of online chess games are played at fast time controls, and therefore is more challenging to use data-driven methods model human behavior in slower games. Future work should explore methods to model human reasoning in slower games, where players have more time to think and make more accurate moves, and test the generalization of our approach different time controls and game formats.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We thank CM Ethan Gu and thousands of Lichess players who tested our systems and provided feedback, well as the Lichess team and contributors for creating free and open platform for chess. We are grateful to Nicholas Carlini, Harshita Diddee, Mourad Heddaya, Xinyue Liu, Vinay Samuels, Chenhao Tan, Barry Wang, and Ximran Zhao for invaluable discussions. YZ is supported by an OpenAI Superalignment Fellowship. YZ and DI are supported by grants from Cisco and CMU Cylab Seed Funding.
REFERENCES
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alee Radford, lya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. hutps//arxiv..rg/abs/000.. 14165v4, May 2020.
Murray Campbell, A. Joseph Hoane, and Feng- boig-hiingg Hsu. Deep Blue. Artificial Intelligence, 134 (1):57-83, January 2002. ISSN 0004-3702.
Nicholas Carlini. Playing chess with large language models, September 2023.
Rémi Coulom. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. In H. Jaap van den Herik, Paolo Ciancarini, and H. H. L. M. (Jeroen) Donkers (eds.), Computers and Games, pp. 72-83, Berlin, Heidelberg, 2007. Springer. ISBN 978-3-540-75538-8
I Liam Dugan, Daphne Ippolito, Arun Kirubarajan, Sherry Shi, and Chris Callison-Burch Real or fake text? investigating human ability to detect boundaries between human- -writee and machine- generated text. In Proceedings of the Shirth-Seventh AAAI Conference Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Sympo- sium on Educational Advances in Artificial Intelligence, volume 37 of pp. 12763-12777 AAAI Press, February 2023. ISBN 978-1-57777 178-1-57733--8800
A.E. Elo. The USCF rating system: Its development, theory, and applications. Chess Life, XXII (8): 242-247, 1967.
Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Henry Mguni, Yali Du, and Jun Wang. ChessGPT: Bridging Policy Learning and I Language Modeling. In Teirth-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, November 2023.
lean-Bastien Grill, Florent Altché, Yunhao Tang. Thomas Hubert, Michal Valko, loannis Antonoglou, and Remi Munos. Monte- Carlo tree search as regularized policy optimization. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 3769-3778, PMLR, July 2020.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, 17, pp. 1321-1330, Sydney, NSW, Australia, August 2017. JMLR.org.
Athul Paul Jacob, David Wu, Gabriele Farina, Adam Lerer, Hengyuan Hu, Anton Bakhtin, Jacob Andreas, and Noam Brown. Modeling strong and human-like gameplay with KL-regularized search. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9695-9728, PMLR, July 2022.
Andy L. Jones. Scaling Scaling Laws with Board Games, April 2021.
Adam Karvonen. Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models. In First Conference Language Modeling, August 2024.
11

---

Published as a conference paper at ICLR 2025
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. CTRL: A Conditional Transformer Language Model for Controllable Generation, September 2019.
Ily Loshchilov and Frank Hutter. SGDR: Stochastic Gradient Descent with Warm Restarts, In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Reid McIlroy- Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning Superhuman Al with Human Behavior: Chess as a Model System. In Proceedings of the 26th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pp. 1677-1687, Virtual Event
CA USA, August 2020. ACM. ISBN 978-1-4503-799844
Reid Mcllroy- Young,, Yu Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Detecting Individual Arcision-Makinn Style: Exploring Behavioral Stylometry Chess. In Advances in Neural Information Processing Systems, volume 34, pp. 24482-24497, Curran Associates, Inc., 2021.
Reid McIlroy- Young, Russell Wang, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Learning Models of Individual Behavior in Chess. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1253-1263, August 2022.
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling Data-Constrained Language Models. In Thirty-Seventh Conference Neural Information Processing Systems, November 2023.
Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI 15, pp. 2901-2907, Austin, Texas, January 2015, AAAI Press. ISBN 778-0-262-51129-1
Isabel Papadimitriou and Dan Jurafsky. Learning Music Helps You Read: Using Transfer to Study Linguistic Structure in Language Models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6829-6839, Online, November 2020. Association for Computational Linguistics.
Gian-Carlo Pascutto and Gary Linscott. Leela chess zero, March 2019.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019.
Tord Romstad et al. Stockfish, November 2008.
Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, Cannada A. Lewis, Joel Veness, and Tim Genewein. Amortized planning with large-scale transformers: A case study on chess. In NeurIPS, 2024.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature, 588(7839):604 609, December 2020. ISSN 0028-0836, 1476-4687.
Claude E. Shannon. XXII. Programming a computer for playing chess. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 41(314))266-75, 1950.
Junhong Shen, Liam Li, Lucio M. Dery, Corey Staten, Mikhail Khodak, Graham Neubig, and Ameet Talwalkar. Cross-Modal Tunn-Tnnng:: Align then Refine. In Proceedings of the 40th International Conference on Machine Learning, pp. 31030-31056. PMLR, July 2023.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap. Karen Simonyan, and Demis Hassabis. Mastering Chess and Shogi by Self-Play with a General Rein- forcement Leaming Algorithm, December 2017.
12

---

Published as a conference paper at ICLR 2025
Zhenwei Tang, Difan Jiao, Reid Mellroy-Young, Jon Kleinberg, Siddhartha Sen, and Ashton Ander- son. Maia-2: Unified Model for Human-AI Alignment in Chess. In The Thirty-eighth Annual Conference Neural Information Processing Systems, November 2024.
Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess Testbed for Language Model State Tracking. Proceedings of the AAAI Conference on Artificial Intelligence, 1 1393, June 2022. ISSN 2374-3468.
Hugo Touvron et Llama 2: Open Foundation and Tine-Tuned Chat Models, July 2023.
Alan Turing. Turochamp, 1948.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t. ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade, and Eran Malach. Transcendence: Generative Models Can Outperform The Experts That Train Them. https//arrixxoor/////4444..11111 117.44000.1111111 June 2024.
13

---

Published as a conference paper at ICLR 2025
A GLOSSARY OF CHESS TERMS
In this section, provide glossary of chess terms that are used throughout the paper. The terms are summarized in Table 6.
Table 6: Chess glossary.
Chess term
Definition
Check
situation which a player's king is under direct attack by an oppo- nent's piece. The player must resolve the check their next move. Check limits the number of valid moves in position.
Castling
A special move involving the king and either rook. The king moves two squares towards the rook, and the rook moves to the square the king crossed.
En passant
A special pawn capture that can occur immediately after pawn makes a double-step move from starting position. The opposing pawn can capture if it had only moved one square.
Pawn promotion
When pawn reaches the opposite end of the board, it can be promoted to any other piece (usually a queen) of the same color, except king.
Threefold repetition
A rule that states player can claim draw if the same position occurs three times during game, with the same player to move each time.
B GPT-3.5 EVALUATION
Following the implementation of Carlini (2023), encode chess move sequences in PGN format (see Figure 7) and feed them as prompt to tprrs.stturoo-insruue for evaluation. Note that we were unable to use the latest OpenAl models like GPT-4 since this evaluation requires access to non-chat language model API. We use greedy decoding to generate the next move, and in the rare case when the model does not output legal move, random move played.
[White "Garry Kasparov"]
[Black "Magnus Carlsen"] [Result "1/2-1/2"] [WhiteElo "2900"] [BlackElo "2800"]
1. e4 e5 2. Nf3
Figure 7: Prompt for
evaluation.
14

---

Published as a conference paper at ICLR 2025
C OFFLINE EVALUATION
C.I LEGAL MOVES
In Table 7, we show that ALLIE not only learns to assign high probability to valid moves human games but also in out-of ofitrriibbitttinnn randomly generated games. Under softmax distribution, the probability mass of all invalid moves low, indicating that the model capable of distinguishing between valid and invalid moves.
Table 7: ALLIE learns to play valid chess moves. 95% confidence intervals are shown.
Evaluation
Top move is valid (%)
Probability mass of all invalid moves (%)
Lichess
100.0 + 0.0
0.2 0.0
Lichess (under check)
100.0 + 0.0
0.2 0.0
Random
99.9 0.0
0.1
Random (under check)
96.6 1 0.0
4.1 = 0.2
C.2 HUMAN MOVE PREDICTION
Overall, we find ALLIE outperform state-of-the-art methods in human move prediction (Table 8). Similar to the findings Jacob et al. (2022), we find that, adding Monte-Caud tree search (ALLIE- ADAPTIVE-SEARCEE improves upon pure imitation learning policy interesting observation is that as the game progresses, human moves become increasingly predictable, shown in Figure 6.
Another
Table ALLIE outperforms state-of-the-art methods in human move prediction. Move prediction accuracy with 95% confidence intervals are reported.
Human plays..
POLICY (%)
ALAPPIADA SEARCH (%)
Maia" (%)
GPT-3.5 (%)
moves
+
55.9 0.1
51.6 0.1
0.1
Castling
74.3 + 0.5
74.3 = 0.5
73.3 + 0.6
72.4 0.6
En passant
70.4 + 4.1
71.0 + 4.0
67.7 + 4.2 4.2
71.4 4.0
Pawn promotion
86.9 1.7
87.5 = 1.6
85.1 1.8
86.0
Threefold repetition
92.0 4.6
90.6 4.9
8.7
92.8
C.3 ANALYSIS ALLIE'S VALUE PREDICTIONS
ALLIE'S supervision on human game outcomes is designed to teach the model to assign high values to positions that a human can convert, as opposed to positions that are theoretically winning under perfect play. This section explores the correlation between ALLIE'S 's value predictions and human game outcomes. We also qualitatively analyze a few positions where ALLIE diverges from Stockfish in its value predictions.
C.3.1 ROLE OF METADATA IN VALUE PREDICTION
To investigate the role of player Elo in value prediction, compared the accuracy of ALLIE'S value predictions against Stockfish evaluations predicting game outcomes at various stages the game. The analysis was conducted decisive games, excluding draws.
Table 9: Game outcome prediction accuracy with a <00 Elo gap between players
Game Phase
Stockfish
ALLIE
Opening 50.4%
55.2%
Midgame
62.9%
65.6%
Endgame
73.9%
74.3%
15

---

Published as a conference paper at ICLR 2025
Table 10: Game outcome prediction accuracy with a 100 Elo gap between players
Game Phase
Stockfish
ALL.I
Opening
65.2%
76.8%
Midgame
69.9%
79.3%
Endgame
82.5%
83.7%
8
6
8
5
4
3
8
8
2
b
9
Figure 8: Opening position where value predictions of ALLIE and Stockfish diverge.
Elo provides significant information about the likely winner. In games with at least a 100 Elo gap, ALLIE outperforms Stockfish by 9.4% in the midgame, compared to 2.3% in games with 10 Elo gap. However, even in games with minimal skill differences, ALLIE consistently outperforms Stockfish across all phases. Taken together, these results suggest that ALLIE'S value predictions are more reliable in human games than Stockfish's. 's even without information about the players' skill level.
C.3.2 QUALITATIVE ANALYSIS OF POSITION CONVERSION
In this section, we qualitatively analyze several positions from the test set, where ALLIE and Stockfish diverged in the sign of their value predictions.
In an opening position (Figure 8). black has a knight for two pawns but cannot castle. Black is objectively winning (Stockfish assigns 77.0% probability of winning to black), while ALLIE favors white with a 78.8% probability, since the position requires a series of precise moves to convert. In the actual game, black misplayed the position and lost the game within five moves, suggesting ALLIE may be able to better value positions that are objectively winning but challenging for humans to convert.
In endgame position (Figure 9), ALLIE surprisingly considers black winning with 75% probabil- ity, despite white having an extra queen (Stockfish assigns 98% to white). ALLIF likely inferred time pressure white from the move history, black's 's predicted advantage began several moves ago when white blundered rook.
However, ALLIE exhibits blindspots with evaluating checkmates and sacrifices, which strong human players can calculate. In a midgame position (Figure 10), ALLIE gives black a 67% probability of winning, but white is objectively winning with the queen sacrifice Qe7. The human player finds this move and wins, as Stockfish predicted. This inability to predict checkmates and sacrifices is not surprising, since reliable position evaluation without search is very challenging.
16

---

Published as a conference paper at ICLR 2025
8
7
5
3
2
b
d
g
Figure 9: Endgame position where value predictions of ALLIE and Stockfish diverge.
8
3
6
4
3
8
2
b
g
Figure 10: Midgame position where ALLIE ignores critical queen sacrifice.
D ONLINE EVALUATION
D.I ESTIMATION OF PERFORMANCE ELO
Our estimation of performance Elo ratings follows guidelines of the International Chess Federation (FIDE). Let denote the average Elo rating the opponents, and p represent the player's average score against these opponents. FIDE provides table of estimated rating differences dp corresponding to various values of p. For example, p = 0.5, then dp 0, and p = 0.75, then dp = 193. These values indicate that player scoring 50% against their opponents is performing level, while a 75% score suggests a performance 193 Elo points above the opposition's 's average. The complete table of estimated rating differences the performance rating, one would add the rating difference dp to the average opponent rating r. This method provides standardized approach estimate player's performance level based their results against opponents of known strength.
the same Elo
be found in the FIDE Handbook 15, To calculate
D.2 SURVEY RESULTS
In Figure 11, we show the results of post-game survey where human players were asked to rate the humanlikeness and enjoyability of the systems. We find that ALLIE is rated as more humanlike
See httts://haaaeeee fi com httpe/oo//caaptee//eeeeeee for the full rating difference table.
17

---

Published as a conference paper at ICLR 2025
(28.9% of participants strongly agree) compared to MAIA (24.8%) and more enjoyable to play against (38.6% vs. 2..5%)
My opponent played like a human player.
enjoyed this game.
Reepoose
Reeponse
Stungly
System
Alli--wiley
Sumindur
deagree
Nevtoll
Crwayy
Neatral
Alle-bercch
Seorghs agore
Semsrbut agree
Alli-Seanch
Semanetat agoas
Stonnly agre
St
Alla- Alla-hdaaptreSann
Percentage participants
Percentage partictpants
Figure II: Survey responses.
D.3 QUALITATIVE FEEDBACK
We provide additional examples of the qualitative feedback we received in our post-game survey in Table 11.
18

---

Published as a conference paper at ICLR 2025
Table 11: Additional examples of the qualitative feedback we received in our post-game survey.
System
Player Elo
Feedback
ALLIE- POLICY
1640
Played very human- like, resigned at the exact time a human would, and got weaker and sort of "demotivated" as she was losing just like human. Amazing chess bot
ALLIE- POLICY
1940 It's very close to being human-liks like. The thing will say is that sometimes it appears to take non-obvious moves with no clear "plan" and have yet to get to resign. It also seems to he quite bit weaker than I am, and don't 't really play Blitz so I can't 't imagine I'm very good.
ALLIE- POLICY
2038 As in the last game I won against Allie, the dropped piece seemed to come out of nowhere. It wasn't 't a missed tactic or anything like that, but bad sacrifice. Sometimes of course this will happen against humans, but both of the games played where this happened to me, it was hard to see any lines (where didn't 't outright blunder two or more moves a row) where the sacrifice would lead to anything. also expected a resignation at the end.
ALLIE- GREEDY
1139
Felt human - sometimes when I play Lichess's 's implementation of Stock- fish at level appropriate for my skill, it makes really bizarre moves, even catastrophic. Maybe they't calculated blunders for noobs like myself, but they're 'e unrealistic. A novice might miss an obvious fork or skewer, but they would never give up their queen for no reason. They would at least try to save it, even if ultimately impossible. Allie doesn't 't seem to do that.
ALLIE- GREEDY
912
It seemed at the end that the bot's goal was to clean out my pieces and promote pawn for second queen to checkmate rather than just go for checkmate. suppose it's possible that the promotion would have been fewer turns I'd have to go back and check.) But I feel like human player would have just gone for a QK v K- K-style checkmate rather than clean out several of my pawns to make easy promotion.
ALLIE- GREEDY
2008 did not take the pawn on e4. Then played what it feels like a pretty accurate series of moves later in the game. From move 21 the bot played all the best moves some of which feel pretty strong.
ALLIE-
1637 The bot is plays very much like human. It understood when it had to move fast and when had take time. The opening was littlc inaccurate but other than that the bot is really good.
ADAPTIVE-
SEARCH
ALLIE-
1998 I'd say all moves up until 26. Qc6 were human. Qc6 is slightly unexpected but not that bad.
ADAPTIVE-
SEARCH
It was bit strange that it took few seconds to take the rook on move 30, because real human would have understood what they were doing by 29... Be3 and taken immediately to finish the combination.
Just like Maia don't 't think it knows what to do the endgame, which probably contrihuted to the blunder.
I slightly expected the bot to do 49... Be4 or Ra2 or something to stop the pawn, but
ALLIE-
2004
In terms of play I think what I found the least human like was it's will- ingness to trade when it was down full piece. My intuition that these very low level concepts like that even very suboptimal moves being prac- tically better because it increases the long term probability of blunders, is something bots of all strengths struggle with.
ADAPTIVE-
SEARCH
However, my opinion is obviously affected by me knowing that I was playing bot and I'm pretty sure wouldn't have suspected anything this was just a normal game! Very cool project!
19

---

Published as a conference paper at ICLR 2025
E TRAINING AND INFERENCE
E.1 PRE-TRAINING HYPERPARAMETERS
ALLIE is GPT-2-stylee (Brown et al., 2020) transformer decoder model with 355M parameters. trained on dataset of 6.6 billion tokens. We use global batch size of 131,072 tokens, a learning rate of 10-4, decaying to 10-6 using cosine annealing (Loshchilov & Hutter, 2017), and maximum sequence length of 512 tokens. The model is trained for 2 million steps, which took approximately weeks on 8 NVIDIA A6000 GPUs using 6 precision.
E.2 MCTS IMPLEMENTATION DETAILS
Our MCTS implementation and heerparraeeters follow variant of AlphaZero (Silver al., 2017) proposed by Grill et al. (2020). A way to view MCTS is KL-regularized policy optimization (Grill al., 2020): in the limit, MCTS produces optimized policy that maximizes search Q values with KL regularization towards the model policy po learned from humans:
π arg max Σ (((,a))((s.0 - ADRI. (= po).
This regularization is key prevent the search from diverging from the model policy (Jacob et al., 2022), and the KL-regularization strength À ~ 2//NNim, where e is hyperparameter. In standard MCTS (ALLIE-SEARCH) with fixed number rollouts, Naim is fixed, and À constant, In adaptive MCTS
we scale by the square root of the search budget to achieve the same effect of a constant regularization strength. We refer the interested reader to (Silver et al., 2017; Grill et al., 2020) for more details the MCTS algorithm and its implementation.
20

---

Published as a conference paper at ICLR 2025
F ABLATIONS
To assess the impact of the training, data, and model decisions on ALLIE'S capability to play humanlike chess, we conduct ablation studies with the following scenarios:
Half data: ALLIE trained on 50% of the dataset for the same number of steps.
Half' compute ALLIE trained on the full dataset for 50% of the steps.
Half parameters: smaller ALLIE model (124M) with roughly half the parameters, keeping the training data and steps unchanged.
Double parameters: A larger ALLIE model (774M) with roughly double the parameters, keeping the training data and steps unchanged.
3.5
Selup
67%
System
Alli
Alle
36-
Half Falffcompuut
Move-matching...mmmmm acuuacy
55%
Half compute
Hall data
Half data
34-
Double punametess
15%
Double paramatur
paramelerss
Validation
54%
822
82%
90%
20%
1,200
2,000
Training progress (%)
Game ratling
Figure 12: Left: Validation loss of ALLIE and ablations throughout training. Right: Move-matching accuracy of ALLIE and ablations on the evaluation set.
Training data and compute. We find that the size of the training dataset has a measurable impact on the final loss and move-matching accuracy of the model. Halving the training data leads 1.0% decrease in move-matching accuracy over the entire dataset, with signs of overfitting emerging towards the end of training. Conversely, halving the compute (training tokens) minimally affects the final model performance, likely because the model still undergoes approximately 20 epochs of training over the dataset. These observations suggest that the scaling of our training setup is data-constrained (Muennighoff et al., 2023), making substantial gains challenging without additional data. Notably, our dataset contains an entire year of blitz games on Lichess, representing substantial portion of publicly available internet games, thus creating 10x larger dataset would be difficult.
Model size. Another factor affecting ALLIE'S performance is the model size. Halving the model size moderately impacts performance, resulting 1.2% decrease in move-matching accuracy. Con- versely, doubling the model size yields minimal gains, with only a 0.3% increase in mate-nng accuracy. The diminishing returns on model size suggest that further performance improvements through scaling up the model size may be limited without additional data.
We report individual validation losses of ALLIE and ablations throughout training in Figure 13. We find that the language modeling loss and the time prediction loss are stable across ablations and decrease throughout training. Notably when trained on half the data, the model overfits to the value prediction objective towards the end of training.
"'Oveffitting is only observed in the value prediction objective (Appendix F).
21

---

Published as a conference paper at ICLR 2025
2.2
Setup
Validation language modeling loss
Allie
Half compute
2.0
data
Half parameters
Double parameters
1.8
0%
20%
40%
60%
80%
100%
Training progress (%)
0.84
Setup
Validation value prediction loss
Allie
0.82
Half compute
Half data
Half parameters
0.80
Double parameters
0.74
20%
40%
100%
Training progress (%)
Setup
Validation time prediction loss
Allie
Half compute
Half data
Half parameters
Double parameters
0.72
0.70
0.68
0.66
0%
20%
40%
60%
80%
100%
Training progress (%)
Figure 13: Validation losses of ALLIE and ablations throughout training. Top: language modeling loss. Middle: value prediction loss. Bottom: time prediction loss.
22


Reviews:
[{"review_id": "ARMtTrVkCf", "reviewer": "Reviewer_onXn", "summary": "This paper presents Allie, a chess-playing AI model designed to imitate human behavior across a spectrum of skill levels. Allie is a transformer-based model trained on human games from Lichess that, given a board position and the Elo strength of the two players, predicts the next move the player will play (policy head), the think time, and the game outcome (value head). During inference time, Allie uses these prediction heads to add some MCTS search, where the search rollouts are guided by the (human-like) policy and value heads and the number of rollouts is proportional to the (human-like) think time. The main prior work in this area is the Maia project by McIlroy-Young et al., which also trained on human games to create models at different skill levels (KDD '20) and at the individual level (NeurIPS '21, KDD '22). Allie differs from Maia in three main ways: (1) it is a larger, transformer-based model that represents chess positions as a sequence of move tokens; (2) it models think time and resignation; and (3) it adds MCTS search during inference based on the human-like predictions.\n\nAllie achieves ~4% higher move-matching accuracy on average than Maia (a much smaller model) and ~2% higher accuracy than GPT 3.5 (a much bigger model). More importantly, it exhibits better skill calibration across a wide range of Elo ratings (1000 - 2600), as verified by a human study of 7,483 games which shows that Allie incurs a skill gap of only 49 Elo points on average (provided adaptive search is used). Notably, Allie can play at a grandmaster level (2500 Elo) and still retain human-like qualities.", "questions": "My questions and suggestions are interspersed in the comments above. I summarize them here for convenience:  \n\n1. There is some missing related work. I realize you may not have seen some of these papers in time, but hope they are helpful in revising your paper.  \n2. The size difference between Allie and Maia's architectures (and its connection to increased move-matching accuracy).\n3. The non-human-like aspects of MCTS search. Can you report move-matching accuracy as a function of search time?\n4. How sensitive are your results (e.g., move-matching accuracy) to the constant 'c' in the rollouts function?\n5. The difference between Allie's value head and Stockfish's evaluation. How accurately does Allie's value head predict game outcomes?\n6. Explain why Allie with adaptive search does better than AlphaZero-like MCTS search against 2500 Elo opponents.\n7. Are soft tokens encoded in a way that also suffers from sparsity? \n8. Any normalization/weighting applied to the loss function terms?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "There are several aspects of Allie's methodology and results that I find compelling and believe advance the state of the art.\n\nAllie models human chess playing more holistically by also predicting think time and special decisions such as resignation, which were not modeled in prior chess-playing agents work. Granted, these are just additional metadata recorded with Lichess games that prior methods could also train on, but what I found compelling is the authors' observation that these prediction outputs can be used to parameterize an MCTS search. This time-adaptive search proves to be crucial for closing the skill gap between Allie and the human opponent, as shown in the human study the authors conducted.\n\nAllie achieves good move-matching accuracy across a wider range of Elo ratings (Fig. 2), as well as closer skill calibration to human players (Fig. 5), than prior work. In particular, the Maia models achieve good move-matching accuracy (so they feel human-like) but are not as well calibrated to the Elo of human opponents. Additionally, the fact that Allie can play at a grandmaster level (2500 Elo) is an impressive achievement given that behavior cloning on human data typically yields sub-expert policies.\n\nI like that the authors include GPT 3.5 as a challenging baseline. GPT 3.5's move-matching accuracy is remarkable, given that it wasn't necessarily intended to play chess. Allie's performance is still slightly better, but its main advantage is in enabling search (and requiring much fewer parameters). \n\nAllie's architecture is quite sensibly designed. The use of move tokens and history is a natural way of representing a board position for a transformer model, and avoids the need for a visual representation (as used in AlphaZero, Maia, etc.) -- though it is an interesting question if adding such a visual representation would help, given how visually humans process a chess board. I liked the use of \"soft tokens\" to encode each player's Elo ratings, allowing Allie to condition its output on skill level. I also liked the design of taking the final decoder layer output and parameterizing each prediction using separate linear layers. \n\nThe user study was a nice addition to the paper, as it demonstrated Allie's skill calibration in real human games and also allowed the authors to collect survey results on how enjoyable and human-like Allie (and the other baselines) felt.", "weaknesses": "There a several things that concern me about the paper's presentation and claims, which I hope can be clarified through the rebuttal. \n\nFirst, the discussion of Maia in the introduction and related work seems lacking/misleading to me. I don't think it's fair to say that existing systems are not \"human-aligned\" -- this was precisely the goal of Maia and they did create both skill-level and individual-level (KDD '22) models using the same basic approach (behavior cloning on human data). The individual-level models improve move-matching accuracy over Maia by a similar margin to your improvement over Maia. McIlroy-Young et al. also did some work on style identification (stylometry) that achieved very high accuracy (NeurIPS '21). I suggest citing and mentioning the NeurIPS '21 and KDD '22 papers.\n\nThe fact that the published Maia models do not include think time, resignation, or history does not seem that fundamental to me -- there are archived submissions on OpenReview that extend Maia in some of these ways, e.g., deepening the model and achieving move-matching accuracy similar to Allie's. One of these submissions was recently accepted to NeurIPS '24 and should be discussed in this paper: \"Maia-2: A Unified Model for Human-AI Alignment in Chess\", which presents a unified version of Maia that conditions on the player's Elo rating and achieves better calibration across skill levels. It is worth noting that Allie's architecture, at 355M parameters, is more than an order of magnitude bigger than these Maia models -- you point out size difference as a reason the comparison to GPT 3.5 is unfair, but you do not point out the size difference between Allie and Maia. \n\nInstead of focusing on the above differences, I think a better framing of this paper is to focus on how predicting human-like outputs (including think time) enables you to augment a behavior-cloned model with search, closing the gap between Allie and human players, particularly stronger ones.\n\nIt is important to note that adding MCTS search introduces a non-human-like element to your approach. There is not enough evidence to suggest that humans search the game tree in an MCTS-like manner, and in general our understanding of how humans balance board state evaluation with search (calculation) is quite poor. Thus, I would tone down the claims about only relying on human data, since a search algorithm applied to a game tree implicitly encodes non-human information. One recent piece of work that supports your use of think time to limit MCTS is the paper by Russek et al. called \"Time spent thinking in online chess reflects the value of computation\". It would be helpful if you could report move-matching accuracy as a function of search time; is it the case that when you do more search, move-matching accuracy decreases? Also, how did you choose the constant 'c' in the number of rollouts function? It would be useful to see how sensitive your results (move-matching accuracy, Elo calibration) are to the choice of c.\n\nRelated to the above, it's not clear to me why you would expect Allie's value estimates to match those of Stockfish (Fig. 4). Isn't Allie's value head predicting the outcome a *human* would experience if they continued from that board position? I agree that in many situations this would align with Stockfish's assessment (e.g., if the human is up a lot of material, or if it's close to the end game), but it seems more sensible to me to compare Allie's value estimate to the actual outcome of the game, not what Stockfish thinks the outcome should be.\n\nHow is it that adaptive search does better than non-adaptive (AlphaZero-like MCTS) search against 2500 Elo players? Is the non-adaptive search time bounded, in addition to being the same for each move?\n \nYou observe that using discrete tokens for Elo would be too sparse, but your soft tokens also interpolate between Elos and are represented as tokens. Isn't this representation also sparse, i.e., won't it suffer from a lack of examples associated with each value, or are you representing them differently internally? \n\nThe loss function you use doesn't mention normalization or weights to balance the different loss terms. Is this a potential issue, given that these terms capture very different things (e.g., log-likelihood of next move vs. MSA of think time prediction)?\n\n**Update**: Based on our rebuttal discussion and the additional analysis the authors provided as to why Allie's predictive power of human game outcomes differs from Stockfish's, I am increasing my overall score. I do wish the authors had embraced this difference more intentionally, and reframed their paper around it, but what they provided is a good start.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DSmZDfcvdp", "reviewer": "Reviewer_t49A", "summary": "The paper presents a new human-like chess model that exceeds the current SOTA while adding ponder. The authors look at the model compared to the previous SOTA tests and add some new ones, along with user studies to strongly argue for their model's performance.", "questions": "See the Weaknesses section", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I think this is a good paper presenting an original approach to human-like play modeling and it answered most of my questions in the text.\n\n# originality\n\nThe idea of using a transformer to solve chess predates the current LLM craze (I can find a cite if needed), but this the most successful one so far and shows a real improvement compared to the current slate of LLMs for chess papers. Adding the ponder time is an obvious step from a FEN based dataset, but again is done very well. So while most of the ideas are not novel the combination and execution are.\n\n# quality\n\nThe experiments look to be done well and the paper flows well. Adding the human studies was good move to really show it's an improvement over Maia.\n\n# clarity\n\nI found figure 3 difficult to read, the median is almost impossible to see. I also don't know what the squares in figure 1b mean from the caption, is it supposed to be a chess board?\n\n# significance\n\nChess AI has had a resurgence in the last few years and this could aid that work, it also shows a path for many other turned based games. So I think this has a chance of being significant.", "weaknesses": "I don't think the term _human-aligned_ is good in this paper. An aligned AI system is already implicitly aligned to humans, but the term does not mean that it acts like a human, just that it works in the best interest of humans. I think a different term should be used for this concept to prevent it from being interpreted in the more common AI alignment sense.\n\nI'm not clear how the <resign> token can be used in practice, even with a false positive rate of 0.1%, that would be ~2% over 20 moves, and only takes one bad resign to distribute a game. Was the resign prediction used in testing the models in the user studies? And how does it look in a real gameplay scenario, not just against single game states?\n\nThe lack of code right now is also concerning, my rating is contingent on the authors fulfilling their promises in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wa3wA69hi7", "reviewer": "Reviewer_aMwP", "summary": "This paper focuses on proposing a chess-playing AI (called ALLIE) that is aligned to human gameplay. It does so by training a Transformer-based network on a dataset of human games, taking into account metadata such as the player's skills (Elo score) and their pondering time. By assuming players ponder on critical situations, the authors propose to add a MCTS-based search component proportional to the pondering time. ALLIE's skill level is estimated against numerous human players, and is shown to have an average skill gap of 49 Elo.", "questions": "- ALLIE has been trained exclusively on Blitz games. How does this impact the importance of search compared to standard games? I would expect that Blitz players play mostly instinctively. Blitz might thus be enough to train an agent solely on a human dataset of gameplays, but this might be insufficient for standard games. Do you have any insights as to the types of changes that would be required to scale ALLIE to standard games?\n- Related to the previous question, the authors mention the need to generalize to out-of-distribution positions. How well would ALLIE (trained on Blitz) perform in standard games?\n- Is the Elo score used throughout this paper specific for Blitz? Or is it the player's global Elo score? I would assume the former, but one survey respondent (Table 9, second row) has an Elo of 1940 and mentions he or she \"doesn't really play Blitz so I can't imagine I'm very good\".\n- Since ALLIE is trained on a fixed dataset of different levels of experts, this would be suited to an offline reinforcement learning (RL) setting. More specifically, ALLIE use a GPT-2 like architecture, making it related to Decision Transformers (DT) [1] (execpt of course the search component). Offline RL approaches provide performance and out-of-distribution gains compared to the behavior-cloning-based approach taken by ALLIE. Are there any reasons why the authors chose to behavior cloning?\n\n\n[1] Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., ... & Mordatch, I. (2021). Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34, 15084-15097.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Shows strong and accurate performance across a wide range of Elo scores\n- Strong performance compared to Maia, which is also human aligned\n- Thorough analysis on different aspects of the algorithm (accuracy of move prediction, accuracy of pondering times, accuracy of value estimate)\n- Extensive human study", "weaknesses": "- Entirely based on offline data, based of recording of human gameplays. This limits the scope of this method for human-aligned AI. Moreover, large parts of the data have been removed for training (downsampling low-Elo games, removing transitions under time pressure), so it seems training ALLIE-like agents require a specific training regimen.\n- There does not seem to be any related work on skill calibration. Although I am not an expert in that area, I would be curious about what works have been done in that regard.\n- I find the claim that ALLIE has a \"reliable board value estimate\" a bit misleading. Fig3 (with a Pearson's $r=0.697$) is already quite noisy, and the vast majority of Fig4 is below that number (around 5+ moves from terminal), with 30+ moves from terminal having a weakly correlated $r<0.3$. What would these values be in terms of mean absolute error?\n- GPT3.5 has a comparative move-matching accuracy as ALLIE on higher-Elo games, despite having not been explicitly trained for chess. Although I do not expect it to be included in the human evaluation, it would have been interesting to provide a more in-depth comparison between GPT3.5 and ALLIE. Would a fine-tuned GPT3.5 on the same dataset as ALLIE (but in PGN format) result in a human-aligned agent?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mMe8pXZoP8", "reviewer": "Reviewer_rNsm", "summary": "This paper introduces ALLIE, a chess AI trained exclusively in human chess games with the goal of creating a skill-calibrated and human-aligned chess AI. If the policy is solely employed for matches with humans, the AI struggles to align with human players of corresponding Elo ratings. To address this challenge, ALLIE takes into account the pondering time of each move recorded in human games and proposes a time-adaptive Monte Carlo tree search to mitigate the skill gap, which outperforms search-free and standard MCTS baselines. Furthermore, ALLIE also considers human resignation behavior; analyses show that ALLIE achieves an 86.4% true positive rate, indicating that it demonstrates more human-like behavior.", "questions": "1. Please address the issues mentioned in the Weakness, especially major comments.\n2. Why do transformer-based models perform better than MAIA? Is this due to the architecture itself or the parameter size? Does the transformer-based model in ALLIE have a longer inference time than MAIA*?\n3. Can this method be applied to other games or domains? Any further experiments on other games?\n4. Do you enable resignation for ALLIE as mentioned in Section 5.2 for the LARGE-SCALE HUMAN STUDY?\n5. Do ALLIE-SEARCH and ALLIE-ADAPTIVE-SEARCH use the softmax function for final move decisions, as ALLIE-policy does? If not, the experiment may be unfair.\n6. Is the bin range in Figure 5 set to 200? The datasets are separated every 100 Elo ratings. Why does the bin range differ?\n7. In section 4.3, it claims \"Due to dependency on the textual PGN notation, this approach\nis not compatible with OpenAI’s latest chat-based LLMs (e.g., GPT-4)\". However, the explanation of why we cannot use GPT-4 for evaluation is unclear.\n8. Could you incorporate the remaining time as input to the model? Maybe it might make the agent more human-like.\n9. How to determine the constant c in the time-adaptive MCTS procedure? It seems it is not explicitly stated in the main text.\n10. For the resignation conditions, you consider both the resignation token and values. What if it only thinks of resign tokens? Would the result be better or worse than TPR of 86.4%?\n11. In Maia’s paper, they state that MCTS tends to degrade move prediction performance in their setting. Why does MCTS improve move accuracy, especially in expert-level human games in this work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work proposes some novel ideas for developing human-like agents by considering not only move accuracy but also non-move behaviors of human players, such as pondering time and when to resign. It helps bridge the gap in developing human-aligned AIs, as strong AI agents may exhibit non-interpretable behaviors. Additionally, it compares favorably with state-of-the-art agents, MAIA and GPT-3.5, achieving better results.", "weaknesses": "## Major Comments:\nFirst, the novelty of this paper is very limited. Specifically, it directly applies the transformer architecture to chess games. The only novel thing is that the approach introduces some new outputs, such as predicting pondering time or resignation. Moreover, for the network architecture, the reason why the transformer architecture is better than MAIA, which uses residual networks, is not explicitly explained. For example, if the parameter size is much larger than MAIA, the inference time might be also much longer, and the fairness of the experiments should be reconsidered. It is better to compare the inference time and the number of parameters under the same hardware.\n\nSecond, the evaluation set excludes moves made under time pressure (when the remaining time is less than 30 seconds) to avoid situations where players tend to make random moves. However, these blitz moves under time pressure might reflect human intuitions. Although MAIA’s evaluation also excludes these moves, but since this paper aims to create human-like AIs, it is recommended to include these moves in the evaluation, as they might yield interesting and more convincing results. Furthermore, while non-move behaviors consider the pondering time for each move, human players often decide the thinking time for the next move according to the remaining time. How about incorporating the remaining time as input to models? By doing this, the method might not need to exclude moves made under time pressure in the evaluation set.\n\nThird, for the generality of the proposed method, the experiment in this paper is only applied to chess, it is not sure whether this method can be generalized to other games. The author should include other games to demonstrate the effectiveness of their approach.\n\n## Minor Comments:\n- In abstract, “... resignations In offline evaluations,” -> “... resignations in offline evaluations,” \n- The time spent for each move should be recorded in the game history, but not explicitly stated in the main text, please address it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nvRx1tC59Y", "reviewer": "Reviewer_wEKh", "summary": "This paper presents ALLIE, a skill-calibrated and human-aligned chess AI trained purely from human games. ALLIE uses the transformer architecture for sequence modelling to predict the distribution of the next move, the pondering time humans spend at the current position, and the winner of the game. Through a test against a database of human games and an online user study, the authors find ALLIE better at predicting human moves than its counterparts. They also propose an adaptive Monte Carlo Tree Search (MCTS) mechanism at inference time, leveraging the predicted pondering time to model human behaviours better and calibrate the skill.", "questions": "- I wonder how the hyperparameter $c$ for deciding $N_{sim}$ was selected. Was it determined via trial and error?\n- Could you explain why the move-matching accuracy is a good metric for human alignment evaluation? For instance, I can imagine that amateur players can make blunder moves in many different ways given the same board position. Thus, ALLIE may get a low move-matching accuracy even with strength conditioning, while the human player feels their opponent is human-like. Continuing with this point, I believe chess beginners are less likely to tell if their opponent is a human simply because they see fewer games and don't have the same level of experience. Therefore, I believe it would be interesting to report the human alignment survey grouped by Elo ratings.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The objectives of the paper are clearly defined, and the paper sticks close to these objectives.\n- The writing and presentation of the paper are clear and of good quality. I do not spot any visible typos in the main text.\n- Though I am not entirely sure, I believe modelling the pondering time and resignation and utilizing the time prediction for adaptive MCTS are novel ideas in this line of work. In addition, the invention of soft tokens for strength conditioning is a fresh idea.\n- The presented evaluation methods and metrics seem rigorous. Furthermore, this work complements the offline human alignment evaluation with surveys consisting of quantitative and qualitative feedback from online users.", "weaknesses": "- The design of ALLIE is akin to AlphaGo [1] without the self-play phase. Therefore, I do not view training the neural networks entirely on human data and applying MCTS for policy improvement as an algorithmic advancement.\n- The authors stress the necessity of adaptive search in skill calibration. However, failure to match the grandmaster level can be interpreted merely as a lack of strength, and incorporating a planning phase improves ALLIE's strength to close that gap. The authors do not explain why adaptive search attains a better skill calibration error even against mid-level opponents.\n- As the authors point out, the users knew they were playing against an AI before they filled out the survey. Therefore, the human alignment result from the user study might not be as objective as if they conducted a Turing test.\n- It would better support the superiority of adaptive search in skill calibration if ALLIE with adaptive search were compared against an AlphaZero-like agent, where the skill level is adjusted using a temperature parameter.\n\n[1] Silver, D., Huang, A., Maddison, C. et al. Mastering the game of Go with deep neural networks and tree search. Nature 529, 484–489 (2016). https://doi.org/10.1038/nature16961", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)