Amortized Planning with Large-Scale Transformers: A Case Study on Chess

Authors: anian  ruoss
grégoire  delétang
sourabh  medapati
jordi  grau-moya
kevin  li
  wenliang
elliot  catt
john  reid
cannada a lewis
joel  veness
tim  genewein
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 11:53:33

Abstract: This paper uses chess, a landmark planning problem in AI, to assess transformers’ performance on a planning task where memorization is futile — even at a large scale. To this end, we release ChessBench, a large-scale benchmark dataset of 10 million chess games with legal move and value annotations (15 billion data points)  provided by Stockfish 16, the state-of-the-art chess engine. We train transformers with up to 270 million parameters on ChessBench via supervised learning and perform extensive ablations to assess the impact of dataset size, model size, architecture type, and different prediction targets (state-values, action-values, and behavioral cloning). Our largest models learn to predict action-values for novel boards quite accurately, implying highly non-trivial generalization. Despite performing no explicit search, our resulting chess policy solves challenging chess puzzles and achieves a surprisingly strong Lichess blitz Elo of 2895 against humans (grandmaster level). We also compare to Leela Chess Zero and AlphaZero (trained without supervision via self-play) with and without search. We show that, although a remarkably good approximation of Stockfish’s search-based algorithm can be distilled into large-scale transformers via supervised learning, perfect distillation is still beyond reach, thus making ChessBench well-suited for future research.

Main paper: 
 introduction: 
 The ability to plan ahead and reason about long-term consequences of actions is a hallmark of rationality and human intelligence. Its replication in artificial systems has been a central goal of AI research since the field's inception. Perhaps the historically most famous planning problem in AI is chess, where naive search is computationally intractable and brute-force memorization is futile -even at scale. Conceptually, the ability to plan and reason about consequences of actions is implemented via search algorithms and value computation [1, 2] . To implement such algorithms at scale, feed-forward neural networks have been playing an increasingly important role. For instance, DQN [3] and AlphaGo [4] popularized the use of neural value estimators to scale RL and (Monte Carlo) tree search. Similarly, today's strongest chess engines also employ a combination of search and amortized neural value estimation. For example, the state-of-the-art Leela Chess Zero [5, 6] , which builds upon AlphaZero [7] , augments Monte Carlo tree search with neural value predictions. Likewise, Stockfish 16, currently the strongest chess engine, uses efficiently updatable neural network evaluation [8] , a highly specialized neural architecture to obtain fast evaluations. It is trained on value estimates of an earlier Stockfish version that uses human chess heuristics. All these chess engines tweak value-estimation networks towards fast evaluations that are to be combined with search We extract all boards from N randomly drawn games from Lichess, discard duplicate board states, and compute the state-value for each board as the win-probability via Stockfish. We compute action-values and the best action for all legal moves of a board state in the same way. Bottom left (Dataset creation): We construct training and test sets of various sizes (see Table A1 ). Our largest training set has 15.3B action-values. Drawing games i.i.d. from the game database for our test set leads to 14.7% of test boards appearing in the largest training set (mostly very early game states). We also use a test set of 10K chess puzzles that come with a correct sequence of moves. Bottom right (Policies): We train predictors on three targets (state-or action-values, or oracle actions), each of which can be used for a chess policy. Our value predictors are discrete discriminators (classifiers) that predict into which bin z i ∈ {z 1 , . . . , z K } the oracle value falls. over many possible future move sequences. As a result, the value estimates themselves may not necessarily be optimal, which leaves open the question of how far amortized planning with modern neural networks can be pushed with scale and whether the resulting "searchless" engines can match engines that use search at test time, at least in principle (putting aside computational efficiency).To scientifically address this question, we create ChessBench (https://github.com/ google-deepmind/searchless_chess), a large-scale chess dataset created from 10 million human games that we annotate with Stockfish 16. We use ChessBench to train transformers of up to 270 million parameters via supervised learning to predict action-values given a board-state. We also construct searchless chess policies from these predictors, where the playing strength depends entirely on the quality of the value predictions. We find that predictions by our largest trained models generalize well and non-trivially to novel board states. The resulting policies are capable of solving challenging chess puzzles and playing chess at a high level (grandmaster) against humans. Due to the combinatorial explosion of chess board states, virtually every new game involves many board states that were never seen during training (see Figure 1 ). This rules out memorization as a possible explanation and suggests that an approximate version of Stockfish's (search-based) value estimation algorithm can indeed be distilled into large transformers. Nonetheless, we also find that the performance gap cannot be fully closed, which may indicate that architectural innovations (or better optimization, data augmentation, etc.) might be needed instead of even larger scale.Our main contributions are:  
methodology: 
 We now describe the dataset creation, the neural predictors and how to construct policies from them, and our evaluation methodology (see Figure 1 for an overview; full details in Appendix A).To construct a training dataset for supervised learning we downloaded 10 million games from Lichess (lichess.org) from February 2023 1 . We extract all board states s from these games and estimate the state-value V SF (s) for each state with Stockfish 16 (our "value oracle") using a time limit of 50ms per board (unbounded depth and maximum skill level). The value of a state is the win percentage estimated by Stockfish, lying between 0% and 100%.2 Note that Stockfish does not provide a centipawn score when it detects a mate-in-k, so we map all of these cases to a win percentage of 100%. We also use Stockfish to estimate action-values Q SF (s, a) for all legal actions a ∈ A legal (s) in each state. Here we use a time limit of 50ms per state-action pair (unbounded depth and max skill level), which corresponds to a Lichess blitz Elo of 2713 for our action-value oracle (see Section 3.1).The action-values (win percentages) also determine the oracle best action a SF :a SF (s) = arg max a∈Alegal(s) Q SF (s, a).In the case where multiple moves are tied in value, we pick a maximizing action arbitrarily. Since we train on individual boards and not whole games we shuffle the dataset after annotation. For our largest training dataset, based on 10M games, we thus obtain 15.3B action-value estimates (or ≈ 530M state-value estimates and oracle best-actions; cf. Table A1 ) to train on (corresponding to roughly 8864 days of unparallelized Stockfish 16 evaluation time given the limit of 50ms per move).To create our test dataset we follow the same annotation procedure, but on 1K games downloaded from a different month (March 2023), resulting in ≈ 1.8M action-value estimates (or ≈ 62K statevalue estimates and oracle best actions). Since there is only a small number of early-game board states and players often play popular openings, 14.7% of boards in this i.i.d. test are also in the training set. We deliberately chose not remove them, to not introduce a distributional shift and skew test set metrics. We also create a puzzle test set, following Carlini [9] , consisting of 10K challenging board states that have an Elo rating and a sequence of moves to solve the puzzle. Only 1.33% of the puzzle boards appear in the training set (i.e., the initial board states, not the complete solution sequences).Value binning The predictors we train are discrete discriminators (i.e., classifiers). Therefore we convert the win percentages (i.e., the ground truth state-or action-values) into discrete "classes" via binning, akin to distributional RL [10] . We divide the interval from 0% to 100% uniformly into K bins (non-overlapping sub-intervals) and assign a one-hot code to each bin z i ∈ {z 1 , . . . , z K }. If not mentioned otherwise, K = 128. For our behavioral cloning experiments we train to directly predict the oracle actions, which are already discrete. We ablate the number of bins K in Appendix B.1, and we investigate losses that do (i.e., the log loss) and do not (i.e., the HL-Gauss and L2 loss) maintain semantic interconnectedness of the labels (i.e., the dependence between the classes) in Table 2 .Tokenization A FEN [11] string is a standard string-based description of a chess position. It consists of a board state, the current side to move, the castling availability for both players, a potential en passant target, a half-move clock and a full-move counter, all represented in a single ASCII string. Our tokenization uses this representation, except that we flatten any uses of run-length encoding to obtain a fixed length (77) tokenized representation. Actions are stored in UCI notation [12] , for example 'e2e4' for the popular opening move for white. To tokenize we use the index of a move into a lexiographically sorted array of the 1968 possible UCI encoded actions (see Appendix A.1 for more details). Note that this representation technically makes the game non-Markovian, because FENs do not contain the move history and therefore, cannot capture all information for rules such as drawing by threefold repetition (drawing because the same board occurs three times).Network inputs and outputs For all our predictors we use a modern decoder-only transformer backbone [13] [14] [15] to parameterize a categorical distribution by normalizing the transformer's outputs with a log softmax layer. The models thus output log probabilities. The context size is 79 for actionvalue prediction and 78 for state-value prediction and behavioral cloning (see 'Tokenization' above). The output size is K (i.e., the number of bins) for action-and state-value prediction and 1968 (the number of all possible legal actions) for behavioral cloning. We use learned positional encodings [16] as the length of the input sequences is constant. Training protocol We train our predictors by minimizing the cross-entropy loss via mini-batch stochastic gradient descent using Adam [17] . The target labels are either bin-indices for state-or action-value prediction or action indices for behavioral cloning. For state-and action-value prediction, we additionally apply label smoothing via the HL-Gauss loss [18] , using a Gaussian smoothing distribution with the mean given by the label and a standard deviation of σ = 0.75/K ≈ 0.05 as recommended by Farebrother et al. [19] . Thus, the labels are no longer one-hot but multi-hot encoded (see Figure 3 in [19] for an overview). We ablate the loss function, i.e., HL-Gauss vs. cross-entropy vs. MSE, in Section 3.4. We train for 10 million steps, which corresponds to 2.67 epochs for a batch size of 4096 with 15.3B data points (cf. Table A1 ; details in Appendices A.2 and A.3).Our predictors are categorical distributions parameterized by neural networks P θ (z|x) that take a tokenized input x and output a predictive distribution over discrete labels {z 1 , . . . , z K }. Depending on the prediction target we distinguish between three tasks (see Figure 1 for an overview).(AV) Action-value prediction The target label is the bin z i into which the ground-truth action-value estimate Q SF (s, a) falls. The input to the predictor is the concatenation of tokenized state and action.The loss for a single input data point (s i , a i ) is:EQUATIONwhere K is the number of bins and HL-Gauss K (x) is the function that computes q i (the smoothed version of label z i ), i.e., a smooth categorical distribution using HL-Gauss [18] rather than the standard one-hot procedure. To use the predictor in a policy, we evaluate the predictor for all legal actions in the current state and pick the action with maximal expected action-value:âAV (s) = arg max a∈Alegal E Z∼P AV θ (•|s,a) [Z]Qθ (s,a).(SV) State-value prediction The target label is the bin z i that the ground-truth state-value V SF (s) falls into. The input to the predictor is the tokenized state. The loss for a single data point is:EQUATIONTo use the state-value predictor as a policy, we evaluate the predictor for all states s ′ = T (s, a) that are reachable via legal actions from the current state (where T (s, a) is the deterministic transition of taking action a in state s). Since s ′ implies that it is now the opponent's turn, the policy picks the action that leads to the state with the worst expected value for the opponent:âSV (s) = arg min a∈Alegal E Z∼P SV θ (•|s ′ ) [Z] Vθ (s ′ ).(BC) Behavioral cloning The target label is the (one-hot) action-index of the ground-truth action a SF (s) within the set of all possible actions (see 'Tokenization' in Section 2.2). The input to the predictor is the tokenized state, which leads to the loss for a single data point:EQUATIONThis gives a policy that picks the highest-probability action: âBC (s) = arg max a∈Alegal P BC θ (a|s).We use the following metrics to evaluate our models and/or measure training progress.The test set percentage where the policy picks the best action: â(s) = a SF (s).Action ranking (Kendall's τ ) The average test set rank correlation [20] of the predicted action distribution with the ground truth given by Stockfish. Kendall's τ ranges from -1 (exact inverse order) to 1 (exact same order), with 0 indicating no rank correlation. The ranking is obtained by evaluating for all legal actions Qθ (s, a), -Vθ (T (s, a)), or P BC θ (a|s), in the case of AV, SV, or BC prediction, respectively. The ground-truth ranking is given by evaluating Q SF (s, a) for all legal actions.Puzzle accuracy We evaluate our policies on their capability of solving puzzles from a collection of Lichess puzzles that are rated by Elo difficulty from 399 to 2867 based on how often each puzzle has been solved correctly. We use puzzle accuracy as the percentage of puzzles where the policy's action sequence exactly matches the entire solution action sequence. For our main results in Sections 3.1 and 3.2 we use 10K puzzles; otherwise, we use the first 1K puzzles to speed up evaluation.We evaluate the playing strength (measured as an Elo rating) of the policies in two ways: (i) we run an internal tournament between all the agents from Table 1 except for GPT-3.5-turbo-instruct, and (ii) we play Blitz games on Lichess against either only humans or only bots. For the tournament we play 400 games per agent pair, yielding 22K games in total, and compute the Elo with BayesElo [21] using the default confidence parameter of 0.5. To ensure variability in the games, we use the openings from the Encyclopaedia of Chess Openings (ECO) [22] . When playing on Lichess, we use a softmax policy with a low temperature of 0.005 for the first five full-moves instead of the arg max policy to create variety in games and prevent simple exploits via repeated play. We anchor the relative BayesElo values to the Lichess Elo vs. bots of our largest (270M) model.We compare the performance of our models against the following engines: Stockfish 16 We consider two variants: (i) a 50ms time limit per legal move (i.e., the dataset oracle), and (ii) a 1.5s limit per board, which is roughly the amount of time the first variant (i) takes on average per board (i.e., there are roughly 30 legal moves per board on average).AlphaZero We consider three variants of AlphaZero [7] : (i) with 400 MCTS simulations, (ii) only the policy network, and (iii) only the value network (where (ii) and (iii) perform no additional search). AlphaZero's networks have 27.6M parameters and are trained on 44M games (full details in [23] ).Leela Chess Zero We consider three variants: (i) with 400 MCTS simulations, (ii) only the policy network, and (iii) only the value network (where (ii) and (iii) perform no additional search). We use the T82 network, which is a convolutional network with 768 filters, 15 blocks, and mish activations [24] -the largest network available on the official Leela Chess Zero website. At the time of writing, the precise network architecture, training data, and training protocol for the T82 network were not reported on the Leela Chess Zero website or other source. Concurrently to our paper, Monroe and Leela Chess Zero Team [6] published a rigorous tech report which now provides many of these details about for a slightly different (state-of-the-art) Leela Zero architecture called ChessFormer, and includes a comparison against our vanilla networks.Table 1 : Comparison of our action-value models against Stockfish 16, variants of Leela Chess Zero and AlphaZero (with and without Monte Carlo tree search), and GPT-3.5-turbo-instruct. Tournament Elo ratings are obtained by making the agents play against each other and cannot be directly compared to the Lichess Elo. Lichess (blitz) Elo ratings result from playing against either human opponents or bots on Lichess. Stockfish 16 with a time limit of 50ms per move is our data-generating oracle. Models operating on the PGN observe the full move history, whereas FENs only contain very limited historical information (sufficient for the fifty-move rule). Unlike all other engines, our policies were trained with supervised learning and use no explicit search at test time (except for GPT-3.5-turbo-instruct, which was trained via self-supervised learning and then instruction tuned). GPT-3.5-turbo-instruct We follow Carlini [9] and encode the entire game with the Portable Game Notation (PGN) [11] to reduce hallucinations. Since Carlini [9] found that (at that time) GPT-4 struggled to play full games without making illegal moves, we do not consider GPT-4.In contrast to our models, all of the above engines rely on the entire game's history (via the PGN; we use the FEN, which only contains the game's current state and very limited historical information).Observing the full game's history helps, for instance, detecting and preventing draws from threefold repetition (games are drawn if the same board state appears three times throughout the game). Our engines require a workaround to deal with this problem (described in Section 4).A direct comparison between all engines comes with a lot of caveats since some engines use the game history, some have very different training protocols (i.e., RL via self-play instead of supervised learning), and some use search at test time. We show these comparisons to situate the performance of our models within the wider landscape, but emphasize that some conclusions can only be drawn within our family of models and the corresponding ablations that keep all other factors fixed. 
case-study results: 
 We use two settings for our experiments: (i) a large-scale setting for our main results (Sections 3.1 and 3.2; details in Appendix A.2), and (ii) a setting geared towards getting representative results with better computational efficiency for our ablations (Sections 3.3 and 3.4; details in Appendix A.3).Table 1 shows the playing strength (internal tournament Elo, external Lichess Elo, and puzzle accuracy) of our large-scale transformers trained on the full (10M games) training set. We compare models with 9M, 136M, and 270M parameters (none of them overfit the training set as shown in Appendix B.4). The results show that all three models exhibit non-trivial generalization to novel boards and can successfully solve a large fraction of puzzles. Across all metrics, increasing model size consistently improves the scores, confirming that model scale matters for strong chess performance.Our largest model achieves a blitz Elo of 2895 against human players, which places it into grandmaster territory. However, the Elo drops when playing against bots on Lichess, which may be a result of having a significantly different player pool [25] , minor technical issues, or a qualitative difference in how bots exploit weaknesses compared to humans (see Section 4 for a detailed discussion). Step 1e6 100'000 Games0 1 2 3 4 5Step 1e6 1'000'000 Games Figure 3 : Puzzle accuracy for different training set sizes (stated above panels) and model sizes (color-coded), evaluated on our small set of 1K puzzles. Generally, larger models trained on larger datasets lead to higher accuracy (which strongly correlates with test set performance and general chess playing strength), highlighting the importance of scale for strong chess play. This effect cannot be explained by memorization since < 1.41% of the initial puzzle board states appear in our training set.If the model is too large in relation to the training set it overfits (left panel; loss curves in Figure A3 ).In Figure 2 we compare the puzzle performance of our 270M parameter model against Stockfish 16 (50ms limit per move), GPT-3.5-turbo-instruct, AlphaZero, and Leela Chess Zero. We use our large puzzle set of 10K puzzles, grouped by their assigned Elo difficulty from Lichess. Stockfish 16 performs the best across all difficulty categories, followed by Leela Chess Zero, AlphaZero, and our model. Impressively, our model, which does not use any explicit search at test time, nearly matches the performance of AlphaZero with (Monte Carlo) tree search. GPT-3.5-turbo-instruct achieves non-trivial puzzle performance but significantly lags behind our model. We emphasize that solving the puzzles requires a correct move sequence, and since our policy cannot explicitly plan ahead, solving the puzzle sequences relies entirely on having good value estimates that can be used greedily.Figure 3 shows a scaling analysis over the dataset and model size. We visualize the puzzle accuracy (train and test losses in Figure A3 ), which correlates well with the other metrics and the overall playing strength. For small training set size (10K games, left panel) larger architectures (≥ 7M) start to overfit. This effect disappears as the dataset size is increased to 100K (middle panel) and 1M games (right panel). The results also show that the final accuracy of a model increases as the dataset size is increased (consistently across model sizes). Similarly, we observe the general trend of increased architecture size leading to increased overall performance (as in our main result in Section 3.1).We perform extensive ablations using the 9M-parameter model and show the results in Tables 2 and A2 . We use the results and conclusions drawn to inform and justify our design choices and determine the default model-, data-, and training configurations. Predictor targets By default we learn to predict action-values given a board state.Here we compare against using state-values or oracle actions (behavioral cloning) as the prediction targets (Section 2.3 and Figure 1 describe how to construct policies from the predictors). Table 2 and Figure A4 show that the action-value predictor is superior in terms of action-ranking (Kendall's τ ), action accuracy, and puzzle accuracy. This superior performance might stem primarily from the significantly larger actionvalue dataset (15.3B state-action pairs vs. ≈ 530M states for our largest training set).We thus run an ablation where we train all three predictors on the same amount of data (results in Table A4 and Figure A5 , which largely confirm this hypothesis). Appendix B.5 contains detailed discussion, including why the performance discrepancy between behavioral cloning and the statevalue prediction policy may be largely due to training only on expert actions rather than the full action distribution.We treat learning Stockfish action-values as a classification problem and train by minimizing the HL-Gauss loss [18] . Here we compare this to the cross-entropy loss (log-loss), which is as close as possible to the (tried and tested) standard LLM setup. Another alternative is to treat the problem as a scalar regression problem by parameterizing a fixed-variance Gaussian likelihood model with a transformer and performing maximum (log) likelihood estimation, i.e., minimizing the mean-squared error (L2 loss). To that end, we modify the architecture to output a scalar (without a final log-layer or similar). Table 2 shows that the HL-Gauss loss outperforms the other two losses.Network depth Table 2 shows the influence of increasing the transformer's depth (i.e., the number of layers) for a fixed number of parameters (we vary the embedding dimension and the widening factor such that all models have the same number of parameters). Since transformers may learn to roll out iterative computation (which arises in search) across layers, deeper networks may hold the potential for deeper unrolls. The performance of our models increases with their depth but saturates at around 16 layers, indicating that depth is important, but not beyond a certain point.Value binning Table 2 shows the impact of varying the number of bins used for state-and actionvalue discretization (from 16 to 256), demonstrating that more bins generally lead to improved performance (up to a certain point). We therefore use K = 128 bins for our experiments. 
discussion: 
 When using our state-based policies to play against humans and bots, two minor technical issues appear that can only be solved by having (some) access to the game's history. In this section, we discuss both issues and present our corresponding workarounds.Blindness to threefold repetition By construction, our state-based predictor cannot detect the risk of threefold repetition (drawing because the same board occurs three times), since it has no access to the game's history (FENs contain minimal historical info, sufficient for the fifty-move rule). To reduce draws from threefold repetitions, we check if the agent's next move would trigger the rule and set the corresponding action's win percentage to 50% before computing the softmax. However, our agents still cannot plan ahead to mimize the risk of being forced into threefold repetition. Figure 4 : Two options to win the game in 3 or 5 moves, respectively (more options exist). Since they both map into the highest-value bin our bot ignores Nh6+, the fastest way to win (in 3), and instead plays Nd6+ (matein-5). Unfortunately, a state-based predictor without explicit search cannot guarantee that it will continue playing the Nd6+ strategy and thus might randomly alternate between different strategies. Overall this increases the risk of drawing the game or losing due to a subsequent (low-probability) mistake, such as a bad softmax sample. Board from a game between our 9M Transformer (white) and a human (blitz Elo of 2145).Indecisiveness in the face of overwhelming victory If Stockfish detects a matein-k it outputs k and not a centipawn score. When annotating our dataset, we map all such outputs to a win percentage of 100%.Similarly, in a very strong position, several actions may end up in the maximum value bin. Thus our agent sometimes plays somewhat randomly rather than committing to a plan that finishes the game (the agent has no knowledge of its past moves).Paradoxically, this means that our agent, despite being in a position of overwhelming win percentage, sometimes fails to take the (virtually) guaranteed win (see Figure 4 ) and might draw or even end up losing since small chances of a mistake accumulate with longer games. To alleviate this, we check whether the predicted scores for all top five moves lie above a win percentage of 99% and double-check this condition with Stockfish, and if so, use Stockfish's top move (out of these) to have consistency in strategy across time-steps.Elo: Humans vs. bots We have three plausible hypotheses for why the Lichess Elo in Table 1 differs when playing against humans vs. bots: (i) humans tend to resign when our bot is in an overwhelmingly winning position but bots do not (i.e., the previously described problem gets amplified against bots);(ii) most humans on Lichess rarely play against bots, i.e., the two player pools (humans and bots) are hard to compare and their Elo ratings may be miscalibrated [25] ; and (iii) based on anecdotal analysis by a chess National Master, our models make the occasional tactical mistake which may be penalized more severely by bots than humans (analysis in Appendices B.7 and B.8). While investigating this Elo discrepancy is interesting, it is not central to our paper and does not impact our main claims.Our primary goal was to investigate whether a complex search algorithm such as Stockfish 16 can be approximated with a feedforward neural network on our dataset via supervised learning. While our largest model achieves good performance, it does not fully close the gap to Stockfish 16, and it is unclear whether further scaling would close this gap or whether other innovations are needed.While we produced a strong chess policy, our goal was not to build a state-of-the-art chess engine. Our models are impractical in terms of speed and would perform poorly in computer chess tournaments with compute limitations. Therefore, we calibrated our policy's playing strength via Lichess, where the claim of "grandmaster-level" play currently holds only against human opponents. In addition, we only evaluated our biggest model against humans on Lichess due to the extensive amount of time required. We also cannot rule out that opponents, through extensive repeated play, may find weaknesses due to the deterministic nature of our policy. Finally, we reiterate that we compare to other engines to situate our models within the wider landscape, but that a direct comparison between all the engines comes with a lot of caveats due to differences in their inputs (FENs vs. PGNs), training protocols (RL vs. supervised learning), and the use of search at test time.Leela Chess Zero's networks, which are trained with self-play and RL, achieve higher Elo ratings without using explicit search at test time than our transformers, which we trained via supervised learning. However, in contrast to our work, very strong chess performance (at low computational cost) is the explicit goal of this open source project (which they have clearly achieved via domain-specific adaptations). We refer interested readers to [6] (which was published concurrently to our work) for details on the current state-of-the-art and a comparison against our networks. 
related work: 
 Unsurprisingly given its long history in AI, there is a vast literature on applying algorithmic techniques to chess. Earlier works predominantly focused on methods which would increase the strength of chess playing entities, but as time has progressed, the role of chess as a problem domain has changed more from a challenge area to that of an illuminating benchmark, and our work continues this tradition.Early computer chess research focused on designing explicit search strategies coupled with heuristics, as evidenced by Turing's initial explorations [26] and implementations like NeuroChess [27] . This culminated in systems like Deep Blue [28] and early versions of Stockfish [29] .AlphaZero [7] and Leela Chess Zero [5] marked a paradigm shift: They employed deep RL with Monte Carlo tree search to learn heuristics (policy and value networks) instead of manually designing them [30, 31] . Several works built upon this framework [32] , including enhancements to AlphaZero's self-play mechanisms [33] and the use of model-free RL [23] . At the time of writing, Leela Zero's T82 networks were state-of-the art, but their training protocol, training data, and some architectural details were not fully reported. In the meantime, Monroe and Leela Chess Zero Team [6] have published a detailed tech report on another transformer-based Leela Zero architecture called the ChessFormer.Their comparison shows that ChessFormers comparable in size to our models outperform our vanilla transformers while requiring fewer FLOPS thanks to clever domain-specific adaptations.Another line of work moved away from explicit search methods by leveraging large-scale game datasets for (un)supervised learning, both for chess [34] [35] [36] [37] and Go [38] . Most closely related to our work, project Maia [35] used behavioral cloning on 12M human games but, rather than maximizing performance (i.e., distilling Stockfish), focused on predicting human moves at different levels of play.The rise of large (pretrained) foundation models also led to innovations in various areas of computer chess research: learning the rules of chess [39, 40] , evaluating move quality [41] , evaluating playing strength [9, 42] , state tracking [43, 44] , and playing chess from visual inputs [45] . Fine-tuning on chess-specific data sources (e.g., chess textbooks) has further improved performance [46] [47] [48] . 
conclusion: 
 Our paper introduces ChessBench, a large-scale, open source benchmark dataset for chess, and shows the feasibility of distilling an approximation of Stockfish 16, a complex planning algorithm, into a feed-forward transformer via standard supervised training. The resulting predictor generalizes well to unseen board states, and, when used in a policy, leads to strong chess play. We show that strong chess capabilities from supervised learning only emerge at sufficient dataset and model scale. Our work thus adds to a rapidly growing body of literature showing that sophisticated algorithms can be distilled into feed-forward transformers, implying a paradigm shift to viewing large transformers as a powerful technique for general algorithm approximation rather than "mere" statistical pattern recognizers. Nevertheless, perfect distillation of Stockfish 16 is still beyond reach and closing the performance gap might need other (e.g., architectural) innovations. Our open source benchmark dataset, ChessBench, thus provides a solid basis for scientific comparison of any such developments. 
impact statement: 
 While the results of training transformer-based architectures at scale in a (self-)supervised way will have significant societal consequences in the near future, these concerns do not apply to closed domains, such as chess, that have limited real-world impact. Moreover, chess has been a domain of machine superiority for decades. Another advantage of supervised training on a single task over other forms of training (particularly self-play or reinforcement learning and meta-learning) is that the method requires a strong oracle solution to begin with (for data annotation) and is unlikely to significantly outperform the oracle, which means that the potential for the method to rapidly introduce substantial unknown capabilities (with wide societal impacts) is very limited. A1 ). We use 50 buckets to generate the histograms. The win percentage distribution is skewed towards 0 as we consider all legal moves per board and most actions are not advantageous for the player.played the following numbers of games for the different policies shown in  
 
 Appendix: We use Stockfish 16 (the version from December 2023) throughout the paper. When we play, we use the oracle we used for training, which is an unconventional way to play with this engine: We evaluate each legal move in the position for 50ms, and return the best move based on these scores. This is not entirely equivalent to a standard thinking time of 50ms multiplied by the number of legal moves per position, as we force Stockfish to spend 50ms on moves that could be uninteresting and unexplored. We chose to keep this setup to have a comparison to the oracle we train on. Note that, when comparing the legal moves in a given position, we do not clear Stockfish's cache between the moves. Therefore, due to the way the cache works, this biases the accuracy of Stockfish's evaluation to be weaker for the first moves considered. Finally, due to the nature of our internal hardware setup, we use two different kinds of chips to run Stockfish: (i) to compute the Lichess Elo, we use a 6-core Intel(R) Xeon(R) W-2135 CPU @ 3.70GHz, and (ii) to compute the tournament Elo, we use a single Tensor Processing Unit (V3), as for all the other agents.We use the AlphaZero [7] version from 2020, with a network trained at that time [23] . We investigate three different versions: (i) policy network only, (ii) value network only and (iii) standard version with search. For (i) we do a search limited to 1 node. For (ii) we do a search limited to 1 node for every legal action and then take the arg max. For (iii), we use the standard search from the paper, with 400 MCTS simulations (confirmed via personal correspondence with Schrittwieser et al. [23] ) and the exact same UCB scaling parameters. We also take the arg max over visit counts. Note that AlphaZero's policy and value network have been trained on 44M games, whereas we trained our largest models on only 10M games.We use the latest release version (v31.0) of Leela Chess Zero [5] with the largest network officially supported at the time of submission (i.e., the 'Large' T82 network from https://lczero.org/ play/networks/bestnets). We investigate three different versions: (i) policy network only, (ii) value network only and (iii) standard version with search. For (i) we do a search limited to 1 node. For (ii) we do a search limited to 1 node for every legal action and then take the arg max. For (iii), we use 400 MCTS simulations. The network architecture and training protocol for T82 were not fully available; more details are now provided in Monroe and Leela Chess Zero Team [6] (written concurrently to our work) about a slightly different architecture, called the ChessFormer.Our codebase is based on JAX [51] and the DeepMind JAX Ecosystem [52, 53] . We used 4 Tensor Processing Units (V5) per model for the ablation experiments. We used 128 Tensor Processing Units (V5) per model to train our large (9M, 136M and 270M) models. We used a single Tensor Processing Unit (V3) per agent for our Elo tournament.We perform additional ablations using the 9M parameter model. As in Section 3.4, the results and conclusions drawn are used to justify our design choices and determine default configuration for model, data, and training. Table A2 shows the results. We report scores on the puzzle set (same for all models) and a test set created using the same time limit as the training set (different for all models). Table 2 shows that a basic time limit of 0.05 seconds gives only marginally worse puzzle performance. As a compromise between computational effort and final model performance we thus choose this as our default value (for our 10M games dataset we need about 15B action-evaluation calls with Stockfish, i.e., roughly 8680 days of unparallelized Stockfish evaluation time).We remove duplicate board states during the generation of the training and test sets. This increases data diversity but introduces distributional shift compared to the "natural" game distribution of boards where early board states and popular openings occur more frequently. To quantify the effect of this shift we use an alternative "weighted" data sampler that draws boards from our filtered training set according to the distribution that would occur if we had not removed duplicates. Results in Table A2 reveal that training on the natural distribution (via the weighted sampler) leads to significantly worse results compared to sampling uniformly randomly from the filtered training set (both trained models are evaluated on a filtered test set with uniform sampling, and the puzzle test set). We hypothesize that the increased performance is due to the increased data diversity seen under uniform sampling. As we train for very few epochs, the starting position and common opening positions are only seen a handful of times during training under uniform sampling, making it unlikely that strong early-game play of our models can be attributed to memorization.Model architecture To investigate the performance of architectures other than transformers on our benchmark, we perform a preliminary investigation with an AlphaZero-like convolutional neural network. We therefore train a residual tower with 8 layers and 256 channels (no pooling, broadcasting, or bottleneck layers), followed by a linear layer and a log softmax. We input the 8-by-8 board in a single channel and treat the other scalars in the FEN as additional channels. Table A2 shows that this particular model-and training configuration does not lead to strong results for the convolutional architecture we consider. Note that we did not extensively tune our experiment setup and would therefore expect highly domain-specific setups that more closely resemble those of AlphaZero or Leela Chess Zero to achieve much higher performance. Thus, the purpose of this experiment is to show that the vanilla transformer setup seems to achieve better performance than an "equivalently vanilla" convolutional setup.Table A3a shows the inference times in milliseconds on 1000 random boards from the Encyclopedia of Chess Openings [22] using a single TPU (V5). Unsurprisingly, the "searchless" agents (i.e., our models and the policy networks) are orders of magnitudes faster than the agents that rely on tree search at test time. Note that the value networks are slow to evaluate since we compute every legal action individually (and sequentially) and then take the arg max over the values.A natural question that arises when training models to play chess is whether they have actually learned the rules of the game. However, the problem of generating legal moves is tangential to our paper since we use a legal move generator (given by the environment) for our policies, as is standard in the chess AI literature (see, e.g., AlphaZero, Stockfish, or Leela Chess Zero). In fact, computing the legal move accuracy does not actually make sense for action-value prediction since the action-value function is only defined for legal moves. In contrast, for behavioral cloning the question of legal move accuracy is potentially more interesting (throughout this work, we normalize the predictive distribution to support only the legal moves except when evaluating the legal move accuracy) since the model is trained to output the oracle (and therefore a valid) action.We investigate the legal move accuracy for action-value and behavioral cloning in Table A3b (the legal move accuracy is undefined for state-value prediction since we can only evaluate boards reachable via legal moves). Unsurprisingly, Table A3b shows that the action-value predictor has a very low legal We observe no overfitting, which justifies always using the fully trained model in our evaluations. Step 1e6 100'000 Games0 1 2 3 4 5Step 1e6 move accuracy, since its outputs for illegal moves are undefined (i.e., never adjusted during training). In contrast, our behavioral cloning policy has learned to almost always predict a legal move, even for highly out-of-distribution puzzle boards, showing that our networks have the capacity to learn the rules given our training protocol and dataset -if explicitly trained to do so (i.e., unlike action-value prediction).In Figure A2 we show the train and test loss curves (and the evolution of the puzzle accuracy) for the large models from Section 3.1. We observe that none of the models overfit and that larger models improve both the training and the test loss.In Figure A3 we visualize the train and test loss curves for the scaling experiment from Section 3.3.In line with the results shown in the main paper we observe that on the smallest training set, models with ≥ 7M parameters start to overfit but not for the larger training sets. Except for the overfitting cases we observe that larger models improve both the training and test loss, regardless of training set size, and that larger training set size improves the test loss when keeping the model size constant. A5 . As the results show, the state-value policy in this case slightly outperforms the action-value policy, except for action-ranking (Kendall's τ ), which makes sense since the action-value predictor is implicitly trained to produce good action rankings. To see how this translates into playing-strength, we pit all three policies (AV, SV, BC) against each other and determine their relative Elo rankings. Table A4 shows that when not controlling for the number of training data points, the action-value policy is strongest (in line with the findings in Table 2 and Figure A4 ), but when controlling for the number of training data points the action-value and state-value policy perform almost identical (in line with Figure A5 ).Throughout all these results we observe lower performance of the behavioral cloning policy, despite being trained on a comparable number of data points as the state-value policy. The main hypothesis for this is that the amount of information in the behavioral cloning dataset is lower than the state-value dataset, since we throw away any information in the state-or action-values beyond the index of the oracle action. We suspect that training on the full action distribution of the oracle (with cross-entropy loss), rather than the best action only would largely close this gap, but we consider this question beyond the scope of this paper and limit ourselves to simply reporting the observed effect in our setting.As a measure of out-of-distribution generalization, we evaluate the performance of our 9M transformer on Fischer random chess3 (Chess960) via Lichess (using the same setup as for Table 1 ). Fischer is a variation of the game of chess, which employs the same board and pieces as classical chess, but the starting position of the pieces on the players' home ranks is randomized, following certain rules. The random setup makes gaining an advantage through the memorization of openings impracticable; players instead must rely more on their skill and creativity over the board. Thus, Fischer random chess can be viewed as a measure of whether our models have learned general tactics (for the early stages of the game). While our bot attains a blitz Elo of 2054 (cf. Table 1 ), it only manages to play Fischer random chess at 1539 Elo (against bots), suggesting that the distribution shift is too high for our models to handle.In Figure A6 , we analyze the tactics learned by our 270M transformer used against a human with a blitz Elo of 2145. We observe that our model has learned to sacrifice material when it is advantageous to build a longer-term advantage.We recruited chess players of National Master level and above to analyze our agent's games against bots and humans on the Lichess platform. They made the following qualitative assessments of its playing style and highlighted specific examples (see Figure A7 ).The agent has an aggressive enterprising style where it frequently sacrifices material for long-term strategic gain. The agent plays optimistically: it prefers moves that give opponents difficult decisions to make even if they are not always objectively correct. It values king safety highly in that it only reluctantly exposes its own king to danger but also frequently sacrifices material and time to expose the opponent's king. For example 17 .. Bg5 in game B.8.1 encouraged its opponent to weaken their king position. Its style incorporates strategic motifs employed by the most recent neural engines [7, 54] . For example it pushes wing pawns in the middlegame when conditions permit (see game B. motif of a pin on the back rank to justify a pawn sacrifice for long term pressure. Game B.8.5 features a piece sacrifice to expose its opponent's king. The sacrifice is not justified according to Stockfish although the opponent does not manage to tread the fine line to a permanent advantage and blunders six moves later with Bg7.The agent has a distinct playing style to Stockfish: one analyzer commented "it feels more enjoyable than playing a normal engine", "as if you are not just hopelessly crushed". Indeed it does frequently agree with Stockfish's move choices suggesting that the agent's action-value predictions match Stockfish's. However the disagreements can be telling: the piece sacrifice in the preceding paragraph is such an example. Also, game B.8.6 is interesting because the agent makes moves that Stockfish strongly disagrees with. In particular the agent strongly favours 18 .. Rxb4 and believes black is better, in contrast Stockfish believes white is better and prefers Nd4. Subsequent analysis by the masters suggests Stockfish is objectively correct in this instance. Indeed on the very next move the agent has reversed its opinion and agrees with Stockfish.The agent's aggressive style is highly successful against human opponents and achieves a grandmasterlevel Lichess blitz Elo of 2895. However, against other engines its estimated Elo is far lower, i.e., 2299. Its aggressive playing style does not work as well against engines that are adept at tactical calculations, particularly when there is a tactical refutation to a sub-optimal move. Most losses against bots can be explained by just one tactical blunder in the game that the opponent refutes. For example Bxh3 in game B.8.7 loses a piece to g4.Finally, the recruited chess masters commented that the agent's style makes it very useful for opening repertoire preparation. It is no longer feasible to surprise human opponents with opening novelties as all the best moves have been heavily over-analyzed. Modern opening preparation amongst professional chess players now focuses on discovering sub-optimal moves that pose difficult problems for opponents. This aligns extremely well with the agent's aggressive, enterprising playing style which does not always respect objective evaluations of positions.Lichess games and puzzles are released under the Creative Commons CC0 license.Stockfish returns a score in centipawns that we convert to the win percentage with the standard formula win% = 100/(1 + exp(-0.00368208 • centipawns)) from https://lichess.org/page/accuracy.https://en.wikipedia.org/wiki/Fischer_random_chess 


Decision: This paper produces a benchmark dataset constisting of chess positions together with Stockfish position evaluations. The main idea is to investigate the ability of transformers to learn this mapping, thus "distilling" Stockfish's search process into a single pass of a transformer. Various dimensions and ablations are investigated, with results showing that a Transformer-only model is capable of playing chess at a high level (without search), although not as well as Stockfish.

The work is solid and was complimented by the reviewers. One strange thing about this paper is that many of the results may indeed be known to the general Leela-chess community, but this would be the first paper that provides documentation and rigour surrounding the idea of using a neural network (in this case, a transformer) to evaluate positions. As such, many of the results will likely be novel to the general machine learning community. On the other hand, one could argue that the audience for this is rather niche. The idea of "distilling" an algorithm or search process into a transformer is generally useful to a wider audience, but because it is so tied to the domain of chess here (naturally, since the benchmark itself focuses on chess), it's unclear whether these results might be of interest outside the domain (and computer chess playing community).

As a minor note, the issue of indeciseveness is tackled in a rather inelegant way, and I can't help but wonder if it reveals an issue with the way the benchmark is constructed. Instead of mapping all guaranteed checkmating sequences to the same value, perhaps they can be mapped to slightly different (high) values to differentiate and encourage the transformer to follow shorter paths to victory.

Reviews: review: paper_summary: paper_summary: The paper is well-written and demonstrates that supervised learning with Stockfish evaluations can create a strong chess engine. The methodology is solid, and the inclusion of an LLM as a baseline is noteworthy. However, the motivation is ambiguous and the use of transformers over CNNs is questioned. Additionally, the paper lacks clarity on certain statements, and the technical contribution is limited. Ablation studies on training objectives are also recommended.
strength_weakness: strength_weakness: - W1: The motivation for this work is somewhat ambiguous. On the one hand, the authors claimed that the main idea is to approximate the search engine Stockfish, by distilling its board evaluation results into NNs (in particular transformers), but the evaluation is on chess performance instead of directly comparing the predicted and ground truth centipawn advantages. Although the action prediction evaluation is kind of doing that, it is still discretized to actions before comparing the rankings, not comparing the continuous values directly. Therefore the claim to approximate stockfish is invalid. On the other hand, the proposed system did really well in chess, it seems that the authors are trying to build a strong chess engine without searching. But it is still underperforming lc0 inference without search.
- W2: Why transformers are particularly interesting for this study? Many prior works have shown that the multi-channel encoding of chess boards combined with CNNs can perform really well, including alphazero, lc0, maia, etc. Essentially, chessboards are 2d and the used inputs are sequential. Is it possible that the tokenization of FEN and the direct inputs to transformers are limiting the performance?
- W3: It is unclear what this statement means: “Although the source code is available, there is no (peer-reviewed) research report explaining and comparing Lc0’s methods”. Does this mean lc0 may include Stockfish eval already? Is it still fair to compare with lc0? It would be good if the authors could clarify.
- W4: The labels of the value prediction are ranges of centipawn advantages predicted by Stockfish. Is there any design to balance the labels? And is there any mechanism to make sure the categorical labels are semantically interconnected?
- W5: The methodological contribution of this work is limited. If I understand correctly, the method basically incorporates stockfish evals as training targets without task-oriented designs.
- W6: Ablation studies on the training objectives should be conducted and presented.
questions: questions: Please refer to W2-W4.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces a large dataset of chess board states along with annotations for best moves and state-(action)-values. It demonstrates that transformers trained on this dataset with supervised learning can achieve significant performance and generalization. This adds to the growing evidence showing that neural nets can implement complex behaviors (such as imitating search-based chess engines).
strength_weakness: strength_weakness: EDIT: see my reply (https://openreview.net/forum?id=XlpipUGygX&noteId=UsMmll2N3a) for my updated stance.

The paper does not add too much that didn't already exist outside peer-reviewed ML venues. As the authors discuss (line 312), the existing Leela Chess Zero network is stronger than the network introduced by this paper. The T82 version of Leela (which this paper compares against) is also trained using supervised learning and uses a transformer architecture similar to the one in this paper. Similarly, datasets of chess board states with engine annotations exist (e.g. at https://database.lichess.org/). I'm not aware of datasets with state-action-value annotations for all moves in a state, but it's not clear that this is even necessary to train a state-action-value predictor.

In my mind, the main contribution of this paper is thus creating a well-documented and peer-reviewed chess-playing neural network and dataset, as well as running ablations (which likely don't come as a surprise to the chess engine community, but seem interesting for the broader NeurIPS audience). This is still a nice contribution (though I wish it was made clearer from the start that this is what the paper does, e.g., in the intro/"contributions" paragraph. EDIT: the authors' changes do make this clear now).

Minor notes:
- Regarding why BC does worse than value predictions (appendix B.5): I don't think the richer training signal is the full explanation, since the value nets of Leela/AlphaZero also outperform their policy nets, even though those policy nets are trained to imitate a full distribution (rather than the single best move). My guess is that another reason is the weak form of "1-ply search" inherent in letting the network evaluate many different actions/follow-up states and taking an argmax.
- Line 177: T82 is a transformer, not a ConvNet (see more below)
- Nitpick for table 1 and elsewhere: Leela and AlphaZero get the past eight board states as input, rather than the full move history (though the lack of even earlier states shouldn't matter much). I think the most notable difference is in fact between GPT-3.5 vs all the other networks, since GPT-3.5 gets *only* moves and so has learn to keep track of the board state itself. So in the "Input" column, I might distinguish between "PGN" (for GPT-3.5), "FEN" (for the new models), and "FEN + past states" (for Leela and AlphaZero).
- On the claim that the network plays at grandmaster level: this is a difficult comparison to make, since human players play better when given more time per move, whereas the neural network of course doesn't improve from time beyond what's needed for one or a fixed number of forward passes. I agree that the network seems as strong as grandmasters with Blitz time controls, but it might still be significantly weaker than grandmasters at classical time controls. I think this distinction isn't always clear (e.g. line 44).

Detailed notes on Leela vs the new model in this paper: The T82 version of Leela that this paper compares against was trained using supervised learning on MCTS data, rather than trained with online RL (this isn't well-documented, but see https://lczero.org/play/networks/basics/ and note that T82 is one of the "contrib runs"). It is also a transformer with an input encoding not too different from the one used in this paper (one token position per square of the chess board). The main differences as far as I can tell are that Leela has some domain-specific improvements to its architecture (https://lczero.org/blog/2024/02/transformer-progress/, Smolgen is part of T82), uses fewer parameters, and was trained on different (and probably more) data.
questions: questions: (pretty minor)
- If I understand correctly, all three types of predictors use entirely separate models. I'm wondering whether training a single model with a shared body and three small prediction heads would lead to transfer between predictors (e.g. let the BC policy profit from the larger amount of data used to train the state-action-value predictor). Is this something you considered/tried?
- Why is the state-value predictor trained only on the smaller amount of data (compared to the state-action-value predictor)? Couldn't you use the state-action-value annotations as state-value annotations for the follow-up state, which would mean you'd already have all the training data needed to also train the state-value predictor on more data?
limitations: limitations: Issues are discussed well in the paper
paper_summary: paper_summary: This paper aims to solve the challenging problem in the chess game. The main contributions include building a large-scale dataset and training a large-scale transformer model with the collected dataset. The proposed approach has shown a significant outperformance over the baselines.
strength_weakness: strength_weakness: The technique novelty of this work is incremental, as the techniques used in the proposed approach have been previously developed. The usage of transformer in amortized planning is a contribution, but not significant enough.
questions: questions: Can the authors elaborate on whether the transformer structure is more important or the dataset is more important for performance improvement?
limitations: limitations: The authors have discussed the potential limitations.
paper_summary: paper_summary: This paper introduces an open dataset comprised of real chess positions annotated with a evaluation score, the best move to play, and a score for each legal move according to StockFish 16 using 50ms for each action. They then train large transformers over this dataset, and include in-depth analysis through making the model play on Lichess, and using ablation studies over the hyperparameters of the network.
strength_weakness: strength_weakness: - The dataset might be of relatively poor quality, as it uses Stockfish 16 with only 50ms for each evaluation. This could make a significant proportion of the evaluations be wrong, and potentially limit the playing strength of models trained using this data. (Table 1. show a significant (>200 elo) disparity in playing strength between 0.05s and 1.5s for stockfish)
- The goal of the paper is not immediately obvious. A first reading of the abstract and the introduction seem to imply with the title that the paper is concerned about the general abilities of Transformers to tackle planning problems, while the contributions are more focused on providing a dataset (as well as an initial case study on it) for future research on this topic, focused on the case of chess.
questions: questions: Section 2.2 mentions that the problem was seen as a classification problem, binning the win percentages into K uniform bins. Was augmenting the density of bins around 50% in order to improve accuracy in critical positions considered?

Table A3(a) in the appendix seems to show that stockfish with 0.05s per move takes 1.5s per move on average. What does this mean? Wasn't the 50ms limit a strict limit?

Appendix A9 mentions the hardware used for the experiments, but not how long it took to train each model. Would it be possible to add this information?
limitations: limitations: The limitations are adequately addressed.

Comments: comment: We are pleased to hear that the rebuttal has solved most of the reviewer's concerns and thank the reviewer for adjusting their evaluation accordingly!

We also thank the reviewer for clarifying the term "semantic interconnectedness". We actually spent considerable time addressing the semantic interconnectedness of the classes, and as the reviewer rightly suspected, it turned out to be important for performance. This was precisely the reason for using the HL-Gauss loss [17], which extends the classical cross-entropy loss with label smoothing (see Figure 3 in [18] for an intuitive visualization). We ablated three different loss functions in Table 2: two that model semantic interconnectedness (L2 and HL-Gauss) and one that does not take it into account (i.e., the log loss), with the HL-Gauss loss achieving the highest performance.

We admit that this important point was not given enough attention in the current manuscript, and we will clarify it in the next revision of our paper. 

We hope that this addresses the reviewer's remaining concerns.
Thanks for the detailed explanations and pointers to the resources in the Appendix. 

However, I didn't find the answers to "W5: The methodological contribution of this work is limited. If I understand correctly, the method basically incorporates stockfish evals as training targets without task-oriented designs.", which remains my main concern.

By "semantic interconnectedness" I mean that: the training targets are stockfish evals and you used "value binning" to cut the value ranges into "classes". However, the "classes" are dependent. For example, the range 0-10 centipawn advantage would be similar to the range 10-20 but significantly to the range 290-300. The method seems without any design to model such dependencies.

Nevertheless, the rebuttal has solved most of my concerns. I have adjusted my evaluation.
I acknowledge that I have read the rebuttal. The revisions and clarifications are welcome, and I will maintain my evaluation.
Thank you for those changes! They clear up all my concerns about the presentation of the motivation/contributions and other details. After more consideration, I've decided to increase my score to 6. In my mind, the main reason against acceptance is still that the paper's main findings aren't very surprising to anyone sufficiently familiar with Lc0, but I think the reasons in favor of acceptance outweigh that:
* Most of the ML community is not, in fact, familiar with Lc0, in part because of a lack of scientific/peer-reviewed writings on it. And for this majority of readers, I think the findings are quite interesting and important.
* As mentioned, I think this paper is very well done in terms of writing and experiments.
* Having a simpler and well-documented/reproducible chess-playing transformer could enable research that would have required significantly more effort on Lc0. So I think this paper has good potential for indirect impact via future work using these models and/or datasets.