Enhancing Chess Reinforcement Learning with Graph Representation

Authors: tomas  rigaux
hisashi  kashima
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 09:21:21

Abstract: Mastering games is a hard task, as games can be extremely complex, and still fundamentally different in structure from one another. While the AlphaZero algorithm has demonstrated an impressive ability to learn the rules and strategy of a large variety of games, ranging from Go and Chess, to Atari games, its reliance on extensive computational resources and rigid Convolutional Neural Network (CNN) architecture limits its adaptability and scalability. A model trained to play on a $19\times 19$ Go board cannot be used to play on a smaller $13\times 13$ board, despite the similarity between the two Go variants.
In this paper, we focus on Chess, and explore using a more generic Graph-based Representation of a game state, rather than a grid-based one, to introduce a more general architecture based on Graph Neural Networks (GNN). We also expand the classical Graph Attention Network (GAT) layer to incorporate edge-features, to naturally provide a generic policy output format.
Our experiments, performed on smaller networks than the initial AlphaZero paper, show that this new architecture outperforms previous architectures with a similar number of parameters, being able to increase playing strength an order of magnitude faster. We also show that the model, when trained on a smaller $5\times 5$ variant of chess, is able to be quickly fine-tuned to play on regular $8\times 8$ chess, suggesting that this approach yields promising generalization abilities.
Our code is available at https://github.com/akulen/AlphaGateau.

Main paper: 
 introduction: 
 In the past decade, combining Reinforcement Learning (RL) with Deep Neural Networks (DNNs) has proven to be a powerful way to design game agents for a wide range of games. Notable achievements include AlphaGo's dominance in Go [17] , AlphaZero's human-like style of play in Chess and Shogi [18] , and MuZero's proficiency across various Atari games [15] . They use self-play and Monte Carlo Tree Search (MCTS) [6] to iteratively improve their performance, mirroring the way humans learn through experience, or intuition, and game-tree exploration.Previous attempts to make RL-based chess engines were unsuccessful as the MCTS exploration requires a precise position heuristic to guide its exploration. Handcrafted heuristics such as the ones used in traditional minimax exhaustive tree searches were too simplistic, and lacked the degree of sophistication that the random tree explorations of MCTS expects to be able to more accurately evaluate a complex chess position. By combining the advances in computing powers with the progress of the field of Deep Learning, AlphaZero was able to provide an adequate heuristic in the form of a Deep Neural Network that was able to learn in symbiosis with the MCTS algorithm to iteratively improve itself.However, these approaches rely on rigid, game-specific neural network architectures, often representing games states using grid-based data structures, and process them with Convolutional Neural Networks (CNNs), which limits their flexibility and generalization capabilities. For example, a model trained on a standard 19 × 19 Go board cannot easily adapt to play on a smaller 13 × 13 board without significant changes to its internal structure, manual parameter transfer, and retraining, despite the underlying similarity of the game dynamics. This inflexibility is further compounded by the extensive computational resources required for training these large-scale models from scratch for each specific game or board configuration. If it was possible to make a single model train of various variants of a game, and on various games at the same time, it would be possible to speed up the training by starting to learn the fundamental rules on a simplified and smaller variant of a game, before presenting the model with the more complex version. Similarly, if a model learned all the rules of chess, it could serve as a strong starting point to learn the rules of Shogi, for example.It could be possible to design a more general architecture for games such as Go, where moves can be mapped one-to-one with the board grid, so that a model could still use CNN layers and handle differently sized boards simultaneously, but this solution is no longer feasible when the moves become more complex, including having to move pieces between squares, or even dropping captured pieces back onto the board in Shogi.Those moves evoke a graph-like structure, where pieces, initially positioned on squares, are moved to different new squares, following an edge between those two squares, or nodes. As such, it is natural to consider basing an improved model on a graph representation, instead of a grid representation. We explore replacing CNN layers with GNN layers to implement that idea, and more specifically consider in this paper attention-based GNN layers, reflecting how chess players usually remember the structures that the pieces form, and how they interact with each other, instead of remembering where each individual piece is placed, when thinking about a position.Representing moves as edges in a graph also introduces the possibility to link the output policy size with the number of edges, to make the model able to handle different game variants with different move structures simultaneously. To do so, it becomes important to have edge features as well as node features, as they will be used to compute for each edge the equivalent move logit. As the classical attention GNN layer, the Graph Attention Network [19] (GAT) only defines and updates node-features, we propose a novel modification of the GAT layer, that we call Graph Attention neTwork with Edge features from Attention weight Updates (GATEAU), to introduce edge-features. We also describe the full model architecture integrating the GATEAU layer that can handle differently sized input graphs as AlphaGateau.Our experimental results demonstrate that this new architecture, when implemented with smaller networks compared to the original AlphaZero, outperforms previous architectures with a similar number of parameters. AlphaGateau exhibits significantly faster learning, achieving a substantial increase in playing strength in a fraction of the training time. Additionally, our approach shows promising generalization capabilities: a model trained on a smaller 5 × 5 variant of chess can be quickly fine-tuned to play on the standard 8 × 8 chessboard, achieving competitive performance with much less computational effort.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The results discussed in the experiments and illustrated in Figures 5, 6 , and 7 support the claims in the abstract and the introduction. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Reinforcement Learning. AlphaGo [17] , AlphaZero [18] , MuZero [15] , and others have introduced a powerful framework to exploit Reinforcement Learning techniques in order to generate self-play data used to train from scratch a neural network to play a game. However, those frameworks use rigid neural networks, that have to be specialized for one specific game. As such, the training process requires a lot of computation resources. It is also not possible to reuse the training on one type of game to train for another one, or to start the training on a smaller and simpler variant of the game, before introducing more complexity. Scalable AlphaZero. In the research of Ben-Assayag and El-Yaniv [2] , using Graph Neural Networks has been investigated as a way to solve those issues. Using a GNN-based model, it becomes possible to feed as input differently-sized samples, such as Othello boards of size between 5 and 16, enabling the model to learn how to play in a simpler version of the game. This approach had promising results, with 10 times faster training time than the AlphaZero baseline. It was however limited to Othello and Gomoku, and using the GNN layers (GIN layers [20] ) only as a scalable variant of CNN layers, keeping a rigid grid structure.Edge-featured GNNs. There exists a large variety of GNN variants, specialized for different use case s and data properties. For this work, a simple layer was enough to experiment with the merits of the proposed approach, except it was critical that the chosen layer handled both node-features and edge-features. We chose to use an attention-based layer.Gong and Cheng [10] introduce the EGNN(a) layer, where each dimension of an edge-feature is used for a different attention head. We wanted edge features to be treated as a closer equivalent to node-features, so we did not use this layer.The EGAT layer introduced by Chen and Chen [4] is better for our case, as they construct a dual graph where edges and nodes have reversed roles, so the node features in the dual graph are edge features for the initial graph. However this method requires building the dual graph, and is quadratic in the maximal node degree. As this was quite complex, we decided to introduce GATEAU, which solves the problem in a simpler and more natural way.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Our main limitations are that we had limited computing resources and had models of depth 5 or 10 when 40 would be better, and we only focused on chess and not other games. We could also only run the experiments once in the time available to us, so the confidence intervals only apply to the rating estimations, and were not extimated over several training runs. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
setting: 
 Our architecture is based on the AlphaZero framework, which employs a neural network f θ with parameters θ that is used as an oracle for a Monte-Carlo Tree Search (MCTS) algorithm to generate self-play games. When given a board state s, the neural network predicts a (value, policy) pair (v(s), π(s, ⋅)) = f θ (s), where the value v ∈ [-1, 1] is the expected outcome of the game, and the policy π is a probability distribution over the moves.We utilize Algorithm 1 to train the models in this paper, with modifications to incorporate Gumbel MuZero [7] with a gumbel scale of 1.0 as our MCTS variant.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: The only theoretical result is the closed form derivation of the confidence interval for the estimated Elo ratings, included in the appendix. 
motivation: representing the game state as a graph: 
 Many games, including chess, are not best represented as a grid. For example, chess moves are analogous to edges in a grid graph, and games like Risk naturally form planar graphs based on the map. As such, it makes natural sense to encode more information through graphs in the neural network layers that are part of the model. This research focuses on implementing this idea in the context of chess. This requires to answer two questions: how to represent a chess position as a graph, and how to output a policy vector that is edge-based, and not node-based.The architecture presented in this paper is based on GNNs, but using node features to evaluate the value head, and edge features to evaluate the policy head. As such, a GNN layer that is able to handle The piece on the square, as a 12-dimensional soft one-hot 2Whether the position was repeated before 98 For each of the previous 7 moves, we repeat the last 14 dimensions to describe the corresponding positions 1The current player 1The total move count 4Castling rights for each player 1The number of moves with no progress both node and edge features is required. This paper will introduce the GATEAU layer, that is a natural extension of the GAT layer [19] to edge features.In AlphaZero, a chess position is encoded as a n×n×119 matrix, where each square on the n×n chess board is associated to a feature vector of size 119, containing information about the corresponding square for the current position, as well as the last 7 positions, as described in Table 1 .We will instead represent the board state as a graph G(V, E), with the n × n squares being nodes V , and the edges E being moves, based on the action design of AlphaZero. Each AlphaZero action is a pair (source square, move), with n × n possible source squares. In 8 × 8 chess, the 73 moves (resp. 49 in 5 × 5 chess) are divided into 56 queen moves (resp. 32), 8 knight moves (resp. 8), and 9 underpromotions (resp. 9) for a total of 4672 actions (resp. 1225). The edge associated with an action is oriented from the node corresponding to the source square, to the destination square of the associated move. In 8 × 8 chess, castling is represented with the action going from the king's starting square going laterally 2 squares. As this action encoding is a little too large, containing moves ending outside of the board that do not correspond to real edges, the constructed graph only contains 1858 edges (resp. 455), corresponding only to valid moves.The node and edge features, of initial size 119 and 15, are detailed in Tables 1 and 2 , respectively. Node features are based on AlphaZero's features, including piece type, game state information, and historical move data. Edge features encode move legality, direction, potential promotions, and piece-specific move capabilities. In the case of 5×5 chess, we include the unused castling information, in order to have the same vector size of the 8 × 8 models. It would be possible to preprocess the node and edge features differently for different games or variants, but for simplicity we didn't do it.The starting positions for all games played in our experiments were either the classical board setup in 8 × 8 chess, or the Gardner setup for 5 × 5 chess, illustrated in Figure 1 .The Graph Attention Network layer introduced by Veličković et al. [19] updates the node features by averaging the features of the neighboring nodes, weighted by some attention coefficient. To be more precise, given the node features h ∈ R N ×K , attention coefficients are defined asEQUATIONwith parameters W u , W v ∈ R K×K ′ . In the original paper, W u = W v , but as we are working with a directed graph, we differentiate them to treat the source and destination node asymmetrically. Then we can compute attention weights α, and use them to update the node features:EQUATIONEQUATIONwith parameters W ∈ R K×K ′′ and a ∈ R K ′ .The main observation motivating GATEAU is that in this process, the attention coefficients e ij serve a role similar to node features, being a vector encoding some information between nodes i and j. As such, we propose to introduce edge features in place of those attention coefficients.Our proposed layer, called Graph Attention neTwork with Edge features from Attention weight Updates (GATEAU) takes the node features h ∈ R N ×K and edge features g i,j ∈ R N ×N ×K ′ as inputs. We start by simply updating the edge features similarly to Eq. 1:g ′ ij = W u h i + W e g ij + W v h j (4) with parameters W u , W v ∈ R K×K ′ and W e ∈ R K ′ ×K ′ .Then the attention weights are obtained as in Eq. 2, by substituting the attention coefficients with our new edge features:EQUATIONwith parameter a ∈ R K ′ . Finally, we update the node features as in Eq. 3:EQUATIONwith parameters W 0 , W h ∈ R K×K ′′ and W g ∈ R K ′ ×K ′′ . We add the self-edges manually as it is inconvenient for the policy head if they are included in the graph, and we mix back the values of the edge features back in the node features.Following the structure of the AlphaZero neural network, we introduce AlphaGateau, combining AlphaZero with GATEAU instead of CNN layers, and redesign the value and policy head to be able to exploit node and edge features respectively to handle arbitrarily sized inputs with the same number of parameters.We define the following layers, which are used to describe AlphaGateau in Figure 2 .Attention Pooling. In order to compute a value for a given graph, we need to pool the features together. Node features seem to be the more closely related to positional information, so we pool them instead of edge features. For this, we use an attention-based pooling layer, similar to the one described in Eq. 7 by Li et al. [14] , which, for node features h ∈ R N ×K and a parameter vector a ∈ R K , outputs EQUATION8 rmblkans 7 opopopop 6 0Z0Z0Z0Z 5 Z0Z0Z0Z0 4 0Z0Z0Z0Z 3 Z0Z0Z0Z0 2 POPOPOPO 1 SNAQJBMR a b c d e f g h 5 snaqj 4 popop 3 Z0Z0Z 2 POPOP 1 SNAQJ a b c d ePolicy head ×L where H ∈ R K is a global feature vector.Batch Normalization and Non-linearity (BNR). As they are a pair of operations that often occur, we group Batch Normalization and a ReLU layer together under the notation BNR:EQUATIONResidual GATEAU (ResGATEAU). Mirroring the AlphaZero residual block architecture, we introduce ResGATEAU, which similarly sums a normalized output from two stacked GATEAU layers to the input:EQUATIONQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We included all the experimental hyperparameters and methodology in the paper, and include all the code that was used in the GitHub repository mentioned in the abstract. However, as the parallelized GPU code is not fully deterministic, it is not possible to replicate the exact training results we present, so we will also publish in the git repository some important model parameter checkpoints that we saved. 
experiments: 
 We evaluate AlphaGateau's performance in learning regular 8 × 8 chess from scratch and generalizing from a 5 × 5 variant to the standard 8 × 8 chessboard. The metric used to evaluate the models is the Elo rating, calculated through games played against other models (or players) with similar ratings. Due to computational constraints, we couldn't replicate the full 40 residual layers used in the initial AlphaZero paper, and experimented with 5 and 6 layer models. We also started exploring 8 layers, but these models required to generate a lot more data, which would make the experiment run an order of magnitude longer. Our results indicate that AlphaGateau learns significantly faster than a traditional CNN-based model with similar structure and depth, and can be efficiently fine-tuned from 5 × 5 to 8 × 8 chess, achieving competitive performance with fewer training iterations. 2 All models used in these experiments are trained with the Adam optimizer [12] with a learning rate of 0.001. All feature vectors have an embedding dimension of 128. The loss function is the same as for the original AlphaZero, which is, for f θ (s) = π, ṽ,EQUATION)Jax and PGX. As the MCTS algorithm requires a lot of model calls weaved throughout the tree exploration, it is essential to have optimized GPU code running both the model calls, and the MCTS search. In order to leverage the MCTX [8] implementations of Gumbel MuZero, all our models and experiments were implemented in Jax [3] and its ecosystem [11] [9] . PGX [13] was used for the chess implementation, and we based our AlphZero implementation on the PGX implementation of AZNet. We used Aim [1] to log all our experiments.To estimate the Elo ratings, we use the statsmodels package [16] implementation of Weighted Least Squares (WLS).Hardware. All our models were trained using multiple Nvidia RTX A5000 GPUs (Learning speed used 8 and Fine-tuning used 6), and their Elo ratings were estimated using 6 of those GPUs.As each training and evaluation lasted a little under a week, we were not able to train each model configuration several times so far. As such, each model presented in the results was trained only once, and the confidence intervals that we include are on the Elo rating that we estimated for each of them, as described in the following.During training, at regular intervals (each 2, 5, or 10 iterations), the model parameters were saved, and we used this dataset of parameters to evaluate Elo ratings. In this section, we will call a pair (model, parameters) a player, and compute a rating for every player.We initially chose 10 players, and simulated 60 games between each pair of players, to get initial match data M . For each pair of players that played a match, we store the number of wins w ij , draws d ij , and lossesl ij : M ij = (w ij , d ij , l ij ).Using this data, we can roughly estimate the ratings r ∈ R N players of the players present in M using a linear regression on the following set of equations:EQUATIONWe artificially add one draw to avoid extreme cases where there are only wins for one player and no losses, in which case the rating difference would theoretically be infinite. This is equivalent to a Jeffreys prior. The last equation fixes the average rating to 1000, as the Elo ratings are collectively invariant by translation.We then ran Algorithm 2 to generate a densely connected match data graph, where each player played against at least 5 other players of similar playing strength. Finally, we used this dataset to fit a linear regression model (Weighted Least Squares) to get Elo ratings that we used in the results figures for the experiments. The confidence intervals were estimated by assuming that the normalized match outcomes followed a Gaussian distribution.If pij = wij + d ij +1 2 wij +dij +lij +1is the estimated probability that player i beats player j, we approximate the distribution that p ij follows as a Gaussian, and using the delta method, we derive that r j -r j asymptotically follows a Gaussian distribution of mean 400 log( 10) log ( 1 pij -1) and variance ( 400 log(10) )2 1(wij +dij +lij )pij (1-pij ) .The proof is detailed in Appendix A.2. Using the WLS linear model of statsmodels [16] , we get Elo ratings for every player, as well as their standard deviations, which we use in the following to derive 2-sigma confidence intervals.Learning Speed. Our first experiment compares the baseline ability of AlphaGateau to learn how to play 8 × 8 chess from scratch, and compares it with a scaled down AlphaZero model. The AlphaZero model has 5 residual layers (containing 10 CNN layers) and a total of 2.2M parameters, and the AlphaGateau model also has 5 ResGATEAU layers (containing 10 GATEAU layers) and a total of 1.0M parameters, as it doesn't need a N nodes × hs × N actions fully connected layer in the policy head, and the GATEAU layers use 2/3 of the parameters a 3 × 3 CNN does. For this experiment, we generated 256 games of length 512 at each iteration, totalling 131 072 frames, and kept a frame window of 1M frames (all of the newly generated frames, and uniform sampling over the frame window of the previous iteration), over 500 iterations. During the neural network training, we used a batch size of 256. The training for AlphaGateau lasted 13 days and 16 hours, while AlphaZero took 10 days and 3 hours.We report the estimated Elo ratings with a 2-sigma confidence interval in Figure 5 . AlphaZero was only able to reach an Elo of 667 ± 38 in 500 iterations, and would likely continue to improve with more time, while AlphaGateau reached an Elo of 2105 ± 42, with an explosive first 50 iterations, and achieving results comparable to the final Elo of AlphaZero after only 10 iterations.Although those results are promising, it is important to note that we only compared to a simplified version of AlphaZero, using only 5 layers instead of the original 40, and without spending large efforts to optimize the hyper-parameters of the model. As such, it is possible that the performance of AlphaZero could be greatly improved in this context with more parameter engineering. Both AlphaGateau and AlphaZero have not reached a performance plateau after 500 iterations, showing slow but consistent growth.Fine-tuning. In our second experiment, we trained a AlphaGateau model with 10 residual layers on 5 × 5 chess for 100 iterations, then fine-tuned this model on 8 × 8 chess for 100 iterations. This model has a total of 1.7M parameters.For this experiment, we generated 1024 games of length 256 at each iteration on 5 × 5 chess, and 256 games of length 512 while fine-tuning on 8 × 8 chess, and kept a frame window of 1M frames. The initial training lasted 2 days and 7 hours, and the fine-tuning 5 days and 15 hours. We report the estimated Elo ratings with a 2-sigma confidence interval in Figure 6 . The initial training ended with an Elo rating of 807 ± 46 when evaluated on 8 × 8 chess games, which suggests that it was able to learn general rules of chess on 5 × 5, and apply them with some success on 8 × 8 chess without having seen any 8 × 8 chess position during its training. Once the fine-tuning starts, the model jumps to an Elo of 1181 ± 50 after a couple iterations, suggesting the baseline learned on 5 × 5 was of high quality. After fine-tuning, the model had an Elo of 1876 ± 47, reaching comparable performances to the smaller model using roughly the same amount of iterations and GPU-time, despite being twice as big. Preliminary testing suggests that this model would become stronger if more data was generated at each iteration, but that would linearly increase the training time, as generating self-play games took half of the 5 days of training.In order to train deeper networks, we experimented with the number of self-play games generated at each generation, and with the size of the frame window. if seems from our results in Figure 7 that having more newly generated data helps the model learn faster. However, the time taken for each iteration scales linearly with the number of generated games, and the model is still able to improve using older data. As such, keeping a portion of the frame window from previous iterations makes for a good compromise. There are however options to improve our frame window selection: The fine-tuned model starts with a good baseline, and reaches comparable performances to the 5-layer model despite being undertrained for its size.• Which previous samples should be selected? We selected uniformly at random from the previous frame window to complement the newly generated samples, but it might be preferable to select fewer samples, but chosen as to represent a wide range of different positions.• Past samples are by design of dubious quality. As they come from self-play games with previous model parameters, they correspond to games played with a lower playing strength, and the policy output by the MCTS is also worse. Keeping a sample that is too old might cause a drop of performance rather than help the model learn. We experimented a little with keeping the 1M most recent samples, but with little success.We also initially tried to increase the number of epochs in one iteration, but only saw marginal gains, suggesting that mixing new data among the previous frame window helps the model extract more training information from previous samples. The two models with a frame window of size 131 072 only kept the latest generated games in the frame window. The model keeping no frame window and generating 256 games was trained in only 39 hours, but had the worst performance. Adding a 1M frame window improved the performance a little and lasted 60 hours, while increasing the number of self-play games to 1024 performed the best, but took 198 hours.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: The full code is published in the GitHub repository Akulen/AlphaGateau.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we introduce AlphaGateau, a variant on the AlphaZero model design that represents a chess game state as a graph, in order to improve performance and enable it to better generalize to variants of a game. This change from grid-based CNN to graph-based GNN yields impressive increase in performance, and seems promising to enable more research on reinforcement-learning based game-playing agent research, as it reduces the resources required to train one.We also introduce a variant of GAT, GATEAU, that we designed in order to handle edge features in a simple manner performed well, and efficiently.Future Work. As our models were relatively shallow when compared to the initial AlphaZero, it would be important to confirm that AlphaGateau still outperforms AlphaZero when both are trained with a full 40-deep architecture. This will require a lot more computing time and resources.As discussed in Section 5.4, our design of the frame window is a little unsatisfactory, and a future improvement would be to define an efficiently computable similarity metric between chess positions, that helps the neural network generalize.We focused on chess for this paper, but there are other games that could benefit from this new approach. The first one would be shogi, as it has similar rules to chess, and the promising generalization results from AlphaGateau could be used to either train a model on one game, and fine-tune it on the other, or to jointly train it on both games, to have a more generalized game-playing agent. As alluded to in the Graph Design 4.2, more features engineering would be required to have node and edge features compatibility between chess graphs, and shogi graphs. It could also be possible to change the model architecture to handle games with more challenging properties, such as the game Risk, which has more than 2 players, randomness, hidden information, and varying maps, but is even more suited to being represented as a graph.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The full training procedure is described in the paper, and the provided code contains the implementation details, such as the gpu data split procedure.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We provide confidence intervals on our estimated Elo ratings, however, as each model was only trained once, they are incomplete confidence intervals.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We include the number and type of GPU, as well as the experiment run time.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We introduce an improvement to an existing DRL architecture to train a game agent. There should be no direct societal impact from this, as the results are currently limited to making research on this topic more accessible, due to the training efficiency gains.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The released models are way less powerful than current existing chess engines, as they are limited in size and depth for compute reasons, so there is no risk in publishing them.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite every non-standard python library used to develop and run the experiments presented in this paper. We do not use any previous dataset or asset besides those libraries.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The Git repo that will be published along the paper will have well structured code.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 
 
 Appendix: A.1 Elo The Elo rating system was initially introduced as a way to assign a value to the playing strength of chess players. The rating of each player is supposed to be dynamic and be adjusted after each game they play to follow their evolution.The Elo ratings are defined to respect the following property: If two players with Elo rating R A and R B played a game, the probability that player A wins isEQUATIONWe assume that the outcome of a game between two players of Elo R A and R B is a bernouilli trial, with a probability that player A win being given by the central Elo equation 12. If we want to estimate the Elo of both players, we need to estimate that probability p. To do so, we can make the two players play n games, and sum the wins of player A, as well as half his draws, to get x = w + d 2 . From this, we can estimate the value of p using the estimator p = x n . In the case that one of the two players is significantly stronger than the other, p could be close to 0 or 1, in which case this estimator is wildly inacurrate. To remedy this, we will instead rely on a Jeffreys prior, to get the estimator pJeffreys = x+1/2 n+1 . We will note this estimator p in the following. From our assumptions, we have that x follows a binomial distribution B(n, p), and we will approximate the distribution that p follows by a normal distribution p ∼ N (p, p(1-p) n ).By inverting the Elo equation 12, we can get the rating difference from the probability that A wins asEQUATIONTherefore, posing g(y) = 400 log(10) log ( 1 y -1), which is differentiable, we can use the delta method to get that g(p) is asymptotically Gaussian. The derivative of g is g ′ (y) = 400 log (10) (-EQUATIONwhich givesRB -RA = g(p) ∼ N (g(p), p(1 -p) n (g ′ (p)) 2 )∼ N ⎛ ⎝ 400 log (10) log( 1 p -1) , p(1 -p) n (- 400 log(10) 1 p(p -1) ) 2 ⎞ ⎠ ∼ N ( 400 log(10) log ( 1 p -1) , 400 2 log(10) 2 p(1 -p) n 1 (p(p -1)) 2 )∼ N ( 400 log (10) log( 1 p -1) , 400 2 log(10) 2 1 np(p -1)) .We also used BayesElo [5] to evaluate the Elo ratings of our models using all the PGN files that were generated recording the games played between all our models. In Figure 8 , we plotted a point Figure 8 : The difference between BayesElo ratings and the Elo ratings according to our method. We removed to each Elo the average Elo of all players in its respective method, such that the average effective Elo for both BayesElo and our method is 0 on (x, y) for each player where x is its Elo rating following our method and y is the difference in predicted Elo between our method and BayesElo.In practice, weaker models tend to be overrated by our method compared with BayesElo, while stronger models are underrated. As this relation is smooth and monotone, this suggest that both methods order the relatives strength of all players similarly, with small differences only being due to noise. The main difference being that our range of Elo ratings is compacted towards the extremes, which we assume is due to our practical Jeffreys prior, that implies that strong models drew at least one game against all their opponents, handicapping their Elo, and similarly weak models always manage to not lose at least one game.We didn't record the FLOPs of our models, however, we did record the time taken for each iteration. In Figure 10 , 12, and 14, we plotted the three main plots of the main paper using that data on the X-axis. As some experiments were run using 6 GPUs instead of *, we multiplied their recorded time by a factor 6 8 for better comparison.We include with Figure 15 , 16, and 17 a few PGNs showing games played by our models. Figure 16 and 17 contains the PGNs of games played by the last iteration (iteration 499) of the full AlphaGateau model of our first experiment as white and black respectively, selected among the games played against other models to evaluate all the Elo rankings, selected using the script book.py in the GitHub repo, following the most played move each ply. The game played as white in Figure 16 is played against the iteration 61 of a fine-tuned AlphaGateau model using 8 layers, 1024 generated games per iteration, but only 32 MCTS simulations, with an estimated Elo rating of 2124. The game played as black in Figure 17 is played against the iteration 439 of the same model.We ran the final iteration of the 5-layer AlphaGateau model on Lichess, as the bot AlphaGateau (https://lichess.org/@/AlphaGateau). We let it play against other bots and some human players in bullet, blitz, and rapid time formats by varying the number of MCTS simulations to adjust the time required to play each move. This was implemented using the default Lichess bot bridge (https://github.com/lichess-bot-devs/lichess-bot), and the relevant code is in the lichess folder of our GitHub repo.At the end of October 2024, after between 100 and 200 games per time format, AlphaGateau was able to reach a bullet Elo of 1991, a blitz Elo of 1829, and a rapid Elo of 1884.• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).In Silver et al.[18], the training of AlphaZero is described in terms of steps, which consists of one minibatch of size 4096, while the generation of games through self-play is done in parallel by other TPUs. In this experiment, an iteration consists of generating 256 games through self-play, then doing one epoch of training, splitinto 3904 mini-batches of size 256, after 7 iterations once the frame window is full. In terms of positions seen, one iteration is equivalent to 244 steps.Figure17: AlphaGateau starts playing a Berlin defense, without any book, and diverts by move 4 into a popular line, with a rare bishop move on move 7. The midgame revolves around black strong knight, until white is forced to give up its remaining rook to stop the attack, leaving black up a rook for a pawn. 


Decision: The reviewers agree that the application of graph neural networks to chess is interesting and worth studying. The reviewers note some weaknesses in the evaluation of the method and the comparison to AlphaZero, but, during the discussion period, the most serious of the concerns were addressed, and the authors have offered additional results and clarifications that will strengthen the paper. Though the reviewers uniformly agree that the paper is acceptable, the somewhat limited scope/impact of the work and the limitations on the conclusions that can be drawn from the experimental evaluations suggest that this work is best presented as a poster, rather than as a spotlight or oral presentation.

Reviews: review: paper_summary: paper_summary: The authors demonstrate a GNN that works on chess and is amenable to generalization.
strength_weakness: strength_weakness: The authors only look at self-play for model training, the original AlphaGo models were trained on human games to establish that the algorithm performed well, before starting the much more expensive self-play training. The lack of supervised training evaluations is makes the claims of strong performance limited to a single axis (sample efficiency).

This result mostly seems to arise from chess specific optimizations applied to an existing algorithm [18], while the authors suggest that it could be applied to other games they only provide evidence in chess.

The fine tuning results require more training time (and thus I'm assuming compute) than the initial training, this suggests the pre-training while beneficial is of limited efficacy unless it can be made more efficient.

I'm also concerned with the  attention-based pooling leading degrading in performance when the graph gets larger as chess is a game that cares much more about the max than the average.

I would have liked to see a more complete training run, but I understand that RL is very compute intensive so am not using this lack to negatively impact my score.
questions: questions: Is the MCTS process identical between AlphaGateau and AlphaZero models? Or does the graph representation change it?

Could the graph representation be leveraged to allow a different search algorithm?

Could the authors provide PGNs with model predictions and some discussion of the difference in the AlphaGateau and AlphaZero models? There are patterns that CNN based models are good at learning and some that are more difficult (see [1] for examples) and showing differences in the models would strengthen this result.

Does the model handle the white bishop well when scaled up? If it does that would be additional evidence of generalization when scaling up since the moves would not have been seen before.

Can you provide more details on the Elo calculation? You use the terms ELO and elo and seem to refer to a varation of the Elo rating system created by Arpad Elo, but don't give much details beyond the optimizer. Are you using and Elo or Glicko (2) type system or something else? Section A has not citations so it's unclear. 

[1] McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Wattenberg, M., Hassabis, D., Kim, B., Paquet, U. and Kramnik, V., 2022. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47), p.e2206625119.
limitations: limitations: Discussed above
paper_summary: paper_summary: The paper explores a novel approach to  reinforcement learning for Chess by utilizing a graph-based representation of the game state instead of the traditional grid-based representation. This method is based on GNNs  and aims to overcome the limitations of CNNs used in previous models like AlphaZero. Specifically, the paper introduces the Graph Attention neTwork with Edge features from Attention weight Updates (GATEAU) layer, which enhances the classical GAT layer by incorporating edge features.

The primary contributions of the paper include demonstrating that the new architecture outperforms previous models with a similar number of parameters and significantly accelerates the learning process. Additionally, the paper provides evidence that the model trained on a smaller variant of chess (5x5) can be quickly fine-tuned to perform well on the standard 8x8 chessboard, indicating promising generalization capabilities. The authors have made their code available, supporting the reproducibility of their experiments and findings.
strength_weakness: strength_weakness: **Limited Generalization Evidence**: The paper claims promising generalization capabilities, but the experimental evidence is limited to a small variant of chess (5x5) and standard chess (8x8). It would strengthen the paper to include additional experiments on other games or variants, such as Shogi or other board games with similar complexity, to demonstrate the broader applicability of the proposed architecture.

**Comparative Analysis**: The paper lacks a thorough comparative analysis with other state-of-the-art models beyond AlphaZero. Including comparisons with recent advancements in graph-based reinforcement learning models would provide a clearer picture of the relative performance and innovations of the proposed method.

**Real-world Applicability**: While the focus is on reinforcement learning in chess, discussing potential real-world applications of the proposed graph-based approach would broaden the impact of the work. Highlighting areas where this approach could be applied, such as other strategic games or decision-making problems in different domains, would make the contributions more compelling.
questions: questions: 1. How does AlphaGateau compare to other graph neural network-based reinforcement learning models, such as those presented by Ben-Assayag and El-Yaniv (2021) [1]?
2. Have the authors considered applying AlphaGateau to other strategic games beyond chess? If so, what preliminary results or observations can the authors share?
3. What potential real-world applications besides chess do the authors envision for the graph-based approach proposed in this paper?

[1] Ben-Assayag, S., & El-Yaniv, R. (2021). Train on small, play the large: Scaling up board games with alphazero and gnn. *arXiv preprint arXiv:2107.08387*.
limitations: limitations: The authors have acknowledged some limitations of their work, such as the restricted computational resources which prevented the training of deeper networks, and their focus solely on chess without extending experiments to other games. They also discussed challenges related to the reproducibility of their results due to the non-deterministic nature of parallelized GPU code. However, the paper could benefit from a deeper discussion on the broader implications of these limitations, particularly how they might affect the generalizability and robustness of the proposed method. Regarding potential negative societal impacts, the paper does not explicitly address this aspect.
paper_summary: paper_summary: The paper introduces a new variant of AlphaZero, AlphaGateau, with a neural network architecture based on GNNs, that allows the model to generalize across board sizes. The new architecture outperforms the original AlphaZero architecture under certain conditions.
strength_weakness: strength_weakness: Evaluation & comparison with AlphaZero: 
While figure 5 clearly shows that AlphaGateau learns much faster than AlphaZero at early training, the paper does not provide any information on how the two models compare when they are fully trained. AlphaGateau seems to reach a performance plateau within 100 steps, but the AlphaZero model trained by the authors would likely keep on improving for orders of magnitude more training steps. For comparison, the authors used a model size roughly 10% of Silver et. al.'s AlphaZero, but trained for 100 steps compared to the original 700,000 steps. It is not clear if AlphaGateau's performance will be comparable with AlphaZero, given enough training steps.
I am aware that training the models for 10^5 steps is extremely expensive and not feasible for this paper. Even so, the correct solution would have been to fit the model size to the training budget and train a significanly smaller AlphaZero model, for longer time and on smaller batches (or to heavily reuse training data between optimization steps). Doing a comparison of smaller-scale models could show if AlphaGateau is a comparable model to AlphaZero. 

In the paper's current form, the reported Elo gap does not give the reader any knowledge about how AlphaGateau compares to a fully-trained AlphaZero, since Elo calculation is unreliable when using such large gaps. For example, the gap between the best performing AlphaGateau and best AlphaZero is about 1,500 Elo, meaning that AlphaZero would win about 1 game in every 7,500 games. This implies that the AlphaGateau Elo score is based purely on comparisons with earlier AlphaGateau training checkpoints. 
Another example is the gap between the first and second checkpoints, which is hard to measure by eye due to the plot format but looks roughly equal to 800. This gap implies the weaker model wins 1 game in 100, suggesting that the error of this gap is very high unless the authors use thousands of test games between this specific checkpoint pair to calculate Elo. I couldn't find the number of Elo games in the paper, and would recommend stating it clearly.

Considering that the best model's Elo is probably based purely on comparisons to its earlier versions, and considering the possible error of the Elo estimate, it is possible that AlphaGateau's Elo rating will drop significantly if fully-trained AlphaZero models are added to the pool of players.

This paper will benefit significantly from fitting the size of the experiments to the compute resources available to the authors. I would suggest training significanly smaller models for 100x more training steps, using the same compute budget. In its current form, the paper only showcases the advantage AlphaGateau has in terms of data-efficiency at early training, in a regime where AlphaZero is severely undertrained.
questions: questions: - What hyperparameters are used to calculate Elo scores? (number of models tested, number of games between each model pair, who played against who)

Three minor suggestions:
- It would be helpful for the reader if figures 5 and 6 had FLOPs on the x-axis instead of steps, or if training compute was reported. Plotting only steps, it is hard to understand if one training step of AlphaGateau is comparable to a training step of AlphaZero in terms of compute.
- It would be easier and more convincing to use traditional methods for calculating Elo scores, such as BayesElo which was used by Silver et. al. That said, the authors provide clear explanations on their Elo calculation method.
- The correct spelling of the metric is "Elo", named after Arpad Elo, rather than "ELO", which is a common typo.
limitations: limitations: No limitations other than those specified above regarding evaluation.

Comments: title: AlphaZero Elo
Discussion Period
comment: Thank you for your detailed response. As I said in my review I am not judging the work by the amount of compute. I think this is an informative paper on GNNs for game play by itself and while I would like a more detailed examination of it's scaling with respect to other models I don't think that is a requirement for acceptance, but the other reviewers disagree so I will continue to watch the discussion. For now I maintain my score.

Also to explain my point about scaling. I the GNN presented here does not use max pooling, instead it uses a weighted average (via attention). This will limit the depth the model can "see" important nodes since information about them will be lost as it passes up network. This is different from the PUCT algorithm which maintains information about the best lines even if they are very deep via counting the number of times each node has been searched and using that as its deciding factor instead of the approximated Q value which is an average. Thus I suspect that there will be scaling issues with this model as the networks increase in scale.
Thank you for clarifying the details, this helps in understanding what went wrong with AlphaZero. Doing only 100 model updates is most probably the reason why it learns so slowly in figure 5. My guess is that 128 MCTS steps is also too small, given that chess has an average branching factor of 35. I honestly cannot tell you how to adapt your hardware for training with more model updates, I'm not familiar with the PGX implementation. But if there is a way you can reduce the quality of epochs in order to do x100 more model updates, that should probably lead to a good AlphaZero agent.

I will change my score from reject to a borderline accept (5), for the following reasons:
- This is a good paper considering anything other than evaluation.
- The internal evaluation (tracking AlphaGateau's improvement against itself) in the PDF is a significant improvement on the one in the paper draft, and proves the model indeed improves fast during training.
- The external evaluation (tracking AlphaGateau's improvement against AlphaZero) is not good. This is through no fault of the authors. Training AlphaZero is expensive and requires a lot of experience, the authors were just unlucky to choose bad training hyperparameters.
- The results the authors report on Lichess increase the likelyhood that their models indeed learned well.

I expect the authors to follow through with the changes they promised. Specifically, doing the following in the final version will minimize the damage of the evaluation issue:
- Add a clear discussion of the limitations of their comparison to AlphaZero.
- Train AlphaZero for longer and recalculating Elo for all agents. Hopefully this agent will improve and give more credibility to the results, but I cannot ask the authors to promise results they cannot control. If using BayesElo, I would suggest playing more matches with AlphaZero to increase these agents' influence on the final Elo scores.
- Add the Lichess results to the paper or appendix. Using more MCTS steps is completely fine, especially given the low number used in training.

I would have given this paper a much higher score if it could show significant improvement on AlphaZero, it's a shame that this paper suffers just because training the baseline is so hard.
Thanks for your rebuttal and I will maintain my score!
With regards to our assessment of AlphaZero, we could have been clearer in the previous answer. It doesn't saturates after only 100 iterations, but around 400. 'with marginal improvements after 100 steps' was a poorly worded way of saying that the model stayed significantly worse than AlphaGateau, eventually reaching a rating between 300 and 500 Elo, compared to the more than 1500 of AlphaGateau. Seeing these results, we decided to only train AlphaGateau for 100 iterations as it seemed that the behavior of the model during these early iterations was the most important part, with both AlphaZero and AlphaGateau increasing significantly less afterwards.

However, we do now agree with your criticism that the asymptotic behavior is also important to include. We will include in the revised version of the paper 500 steps of training of AlphaZero. We are currently running this experiment with our up-to-date code, with Elo ratings going up to iteration 300, where it still only goes from 100 elo at iteration 100 to around 450 by iteration 300.

We will also note that AlphaGateau models don't saturate either by iteration 100. This was not clear in our initial figures as we only evaluated Elo rating once every 5 iterations, but it is clearer in our new figures evaluated every second iteration, such as the one in the global rebuttal pdf. We would further like to extend the runs of the AlphaGateau models to at least iteration 200, and will include those new runs in the revised paper.

Yes, we only updated the data generating model once every step, and chose how many games to generate in one step to saturate our GPU. We could generate less games per step, but we would waste computing resources by doing so. We experimented trying to run more than one full epoch between each step of generation, but didn't notice any significant improvement behind the first 10 steps, so we decided to stick to only one epoch per step (and around 500-1000 batches) as the training already took the majority of our compute time.

For the details of the training of AlphaZero, we generated 256 games at each step, and used batches of 2048 positions, for a total of 488 batches each step (except the first 6-7 steps, while the frame buffer is filling up). Each set of 256 generated games used newly updated network parameters. We also only used 128 MCTS simulations in each of our experiments.

We didn't attempt to anchor our Elo ratings to other real agents, as our main results are the improvements of our architecture when compared to its AlphaZero basis. We however did let our latest fine-tuned for 20 steps 6-layer AlphaGateau model play around 600 blitz and bullet games on lichess, against mostly other bots, ending with an approximate Elo of 1800 in blitz, and 2000 in bullet. However, we had to adjust the number of MCTS simulations in order to make our model take an appropriate mostly constant amount of time to evaluate each move, which was often different to the number used during training, so these ratings are only indicative of the approximate abilities of that model.
Thank you for the detailed reply, I still need to go through all of it but wanted to clarify one point first. Reading the match-related numbers you specify, together with figure 1 in the PDF, I tend to believe your reported Elo scores are reliable, at least for the pool of AlphaGateau agents. It's also nice to see your Elo is not that different than that of BayesElo, making your results more robust.

My problem with the comparison to AlphaZero still stands though. You claim (and I believe you) that your AlphaZero agents showed signs of saturation already after 100 'steps', meaning that according to figure 5 in the paper your AlphaZero models can only improve <300 Elo above an agent at 'step' 1, which is almost a purely random agent (right?). That means your AlphaZero still loses 15% of the time against an almost random agent, so clearly it failed to learn well. 

It is not necessary to train with a DeepMind-sized compute budget to get a good chess agent. One can train a small model tuned to the budget size to get very quick gains in Elo, that will saturate quicker the smaller the model is. I honestly don't know what went wrong with your training, you clearly have enough compute to go through a sufficient number of batches. Is the data-generating model only updated each 'step'? That could partially explain the problem, doing only 100 model updates is crucially little in my experience. Although it doesn't explain why the model would saturate so quickly. Could you specify how AlphaZero was trained? specifically how many games were played between optimization steps, how many batches used each optimization step, how often were the model weights updated (for the data-generation agent)?

To summarize, I think your internal Elo rating is solid, but these Elo numbers are not anchored to any competent non-AlphaGateau agent, which makes it impossible to make any claim about AlphaGateau's performance.
Hi all! Just your friendly area chair checking in.

First, thanks to the reviewers for your work so far on this paper. The discussion period has begun, which includes both reviewers and authors. This is our opportunity to clear up misunderstandings, get answers to questions, and generally gather enough information to make an informed decision grounded in the acceptance criteria.

This paper has a sizable spread of overall assessments from the reviewers. It seems there is general agreement that the approach in the paper is interesting and promising. The main critiques raised seem to concern lack of support for claims of generality, the size of the model and extent of training (which may not cover asymptotic performance comparisons), and the validity of the ELO ratings and comparisons. Does this seem like a fair summary?

Reviewers: please carefully read the other reviews and the authors' responses (with included PDF) and let us know what follow-up questions you have and to what extent your evaluation might change as a result of the additional context. Please especially raise any points of disagreement with other reviewers or the authors, as these are opportunities for us to clarify misunderstandings and detect misconceptions.