Enhancing Chess Reinforcement Learning with Graph Representation
Authors: Tomas Rigaux
Hisashi Kashima
Venue: NeurIPS 2024 poster (2024)
Keywords: deep learning
GNN
Chess
RL
Primary area: graph_neural_networks
TL;DR: We introduce an edge-featured Neural Network AlphaGateau to improve Reinforcement Learning for Chess as a Graph-structured game.

Abstract: Mastering games is a hard task, as games can be extremely complex, and still fundamentally different in structure from one another. While the AlphaZero algorithm has demonstrated an impressive ability to learn the rules and strategy of a large variety of games, ranging from Go and Chess, to Atari games, its reliance on extensive computational resources and rigid Convolutional Neural Network (CNN) architecture limits its adaptability and scalability. A model trained to play on a $19\times 19$ Go board cannot be used to play on a smaller $13\times 13$ board, despite the similarity between the two Go variants.
In this paper, we focus on Chess, and explore using a more generic Graph-based Representation of a game state, rather than a grid-based one, to introduce a more general architecture based on Graph Neural Networks (GNN). We also expand the classical Graph Attention Network (GAT) layer to incorporate edge-features, to naturally provide a generic policy output format.
Our experiments, performed on smaller networks than the initial AlphaZero paper, show that this new architecture outperforms previous architectures with a similar number of parameters, being able to increase playing strength an order of magnitude faster. We also show that the model, when trained on a smaller $5\times 5$ variant of chess, is able to be quickly fine-tuned to play on regular $8\times 8$ chess, suggesting that this approach yields promising generalization abilities.
Our code is available at https://github.com/akulen/AlphaGateau.

Enhancing Chess Reinforcement Learning with Graph Representation
Tomas Rigaux" Kyoto University Kyoto, Japan tomas@rigaux.cmm com com
Hisashi Kashima Kyoto University Kyoto, Japan
kashima@i.kyotouu.com
Abstract
Mastering games is hard task, games be extremely complex, and still fundamentally different in structure from one another. While the AlphaZero algorithm has demonstrated an impressive ability to learn the rules and strategy of a large variety of games, ranging from Go and Chess, to Atari games, its reliance extensive computational resources and rigid Convolutional Neural Network (CNN) architecture limits its adaptability and scalability. A model trained to play on 19 19 Go board cannot be used to play on smaller 13 x 13 board, despite the similarity between the two Go variants. In this paper, we focus on Chess, and explore using more generic Graph-based Representation of game state, rather than a grid-based one, to introduce more general architecture based on Graph Neural Networks (GNN). We also expand the classical Graph Attention Network (GAT) layer to incorporate tigtufee,uurs, to naturally provide generic policy output format. Our experiments, performed on smaller networks than the initial AlphaZero paper, show that this new architecture outperforms previous architectures with similar number of parameters, being able increase playing strength an order of magnitude faster. We also show that the model, when trained on smaller 5 x variant of chess, able to be quickly fine-tuned to play on regular 8 x 8 chess, suggesting that this approach yields promising generalization abilities. Our code is available at httts:tp// //gi thub. com/ akulen/ AlphaGateau
Introduction
In the past decade, combining Reinforcement Learning (RL) with Deep Neural Networks (DNNs) has proven to be powerful way design game agents for wide range of games. Notable achievements include AlphaGio's 's dominance in Go 17]. AlphaZero's human-like style of play in Chess and Shogi [8]. and MuZero's proficiency across various Atari games (15). They use self-play and Monte Carlo Tree Search (MCTS) 6 to iteratively improve their performance, mirroring the way humans learn through experience, or intuition, and game-tree exploration.
Previous attempts to make RL-based chess engines were unsuccessful as the MCTS exploration requires precise position heuristic to guide its exploration. Handcrafted heuristics such the ones used traditional minimax exhaustive tree searches were too simplistic, and lacked the degree of sophistication that the random tree explorations of MCTS expects to be able to more accurately evaluate a complex chess position. By combining the advances in computing powers with the progress of the field of Deep Learning, AlphaZero was able provide adequate heuristic in the form of Deep Neural Network that was able to learn in symbiosis with the MCTS algorithm to iteratively improve itself.
tomas. Comas.rigaux wiw.ux.gc..oooo com
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

However, these approaches rely on rigid, game-specific neural network architectures, often repre- senting games states using grid-based data structures, and process them with Convolutional Neural Networks (CNNs), which limits their flexibility and generalization capabilities. For example, a model trained on a standard 19 x 19 Go board cannot easily adapt to play on a smaller x 13 board without significant changes to its internal structure, manual parameter transfer, and retraining, despite the underlying similarity of the game dynamics. This inflexibility is further compounded by the extensive computational resources required for training these large-scale models from scratch for each specific game or board configuration. If it was possible to make a single model train of various variants of a game, and various games at the same time, it would be possible to speed up the training by starting to learn the fundamental rules on a simplified and smaller variant of a game, before presenting the model with the more complex version. Similarly, if a model learned all the rules of chess, it could serve as a strong starting point to learn the rules of Shogi, for example.
It could be possible to design more general architecture for games such as Go, where moves can be mapped one-to-one with the board grid, so that a model could still use CNN layers and handle differently sized boards simultaneously, but this solution is no longer feasible when the moves become more complex, including having to move pieces between squares, or even dropping captured pieces back onto the board in Shogi.
Those moves evoke graph-like like structure, where pieces, initially positioned on squares, are moved to different new squares, following an edge between those two squares, or nodes. As such, it is natural to consider basing an improved model on a graph representation, instead of grid representation. We explore replacing CNN layers with GNN layers to implement that idea, and more specifically consider this paper attention- based GNN layers, reflecting how chess players usually remember the structures that the pieces form, and how they interact with each other, instead of remembering where each individual piece is placed, when thinking about a position.
Representing moves as edges in a graph also introduces the possibility to link the output policy size with the number of edges, make the model able to handle different game variants with different move structures simultaneously. To do so, it becomes important to have edge features as well as node features, as they will be used to compute for each edge the equivalent move logit. As the classical attention GNN layer, the Graph Attention Network [19] (GAT) only defines and updates node-features, we propose novel modification of the GAT layer, that we call Graph Attention neTworrk with Edge features from Attention weight Updates (GATEAU), to introduce degtures We also describe the full model architecture integrating the GATEAU layer that can handle differently sized input graphs as AlphaGateau.
Our experimental results demonstrate that this new architecture, when implemented with smaller networks compared to the original AlphaZero, outperforms previous architectures with a similar number of parameters. AlphaGateau exhibits significantly faster learning, achieving a substantial increase in playing strength in fraction of the training time. Additionally. our approach shows promising generalization capabilities: a model trained on a smaller 5 5 variant of chess can be quickly fine-tuned to play on the standard 8 8 chessboard, achieving competitive performance with much less computational effort.
2 Related Work
AlphaZero 18) MuZero (15), and others have introduced a powerful framework to exploit Reinforcement Learning techniques in order to generate self-play data used train from scratch a neural network to play a game
Reinforcement Learning. AlphaGo (17)
However, those frraeworks use rigid neural networks, that have to be specialized for one specific game. As such, the training process requires lot of computation resources. It is also not possible to reuse the training on one type of game to train for another one, or to start the training on a smaller and simpler variant of the game, before introducing more complexity.
Scalable AlphaZero. In the research of Aen-assayag and EI- Yaniv 2 using Graph Neural Networks has been investigated as way to solve those issues. Using a GNN based model, it becomes possible to feed as input differently-sized samples, such as Othello boards of size between and 16, enabling the model to learn how to play in simpler version of the game.
2

---

Algorithm 1: Self-Play Training
Parameters: Niter 100, Ngames = 256, Nsm 128, ws = 10", .Nirain = 1, 2048 A - --mdellinnn():: for i 4 to Niter do
data - selfplay(f Ngames; Naim):
/* We generate self-play data +/ trame_windone - (datal [ttnne__iidoo]]]__ ws] /* The new frame window consists of the newly generated data and a uniform sample of the previous window */
for j 4 Nerroi do
frame frame_windde window frame wwinow.shufffee for batch in frame window batches(bs) do - apply(8; gradientt (f,banh))
end
end
end
This approach had promising results, with 10 times faster training time than the AlphaZero baseline. was however limited to Othello and Gomoku, and using the GNN layers (GIN layers (20 only as scalable variant of CNN layers, keeping rigid grid structure.
Edge-featured GNNs. There exists large variety of GNN variants, specialized for different use case and data properties. For this work, a simple layer was enough to experiment with the merits of the proposed approach, except it was critical that the chosen layer handled both node-features and edge-features, We chose to use attention-based layer.
Gong and Cheng [00] introduce the EGNN(a) layer, where each dimension of an adge-feature is used for different attention head. We wanted edge features to be treated as closer equivalent to node-features, so we did not use this layer.
The EGAT layer introduced by Chen and Chen is better for our case, as they construct a dual graph where edges and nodes have reversed roles, the node features in the dual graph edge features for the initial graph. However, this method requires building the dual graph, and is quadratic in the maximal node degree. As this quite complex, we decided to introduce GATEAU, which solves the problem in simpler and more natural way.
3 Setting
Our architecture is based on the AlphaZero framework, which employs neural network fo with parameters 9 that is used as an oracle for Monte-Cart Carlo Tree Search (MCTS) algorithm to generate self-play games. When given a board state s, the neural network predicts a (value, policy) pair (n(s), (), fs(),, where the value E [-1,1] is the expected outcome of the game, and the policy æ is probability distribution over the moves.
We utilize Algorithm to train the models this paper, with modifications to incorporate Gumbel MuZero 7 with gumbel scale of 1.0 our MCTS variant.
4 Proposed Models
4.1 Motivation: Representing the Game State as a Graph
Many games, including chess, are not best represented as a grid. For example, chess moves are analogous to edges in a grid graph, and games like Risk naturally form planar graphs based on the map. As such, makes natural sense encode more information through graphs in the neural network layers that are part of the model.
This research focuses implementing this idea in the context of chess. This requires to answer two questions: how to represent chess position as a graph, and how to output policy vector that is edge-based, and not node-based.
The architecture presented in this paper is based on GNNs, but using node features to evaluate the value head, and edge features evaluate the policy head. As such, a GNN layer that is able to handle

---

Table 1: Node features
Dimensions
Description
12
The piece the square, 12-dimensional soft one-hot Whether the position was repeated before
2
For cach of the previous moves, we repeat the last dimensions to describe the corresponding positions
98
1
The current player
1
The total move count Castling rights for each player
4
The number moves with no progress
Table 2: Edge features
Dimensions
Description
Is this move legal in the current position?
2
How many squares {to the left/up) does this edge move?
4
Would pawn promote to (kmighhhishopprooo/qqenn)) if did this move?
Could [white/black] pawn this move?
Could [knighhhbshopprrok/qqeen do this move?
2 Could [white/black] king do this move?
both node and edge features is required. This paper will introduce the GATEAU layer, that is natural extension of the GAT layer [9 edge features.
4.2 Graph Design
In AlphaZero, chess position is encoded as an x n 119 matrix, where each square on the n chess board associated to feature vector of size 119, containing information about the corresponding square for the current position, as well as the last 7 positions, as described in Table []
We will instead represent the board state as a graph G(V, E), with the n squares being nodes V, and the edges being moves, based on the action design of AlphaZero. Each AlphaZero action pair (source square, move), with n possible source squares. In 8 8 chess, the 73 moves (resp. 49 in 5 x 5 chess) are divided into 56 queen moves (resp. 32), 8 knight moves (resp. 8), and underpromotions (resp. 9) for total 4672 actions (resp. 1225). The edge associated with an action is oriented from the node corresponding to the source square, to the destination square of the associated move. In 8 8 chess, castling is represented with the action going from the king' starting square going laterally 2 squares. As this action encoding is a little too large, containing moves ending outside of the board that do not correspond to real edges, the constructed graph only contains 1858 edges (resp. 455), corresponding only to valid moves.
The node and edge features, of initial size 119 and 15, detailed in Tables and respectively. Node features are based on AlphaZero's 's features, including piece type, game state information, and historical move data. Edge features encode move legality, direction, potential promotions, and piece-specific move capabilities. In the case of 5x5 chess, we include the unused castling information, order have the same vector size of the 8 models. would be possible to preprocess the node and edge features differently for different games or variants, but for simplicity we didn't do it.
The starting positions for all games played in experiments either the classical board setup in 8 chess, or the Gardner setup for 5 x chess, illustrated in Figure
4.3 GATEAU: A New GNN Layer, with Edge Features
The Graph Attention Network layer introduced by Veličković et al. 19 updates the node features by averaging the features of the neighbouring nodes, weighted by some attention coefficient. To be more precise, given the node features h E R attention coefficients are defined as
efj h4 + W.hy
(1)
with parameters W., W. RKKK", In the original paper, W,, W., as we are working with directed graph, differentiate them to treat the source and destination node asymmetrically. Then

---

we can compute attention weights and use them to update the node features:
exploabyRellU(o cz))
oij =
(a"e:)) Σx exploalyRelUU@
(2)
1/4 Σ JeN
U
Wh
with parameters W RRKK and a ε R.K.
The main observation motivating GATEAU is that in this process, the attention coefficients eij serve a role similar to node features, being a vector encoding some information between nodes and j. As such, we propose to introduce edge features in place of those attention coefficients.
Our proposed layer, called Graph Attention Twokkrk with Edge features from Attention weight Updates (GATEAU) takes the node features E N and edge features Ds.j RNNNKK as inputs. We start by simply updating the edge features similarly to Eq.
Dij W. + Weltj W.h
with parameters W., W IR Kxk' and W. e R'''' Then the attention weights are obtained as in Eq by substituting the attention coefficients with our new edge features:
Oij softmax, (LeakyReL.U g.))
with parameter RRK. Finally, we update the node features in Eq.
h4 Woh
Σ Ozi (Wnhg W Wggit)
(6)
JEN.
with parameters Wo, WA E IR Kxk" and W K"=K". We add the self-edges manually as it is inconvenient for the policy head they are included in the graph, and we mix back the values of the edge features back in the node features.
4.4 AlphaGateau: A Full Model Architecture Based on AlphaZero and GATEAU
Following the structure of the AlphaZero neural network, introduce AlphaGateau, combining AlphaZero with GATEAU instead of CNN layers, and redesign the value and policy head to be able exploit node and edge features respectively to handle arbitrarily sized inputs with the same number of parameters.
We define the following layers, which are used to describe AlphaGateau in Figure 2
Attention Pooling. In order compute a value for given graph, need to pool the features together. Node features seem to be the more closely related to positional information, we pool them instead of edge features, For this, we use attention sirentio--based pooling layer, similar to the one described in Eq. 7 by Li et al. [14) which, for node features h IR Nx and parameter vector R outputs
0 = (Team (nF1.))
H Eaph
A
A
A
A
A
4
2
6
Figure 1: The starting positions of 8 8 and 5 chess games
5

---

Linear hs
Nodes (squares)
h
h
Value head
ResGATEAU
h8
Linear
Edges (moves)
g
X
g
Policy head
hs
Figure 2: The AlphaGateau network, ht is the inner size of the feature vectors, and L is the number of residual blocks.
h hs
Linear hs
Linear
Value
BNR
BNR
Att Pool
ReLU
tanh
Figure 3: Value head
Linear hs
Linear
Policy #actions
BNR
BNR
hs
Figure Policy head
where H R is global feature vector.
Batch Normalization and Non-linearity (BNR). As they are pair of operations that often occur, we group Batch Normalization and a ReLU layer together under the notation BNR:
BNR(x) (x) ReLU((BttcNorm(z))
(8)
Residual GATEAU (ResGATEAU). Mirroring the AlphaZero residual block architecture, we introduce ResGATEAU, which similarly sums a normalized output from two stacked GATEAU layers to the input:
ResGATEAU(h.g)- (h,g) + AATTTAATTAEEEERRAATTTTTNNL....L.
(9)
Experiments
We evaluate AlphaGateau's performance in learning regular 8 8 chess from scratch and generalizing from x variant to the standard chessboard. The metric used to evaluate models Elo rating, calculated through games played against other models (or players) with similar ratings. Due to computational constraints, we couldn't replicate the full 40 residual layers used in the initial AlphaZero paper, and experimented with and layer models. We also started exploring layers, but these models required to generate a lot more data, which would make the experiment run an order of magnitude longer. Our results indicate that AlphaGatcau learns significantly faster than traditional CNN-based model with similar structure and depth, and can be efficiently fine-tuned from 5 5 to 8 chess, achieving competitive performance with fewer training iterations. F All models used in these experiments are trained with the Adam optimizer [12] with a learning rate of 0.001. All feature vectors have embedding dimension of 1 128. The loss function is the same for the original AlphaZero, which is, for fo(x) म, C.
L L( = - (v-
(10)
"In Silver al. [8]. the training of AlphaZero is described in terms of steps, which each consists of mini-batch size 4096, while generation of games through self elay done in parallel by other TPUs. For our experiments, an iteration consists of generating 256 games through self-play, then doing one epoch of training. split into 3904 mini-batches of size 256, after 7 iterations once the frame window is full. In terms of positions iteration equivalent to 244 steps.

---

5.1 Implementation
Jax and PGX. As the MCTS algorithm requires lot of model calls weaved throughout the tree exploration, it is essential to have optimized GPU code running both the model calls, and the MCTS implementations of Gumbel MuZero, all our models and
search. In order to leverage the MCTX g
PGX [3] was used for the chess implementation, and we based our AlphaZero implementation on the PGX implementation of AZNet. We used Aim [ to log all our experiments.
experiments were implemented in Jax 3
and its ecosystem [[]] [
To estimate the Elo ratings, we use the statsmodels package [16 implementation of Weighted Least Squares (WLS).
Hardware. All our models were trained using multiple Nvidia RTX A5000 GPUs (Learning speed used 8 and Fine-tuming used 6), and their Elo ratings were estimated using 6 of those GPUs.
5.2 Evaluation
As each training and evaluation lasted a little under a week, we were not able to train each model configuration several times so far. As such, each model presented in the results was trained only once, and the confidence intervals that we include are on the Elo rating that we estimated for each of them, as described in the following.
During training, at regular intervals (each 2, 5, or 10 iterations), the model parameters were saved, and we used this dataset of parameters to evaluate Elo ratings. In this section, will call pair (model, parameters) a player, and compute a rating for every player.
We initially chose 10 players, and simulated 60 games between each pair of players, to get initial match data M. For each pair of players that played a match, we store the number of wins whys draws dij. and losses 4y: Mij (Wiy. dej. 4.). Using this data, we can roughly estimate the ratings r E I N Nloggers of the players present in M using linear regression on the following set of equations:
400
Wii + dis + L4g 1
1)
r3 - T2
log
for i M,j E Ma.
l10(10)
wiy + d1+1
(11)
2
Σ rr = | x 1000.
SEM
We artificially add one draw to avoid extreme cases where there are only wins for one player and no losses, in which case the rating difference would theoretically be infinite. This is equivalent to Jeffreys prior. The last equation fixes the average rating to 1000, as the Elo ratings are collectively invariant by translation.
We then ran Algorithm 2 to generate a densely connected match data graph, where each player played against at least 5 other players of similar playing strength. Finally, we used this dataset to fit a linear regression model (Weighted Least Squares) to get Elo ratings that we used in the results figures for the experiments. The confidence intervals were estimated by assuming that the normalized
match outcomes followed a Gaussian distribution. If fug way+ "sp"
+ley++ is the estimated probability that player i beats player j. we approximate the distribution that Pij follows as a Gaussian, and using the delta method, we derive that ry .r asymptotically follows a Gaussian distribution of L- - and variance log(10)
400 mean log(10) log pi
The proof is detailed in Using the WL.S linear model of statsmodels (16), we get Elo ratings for every player,
400
Appendix A.2 A.2
as well as their standard deviations, which we use in the following to derive 2-sigma confidence intervals.
5.3 Experiments
Learning Speed. Our first experiment compares the baseline ability of AlphaGateau learn how to play 8 x 8 chess from scratch, and compares it with a scaled down AlphaZero model. The AlphaZero model has 5 residual layers (containing CNN layers) and a total of 2.2M parameters, and the AlphaGateau model also has 5 ResGATEAU layers (containing 10 GATEAU layers) and total of 1.0M parameters, as it doesn't need an Nnodes x hs x Nactions fully connected layer in the policy head, and the GATEAU layers use 2/3 of the parameters a 3 x 3 CNN does.
7

---

Algorithm 2: Matching Players
Parameters: Ngames 60, Nam 128
Input: M
for player unmatched players do for <- to do
r <- (l)__R())
/+ The Linear Regression is run Eq. 11 */
opponent argmin jemr Ir, ppeyer ;
/+ If player M, set 'pleyer 1000 +/
(w, d, play (player, opponent, Noames; Naon);
Mp Mplayer Mplayer.opponent opporent Mlaaee Mplayer.opponeet apponent [w. d,));
M Mopponent.player M ppponent..player (l,d,w).
end
end
For this experiment, generated 256 games of length 512 at each iteration, totalling 131 072 frames, and kept a frame window of 1M frames (all the newly generated frames, and uniform sampling over the frame window of the previous iteration), over 500 iterations. During the neural network training, we used a batch size of 256. The training for AlphaGateau lasted 13 days and 16 hours, while AlphaZero took 10 days and hours.
We report the estimated Elo ratings with a 2-sigma confidence interval in Figure S AlphaZero was only able to reach an Elo of 667 + 38 in 500 iterations, and would likely continue to improve with more time, while AlphaGateau reached an Elo of 2105 + 42, with an explosive first 50 iterations, and achieving results comparable to the final Elo of AlphaZero after only 10 iterations.
Although those results are promising, it is important to note that we only compared to a simplified version of AlphaZero, using only 5 layers instead of the original 40, and without spending large efforts to optimize the hyperparameters of the model. As such, of AlphaZero could be greatly improved in this context with more parameter engineering. Both AlphaGateau and AlphaZero have not reached a performance plateau after 500 iterations, showing slow but consistent growth.
possible that the performance
Fine-tuning. our second experiment, we trained AlphaGateau model with 10 residual layers on 5 5 chess for 100 iterations, then fine-tuned this model on 8 x chess for 100 iterations. This model has total of 1.7M parameters.
For this experiment, we generated 1024 games length 256 at each iteration on x 5 chess, and 256 games of length 512 while fine-tuning on 8 chess, and kept a frame window of 1M frames. The initial training lasted 2 days and 7 hours, and the fine-tuning 5 days and 15 hours.
We report the estimated Elo ratings with 2-sigma confidence interval in Figure The initial training ended with an Elo rating of 807 + 46 when evaluated on 8 8 chess games, which suggests that it was able to learn general rules of chess on 5 5, and apply them with some success on 8 x 8 chess without having seen any x 8 chess position during its training. Once the fine-tuning starts, the model jumps to an Elo of 1181 + 50 after couple iterations, suggesting the baseline learned on 5 5 was of high quality. After fine-tuning, the model had an Elo of 1876 + 47, reaching comparable performances to the smaller model using roughly the same amount of iterations and GPU-time, despite being twice as big. Preliminary testing suggests that this model would become stronger if more data was generated at each iteration, but that would linearly increase the training time, generating self-play games took half of the 5 days of training.
5.4 Impact of the Frame Window and the Number of Self-play Games
In order to train deeper networks, we experimented with the number of self-play games generated at each generation, and with the size of the frame window. It seems from our results in Figure that having more newly generated data helps the model learn faster. However, the time taken for each iteration scales linearly with the number of generated games, and the model is still able to improve using older data. As such, keeping a portion of the frame window from previous iterations makes for good compromise. There are however options to improve frame window selection:
8

---

2000
1500
520
phalaazey
byye
50
150
150
2.00
2550
300
35.0
403
450
500
20
60 100
Btrraion
Itersso
Figure 5: The Elo ratings of AlphaZero and Alpha- Gateau with 5 residual layers trained over iter- ations. The AlphaGateau model initially learns ~10 times faster than the AlphaZero model, and settles after 100 iterations to comparable speed of growth to that of AlphaZero.
Figure 6: The Elo ratings of the first 100 iterations of AlphaGateau model from Figure was included for comparison. The initial training on 5 5x5 chess is able to increase its rating while evaluated on 8 8 chess during training, even without secing any 8 x 8 chess position. The fine-tuned model starts with a good baseline, and reaches comparable performances to the 5-layer model despite heing undertrained for its size.
Which previous samples should be selected? We selected uniformly at random from the previous frame window complement the newly generated samples, but it might be preferable to select fewer samples, but chosen as to represent a wide range of different positions.
Past samples are by design of dubious quality. As they come from self-play games with previous model parameters, they correspond to games played with a lower playing strength, and the policy output by the MCTS is also worse. Keeping sample that is too old might cause drop of performance rather than help the model learn. We experimented little with keeping the 1M most recent samples, but with little success.
We also initially tried to increase the number of epochs in one iteration, but only saw marginal gains, suggesting that mixing new data among the previous frame window helps the model extract more training information from previous samples.
2000
1500
Elo Razing
500
games/1Blk
Alsh aditees gamee/1M window
Alph aCateau 1024 gamexy52 4k wit wirdom
-500
20
60
100
Beeation
Figure The two models frame window. The model keeping no frame window and generating 256 games was trained in only 39 hours, but had the worst performance. Adding 1M frame window improved the performance little and lasted 60 hours, while increasing the number of self-pla play games to 1024 performed the best, but took 198 hours.
window
131 only kept latest generated games in

---

Conclusion
In this paper, we introduce AlphaGateau, a variant on the AlphaZero model design that represents a chess game state as a graph, in order to improve performance and enable it to better generalize to variants of game. This change from grid-based CNN to graph-based GNN yields impressive increase in performance, and seems promising to enable more research on reinforcement-learing based game-playing agent research,
reduces the resources required train one.
We also introduce a variant of GAT, GATEAU, that we designed in order to handle edge features in a simple manner performed well, and efficiently.
Future Work. As our models were relatively shallow when compared to the initial AlphaZero, would be important confirm that AlphaGateau still outperforms AlphaZero when both are trained with a full 40-deep architecture. This will require lot more computing time and resources.
As discussed in Section 5.4, our design of the frame window is little unsatisfactory, and a future improvement would be to define an efficiently computable similarity metric between chess positions, that helps the neural network generalize.
We focused on chess for this paper, but there are other games that could benefit from this approach. The first one would be shogi, as it has similar rules to chess, and the promising generalization results from AlphaGiteaa could be used either train model on one game, and fine-tune it the other, to jointly train it on both games, to have more generalized game-playing agent. As alluded to in the Graph Design 4.2 more features engincering would be required to have node and edge features compatibility between chess graphs, and shogi graphs. It could also be possible to change the model architecture to handle games with more challenging properties, such as the game Risk, which has more than 2 players, randomness, hidden information, and varying maps, but is even more suited to being represented as graph.
Acknowledgments and Disclosure of Funding
This work was supported by JST BOOST, Grant Number JPMJBS2407 and JST CREST, Grant Number JPMICR2IDI. We thank Armand Barbot and Jill-Jênn Vie for their helpful comments and insightful discussions.
References
[I] G. Arakelyan, G. Soghomonyan, and The Aim team. Aim, June 2020. URL com/ Com/aimhubio/attn
[2] S. Ben-Assayag and R. El-Yaniv. Train on Small, Play the Large: Scaling Up Board Games with AlphaZero and GNN, July 2021. Uttp wwwpw//.wrrgaaaa//.....o. 08387 arXiv:2107.08387 [es].
[3] Bradbury, Frostig, P. Hawkins, M. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman- Minne,, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL pettt:://ithhn httggg/tuub..oooooggeeeeeeeeeeee
com/google/jaa
[4] Chen and H. Chen. Edge-Featured Graph Attention Network, Jan. 2021. URL http
7/arxiv.
www./ogg///b//eeeeeeeoo aa//abbb/21101 11111.00777111 arXiv:2101.0761 [cs].
[5] R. Coulom. Whole- History Rating: A Bayesian Rating System for Players of Time- Varying Strength. In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, Mattern, J. C. Mitchell, M. Naor, O. Nierstrasz, C. Pandu Rangan, B. Steffen, M. Sudan, D. Terzopoulos, D. Tygar, M. Y. Vardi, G. Weikum, H. Van Den Herik, X. Xu, 7.. Ma, and M.
Winands, editors, Comput- ers and Games, volume 5131, pages 13-124. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. ISBN 978-3-540-8788
778-3-54008760888 doi: 10. 1007/978-33 87608-3 11. Series Title:
URL hnttps://ling Lecture Notes Computer Science.
[6] Czech, Korus, and K. Kersting. Monte-Carlo Graph Search for AlphaZero, Dec. 2020.
URL TRRLhtps::/ httts::/ta/ariiii org/ abtp 11045v1
10

---

[7] I. Danihelka, A. Guez, J. Schrittwieser, and D. Silver. Policy improvement by planning with Gumbel. Oct. 2021. URL / htt.:eeeeee/eoorrriooaaaaaaoooo
[8] DeepMind, Babuschkin, K. Baumli, A. Bell, Bhupatiraju, J. Bruce, P. Buchlovsky, D. Bud- den, T. Cai, A. Clark, I. Danihelka, A. Dedieu, C. Fantacci, J. Godwin, C. Jones, R. Hemsley, Henmigan, M. Hessel, S. Hou, S. Kapturowski, T. Keck, I. Kemaev, M. King, M. Kunesch, L. Martens, H. Merzic, V. Mikulik, T. Norman, G. Papamakarios, J. Quan, R. Ring. F. Ruiz, A. Sanchez, L. Sartran, R. Schneider, E. Sezener, Spencer, Srinivasan, M. Stanojević, W. Stokowiec. L. Wang. G. Zhou, and F. Viola. The DeepMind JAX Ecosystem, 2020. URL http://github. com/ deepmind
[9] J. Godwin T. Keck* P. Battaglia, V. Bapst, T. Kipf, Y. Li, K. Stachenfeld, P. Veličkovié, and A. SanchzzGonnzllez. Jraph: A library for graph neural networks in jax., 2020. URL
http http://tthubb.com//eepmmm.dd/ddd
[10] L. Gong and Q. Cheng. Exploiting Edge Features in Graph Neural Networks, Jan. 2019. URL http::/arxivv.rr// w8tp:/asss1110.000 Ttt.//18990000000
NTf1::809.027000 [cs, stat].
[11] Heek, Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. v. Zee. Flax:
A neural network library and ecosystem for JAX, 2023. URL http://github. com/ google/
flax
[12] D. P. Kingma and Ba. Adam: A Method for Stochastic Optimization, Jan. 2017. URL http://arxiv.. org/ abs 1412. 6980 arXiv:1412.699 [es].
[13] Koyamada, Okano, S. Nishimori. Murata, K. Habara. H. Kita, and Ishii Pgx: Hardware--Accllrrated Parallel Game Simulators for Reinforcement Learning, Jan. 2024. URL 17503 arXiv:2303.17500 [es].
[14] Y. Li, D. Tarlow, M. Brockschmidt, and Zemel. Gated Graph Sequence Neural Networks, Sept. 2017. URL
htw.aa////1111111....... arXiv: 151 151v:1511.00593 [cs, stat].
[15] J. Schritwieseer, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering Atari, Go, Chess and Shogi by Planning with Learned Model. Nature. 588(7839):604-600 Dec. 2020. ISSN 0028-0836. 1476-4687. doi: 10. 1033/04/ 1581-0286---000 -286-000-0001-4,, URL BtLh:tp::/ arxiv awaaaass9999 08265 arXiv:1911.088. 1.0222655 [es, stat].
[16] S. Seabold and Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference, 2010.
[17] D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalch- brenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature, 529:484-449 Jan. 2016. doi: 10. n..103/naaure1690
[18] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap. K. Simonyan, and D. Hassabis. Mastering Chess and Shogi by Self-Play with General Reinforcement Learning Algorithm, Dec. 2017. URL https://arxiv. wwwrrxivvorrgreee abs/ 1712. 01815v11
[19] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph Attention Networks, Oct. 2017. URL https: wwwpexivivoorvaaeeoe /alb///////0 10903v3
[20] K. Xu, W. Hu, J. Leskovec, and Jegelka. How Powerful are Graph Neural Networks?, Oct. 2018. http:s///araii.oor//////11.000.

---

A Appendix supplemental material
A.1
Elo
The Elo rating system was initially introduced as a way to assign a value to the playing strength of chess players. The rating of each player supposed be dynamic and be adjusted after each game they play to follow their evolution.
The Elo ratings are defined to respect the following property: If two players with Elo rating RA and Ry played a game, the probability that player wins
Ea
(12)
Rg RA
1
10
100
A.2 Variance of estimated Elo rating difference
We assume that the outcome of game between two players of Elo RA and R3 is a Bernoulli trial, with a probability that player win being given by the central Elo equation [2] If we want to estimate the Elo of both players, we need to estimate that probability p. To do so, we can make the two d In the case that one of the two
players play games, and sum the wins of player A, as well as half his draws, to get From this, we can estimate the value of p using the estimator ý x. players is significantly stronger than the other, could be close to O or 1, in which case this estimator wildly inaccurate. To remedy this, we will instead rely on a Jeffreys prior, to get the estimator
n'
m+1 We will note this estimator P in the following. n=1
PJef frefsreys
From our assumptions, we have that x follows a binomial distribution B(n,p), and we will approxi- mate the distribution that ý follows by a normal distribution ý N>N(p. p(1-2)).
By inverting the Elo equation 12 can get the rating difference from the probability that A wins as
400 [_ -1
R RA
log(10) log
(13)
Therefore, posing g(v) get that (()) is asymptotically Gaussian. The derivative of is
400 log 1 which is differentiable, we can use the delta method to log(10)
400
g'(v) log(10)
1/2
400
1 y
log( (10) y/2 y
400
1
(14)
log(10) v(y 1)
which gives
Rn - RA= g(() ~ ~N (()).
p( p(1
(o((T))²)
400
1
p(1-p) -p)
400
N
log
-
log(10)
p
n
(0g(10) p(p
400
400
p(1 -1-p) p)
~
log
log(10)
p
log(10)²
(p(p- 1))²
~
400
log
- 1):
400
(15)
log(10)
p log(10) np(p
A.3 Comparison with BayesElo
We also used BayesElo [5] to evaluate the Elo ratings of our models using all the PGN files that were generated recording the games played between all our models. In Figure we plotted point
12

---

Elo difference between our model and elobayes
102
209
102
500
500
1000
1500
2000
2500
Figure The difference between BayesElo ratings and the Elo ratings according to method. We removed each the average all players in respective method, such the average effective Elo for both BayesElo and our method is
2020
2000
being
1000
1000
550
500
Aip 12zro Alphaaaacco0
AlanaZere AbpaaGateau
-500
-500
Figure 9: copy of Figure for comparison.
Figure 10: Using running time instead of iteration for Figure doesn't little bit Taster than AlphaGateau.
doesn'' 't change much, as AlphaZero is only
on (x, .) for each player where its Elo rating following our method and y is the difference in predicted Elo between our method and BayesElo.
In practice, weaker models tend be overrated by method compared with BayesElo, while stronger models are underrated. As this relation is smooth and monotone, this suggests that both methods order the relative strength all players similarly, with small differences only being due to noise. The main difference being that our range of Elo ratings is compacted towards the extremes, which we assume is due our practical Jeffreys prior, that implies that strong models drew least one game against all their opponents, handicapping their Elo, and similarly weak models always manage lose least one game.
A.4 FLOPs comparison
We didn't 't record the FLOPs of our models, however, we did record the time taken for each iteration. In Figure 10 12 and 14 we plotted the three main plots of the main paper using that data on the X-axis. some experiments were run using GPUs instead we multiplied their recorded time 6 by factor g for better comparison.
A.5 PGNs
We include with Figure 15
16 and 17
few PGNs showing games played by our models.
13

---

2020
2000
AlphaGatesu byes layers
phalaazay
lnyers
byyn
530
20
a0
360
20
60
100
Iteratin
Time thour)
Figure 11: A copy of Figure 6 for comparison.
Figure 12: Using running time instead of iteration for Figure[5] Training deeper model takes roughly 40 hours longer, for a similar amount of generated games and training steps.
2000
2000
1,000-
500
500
Alphodattzu gameertB
Alrnaaatizu 256 gumesl window
Abhoaaateau gimesrJM
AlanaGateau gumees
AlttaCaasau gamear5.24 24i window
Alonaaateau garmen12244 wndow
20
200
100
Steration
Time Chour
Figure 13: copy Figure 7 for comparison.
Using rumning time instead of iteration Figure [ shows that although using more newly generated games instead relying on frame window of previous data makes the model improve more per iteration, does result in slower training practice.
Figure 14:
Figure 15 contains the PGN of a game played on x 8 chess by the last iteration of the x 5 AlphaGateau model of fine-tuning experiment, showcase its playing style while having never seen an 8 x 8 board during its training.
Figure 16 and 17 contains the PGNs of games played by the last iteration (iteration 499) of the full AlphaGatcau model of our first experiment as white and black respectively, selected among the games played against other models evaluate all the Elo rankings, selected using the script book.py the GitHub repo, following the most played move each ply. The game played as white in Figure [6] is played against the iteration 61 of fine-tuned AlphaGateau model using 8 layers, 1024 generated games per iteration, but only 32 MCTS simulations, with an estimated Elo rating of 2124. The game played as black in Figure 17 is played against the iteration 439 of the same model.
A.6 Lichess Evaluation
We ran the final iteration of the 5-layer AlphaGateau model on Lichess, as the bot AlphaGateau hottps://lichess www/e///////// Altp/halhaaaaeaa We let it play against other bots and some human players in bullet, blitz, and rapid time formats by varying the number of MCTS simulations to adjust the time required to play each move. This implemented using the default Lichess bot bridge (https: thub. com/ http//lhhhhssseeeeeeeeeee the lichess folder of our GitH repo
--icchss--ot--dweeee bot- btt-dbbs/ddddeeeeeeessseeeeeeeee bot), and the relevant code is in
At the end of October 2024, after between 100 and 200 games per time format, AlphaGateau was able to reach bullet Elo of 1991, blitz Elo of 1829, and rapid Elo of 1884.
14

---

1.e4 3. |ecchh....aaagggaa21077737334444 10. 2xg
11.c
13, h1gaaaaaaaaaaa111111.a11......1.
gxh5 18.
Qg8 19. f6
24. 3g7
25. Exd4 h2+ 26. $3 h1 21 3d3 wg 29. 29. Eg7 30. hhl Se2
757 32. Eg3 8 33.
3d8
Hal 177 36. 3x 38. 38. Rg7
39. c22 7767 40. We 3e8
41.
h22 42. d3 h1 43, Mf4 0xe3 44.
3xe3 Rgl 45. Re2
Exd4 46. f22 Mxf4 47. xxh1 5h4
51. 4h8 412 dd 49. h5 Eg3+ 50. d2 Exc3+
52. b6 d6 53. bxaó 54. d8
55. xh4 g8 56. bxcó Exc6 57. 14 &8. xx6+
d5 59. W15+ 344 60. W3
2b 61. xh5 xaa 62. Mxa6+ 2b 63. 1 Eh8 64. Ea7 2d4
65. +b &b 66. Dd3+ $24
67.
e7 h5 68. 16 28 69. 14+ 5b 70. He5+ 9b 71.
c 72. De4 Ee 73. 3c3
74. d4 c6 75. 3g b6 76. 347 bb 77. 3 Sc 78. $d
h4 79. De2 h3 80. 14 d7 81. 2g3 7 82. h dd 83. Exh8 h2 84. Xxh2 2c7 85. 1 5c8
86. f5+ 67 87. T d7 a 88. 5 48 89. Eh6 2a7 90. Et6 2g 91. Er 88 92. Ers#
Figure 15: Game played by a model fully trained only on 5 x 5 chess as white. White is able to use their white bishop to eliminate black's 's white bishop, but seems to undervalue their knight on move 14, probably because it is worse piece in 5 5 chess duc to being more constrained and harder to effectively employ
e. c5 2. Dblc3 e6 3. ggf b886 4. d4 ffdd g88f 6. 0d4c c6 7. e5 16dd 8, 0c3e4
d8c7 f4 c7b 10. ff 886 11.
2226 6666 12. a3 h5 13. T dle2 w a662 14. $ele?
15 15. De4c3 a5 16. Dc3d5 d5 17. &cle3 ff877 18. c3 Eh8g8 19. $e2B3 g5 20. g3 g1. $B322
21,
h4 22. b4 $37817 23. Halel h3 24. âe3bó b4 25. b4 Ea8a3 26. Eclc7 Eg 27. bb6e5 Ae7c5
28. 77e5 Mb8b4 29. 2c5cc d4 30. Hhidl $17e7 31. $e22 MaM3 32. 822gl d3 33. E22e3
Kh4b2 3c3d3 3f3d 35. Edid3 Xb2c2 36. Ed3d1 Ec2g2 37. @glhl Zg2a2 38, hhigl
#77e8 dd.b1 Ka2g2 40. #glh1 Rg2c2 41. higg Mc2c3 Eb1b8 $e8e7 Eb8b1
Rc3c2 44. 4b177 He2el 45. 1g1f Zelhl 46. ffee2 Ehth2 47. se21 Eh2g2 48, Eb7h3
Bg2d2 49. (flgg 3e717
Hb3b7 6778 5b7b8 $187 52. 5g. Ed2g2 53. Mb8b7
67777 54. Kb7d7 3e7e8
55. 55.03
5g.2
Id3d6 &e887 5d.66 2
58.
58. Me6c7 2e7d8
59. kd8d7 60.
Ec6d6 dd7c7 61. 6h.2h1h2 EBअ 62. gh2h1 1123
63.2 Mf3
64. sh2h1 Rf2g2
6d.
Egle2 66.
Mc2c3 67. 5h1h2 Me3a3
dd4d2 Ma3a6
dd2e Ma6a5
70.
Maaal Hd2b2 Hala8 72. 9h2h1 Ea8c8
hlll Eea88
74. bb2b6 se7d7
7b6d6 dd777 76. &glhl Ea8a3
77. 2h1h2
Ha3a2
78. hh2hl
2aaa7
79. hlll Maaal 80. 8g1h2 Eala2 81. 6h2h1 Hale2 82. dd6d
Ee2g2 83. dd4d3
2e7e8
84. 333a3 Egle2 85, Ea3d3 eelll 86. $h1h2 Eele2 87. hh2h1 4e2h2 88. Ed3e3 Eb2a2
89. 333c #e8d7 90. Eddd6 d777 91. d66c Ha2e2 92. 3c6c3 He2g2 93. 3c3d3
Figure 16: AlphaGateau starts with a closed Sicilian, transposing into the Four Knights Sicilian, following a popular line until move 10, when white moves its white bishop to c2. The pawn structure locks the situation by move 35. Nothing much happens before the game ends in draw, besides an interesting stalemate trick on move 54
1. e4 e5 2. 41 De 3. Abs 11 4. d3 Ac5 5. Åxcó dxc6 6. 6.0- W 7. Ag 0-0 8, 4h4 h6
9. Dbd2 b5 10. eel a5 h3 P6 12. &g3 Ee8 13. 4xe5 a4 14. 2c3 4hs 15. 414 16. h2
17. d1 Mss 18. ext5 xxf5 19. Wel 17 20. g4 Wc5 21, 4e4 da 22. Egl Deb 23. Th4 Øg5
24. 46 gxf6 25. De 26. Eg3 d4 27, e4 2 Eel Hes 29. g2 Exel 30. Axel
Weo 31. A.2 4e2 32. âxb6 cxh6 33.15 We 34. 013 xxg3+ 35. xg3 0xg3 36, 5xg3 2c8
37. 214 Ed7 38. dd2 $17 39, De4 hab
h2g5 42.
fxg5+
hxg5+ 43. 0xg5+ $g7
44. 3e7+ 45. Deó+ $17 46. dd 2c8 47. 4g5+ $1
966 48. De4+ 817 49, 0g5+ 818 f0
Ed7+ 51. 36 04 52, Den+ 777 53. 01 exf6 54, 2d5+ 3e 55. 4xb6 cxd3 56. cxd3 ab7+
57. xbb Exd3 58, 0xa4 59. $44 Th3
2 62. a4 aeb 63. dd ab
a5 kh4 65. $5 Exb4 66. a6 96 67. 2h 4c2
68. 3
ab 69. De2 $e5 70. 44 Ea4
71. Dd3+ 636 72. 214 234 Gg 4xa 74. 414 4c475. 0g2
4b 76. 114 de2 77. De6 Ee4
78. 15 Mel+ 79. 16 dd 80. 244 gg 81.
De2
M11 82, &g5 5e 83. 0g 717 84, 4hs Ad3
85. g3 Eg7+ 86. h4 gg 87. h3 2 88. 4h5+ &g 89. Og3 th8+ 90. 2g2 $14 91. Drl
Eb8 92. 2g #b2+ 93. hh gg 94. ah5+ Axh 95. h4 hh2#
Figure 17: AlphaGateau starts playing a Berlin defence without any book, and diverts by move 4 into popular line, with a rare bishop forced to give up its remaining rook to stop the attack, leaving black up rook for pawn.
move 7. The middlegame revolves around black strong knight, until white is
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The results discussed in the experiments and illustrated in Figures S and support the claims in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Our main limitations are that we had limited computing resources and had models of depth 5 or 10 when 40 would be better, and we only focused chess and not other games. We could also only run the experiments once in the time available to us, so the confidence intervals only apply the rating estimations, and were extimated over several training runs.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
16

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The only theoretical result is the closed form derivation of the confidence interval for the estimated Elo ratings, included in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We included all the experimental hyperparameters and methodology in the paper, and include all the code that was used in the GitHub repository mentioned in the abstract. However, as the parallelized GPU code not fully deterministic, is not possible to replicate the exact training results present, so we will also publish in the git repository some important model parameter checkpoints that we saved.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
17

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The full code published in the GitHub repository Akulen//lppaGGateau.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/guides/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https //nips.ccpputt wttpcccccppccccceee copollccggiieee wwwedeeeeeeeebbmeeeeeeee
ssionollccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: The full training procedure is described in the paper, and the provided code contains the implementation details, such the gpu data split procedure.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide confidence intervals on our estimated Elo ratings, however, as each model was only trained once, they are incomplete confidence intervals.
Guidelines:
18

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the number and type of GPU, as well the experiment run time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://pp....aao....o//oo/////er
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
19

---

We introduce improvement to an existing DRL architecture to train a game agent. There should be no direct societal impact from this, as the results are currenty limited to making research on this topic more accessible, due to the training efficiency gains.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
The released models are way less powerful than current existing chess engines, as they are limited in size and depth for compute reasons, so there is no risk in publishing them.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite every non-standard python library used to develop and run the exper- iments presented in this paper. We do not use any previous dataset or asset besides those libraries.
20

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The Git repo that will be published along the paper will have well structured code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
21

---

Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
22


Reviews:
[{"review_id": "CEIhIXhCTU", "reviewer": "Reviewer_NRN3", "summary": "The authors demonstrate a GNN that works on chess and is amenable to generalization.", "questions": "Is the MCTS process identical between AlphaGateau and AlphaZero models? Or does the graph representation change it?\n\nCould the graph representation be leveraged to allow a different search algorithm?\n\nCould the authors provide PGNs with model predictions and some discussion of the difference in the AlphaGateau and AlphaZero models? There are patterns that CNN based models are good at learning and some that are more difficult (see [1] for examples) and showing differences in the models would strengthen this result.\n\nDoes the model handle the white bishop well when scaled up? If it does that would be additional evidence of generalization when scaling up since the moves would not have been seen before.\n\nCan you provide more details on the Elo calculation? You use the terms ELO and elo and seem to refer to a varation of the Elo rating system created by Arpad Elo, but don't give much details beyond the optimizer. Are you using and Elo or Glicko (2) type system or something else? Section A has not citations so it's unclear. \n\n[1] McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Wattenberg, M., Hassabis, D., Kim, B., Paquet, U. and Kramnik, V., 2022. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47), p.e2206625119.", "limitations": "Discussed above", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "# originality\n\nThis is the first chess GNN approach that performs well that I'm aware of in the literature, and requires some invoations in the edge representation. \n\n# quality\n\nThe results look good, although the computational limitation of the work mean that a larger test would be good to demonstrate that this model scales up.\n\n# clarity\n\nThe paper is clear and readable.\n\n# significance\n\nThis is a new result in chess AI and looks like it could matter to the larger RL community. I'm not sure this will impact much chess engines in practice though as the GNN is likely much harder to optimize and implement than CNNs or heuristics, the memory usage on the attention mechanism for example would make implementing this in a high performance search code-base difficult.", "weaknesses": "The authors only look at self-play for model training, the original AlphaGo models were trained on human games to establish that the algorithm performed well, before starting the much more expensive self-play training. The lack of supervised training evaluations is makes the claims of strong performance limited to a single axis (sample efficiency).\n\nThis result mostly seems to arise from chess specific optimizations applied to an existing algorithm [18], while the authors suggest that it could be applied to other games they only provide evidence in chess.\n\nThe fine tuning results require more training time (and thus I'm assuming compute) than the initial training, this suggests the pre-training while beneficial is of limited efficacy unless it can be made more efficient.\n\nI'm also concerned with the  attention-based pooling leading degrading in performance when the graph gets larger as chess is a game that cares much more about the max than the average.\n\nI would have liked to see a more complete training run, but I understand that RL is very compute intensive so am not using this lack to negatively impact my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Remarks\n\n> The results look good, although the computational limitation of the work mean that a larger test would be good to demonstrate that this model scales up.\n\nWe are aware that the reduced scale compared with the original AlphaZero is an issue, we have since ran a fine-tuning experiment with a model of depth 8 that will replace the model of depth 6 in the initial paper. The corresponding plot is included in the global PDF\n\n\n> the memory usage on the attention mechanism for example would make implementing this in a high performance search code-base difficult.\n\nWe do not compute attention between each pair of node, but only when an edge is already present. As such, it shouldn't represent a significant memory sink, as there is only one attention coefficient per edge, which already have a feature vector of size 128.\n\n\n> This result mostly seems to arise from chess specific optimizations applied to an existing algorithm [18], while the authors suggest that it could be applied to other games they only provide evidence in chess.\n\nYes, this paper only experimented on chess. It would be relatively straight-forward to extend the model to Shogi, as the rules are quite similar besides piece-dropping, which could provide insights to the capacity of the model to learn general game concepts if an hybrid chess/Shogi model was trained in some way.\nWe also plan to look into extending this model to the game of Risk, which is even more suited to graph representations, but several challenges will need to be solved, as Risk has more than 2 players, hidden information, and randomness, which are not currently well handled by AlphaZero (although there exists extensions that focus on those issues, like DeepNash)\n\n\n> The fine tuning results require more training time (and thus I'm assuming compute) than the initial training, this suggests the pretraining while beneficial is of limited efficacy unless it can be made more efficient.\n\nThe fine tuning on 8x8 chess is slower than the initial training on 5x5 chess, but the fine-tuning starts with a model that is around 1000 Elo point higher than a random initial model, which shows that it can transfer the learning from a smaller variant to the regular variant quickly. It still requires a significant amount of training to achieve optimal performances, but it should be possible to train a large model with strong hardware to provide a good baseline that can be used for further fine-tuning with more modest hardware on a wide range of chess variants, such as chess960, king of the hill, or other (in a similar way as how LLM models are used).\n\n\n> I'm also concerned with the attention-based pooling leading degrading in performance when the graph gets larger as chess is a game that cares much more about the max than the average.\n\nI'm not quite sure I understand the issue raised. In our experiments, the graph remains relatively small, containing 64 nodes for 8x8 chess and 1858 edges (25 nodes and 455 edges for 5x5 chess).\n\n\n> I would have liked to see a more complete training run, but I understand that RL is very compute intensive so am not using this lack to negatively impact my score.\n\nThe original AlphaZero paper trained for 700 000 \"steps\". However, this is confusing as we also use the term step in our paper (and will clarify this point in the paper) but applied to something else. AlphaZero's steps are equivalent to batches in our paper, and we evaluate between 500 and 1000 batches in one of our step (depending on the size of the frame window). As such, our 100 steps are comprised of up to 100 000 batches. This is still an order of magnitude less, but our model is also around an order of magnitude smaller. We however are also working on a base AlphaZero run with 500 steps to assess whether it tapers off, or continues to increase in performance, after reading the reviews.\n\n# Questions\n\n> Is the MCTS process identical between AlphaGateau and AlphaZero models? Or does the graph representation change it?\n\n> Could the graph representation be leveraged to allow a different search algorithm?\n\nYes, we use the same MCTS algorithm as for Gumbel MuZero, which uses the value and policy neural network oracle as a black box. We did not find a way to use the graph representation to improve it.\n\n\n> Could the authors provide PGNs with model predictions and some discussion of the difference in the AlphaGateau and AlphaZero models? There are patterns that CNN based models are good at learning and some that are more difficult (see [1] for examples) and showing differences in the models would strengthen this result.\n\nWe did not analyse in detail the AlphaZero model as it performed quite poorly. For illustration, after 170 steps, the AlphaZero model distribution of first moves over 240 games as white is ` b2b3: 48, Ng1f3: 44, f2f3: 35, g2g4: 26, Nb1a3: 25, b2b4: 19, c2c4: 13, Ng1h3: 9, a2a4: 4, c2c3: 4, h2h3: 3, d2d4: 3, d2d3: 3, a2a3: 2, Nb1c3: 1, g2g3: 1`\n\n\n> Does the model handle the white bishop well when scaled up? If it does that would be additional evidence of generalization when scaling up since the moves would not have been seen before.\n\nThe model seems to have a reasonable grasp of 8x8 chess, even when it was only trained on 5x5 positions. We can see from its distribution of first white move ` b2b3: 16, e2e4: 15, g2g3: 13, a2a3: 12, h2h3: 11, f2f4: 11, c2c3: 11, a2a4: 11, e2e3: 9, d2d3: 8, h2h4: 7, f2f3: 7, b2b4: 6, g2g4: 5, c2c4: 3, Ng1f3: 2, d2d4: 2, Nb1c3: 1`, that it is drawn to playing double pawn moves, which weren't available in 5x5 chess. We also included a PGN of a game it played as white where it was able to use and understand the white bishop, but also probably undervalued the knight as it was quite a restrained piece in 5x5\n\n\n> Can you provide more details on the Elo calculation?\n\nWe use the base Elo rating model, but instead of updating the ratings after each game, we periodically run a linear regression following Eq (11) to fit Elo ratings on the players.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "183ppppFnt", "reviewer": "Reviewer_R1p6", "summary": "The paper explores a novel approach to  reinforcement learning for Chess by utilizing a graph-based representation of the game state instead of the traditional grid-based representation. This method is based on GNNs  and aims to overcome the limitations of CNNs used in previous models like AlphaZero. Specifically, the paper introduces the Graph Attention neTwork with Edge features from Attention weight Updates (GATEAU) layer, which enhances the classical GAT layer by incorporating edge features.\n\nThe primary contributions of the paper include demonstrating that the new architecture outperforms previous models with a similar number of parameters and significantly accelerates the learning process. Additionally, the paper provides evidence that the model trained on a smaller variant of chess (5x5) can be quickly fine-tuned to perform well on the standard 8x8 chessboard, indicating promising generalization capabilities. The authors have made their code available, supporting the reproducibility of their experiments and findings.", "questions": "1. How does AlphaGateau compare to other graph neural network-based reinforcement learning models, such as those presented by Ben-Assayag and El-Yaniv (2021) [1]?\n2. Have the authors considered applying AlphaGateau to other strategic games beyond chess? If so, what preliminary results or observations can the authors share?\n3. What potential real-world applications besides chess do the authors envision for the graph-based approach proposed in this paper?\n\n[1] Ben-Assayag, S., & El-Yaniv, R. (2021). Train on small, play the large: Scaling up board games with alphazero and gnn. *arXiv preprint arXiv:2107.08387*.", "limitations": "The authors have acknowledged some limitations of their work, such as the restricted computational resources which prevented the training of deeper networks, and their focus solely on chess without extending experiments to other games. They also discussed challenges related to the reproducibility of their results due to the non-deterministic nature of parallelized GPU code. However, the paper could benefit from a deeper discussion on the broader implications of these limitations, particularly how they might affect the generalizability and robustness of the proposed method. Regarding potential negative societal impacts, the paper does not explicitly address this aspect.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Novel Graph-based Approach**: This paper introduces a creative way to represent game states using a graph-based model instead of the traditional grid-based models. By employing Graph Neural Networks and the Graph Attention neTwork with Edge features from Attention weight Updates, the authors address significant limitations found in Convolutional Neural Networks used in models like AlphaZero.\n\n**Improved Learning Efficiency**: The proposed AlphaGateau model demonstrates much faster learning compared to traditional CNN-based models. Experiments show that AlphaGateau can achieve a significant increase in playing strength in a fraction of the training time required by AlphaZero, which is a notable improvement in learning efficiency.\n\n**Reproducibility**: The authors have made their code publicly available, supporting the reproducibility of their experiments.", "weaknesses": "**Limited Generalization Evidence**: The paper claims promising generalization capabilities, but the experimental evidence is limited to a small variant of chess (5x5) and standard chess (8x8). It would strengthen the paper to include additional experiments on other games or variants, such as Shogi or other board games with similar complexity, to demonstrate the broader applicability of the proposed architecture.\n\n**Comparative Analysis**: The paper lacks a thorough comparative analysis with other state-of-the-art models beyond AlphaZero. Including comparisons with recent advancements in graph-based reinforcement learning models would provide a clearer picture of the relative performance and innovations of the proposed method.\n\n**Real-world Applicability**: While the focus is on reinforcement learning in chess, discussing potential real-world applications of the proposed graph-based approach would broaden the impact of the work. Highlighting areas where this approach could be applied, such as other strategic games or decision-making problems in different domains, would make the contributions more compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1. How does AlphaGateau compare to other graph neural network-based reinforcement learning models, such as those presented by Ben-Assayag and El-Yaniv (2021) [1]?\n\nWe are not aware of GNN-based RL models that can be applied to chess besides a slightly adapted version of ScalableAlphaZero (from Ben-Assayag and El-Yaniv (2021)). ScalableAlphaZero differs from AlphaZero mainly in the fact that it replaced the CNN layers with GIN (Graph Isomorphism Network) layers, using a chess king grid as the graph. This should mainly be less expressive than AZ as the convolution kernel cannot treat the 8 surrounding squares differently, as opposed to how a CNN convolution would. The main advantage of ScalableAlphaZero is that it is able to scale, which allow samples of differently sized board to be given during training and testing. This allows for training on scaled down variants of Gomoku and Othello.\n\nAs such, we believe that ScalableAlphaZero would perform similarly to AlphaZero under the same constraints, or possibly a little worse, due to the more complex functions to evaluate, and lower expressivity. We were not able to perform a fine-tuning experiment with ScalableAlphaZero to compare with AlphaGateau yet as the ScalableAlphaZero code is not available, and the model would require substantial changes to adapt to the action space of chess being linked to the graph edges rather than the graph nodes. We did not compare AG's performances to ScalableAlphaZero's on Othello or Gomoku either as those games do not seem like they would benefit much from a graph based representation.\n\n\n> 2. Have the authors considered applying AlphaGateau to other strategic games beyond chess? If so, what preliminary results or observations can the authors share?\n\nWe have considered applying AlphaGateau to Shogi and Risk. We didn't test Shogi yet as we were more familiar with Chess while lacking knowledge and intuition for Shogi, and so could more easily interpret the results and play patterns of the model in Chess. We plan to address Risk in the future, but several challenges must be solved first, including handling more than 2 players, randomness (which can be handled by improvements to AlphaZero such as DeepNash), hidden information, and more complex turns.\n\n\n> 3. What potential real-world applications besides chess do the authors envision for the graph-based approach proposed in this paper?\n\nWe believe that these kind of methods help handling graph based tasks, and could potentially be used as a basis in this kind of adversarial setting to electric network optimizations, or traffic minimization, for example.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9rLYqq7kfv", "reviewer": "Reviewer_abiC", "summary": "The paper introduces a new variant of AlphaZero, AlphaGateau, with a neural network architecture based on GNNs, that allows the model to generalize across board sizes. The new architecture outperforms the original AlphaZero architecture under certain conditions.", "questions": "- What hyperparameters are used to calculate Elo scores? (number of models tested, number of games between each model pair, who played against who)\n\nThree minor suggestions:\n- It would be helpful for the reader if figures 5 and 6 had FLOPs on the x-axis instead of steps, or if training compute was reported. Plotting only steps, it is hard to understand if one training step of AlphaGateau is comparable to a training step of AlphaZero in terms of compute.\n- It would be easier and more convincing to use traditional methods for calculating Elo scores, such as BayesElo which was used by Silver et. al. That said, the authors provide clear explanations on their Elo calculation method.\n- The correct spelling of the metric is \"Elo\", named after Arpad Elo, rather than \"ELO\", which is a common typo.", "limitations": "No limitations other than those specified above regarding evaluation.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "The paper follows a promising direction, using GNNs in an AlphaZero-like model to utilize the graph nature of many board games. \nAlphaGateau is an original model that shows potential in terms of data efficiency and fast training. It learns much faster than AlphaZero during the first 100 training iterations, saturating at a much higher Elo compared to that of AlphaZero, which does not improve much in this timeframe.\nIt is clear that when both models are tested on equal grounds, in the setting used, AlphaGateau is significantly more data efficient, and improves its Elo score much faster than AlphaZero. These are significant improvements on the original AlphaZero model if they hold under rigorous evaluation.", "weaknesses": "Evaluation & comparison with AlphaZero: \nWhile figure 5 clearly shows that AlphaGateau learns much faster than AlphaZero at early training, the paper does not provide any information on how the two models compare when they are fully trained. AlphaGateau seems to reach a performance plateau within 100 steps, but the AlphaZero model trained by the authors would likely keep on improving for orders of magnitude more training steps. For comparison, the authors used a model size roughly 10% of Silver et. al.'s AlphaZero, but trained for 100 steps compared to the original 700,000 steps. It is not clear if AlphaGateau's performance will be comparable with AlphaZero, given enough training steps.\nI am aware that training the models for 10^5 steps is extremely expensive and not feasible for this paper. Even so, the correct solution would have been to fit the model size to the training budget and train a significanly smaller AlphaZero model, for longer time and on smaller batches (or to heavily reuse training data between optimization steps). Doing a comparison of smaller-scale models could show if AlphaGateau is a comparable model to AlphaZero. \n\nIn the paper's current form, the reported Elo gap does not give the reader any knowledge about how AlphaGateau compares to a fully-trained AlphaZero, since Elo calculation is unreliable when using such large gaps. For example, the gap between the best performing AlphaGateau and best AlphaZero is about 1,500 Elo, meaning that AlphaZero would win about 1 game in every 7,500 games. This implies that the AlphaGateau Elo score is based purely on comparisons with earlier AlphaGateau training checkpoints. \nAnother example is the gap between the first and second checkpoints, which is hard to measure by eye due to the plot format but looks roughly equal to 800. This gap implies the weaker model wins 1 game in 100, suggesting that the error of this gap is very high unless the authors use thousands of test games between this specific checkpoint pair to calculate Elo. I couldn't find the number of Elo games in the paper, and would recommend stating it clearly.\n\nConsidering that the best model's Elo is probably based purely on comparisons to its earlier versions, and considering the possible error of the Elo estimate, it is possible that AlphaGateau's Elo rating will drop significantly if fully-trained AlphaZero models are added to the pool of players.\n\nThis paper will benefit significantly from fitting the size of the experiments to the compute resources available to the authors. I would suggest training significanly smaller models for 100x more training steps, using the same compute budget. In its current form, the paper only showcases the advantage AlphaGateau has in terms of data-efficiency at early training, in a regime where AlphaZero is severely undertrained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Remarks\n\n> the paper does not provide any information on how the two models compare when they are fully trained. AlphaGateau seems to reach a performance plateau within 100 steps, but the AlphaZero model trained by the authors would likely keep on improving for orders of magnitude more training steps\n\nInitial tests on the AlphaZero implementation of PGX seemed to indicate that the model converges in less than 400 steps, with marginal improvements after 100 steps, which is why we limited our experiments to 100 steps. However, we didn't test the runs in the paper for more than 100 steps. We started new longer runs on the AlphaZero model, to include either in the main paper or in the appendix.\n\n\n> For comparison, the authors used a model size roughly 10% of Silver et. al.'s AlphaZero, but trained for 100 steps compared to the original 700,000 steps. It is not clear if AlphaGateau's performance will be comparable with AlphaZero, given enough training steps.\n\n> This paper will benefit significantly from fitting the size of the experiments to the compute resources available to the authors. I would suggest training significanly smaller models for 100x more training steps, using the same compute budget. In its current form, the paper only showcases the advantage AlphaGateau has in terms of data-efficiency at early training, in a regime where AlphaZero is severely undertrained.\n\nIn Silver et. al., they trained for 700,000 steps, however, due to their hardware availability, they were using 5000 TPUs to generate games, and 64 TPUs to train the network. As such, they could have the TPUs in charge of generating the games be always on and constantly generate games, while the TPUs in charge of training could just grab a random batch of positions from the last 1,000,000 generated positions to do one \"step\" of gradient update.\n\nIn contrast, for our experiments, we used the same 8 GPUs alternating between generating data and training the neural network. When training, we sample as many batches as required without replacement as to fully cover the current frame window. What we call in the paper one step are the generation of around 130,000 positions, followed by around 500 to 1000 batches to cover the full frame window (depending on the size of the frame window we used).\n\nAs such, our 100 steps are comprised of up to 100 000 batches. This is still an order of magnitude less than Silver et. al., but our model is also around an order of magnitude smaller. We will update the paper in order to clear up this confusing choice of term on our part.\n\n\n> or to heavily reuse training data between optimization steps\n\nAs mentioned in the previous answer, we used a frame window, such that on average, each data point is reused around 7 times. We mentioned with Figure 7 part of our experiments on the impact of reusing training data, by varying the amount of data generated, and the amount of data reuse.\n\n\n> In the paper's current form, the reported Elo gap does not give the reader any knowledge about how AlphaGateau compares to a fully-trained AlphaZero, since Elo calculation is unreliable when using such large gaps.\n\nEach model is initially assumed to have an Elo of 1000, and plays 5 matches of 60 games each against 5 opponents. After each match the Elo of all the models is recomputed, and each opponent is chosen to be as close as possible to the current Elo evaluation of the model. Models with high difference in Elo are effectively not appropriate to evaluate the Elo ratings, but this is reflected in the plotted confidence intervals. We included in the global PDF response a distribution plot of all the Elo of the models that are included in the linear regression described in Eq (11), which shows that the ratings are quite continuous, without leaving any gaps that could bias the rating estimation. Each player played at least 300 games (150 as white and 150 as black), which was enough to keep the confidence intervals smaller than 100 Elo points.\n\n\n# Questions\n\n> What hyper-parameters are used to calculate Elo scores? (number of models tested, number of games between each model pair, who played against who)\n\nAt the time of submission, we evaluated 255 players, and have since evaluated 977 players (pair of (model, parameters)). Each player played at least 60 games against each of 5 other players that were already evaluated (such that the graph of matches is 5-edge-connected). As the opponents of each player depended on their successive rating evaluations, they are not easy to list. However, the full  list of matches as well as their outcomes (wins, draws, losses) is included in the supplementary material zip in `rankings.json`\n\n\n> It would be helpful for the reader if figures 5 and 6 had FLOPs on the x-axis instead of steps, or if training compute was reported. Plotting only steps, it is hard to understand if one training step of AlphaGateau is comparable to a training step of AlphaZero in terms of compute.\n\nWe do not easily have access to FLOPs data for our previous runs, however we do have access to timing data, which should loosely correlate with FLOPs as the same hardware was used for all experiments. We will include the corresponding plots in the appendix\n\n> It would be easier and more convincing to use traditional methods for calculating Elo scores, such as BayesElo which was used by Silver et. al. That said, the authors provide clear explanations on their Elo calculation method.\n\nThanks, we were not aware of the existence of BayesElo. We compared the Elo from our method with the Elo for BayesElo (see plot in global PDF) and got relatively consistent results, besides the fact that weaker players were rated higher by our model, and similarly, stronger players were rater lower by our model, compressing the Elo range when compared to BayesElo. The main effective difference is a small amount of players were rated around 10 to 20 points out of sync with the players closest to them in Elo.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewer for their time and thoughtful opinions. 

As shown by reviewer abiC, we were unclear in our terminology and cause confusion when compared to Silver et. al. (2017) with regards to what we called steps, leading to a first impression of our experiments being smaller than they actually were. We will clarify this point in our paper.

We have since the initial submission been able to train a deeper model, with 8 layers instead of 5 and 6, which further confirm our initial results, and included a revised version of Figure 6 in the joint pdf. We also retrained the base 5-layers model in order to be able to evaluate it every second iteration rather than the 1 in 5 it was initially. In addition, we found a bug in our implementation that made some of the models presented in the figure have slightly different hyper-parameters than reported. We have since re-run all affected models resulting in similar or slightly improved results. The paper will be updated with the new 8 and 5 layers run, as well as the fixed reruns.

We also plan to include in the appendix a version of each plot with `running time` on the x axis rather than `iteration`, as a proxy to FLOPs, following the recommendation of reviewer abiC.

A pgn of a game played as white on 8x8 chess by a model trained only on 5x5 chess was also included in the joint pdf, to show the model ability to learn general pattern and rules in a simplified version of the game. We will add more pgns to the appendix of the full paper.

We also recognize that our method of computing Elo ratings could have been better justified. We included in the joint pdf 2 plots justifying that the collection of models tested didn't leave gaps in the rating range, and that our method produced results consistent with ratings estimated using BayesElo.

We would be happy to engage in further discussion if you feel there is any remaining questions that we left unanswered, or if our previous answers were lacking in some way.

Meta-review: The reviewers agree that the application of graph neural networks to chess is interesting and worth studying. The reviewers note some weaknesses in the evaluation of the method and the comparison to AlphaZero, but, during the discussion period, the most serious of the concerns were addressed, and the authors have offered additional results and clarifications that will strengthen the paper. Though the reviewers uniformly agree that the paper is acceptable, the somewhat limited scope/impact of the work and the limitations on the conclusions that can be drawn from the experimental evaluations suggest that this work is best presented as a poster, rather than as a spotlight or oral presentation.
Decision: Accept (poster)