Title: Enhancing Chess Reinforcement Learning with Graph Representation
Venue: NeurIPS 2024
Accepted: yes
Keywords: deep learning

GNN

Chess

RL
Abstract: Mastering games is a hard task, as games can be extremely complex, and still fundamentally different in structure from one another. While the AlphaZero algorithm has demonstrated an impressive ability to learn the rules and strategy of a large variety of games, ranging from Go and Chess, to Atari games, its reliance on extensive computational resources and rigid Convolutional Neural Network (CNN) architecture limits its adaptability and scalability. A model trained to play on a $19\times 19$ Go board cannot be used to play on a smaller $13\times 13$ board, despite the similarity between the two Go variants.
In this paper, we focus on Chess, and explore using a more generic Graph-based Representation of a game state, rather than a grid-based one, to introduce a more general architecture based on Graph Neural Networks (GNN). We also expand the classical Graph Attention Network (GAT) layer to incorporate edge-features, to naturally provide a generic policy output format.
Our experiments, performed on smaller networks than the initial AlphaZero paper, show that this new architecture outperforms previous architectures with a similar number of parameters, being able to increase playing strength an order of magnitude faster. We also show that the model, when trained on a smaller $5\times 5$ variant of chess, is able to be quickly fine-tuned to play on regular $8\times 8$ chess, suggesting that this approach yields promising generalization abilities.
Our code is available at https://github.com/akulen/AlphaGateau.

Enhancing Chess Reinforcement Learning with Graph Representation

Abstract

Mastering games is a hard task, as games can be extremely complex, and still fundamentally different in structure from one another. While the AlphaZero algorithm has demonstrated an impressive ability to learn the rules and strategy of a large variety of games, ranging from Go and Chess, to Atari games, its reliance on extensive computational resources and rigid Convolutional Neural Network (CNN) architecture limits its adaptability and scalability. A model trained to play on a 19 × 19 Go board cannot be used to play on a smaller 13 × 13 board, despite the similarity between the two Go variants. In this paper, we focus on Chess, and explore using a more generic Graph-based Representation of a game state, rather than a grid-based one, to introduce a more general architecture based on Graph Neural Networks (GNN). We also expand the classical Graph Attention Network (GAT) layer to incorporate edge-features, to naturally provide a generic policy output format. Our experiments, performed on smaller networks than the initial AlphaZero paper, show that this new architecture outperforms previous architectures with a similar number of parameters, being able to increase playing strength an order of magnitude faster. We also show that the model, when trained on a smaller 5 × 5 variant of chess, is able to be quickly fine-tuned to play on regular 8 × 8 chess, suggesting that this approach yields promising generalization abilities. Our code is available at https://github.com/akulen/AlphaGateau .

1 Introduction

In the past decade, combining Reinforcement Learning (RL) with Deep Neural Networks (DNNs) has proven to be a powerful way to design game agents for a wide range of games. Notable achievements include AlphaGo's dominance in Go [17], AlphaZero's human-like style of play in Chess and Shogi [18], and MuZero's proficiency across various Atari games [15]. They use self-play and Monte Carlo Tree Search (MCTS) [6] to iteratively improve their performance, mirroring the way humans learn through experience, or intuition, and game-tree exploration.

Previous attempts to make RL-based chess engines were unsuccessful as the MCTS exploration requires a precise position heuristic to guide its exploration. Handcrafted heuristics such as the ones used in traditional minimax exhaustive tree searches were too simplistic, and lacked the degree of sophistication that the random tree explorations of MCTS expects to be able to more accurately evaluate a complex chess position. By combining the advances in computing powers with the progress of the field of Deep Learning, AlphaZero was able to provide an adequate heuristic in the form of a Deep Neural Network that was able to learn in symbiosis with the MCTS algorithm to iteratively improve itself.

However, these approaches rely on rigid, game-specific neural network architectures, often representing games states using grid-based data structures, and process them with Convolutional Neural Networks (CNNs), which limits their flexibility and generalization capabilities. For example, a model trained on a standard 19 × 19 Go board cannot easily adapt to play on a smaller 13 × 13 board without significant changes to its internal structure, manual parameter transfer, and retraining, despite the underlying similarity of the game dynamics. This inflexibility is further compounded by the extensive computational resources required for training these large-scale models from scratch for each specific game or board configuration. If it was possible to make a single model train of various variants of a game, and on various games at the same time, it would be possible to speed up the training by starting to learn the fundamental rules on a simplified and smaller variant of a game, before presenting the model with the more complex version. Similarly, if a model learned all the rules of chess, it could serve as a strong starting point to learn the rules of Shogi, for example.

It could be possible to design a more general architecture for games such as Go, where moves can be mapped one-to-one with the board grid, so that a model could still use CNN layers and handle differently sized boards simultaneously, but this solution is no longer feasible when the moves become more complex, including having to move pieces between squares, or even dropping captured pieces back onto the board in Shogi.

Those moves evoke a graph-like structure, where pieces, initially positioned on squares, are moved to different new squares, following an edge between those two squares, or nodes. As such, it is natural to consider basing an improved model on a graph representation, instead of a grid representation. We explore replacing CNN layers with GNN layers to implement that idea, and more specifically consider in this paper attention-based GNN layers, reflecting how chess players usually remember the structures that the pieces form, and how they interact with each other, instead of remembering where each individual piece is placed, when thinking about a position.

Representing moves as edges in a graph also introduces the possibility to link the output policy size with the number of edges, to make the model able to handle different game variants with different move structures simultaneously. To do so, it becomes important to have edge features as well as node features, as they will be used to compute for each edge the equivalent move logit. As the classical attention GNN layer, the Graph Attention Network [19] (GAT) only defines and updates node-features, we propose a novel modification of the GAT layer, that we call Graph Attention neTwork with Edge features from Attention weight Updates (GATEAU), to introduce edge-features. We also describe the full model architecture integrating the GATEAU layer that can handle differently sized input graphs as AlphaGateau.

Our experimental results demonstrate that this new architecture, when implemented with smaller networks compared to the original AlphaZero, outperforms previous architectures with a similar number of parameters. AlphaGateau exhibits significantly faster learning, achieving a substantial increase in playing strength in a fraction of the training time. Additionally, our approach shows promising generalization capabilities: a model trained on a smaller 5 × 5 variant of chess can be quickly fine-tuned to play on the standard 8 × 8 chessboard, achieving competitive performance with much less computational effort.

2 Related Work

Reinforcement Learning. AlphaGo [17], AlphaZero [18], MuZero [15], and others have introduced a powerful framework to exploit Reinforcement Learning techniques in order to generate self-play data used to train from scratch a neural network to play a game.

However, those frameworks use rigid neural networks, that have to be specialized for one specific game. As such, the training process requires a lot of computation resources. It is also not possible to reuse the training on one type of game to train for another one, or to start the training on a smaller and simpler variant of the game, before introducing more complexity.

Scalable AlphaZero. In the research of Ben-Assayag and El-Yaniv [2], using Graph Neural Networks has been investigated as a way to solve those issues. Using a GNN based model, it becomes possible to feed as input differently-sized samples, such as Othello boards of size between 5 and 16, enabling the model to learn how to play in a simpler version of the game.

Algorithm 1: Self-Play Training

This approach had promising results, with 10 times faster training time than the AlphaZero baseline. It was however limited to Othello and Gomoku, and using the GNN layers (GIN layers [20]) only as a scalable variant of CNN layers, keeping a rigid grid structure.

Edge-featured GNNs. There exists a large variety of GNN variants, specialized for different use case s and data properties. For this work, a simple layer was enough to experiment with the merits of the proposed approach, except it was critical that the chosen layer handled both node-features and edge-features. We chose to use an attention-based layer.

Gong and Cheng [10] introduce the EGNN(a) layer, where each dimension of an edge-feature is used for a different attention head. We wanted edge features to be treated as a closer equivalent to node-features, so we did not use this layer.

The EGAT layer introduced by Chen and Chen [4] is better for our case, as they construct a dual graph where edges and nodes have reversed roles, so the node features in the dual graph are edge features for the initial graph. However, this method requires building the dual graph, and is quadratic in the maximal node degree. As this was quite complex, we decided to introduce GATEAU, which solves the problem in a simpler and more natural way.

3 Setting

Our architecture is based on the AlphaZero framework, which employs a neural network f θ with parameters θ that is used as an oracle for a Monte-Carlo Tree Search (MCTS) algorithm to generate self-play games. When given a board state s , the neural network predicts a (value, policy) pair ( v ( s ) , π ( s, ⋅)) = f θ ( s ) , where the value v ∈ [-1 , 1 ] is the expected outcome of the game, and the policy π is a probability distribution over the moves.

We utilize Algorithm 1 to train the models in this paper, with modifications to incorporate Gumbel MuZero [7] with a gumbel scale of 1.0 as our MCTS variant.

4 Proposed Models

4.1 Motivation: Representing the Game State as a Graph

Many games, including chess, are not best represented as a grid. For example, chess moves are analogous to edges in a grid graph, and games like Risk naturally form planar graphs based on the map. As such, it makes natural sense to encode more information through graphs in the neural network layers that are part of the model.

This research focuses on implementing this idea in the context of chess. This requires to answer two questions: how to represent a chess position as a graph, and how to output a policy vector that is edge-based, and not node-based.

The architecture presented in this paper is based on GNNs, but using node features to evaluate the value head, and edge features to evaluate the policy head. As such, a GNN layer that is able to handle

Dimensions

1

2

4

2

4

2

Description

Is this move legal in the current position?

How many squares {to the left/up} does this edge move?

Would a pawn promote to a {knight/bishop/rook/queen} if it did this move?

Could a {white/black} pawn do this move?

Could a {knight/bishop/rook/queen} do this move?

Could a {white/black} king do this move?

both node and edge features is required. This paper will introduce the GATEAU layer, that is a natural extension of the GAT layer [19] to edge features.

4.2 Graph Design

In AlphaZero, a chess position is encoded as an n × n × 119 matrix, where each square on the n × n chess board is associated to a feature vector of size 119, containing information about the corresponding square for the current position, as well as the last 7 positions, as described in Table 1.

We will instead represent the board state as a graph G ( V,E ) , with the n × n squares being nodes V , and the edges E being moves, based on the action design of AlphaZero. Each AlphaZero action is a pair (source square, move), with n × n possible source squares. In 8 × 8 chess, the 73 moves (resp. 49 in 5 × 5 chess) are divided into 56 queen moves (resp. 32), 8 knight moves (resp. 8), and 9 underpromotions (resp. 9) for a total of 4672 actions (resp. 1225). The edge associated with an action is oriented from the node corresponding to the source square, to the destination square of the associated move. In 8 × 8 chess, castling is represented with the action going from the king's starting square going laterally 2 squares. As this action encoding is a little too large, containing moves ending outside of the board that do not correspond to real edges, the constructed graph only contains 1858 edges (resp. 455), corresponding only to valid moves.

The node and edge features, of initial size 119 and 15, are detailed in Tables 1 and 2, respectively. Node features are based on AlphaZero's features, including piece type, game state information, and historical move data. Edge features encode move legality, direction, potential promotions, and piece-specific move capabilities. In the case of 5 × 5 chess, we include the unused castling information, in order to have the same vector size of the 8 × 8 models. It would be possible to preprocess the node and edge features differently for different games or variants, but for simplicity we didn't do it.

The starting positions for all games played in our experiments were either the classical board setup in 8 × 8 chess, or the Gardner setup for 5 × 5 chess, illustrated in Figure 1.

4.3 GATEAU: A New GNN Layer, with Edge Features

The Graph Attention Network layer introduced by Veliˇ ckovi´ c et al. [19] updates the node features by averaging the features of the neighbouring nodes, weighted by some attention coefficient. To be more precise, given the node features h ∈ R N × K , attention coefficients are defined as

with parameters W u , W v ∈ R K × K ′ . In the original paper, W u = W v , but as we are working with a directed graph, we differentiate them to treat the source and destination node asymmetrically. Then

we can compute attention weights α , and use them to update the node features:

with parameters W ∈ R K × K ′′ and a ∈ R K ′ .

The main observation motivating GATEAU is that in this process, the attention coefficients e ij serve a role similar to node features, being a vector encoding some information between nodes i and j . As such, we propose to introduce edge features in place of those attention coefficients.

Our proposed layer, called Graph Attention neTwork with Edge features from Attention weight Updates (GATEAU) takes the node features h ∈ R N × K and edge features g i,j ∈ R N × N × K ′ as inputs. We start by simply updating the edge features similarly to Eq. 1:

with parameters W u , W v ∈ R K × K ′ and W e ∈ R K ′ × K ′ . Then the attention weights are obtained as in Eq. 2, by substituting the attention coefficients with our new edge features:

with parameter a R K

∈ ′ . Finally, we update the node features as in Eq. 3:

with parameters W 0 , W h ∈ R K × K ′′ and W g ∈ R K ′ × K ′′ . We add the self-edges manually as it is inconvenient for the policy head if they are included in the graph, and we mix back the values of the edge features back in the node features.

4.4 AlphaGateau: A Full Model Architecture Based on AlphaZero and GATEAU

Following the structure of the AlphaZero neural network, we introduce AlphaGateau, combining AlphaZero with GATEAU instead of CNN layers, and redesign the value and policy head to be able to exploit node and edge features respectively to handle arbitrarily sized inputs with the same number of parameters.

We define the following layers, which are used to describe AlphaGateau in Figure 2.

Attention Pooling. In order to compute a value for a given graph, we need to pool the features together. Node features seem to be the more closely related to positional information, so we pool them instead of edge features. For this, we use an attention-based pooling layer, similar to the one described in Eq. 7 by Li et al. [14], which, for node features h ∈ R N × K and a parameter vector a ∈ R K , outputs

8

rmblkans

7

opopopop

6

0Z0Z0Z0Z

5

Z0Z0Z0Z0

4

0Z0Z0Z0Z

3

Z0Z0Z0Z0

2

POPOPOPO

1

SNAQJBMR

a

b

c

d

e

f

g

h

5

snaqj

4

popop

3

Z0Z0Z

2

POPOP

1

SNAQJ

a

b

c

d

e

Nodes (squares)

Edges (moves)

Linear

hs

Linear

hs

h

g

ResGATEAU

hs

h

′

g

′

Value head

Policy head

×

L

h

′

hs

BNR

Linear

hs

BNR

Att Pool

ReLU

Linear

1

tanh

Value

1

Figure 3:

Value head

g

′

hs

BNR

Linear

hs

BNR

Linear

1

Policy

#actions

where H ∈ R K is a global feature vector.

Batch Normalization and Non-linearity (BNR). As they are a pair of operations that often occur, we group Batch Normalization and a ReLU layer together under the notation BNR:

Residual GATEAU (ResGATEAU). Mirroring the AlphaZero residual block architecture, we introduce ResGATEAU, which similarly sums a normalized output from two stacked GATEAU layers to the input:

5 Experiments

We evaluate AlphaGateau's performance in learning regular 8 × 8 chess from scratch and generalizing from a 5 × 5 variant to the standard 8 × 8 chessboard. The metric used to evaluate the models is the Elo rating, calculated through games played against other models (or players) with similar ratings. Due to computational constraints, we couldn't replicate the full 40 residual layers used in the initial AlphaZero paper, and experimented with 5 and 6 layer models. We also started exploring 8 layers, but these models required to generate a lot more data, which would make the experiment run an order of magnitude longer. Our results indicate that AlphaGateau learns significantly faster than a traditional CNN-based model with similar structure and depth, and can be efficiently fine-tuned from 5 × 5 to 8 × 8 chess, achieving competitive performance with fewer training iterations. 2 All models used in these experiments are trained with the Adam optimizer [12] with a learning rate of 0.001. All feature vectors have an embedding dimension of 128. The loss function is the same as for the original AlphaZero, which is, for f θ ( s ) = ˜ π, ˜ v ,

5.1 Implementation

Jax and PGX. As the MCTS algorithm requires a lot of model calls weaved throughout the tree exploration, it is essential to have optimized GPU code running both the model calls, and the MCTS search. In order to leverage the MCTX [8] implementations of Gumbel MuZero, all our models and experiments were implemented in Jax [3] and its ecosystem [11] [9]. PGX [13] was used for the chess implementation, and we based our AlphaZero implementation on the PGX implementation of AZNet. We used Aim [1] to log all our experiments.

To estimate the Elo ratings, we use the statsmodels package [16] implementation of Weighted Least Squares (WLS).

Hardware. All our models were trained using multiple Nvidia RTX A5000 GPUs ( Learning speed used 8 and Fine-tuning used 6), and their Elo ratings were estimated using 6 of those GPUs.

5.2 Evaluation

As each training and evaluation lasted a little under a week, we were not able to train each model configuration several times so far. As such, each model presented in the results was trained only once, and the confidence intervals that we include are on the Elo rating that we estimated for each of them, as described in the following.

During training, at regular intervals (each 2, 5, or 10 iterations), the model parameters were saved, and we used this dataset of parameters to evaluate Elo ratings. In this section, we will call a pair (model, parameters) a player, and compute a rating for every player.

We initially chose 10 players, and simulated 60 games between each pair of players, to get initial match data M . For each pair of players that played a match, we store the number of wins w ij , draws d ij , and losses l ij : M ij = ( w ij , d ij , l ij ) . Using this data, we can roughly estimate the ratings r ∈ R N players of the players present in M using a linear regression on the following set of equations:

We artificially add one draw to avoid extreme cases where there are only wins for one player and no losses, in which case the rating difference would theoretically be infinite. This is equivalent to a Jeffreys prior. The last equation fixes the average rating to 1000, as the Elo ratings are collectively invariant by translation.

We then ran Algorithm 2 to generate a densely connected match data graph, where each player played against at least 5 other players of similar playing strength. Finally, we used this dataset to fit a linear regression model (Weighted Least Squares) to get Elo ratings that we used in the results figures for the experiments. The confidence intervals were estimated by assuming that the normalized w + d ij + 1

match outcomes followed a Gaussian distribution. If ˆ p ij = ij 2 w ij + d ij + l ij + 1 is the estimated probability that player i beats player j , we approximate the distribution that p ij follows as a Gaussian, and using the delta method, we derive that r j -r j asymptotically follows a Gaussian distribution of mean 400 log ( 10 ) log ( 1 p ij -1 ) and variance ( 400 log ( 10 ) ) 2 1 ( w ij + d ij + l ij ) p ij ( 1 -p ij ) . The proof is detailed in Appendix A.2. Using the WLS linear model of statsmodels [16], we get Elo ratings for every player, as well as their standard deviations, which we use in the following to derive 2-sigma confidence intervals.

5.3 Experiments

Learning Speed. Our first experiment compares the baseline ability of AlphaGateau to learn how to play 8 × 8 chess from scratch, and compares it with a scaled down AlphaZero model. The AlphaZero model has 5 residual layers (containing 10 CNN layers) and a total of 2.2M parameters, and the AlphaGateau model also has 5 ResGATEAU layers (containing 10 GATEAU layers) and a total of 1.0M parameters, as it doesn't need an N nodes × hs × N actions fully connected layer in the policy head, and the GATEAU layers use 2/3 of the parameters a 3 × 3 CNN does.

Algorithm 2: Matching Players

For this experiment, we generated 256 games of length 512 at each iteration, totalling 131072 frames, and kept a frame window of 1M frames (all the newly generated frames, and uniform sampling over the frame window of the previous iteration), over 500 iterations. During the neural network training, we used a batch size of 256. The training for AlphaGateau lasted 13 days and 16 hours, while AlphaZero took 10 days and 3 hours.

We report the estimated Elo ratings with a 2-sigma confidence interval in Figure 5. AlphaZero was only able to reach an Elo of 667 ± 38 in 500 iterations, and would likely continue to improve with more time, while AlphaGateau reached an Elo of 2105 ± 42 , with an explosive first 50 iterations, and achieving results comparable to the final Elo of AlphaZero after only 10 iterations.

Although those results are promising, it is important to note that we only compared to a simplified version of AlphaZero, using only 5 layers instead of the original 40, and without spending large efforts to optimize the hyperparameters of the model. As such, it is possible that the performance of AlphaZero could be greatly improved in this context with more parameter engineering. Both AlphaGateau and AlphaZero have not reached a performance plateau after 500 iterations, showing slow but consistent growth.

Fine-tuning. In our second experiment, we trained a AlphaGateau model with 10 residual layers on 5 × 5 chess for 100 iterations, then fine-tuned this model on 8 × 8 chess for 100 iterations. This model has a total of 1.7M parameters.

For this experiment, we generated 1024 games of length 256 at each iteration on 5 × 5 chess, and 256 games of length 512 while fine-tuning on 8 × 8 chess, and kept a frame window of 1M frames. The initial training lasted 2 days and 7 hours, and the fine-tuning 5 days and 15 hours.

We report the estimated Elo ratings with a 2-sigma confidence interval in Figure 6. The initial training ended with an Elo rating of 807 ± 46 when evaluated on 8 × 8 chess games, which suggests that it was able to learn general rules of chess on 5 × 5 , and apply them with some success on 8 × 8 chess without having seen any 8 × 8 chess position during its training. Once the fine-tuning starts, the model jumps to an Elo of 1181 ± 50 after a couple iterations, suggesting the baseline learned on 5 × 5 was of high quality. After fine-tuning, the model had an Elo of 1876 ± 47 , reaching comparable performances to the smaller model using roughly the same amount of iterations and GPU-time, despite being twice as big. Preliminary testing suggests that this model would become stronger if more data was generated at each iteration, but that would linearly increase the training time, as generating self-play games took half of the 5 days of training.

5.4 Impact of the Frame Window and the Number of Self-play Games

In order to train deeper networks, we experimented with the number of self-play games generated at each generation, and with the size of the frame window. It seems from our results in Figure 7 that having more newly generated data helps the model learn faster. However, the time taken for each iteration scales linearly with the number of generated games, and the model is still able to improve using older data. As such, keeping a portion of the frame window from previous iterations makes for a good compromise. There are however options to improve our frame window selection:

0

50

100

150

200

250

300

350

400

450

500

Iteration

500

0

500

1000

1500

2000

Elo Rating

AlphaZero

AlphaGateau

0

20

40

60

80

100

500

0

500

1000

1500

2000

Elo Rating

5x5 chess AlphaGateau 10 layers

AlphaGateau 5 layers

Fine-tuned AlphaGateau 10 layers

Iteration

Which previous samples should be selected? We selected uniformly at random from the previous frame window to complement the newly generated samples, but it might be preferable to select fewer samples, but chosen as to represent a wide range of different positions.

Past samples are by design of dubious quality. As they come from self-play games with previous model parameters, they correspond to games played with a lower playing strength, and the policy output by the MCTS is also worse. Keeping a sample that is too old might cause a drop of performance rather than help the model learn. We experimented a little with keeping the 1M most recent samples, but with little success.

We also initially tried to increase the number of epochs in one iteration, but only saw marginal gains, suggesting that mixing new data among the previous frame window helps the model extract more training information from previous samples.

0

20

40

60

80

100

500

0

500

1000

1500

2000

Elo Rating

AlphaGateau 256 games/131k window

AlphaGateau 256 games/1M window

AlphaGateau 1024 games/524k window

Iteration

6 Conclusion

In this paper, we introduce AlphaGateau, a variant on the AlphaZero model design that represents a chess game state as a graph, in order to improve performance and enable it to better generalize to variants of a game. This change from grid-based CNN to graph-based GNN yields impressive increase in performance, and seems promising to enable more research on reinforcement-learning based game-playing agent research, as it reduces the resources required to train one.

We also introduce a variant of GAT, GATEAU, that we designed in order to handle edge features in a simple manner performed well, and efficiently.

Future Work. As our models were relatively shallow when compared to the initial AlphaZero, it would be important to confirm that AlphaGateau still outperforms AlphaZero when both are trained with a full 40-deep architecture. This will require a lot more computing time and resources.

As discussed in Section 5.4, our design of the frame window is a little unsatisfactory, and a future improvement would be to define an efficiently computable similarity metric between chess positions, that helps the neural network generalize.

Wefocused on chess for this paper, but there are other games that could benefit from this new approach. The first one would be shogi, as it has similar rules to chess, and the promising generalization results from AlphaGateau could be used to either train a model on one game, and fine-tune it on the other, or to jointly train it on both games, to have a more generalized game-playing agent. As alluded to in the Graph Design 4.2, more features engineering would be required to have node and edge features compatibility between chess graphs, and shogi graphs. It could also be possible to change the model architecture to handle games with more challenging properties, such as the game Risk, which has more than 2 players, randomness, hidden information, and varying maps, but is even more suited to being represented as a graph.

Acknowledgments and Disclosure of Funding

This work was supported by JST BOOST, Grant Number JPMJBS2407 and JST CREST, Grant Number JPMJCR21D1. We thank Armand Barbot and Jill-Jênn Vie for their helpful comments and insightful discussions.

References

G. Arakelyan, G. Soghomonyan, and The Aim team. Aim, June 2020. URL https://github. com/aimhubio/aim .

S. Ben-Assayag and R. El-Yaniv. Train on Small, Play the Large: Scaling Up Board Games with AlphaZero and GNN, July 2021. URL http://arxiv.org/abs/2107.08387 . arXiv:2107.08387 [cs].

J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax .

J. Chen and H. Chen. Edge-Featured Graph Attention Network, Jan. 2021. URL http: //arxiv.org/abs/2101.07671 . arXiv:2101.07671 [cs].

R. Coulom. Whole-History Rating: A Bayesian Rating System for Players of Time-Varying Strength. In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, O. Nierstrasz, C. Pandu Rangan, B. Steffen, M. Sudan, D. Terzopoulos, D. Tygar, M. Y. Vardi, G. Weikum, H. J. Van Den Herik, X. Xu, Z. Ma, and M. H. M. Winands, editors, Computers and Games , volume 5131, pages 113-124. Springer Berlin Heidelberg, Berlin, Heidelberg, 2008. ISBN 978-3-540-87607-6 978-3-540-87608-3. doi: 10.1007/978-3-540-87608-3_11. URL https://link.springer.com/10.1007/978-3-540-87608-3_11 . Series Title: Lecture Notes in Computer Science.

J. Czech, P. Korus, and K. Kersting. Monte-Carlo Graph Search for AlphaZero, Dec. 2020. URL https://arxiv.org/abs/2012.11045v1 .

I. Danihelka, A. Guez, J. Schrittwieser, and D. Silver. Policy improvement by planning with Gumbel. Oct. 2021. URL https://openreview.net/forum?id=bERaNdoegnO .

DeepMind, I. Babuschkin, K. Baumli, A. Bell, S. Bhupatiraju, J. Bruce, P. Buchlovsky, D. Budden, T. Cai, A. Clark, I. Danihelka, A. Dedieu, C. Fantacci, J. Godwin, C. Jones, R. Hemsley, T. Hennigan, M. Hessel, S. Hou, S. Kapturowski, T. Keck, I. Kemaev, M. King, M. Kunesch, L. Martens, H. Merzic, V. Mikulik, T. Norman, G. Papamakarios, J. Quan, R. Ring, F. Ruiz, A. Sanchez, L. Sartran, R. Schneider, E. Sezener, S. Spencer, S. Srinivasan, M. Stanojevi´ c, W. Stokowiec, L. Wang, G. Zhou, and F. Viola. The DeepMind JAX Ecosystem, 2020. URL http://github.com/deepmind .

J. Godwin*, T. Keck*, P. Battaglia, V. Bapst, T. Kipf, Y. Li, K. Stachenfeld, P. Veliˇ ckovi´ c, and A. Sanchez-Gonzalez. Jraph: A library for graph neural networks in jax., 2020. URL http://github.com/deepmind/jraph .

L. Gong and Q. Cheng. Exploiting Edge Features in Graph Neural Networks, Jan. 2019. URL http://arxiv.org/abs/1809.02709 . arXiv:1809.02709 [cs, stat].

J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. v. Zee. Flax: A neural network library and ecosystem for JAX, 2023. URL http://github.com/google/ flax .

D. P. Kingma and J. Ba. Adam: A Method for Stochastic Optimization, Jan. 2017. URL http://arxiv.org/abs/1412.6980 . arXiv:1412.6980 [cs].

S. Koyamada, S. Okano, S. Nishimori, Y. Murata, K. Habara, H. Kita, and S. Ishii. Pgx: Hardware-Accelerated Parallel Game Simulators for Reinforcement Learning, Jan. 2024. URL http://arxiv.org/abs/2303.17503 . arXiv:2303.17503 [cs].

Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated Graph Sequence Neural Networks, Sept. 2017. URL http://arxiv.org/abs/1511.05493 . arXiv:1511.05493 [cs, stat].

J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature , 588(7839):604-609, Dec. 2020. ISSN 0028-0836, 1476-4687. doi: 10.1038/s41586-020-03051-4. URL http://arxiv.org/abs/1911.08265 . arXiv:1911.08265 [cs, stat].

S. Seabold and J. Perktold. statsmodels: Econometric and statistical modeling with python. In 9th Python in Science Conference , 2010.

D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis. Mastering the game of Go with deep neural networks and tree search. Nature , 529:484-489, Jan. 2016. doi: 10.1038/nature16961.

D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm, Dec. 2017. URL https://arxiv.org/abs/1712.01815v1 .

P. Veliˇ ckovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio. Graph Attention Networks, Oct. 2017. URL https://arxiv.org/abs/1710.10903v3 .

K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How Powerful are Graph Neural Networks?, Oct. 2018. URL https://arxiv.org/abs/1810.00826v3 .

A Appendix / supplemental material

A.1 Elo

The Elo rating system was initially introduced as a way to assign a value to the playing strength of chess players. The rating of each player is supposed to be dynamic and be adjusted after each game they play to follow their evolution.

The Elo ratings are defined to respect the following property: If two players with Elo rating R A and R B played a game, the probability that player A wins is

A.2 Variance of estimated Elo rating difference

We assume that the outcome of a game between two players of Elo R A and R B is a Bernoulli trial, with a probability that player A win being given by the central Elo equation 12. If we want to estimate the Elo of both players, we need to estimate that probability p . To do so, we can make the two players play n games, and sum the wins of player A , as well as half his draws, to get x = w + d 2 . From this, we can estimate the value of p using the estimator ˆ p = x n . In the case that one of the two players is significantly stronger than the other, ˆ p could be close to 0 or 1 , in which case this estimator is wildly inaccurate. To remedy this, we will instead rely on a Jeffreys prior, to get the estimator ˆ p Jeffreys = x + 1 / 2 n + 1 . We will note this estimator ˆ p in the following.

By inverting the Elo equation 12, we can get the rating difference from the probability that A wins as

From our assumptions, we have that x follows a binomial distribution B ( n, p ) , and we will approximate the distribution that ˆ p follows by a normal distribution ˆ p ∼ N( p, p ( 1 -p ) n ) .

Therefore, posing g ( y ) = 400 log ( 10 ) log ( 1 y -1 ) , which is differentiable, we can use the delta method to get that g ( ˆ p ) is asymptotically Gaussian. The derivative of g is which gives

A.3 Comparison with BayesElo

We also used BayesElo [5] to evaluate the Elo ratings of our models using all the PGN files that were generated recording the games played between all our models. In Figure 8, we plotted a point

500

0

500

1000

1500

2000

2500

Elocustom

200

100

0

100

200

300

Elocustom

EloBayes

Elo difference between our model and elobayes

0

50

100

150

200

250

300

350

400

450

500

Iteration

500

0

500

1000

1500

2000

Elo Rating

AlphaZero

AlphaGateau

0

50

100

150

200

250

Time (hour)

500

0

500

1000

1500

2000

Elo Rating

AlphaZero

AlphaGateau

on ( x, y ) for each player where x is its Elo rating following our method and y is the difference in predicted Elo between our method and BayesElo.

In practice, weaker models tend to be overrated by our method compared with BayesElo, while stronger models are underrated. As this relation is smooth and monotone, this suggests that both methods order the relative strength of all players similarly, with small differences only being due to noise. The main difference being that our range of Elo ratings is compacted towards the extremes, which we assume is due to our practical Jeffreys prior, that implies that strong models drew at least one game against all their opponents, handicapping their Elo, and similarly weak models always manage to not lose at least one game.

A.4 FLOPs comparison

We didn't record the FLOPs of our models, however, we did record the time taken for each iteration. In Figure 10, 12, and 14, we plotted the three main plots of the main paper using that data on the X-axis. As some experiments were run using 6 GPUs instead of *, we multiplied their recorded time by a factor 6 8 for better comparison.

A.5 PGNs

We include with Figure 15, 16, and 17 a few PGNs showing games played by our models.

0

20

40

60

80

100

500

0

500

1000

1500

2000

Elo Rating

5x5 chess AlphaGateau 10 layers

AlphaGateau 5 layers

Fine-tuned AlphaGateau 10 layers

Iteration

0

20

40

60

80

100

Iteration

500

0

500

1000

1500

2000

Elo Rating

AlphaGateau 256 games/131k window

AlphaGateau 256 games/1M window

AlphaGateau 1024 games/524k window

0

20

40

60

80

100

500

0

500

1000

1500

2000

Elo Rating

5x5 chess AlphaGateau 10 layers

AlphaGateau 5 layers

Fine-tuned AlphaGateau 10 layers

Time (hour)

0

50

100

150

200

Time (hour)

500

0

500

1000

1500

2000

Elo Rating

AlphaGateau 256 games/131k window

AlphaGateau 256 games/1M window

AlphaGateau 1024 games/524k window

Figure 15 contains the PGN of a game played on 8 × 8 chess by the last iteration of the 5 × 5 AlphaGateau model of our fine-tuning experiment, to showcase its playing style while having never seen an 8 × 8 board during its training.

Figure 16 and 17 contains the PGNs of games played by the last iteration (iteration 499) of the full AlphaGateau model of our first experiment as white and black respectively, selected among the games played against other models to evaluate all the Elo rankings, selected using the script book.py in the GitHub repo, following the most played move each ply. The game played as white in Figure 16 is played against the iteration 61 of a fine-tuned AlphaGateau model using 8 layers, 1024 generated games per iteration, but only 32 MCTS simulations, with an estimated Elo rating of 2124 . The game played as black in Figure 17 is played against the iteration 439 of the same model.

A.6 Lichess Evaluation

We ran the final iteration of the 5-layer AlphaGateau model on Lichess, as the bot AlphaGateau ( https://lichess.org/@/AlphaGateau ). We let it play against other bots and some human players in bullet, blitz, and rapid time formats by varying the number of MCTS simulations to adjust the time required to play each move. This was implemented using the default Lichess bot bridge ( https://github.com/lichess-bot-devs/lichess-bot ), and the relevant code is in the lichess folder of our GitHub repo.

At the end of October 2024, after between 100 and 200 games per time format, AlphaGateau was able to reach a bullet Elo of 1991 , a blitz Elo of 1829 , and a rapid Elo of 1884 .

1. e4 c6 2. h3 h6 3. B e2 N a6 4. g4 g6 5. a4 R h7 6. f4 f6 7. N f3 B g7 8. h4 d5 9. e5 N b4 10. c3 B xg4 11. cxb4 B h5 12. N g1 B h8 13. N c3 d4 14. R h3 a5 15. exf6 N xf6 16. bxa5 e6 17. B xh5 gxh5 18. b4 N g8 19. R d3 N f6 20. N ge2 N g4 21. Q c2 Q xh4 + 22. N g3 Q h2 23. N ge4 Q h1 + 24. K e2 R g7 25. R xd4 Q h2 + 26. K f3 Q h1 + 27. K e2 R f7 28. R d3 Q g2 + 29. K e1 R g7 30. R a2 Q h1 + 31. K e2 K e7 32. R g3 R e8 33. Q d3 R d8 34. Q c4 Q h4 35. R a1 R f7 36. R a2 R xf4 37. d4 R f7 38. B e3 R g7 39. R c2 K f7 40. Q c5 K e8 41. R f3 Q h2 + 42. K d3 Q h1 43. R f4 N xe3 44. K xe3 R g1 45. R e2 R xd4 46. N f2 R xf4 47. N xh1 R h4 48. N f2 K d7 49. b5 R g3 + 50. K d2 B xc3 + 51. K c2 B h8 52. Q b6 K d6 53. a6 bxa6 54. Q d8 + K c5 55. Q xh4 R g8 56. bxc6 K xc6 57. Q f4 B g7 58. R xe6 + K d5 59. Q f5 + K c4 60. Q f3 K b4 61. Q xh5 K xa4 62. R xa6 + K b4 63. Q f5 R h8 64. R a7 B d4 65. R b7 + B b6 66. N d3 + K a4 67. R e7 h5 68. Q f6 B d8 69. Q f4 + K b5 70. R e5 + K b6 71. N f2 K c7 72. N e4 K c6 73. K c3 K b6 74. K d4 K c6 75. N g3 K b6 76. K c4 K b7 77. Q f3 + K c7 78. K d5 h4 79. N e2 h3 80. Q f4 K d7 81. N g3 K c7 82. R h5 + K d7 83. R xh8 h2 84. R xh2 B c7 85. Q f2 K c8 86. Q f5 + K b7 87. Q d7 K a8 88. K e4 B b8 89. R h6 B a7 90. R f6 B g1 91. R f5 K b8 92. R f8#

Figure 15: Game played by a model fully trained only on 5 × 5 chess as white. White is able to use their white bishop to eliminate black's white bishop, but seems to undervalue their knight on move 14, probably because it is a worse piece in 5 × 5 chess due to being more constrained and harder to effectively employ

1. e4 c5 2. N b1c3 e6 3. N g1f3 N b8c6 4. d4 d4 5. N f3d4 N g8f6 6. N d4c6 c6 7. e5 N f6d5 8. N c3e4 Q d8c7 9. f4 Q c7b6 10. B f1e2 B c8a6 11. B e2a6 Q b6a6 12. a3 h5 13. Q d1e2 Q a6e2 14. K e1e2 f5 15. N e4c3 a5 16. N c3d5 d5 17. B c1e3 B f8e7 18. c3 R h8g8 19. K e2f3 g5 20. g3 g4 21. K f3e2 h4 22. b4 K e8f7 23. R a1c1 h3 24. B e3b6 b4 25. b4 R a8a3 26. R c1c7 R g8b8 27. B b6c5 B e7c5 28. R c7c5 R b8b4 29. R c5c2 d4 30. R h1d1 K f7e7 31. K e2f2 R a3f3 32. K f2g1 d3 33. R c2c3 R b4b2 34. R c3d3 R f3d3 35. R d1d3 R b2c2 36. R d3d1 R c2g2 37. K g1h1 R g2a2 38. K h1g1 K e7e8 39. R d1b1 R a2g2 40. K g1h1 R g2c2 41. K h1g1 R c2c3 42. R b1b8 K e8e7 43. R b8b1 R c3c2 44. R b1b7 R c2c1 45. K g1f2 R c1h1 46. K f2e2 R h1h2 47. K e2f1 R h2g2 48. R b7b3 R g2d2 49. K f1g1 K e7f7 50. R b3b7 K f7f8 51. R b7b8 K f8f7 52. K g1h1 R d2g2 53. R b8b7 K f7e7 54. R b7d7 K e7e8 55. R d7d3 R g2f2 56. R d3d6 K e8e7 57. R d6c6 R f2f3 58. R c6c7 K e7d8 59. R c7c6 K d8d7 60. R c6d6 K d7e7 61. K h1h2 R f3f2 62. K h2h1 R f2f3 63. K h1h2 R f3f2 64. K h2h1 R f2g2 65. R d6d3 R g2c2 66. R d3d4 R c2c3 67. K h1h2 R c3a3 68. R d4d2 R a3a6 69. R d2e2 R a6a5 70. R e2d2 R a5a1 71. R d2b2 R a1a8 72. K h2h1 R a8c8 73. K h1g1 R c8a8 74. R b2b6 K e7d7 75. R b6d6 K d7e7 76. K g1h1 R a8a3 77. K h1h2 R a3a2 78. K h2h1 R a2a7 79. K h1g1 R a7a1 80. K g1h2 R a1a2 81. K h2h1 R a2e2 82. R d6d4 R e2g2 83. R d4d3 K e7e8 84. R d3a3 R g2e2 85. R a3d3 R e2e1 86. K h1h2 R e1e2 87. K h2h1 R e2b2 88. R d3c3 R b2a2 89. R c3c6 K e8d7 90. R c6d6 K d7e7 91. R d6c6 R a2e2 92. R c6c3 R e2g2 93. R c3d3

Figure 16: AlphaGateau starts with a closed Sicilian, transposing into the Four Knights Sicilian, following a popular line until move 10, when white moves its white bishop to e2. The pawn structure locks the situation by move 35. Nothing much happens before the game ends in a draw, besides an interesting stalemate trick on move 54

1. e4 e5 2. N f3 N c6 3. B b5 N f6 4. d3 B c5 5. B xc6 dxc6 6. O -O Q e7 7. B g5 O -O 8. B h4 h6 9. N bd2 b5 10. Q e1 a5 11. h3 B b6 12. B g3 R e8 13. B xe5 a4 14. B c3 N h5 15. a3 N f4 16. K h2 f5 17. Q d1 R f8 18. exf5 R xf5 19. Q e1 Q f7 20. g4 R c5 21. N e4 R d5 22. R g1 N e6 23. N h4 N g5 24. N f6 + gxf6 25. f4 N e6 26. R g3 N d4 27. Q e4 B d7 28. R e1 R e8 29. Q g2 R xe1 30. B xe1 Q e6 31. B f2 N e2 32. B xb6 cxb6 33. f5 Q e5 34. N f3 Q xg3 + 35. Q xg3 N xg3 36. K xg3 B c8 37. K f4 R d7 38. N d2 K f7 39. N e4 B a6 40. h4 c5 41. g5 fxg5 + 42. hxg5 hxg5 + 43. N xg5 + K g7 44. K e5 R e7 + 45. N e6 + K f7 46. K d6 B c8 47. N g5 + K f6 48. N e4 + K f7 49. N g5 + K f8 50. f6 R d7 + 51. K c6 c4 52. N e6 + K f7 53. N f4 K xf6 54. N d5 + K e6 55. N xb6 cxd3 56. cxd3 B b7 + 57. K xb5 R xd3 58. N xa4 K d6 59. K c4 R h3 60. N c3 K c6 61. b4 B c8 62. a4 B e6 + 63. K d4 B b3 64. a5 R h4 + 65. K e5 R xb4 66. a6 K b6 67. N b1 B c2 68. N c3 B b3 69. N e2 K c5 70. N f4 R a4 71. N d3 +K c6 72. N f4 B c4 73. N g6 B xa6 74. N f4 B c4 75. N g2 B b3 76. N f4 B c2 77. N e6 R e4 + 78. K f5 R e1 + 79. K f6 K d6 80. N f4 R g1 81. N e2 R f1 + 82. K g5 K e5 83. N g3 R f7 84. N h5 B d3 85. N g3 R g7 + 86. K h4 R g8 87. K h3 K f4 88. N h5 + K g5 89. N g3 R h8 + 90. K g2 K f4 91. N f1 R b8 92. N g3 R b2 + 93. K h3 B g6 94. N h5 + B xh5 95. K h4 R h2#

Figure 17: AlphaGateau starts playing a Berlin defence, without any book, and diverts by move 4 into a popular line, with a rare bishop move on move 7. The middlegame revolves around black strong knight, until white is forced to give up its remaining rook to stop the attack, leaving black up a rook for a pawn.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The results discussed in the experiments and illustrated in Figures 5, 6, and 7 support the claims in the abstract and the introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Our main limitations are that we had limited computing resources and had models of depth 5 or 10 when 40 would be better, and we only focused on chess and not other games. We could also only run the experiments once in the time available to us, so the confidence intervals only apply to the rating estimations, and were not extimated over several training runs.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The only theoretical result is the closed form derivation of the confidence interval for the estimated Elo ratings, included in the appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We included all the experimental hyperparameters and methodology in the paper, and include all the code that was used in the GitHub repository mentioned in the abstract. However, as the parallelized GPU code is not fully deterministic, it is not possible to replicate the exact training results we present, so we will also publish in the git repository some important model parameter checkpoints that we saved.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The full code is published in the GitHub repository Akulen/AlphaGateau.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The full training procedure is described in the paper, and the provided code contains the implementation details, such as the gpu data split procedure.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide confidence intervals on our estimated Elo ratings, however, as each model was only trained once, they are incomplete confidence intervals.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include the number and type of GPU, as well as the experiment run time. Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: We introduce an improvement to an existing DRL architecture to train a game agent. There should be no direct societal impact from this, as the results are currently limited to making research on this topic more accessible, due to the training efficiency gains.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The released models are way less powerful than current existing chess engines, as they are limited in size and depth for compute reasons, so there is no risk in publishing them.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite every non-standard python library used to develop and run the experiments presented in this paper. We do not use any previous dataset or asset besides those libraries.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The Git repo that will be published along the paper will have well structured code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new variant of AlphaZero, AlphaGateau, with a neural network architecture based on GNNs, that allows the model to generalize across board sizes. The new architecture outperforms the original AlphaZero architecture under certain conditions.\n\nSTRENGTHS:\nThe paper follows a promising direction, using GNNs in an AlphaZero-like model to utilize the graph nature of many board games. \nAlphaGateau is an original model that shows potential in terms of data efficiency and fast training. It learns much faster than AlphaZero during the first 100 training iterations, saturating at a much higher Elo compared to that of AlphaZero, which does not improve much in this timeframe.\nIt is clear that when both models are tested on equal grounds, in the setting used, AlphaGateau is significantly more data efficient, and improves its Elo score much faster than AlphaZero. These are significant improvements on the original AlphaZero model if they hold under rigorous evaluation.\n\nWEAKNESSES:\nEvaluation & comparison with AlphaZero: \nWhile figure 5 clearly shows that AlphaGateau learns much faster than AlphaZero at early training, the paper does not provide any information on how the two models compare when they are fully trained. AlphaGateau seems to reach a performance plateau within 100 steps, but the AlphaZero model trained by the authors would likely keep on improving for orders of magnitude more training steps. For comparison, the authors used a model size roughly 10% of Silver et. al.'s AlphaZero, but trained for 100 steps compared to the original 700,000 steps. It is not clear if AlphaGateau's performance will be comparable with AlphaZero, given enough training steps.\nI am aware that training the models for 10^5 steps is extremely expensive and not feasible for this paper. Even so, the correct solution would have been to fit the model size to the training budget and train a significanly smaller AlphaZero model, for longer time and on smaller batches (or to heavily reuse training data between optimization steps). Doing a comparison of smaller-scale models could show if AlphaGateau is a comparable model to AlphaZero. \n\nIn the paper's current form, the reported Elo gap does not give the reader any knowledge about how AlphaGateau compares to a fully-trained AlphaZero, since Elo calculation is unreliable when using such large gaps. For example, the gap between the best performing AlphaGateau and best AlphaZero is about 1,500 Elo, meaning that AlphaZero would win about 1 game in every 7,500 games. This implies that the AlphaGateau Elo score is based purely on comparisons with earlier AlphaGateau training checkpoints. \nAnother example is the gap between the first and second checkpoints, which is hard to measure by eye due to the plot format but looks roughly equal to 800. This gap implies the weaker model wins 1 game in 100, suggesting that the error of this gap is very high unless the authors use thousands of test games between this specific checkpoint pair to calculate Elo. I couldn't find the number of Elo games in the paper, and would recommend stating it clearly.\n\nConsidering that the best model's Elo is probably based purely on comparisons to its earlier versions, and considering the possible error of the Elo estimate, it is possible that AlphaGateau's Elo rating will drop significantly if fully-trained AlphaZero models are added to the pool of players.\n\nThis paper will benefit significantly from fitting the size of the experiments to the compute resources available to the authors. I would suggest training significanly smaller models for 100x more training steps, using the same compute budget. In its current form, the paper only showcases the advantage AlphaGateau has in terms of data-efficiency at early training, in a regime where AlphaZero is severely undertrained.\n\nQUESTIONS:\n- What hyperparameters are used to calculate Elo scores? (number of models tested, number of games between each model pair, who played against who)\n\nThree minor suggestions:\n- It would be helpful for the reader if figures 5 and 6 had FLOPs on the x-axis instead of steps, or if training compute was reported. Plotting only steps, it is hard to understand if one training step of AlphaGateau is comparable to a training step of AlphaZero in terms of compute.\n- It would be easier and more convincing to use traditional methods for calculating Elo scores, such as BayesElo which was used by Silver et. al. That said, the authors provide clear explanations on their Elo calculation method.\n- The correct spelling of the metric is \"Elo\", named after Arpad Elo, rather than \"ELO\", which is a common typo.\n\nLIMITATIONS:\nNo limitations other than those specified above regarding evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores a novel approach to  reinforcement learning for Chess by utilizing a graph-based representation of the game state instead of the traditional grid-based representation. This method is based on GNNs  and aims to overcome the limitations of CNNs used in previous models like AlphaZero. Specifically, the paper introduces the Graph Attention neTwork with Edge features from Attention weight Updates (GATEAU) layer, which enhances the classical GAT layer by incorporating edge features.\n\nThe primary contributions of the paper include demonstrating that the new architecture outperforms previous models with a similar number of parameters and significantly accelerates the learning process. Additionally, the paper provides evidence that the model trained on a smaller variant of chess (5x5) can be quickly fine-tuned to perform well on the standard 8x8 chessboard, indicating promising generalization capabilities. The authors have made their code available, supporting the reproducibility of their experiments and findings.\n\nSTRENGTHS:\n**Novel Graph-based Approach**: This paper introduces a creative way to represent game states using a graph-based model instead of the traditional grid-based models. By employing Graph Neural Networks and the Graph Attention neTwork with Edge features from Attention weight Updates, the authors address significant limitations found in Convolutional Neural Networks used in models like AlphaZero.\n\n**Improved Learning Efficiency**: The proposed AlphaGateau model demonstrates much faster learning compared to traditional CNN-based models. Experiments show that AlphaGateau can achieve a significant increase in playing strength in a fraction of the training time required by AlphaZero, which is a notable improvement in learning efficiency.\n\n**Reproducibility**: The authors have made their code publicly available, supporting the reproducibility of their experiments.\n\nWEAKNESSES:\n**Limited Generalization Evidence**: The paper claims promising generalization capabilities, but the experimental evidence is limited to a small variant of chess (5x5) and standard chess (8x8). It would strengthen the paper to include additional experiments on other games or variants, such as Shogi or other board games with similar complexity, to demonstrate the broader applicability of the proposed architecture.\n\n**Comparative Analysis**: The paper lacks a thorough comparative analysis with other state-of-the-art models beyond AlphaZero. Including comparisons with recent advancements in graph-based reinforcement learning models would provide a clearer picture of the relative performance and innovations of the proposed method.\n\n**Real-world Applicability**: While the focus is on reinforcement learning in chess, discussing potential real-world applications of the proposed graph-based approach would broaden the impact of the work. Highlighting areas where this approach could be applied, such as other strategic games or decision-making problems in different domains, would make the contributions more compelling.\n\nQUESTIONS:\n1. How does AlphaGateau compare to other graph neural network-based reinforcement learning models, such as those presented by Ben-Assayag and El-Yaniv (2021) [1]?\n2. Have the authors considered applying AlphaGateau to other strategic games beyond chess? If so, what preliminary results or observations can the authors share?\n3. What potential real-world applications besides chess do the authors envision for the graph-based approach proposed in this paper?\n\n[1] Ben-Assayag, S., & El-Yaniv, R. (2021). Train on small, play the large: Scaling up board games with alphazero and gnn. *arXiv preprint arXiv:2107.08387*.\n\nLIMITATIONS:\nThe authors have acknowledged some limitations of their work, such as the restricted computational resources which prevented the training of deeper networks, and their focus solely on chess without extending experiments to other games. They also discussed challenges related to the reproducibility of their results due to the non-deterministic nature of parallelized GPU code. However, the paper could benefit from a deeper discussion on the broader implications of these limitations, particularly how they might affect the generalizability and robustness of the proposed method. Regarding potential negative societal impacts, the paper does not explicitly address this aspect.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors demonstrate a GNN that works on chess and is amenable to generalization.\n\nSTRENGTHS:\n# originality\n\nThis is the first chess GNN approach that performs well that I'm aware of in the literature, and requires some invoations in the edge representation. \n\n# quality\n\nThe results look good, although the computational limitation of the work mean that a larger test would be good to demonstrate that this model scales up.\n\n# clarity\n\nThe paper is clear and readable.\n\n# significance\n\nThis is a new result in chess AI and looks like it could matter to the larger RL community. I'm not sure this will impact much chess engines in practice though as the GNN is likely much harder to optimize and implement than CNNs or heuristics, the memory usage on the attention mechanism for example would make implementing this in a high performance search code-base difficult.\n\nWEAKNESSES:\nThe authors only look at self-play for model training, the original AlphaGo models were trained on human games to establish that the algorithm performed well, before starting the much more expensive self-play training. The lack of supervised training evaluations is makes the claims of strong performance limited to a single axis (sample efficiency).\n\nThis result mostly seems to arise from chess specific optimizations applied to an existing algorithm [18], while the authors suggest that it could be applied to other games they only provide evidence in chess.\n\nThe fine tuning results require more training time (and thus I'm assuming compute) than the initial training, this suggests the pre-training while beneficial is of limited efficacy unless it can be made more efficient.\n\nI'm also concerned with the  attention-based pooling leading degrading in performance when the graph gets larger as chess is a game that cares much more about the max than the average.\n\nI would have liked to see a more complete training run, but I understand that RL is very compute intensive so am not using this lack to negatively impact my score.\n\nQUESTIONS:\nIs the MCTS process identical between AlphaGateau and AlphaZero models? Or does the graph representation change it?\n\nCould the graph representation be leveraged to allow a different search algorithm?\n\nCould the authors provide PGNs with model predictions and some discussion of the difference in the AlphaGateau and AlphaZero models? There are patterns that CNN based models are good at learning and some that are more difficult (see [1] for examples) and showing differences in the models would strengthen this result.\n\nDoes the model handle the white bishop well when scaled up? If it does that would be additional evidence of generalization when scaling up since the moves would not have been seen before.\n\nCan you provide more details on the Elo calculation? You use the terms ELO and elo and seem to refer to a varation of the Elo rating system created by Arpad Elo, but don't give much details beyond the optimizer. Are you using and Elo or Glicko (2) type system or something else? Section A has not citations so it's unclear. \n\n[1] McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Wattenberg, M., Hassabis, D., Kim, B., Paquet, U. and Kramnik, V., 2022. Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences, 119(47), p.e2206625119.\n\nLIMITATIONS:\nDiscussed above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}