Scaling Laws for a Multi-Agent Reinforcement Learning Model
Authors: Oren Neumann
Claudius Gros
Venue: ICLR 2023 poster (2023)
Keywords: Neural scaling laws
Multi-agent reinforcement learning
AlphaZero

Abstract: The recent observation of neural power-law scaling relations has made a significant impact in the field of deep learning. A substantial amount of attention has been dedicated as a consequence to the description of scaling laws, although mostly for supervised learning and only to a reduced extent for reinforcement learning frameworks. In this paper we present an extensive study of performance scaling for a cornerstone reinforcement learning algorithm, AlphaZero. On the basis of a relationship between Elo rating, playing strength and power-law scaling, we train AlphaZero agents on the games Connect Four and Pentago and analyze their performance. We find that player strength scales as a power law in neural network parameter count when not bottlenecked by available compute, and as a power of compute when training optimally sized agents. We observe nearly identical scaling exponents for both games. Combining the two observed scaling laws we obtain a power law relating optimal size to compute similar to the ones observed for language models. We find that the predicted scaling of optimal neural network size fits our data for both games. This scaling law implies that previously published state-of-the-art game-playing models are significantly smaller than their optimal size, given the respective compute budgets. We also show that large AlphaZero models are more sample efficient, performing better than smaller models with the same amount of training data.

SCALING LAWS FOR A MULTI-AGENT REINFORCE- MENT LEARNING MODEL
Oren Neumann & Claudius Gros Institute for Theoretical Physics Goethe University Frankfurt
Frankfurt am Main, Germany
{neumann, gros www.uui-ifaauuuut.eee.ee.
ABSTRACT
The recent observation of neural power-law scaling relations has made a signifi- cant impact in the field of deep learning. A substantial amount of attention has been dedicated as a consequence to the description of scaling laws, although mostly for supervised learning and only to reduced extent for reinforcement learning frameworks. In this paper we present an extensive study of performance scaling for cornerstone reinforcement learning algorithm, AlphaZero. On the ba- sis of a relationship between Elo rating, playing strength and power-law scaling, we train AlphaZero agents the games Connect Four and Pentago and analyze their performance. We find that player strength scales as power law in neural network parameter count when not bottlenecked by available compute, and as a power of compute when training optimally sized agents. We observe nearly iden- tical scaling exponents for both games. Combining the two observed scaling laws we obtain power law relating optimal size compute similar to the ones ob- served for language models. We find that the predicted scaling of optimal neural network size fits our data for both games. We also show that large AlphaZero models are more sample efficient, performing better than smaller models with the same amount of training data.
INTRODUCTION
In recent years, power-law scaling of performance indicators has been observed in range of machine-learning architectures (Hestness et al., 2017 Kaplan et al., 2020 Henighan et al. 2020 Gordon et al. 2021 Hernandez et al. 202 Zhai et al. 2022), such as Transtormers, LSTMs, Rout- ing Networks (Clark et al. 2022) and ResNets (Bello et al. 2021). The range of fields investigated include natural language processing and computer vision (Rosenfeld et al. 2019). Most of these scaling laws regard the dependency of test loss on either dataset size, number of neural network parameters, training compute. The robustness of the observed scaling laws across many orders of magnitude led to the creation of large models, with parameters numbering in the tens and hundreds of billions (Brown et 2020 Hoffmann et 2022 Alayrac et al. 2022).
Until now, evidence for pawer-law scaling has come in most part from supervised learning methods. Considerably less effort has been dedicated the scaling of reinforcement learning algorithms, such as performance scaling with model size (Reed et al., 2022 Lee et al., 2022). At times, scaling laws remained unnoticed, given that they show up not as power laws, but as lineir relations when Elo scores are taken as the performance measure in multi-agent reinforcement learning (MARL) (Jones) 2021 Liu et al.) (02T) (see Section 3.2). Of particular interest in this context is the AlphaZero family of models, AlphaGo Zero (Silver al. 2017b). AlphaZero (Silver et al.) 2017a), and MuZero (Schrittwieser et al. 2020), which achieved state-of-the-arr performance on several board games without access human gameplay datasets by applying tree search guided by neural network.
Here we present an extensive study of power-law scaling in the context of two-player open- information games. Our study constitutes, to knowledge, the first investigation of power-law scaling phenomena for a MARL algorithm. Measuring the performance of the AlphaZero algorithm using Elo rating, we follow similar path Kaplan al. (2020) by providing evidence of power-law

---

10
Optimal size (parameters)
10a
10a
AlphaGo Zero
AlphaZero
10a
10
104
10
Connect Four
10
Pentago
10
10a
Mye (C/10*
1011
10
10th
10
10
10
10
10a4
10
10
10a0
10
10a
FLOPs
FLOPs
Figure 1: Left: Optimal number of neural network parameters for different amounts of available compute. The optimal agent size scales for both Connect Four and Pentago as single power law with available compute. The predicted slope a ac/aN Eq. (7) matches the observed data, where ac and ax are the compute and model size scaling exponents, respectively. See Table 3 for numerical values. Right: The same graph zoomed out include the resources used create AlphaZero (Silver et al. 2017a) and AlphaGoZero (Silver et al. 2017b). These models stand well bellow the optimal trend for Connect Four and Pentago.
scaling of playing strength with model size and compute, as well as power law of optimal model size with respect to available compute. Focusing on AlphaZero- agennts that are guided by neural nets with fully connected layers, we test our hypothesis two popular board games: Connect Four and Pentago. These games are selected for being different from each other with respect to branching factors and game lengths.
Using the Madley-Terry model definition of playing strength (Bradley & Terry 1952). we start by showing that playing strength scales as a power law with neural network size when models trained until convergence in the limit of abundant compute. We find that agents trained on Connect Four and Pentago scale with similar exponents.
In a second step we investigate the trade-of off between model size and compute. Similar to scaling observed in the game Hex (Jones) 2021), we observe power-law scaling when compute limited, again with similar exponents for Connect Four and Pentago. Finally we utilize these two scaling laws find scaling law for the optimal model size given the amount of compute available, as shown in Fig. [1] We find that the optimal neural network size scales power law with compute, with an exponent that can be derived from the individual size-scaling and compute-scaling exponents. All code and data used in our experiments available online
2 RELATED WORK
Little work on power-law scaling has been published for MARL algorithms. Schrittwieser et al. (202T) report reward scaling as a power law with data frames when training a data efficient vari- ant of MuZero. Jones (021), the closest work performance with compute, by measuring the performance of AlphaZero agents on small-board vari- ants of the game of Hex. For board-sizes 3-9, log-scaling of Elo rating with compute is found when plotting the maximal scores reached among training runs. Without making an explicit connection to power-law scaling, the results reported by Jones (2021) can be characterized by compute exponent of ag &e 1.3, which can be shown when using Eq 3 In the paper, the author suggests a phenomeno- logical explanation for the observation that an agent with twice the compute of its opponent seems to win with probability of roughly 2/3, which in fact corresponds to a compute exponent of ac = 1. Similarly, Liu et al. (202T) report scores that appear to scale as a log of environment frames for humanoid agents playing football, which would correspond to power-law exponent of roughly 0.5 for playing strength scaling with data. [ee al. (2022) apply the Transformer architecture Atari games and plot performance scaling with the number of model parameters. Due the substantially increased cost of calculating model-size scaling compared to compute or dataset-size scaling, they obtain only limited number of data points, each generated by single training seed. On this ba-
own, shows evidence of power-law scaling of
https:://githbb...m///eeeeeeeeee /ttp//////////////////////////// wcw.snaaaaaaaaaaaaa 1aws
2

---

sis, analyzing the scaling behavior seems difficult. First indications of the model-size scaling law presented in this work can found Neumann & Gros 2022
BACKGROUND
3.1
LANGUAGE MODEL SCALING LAWS
Kaplan et al. (2020) showed that the cross-entropy loss of autoregressive Transformers (Vaswani al., 2017) scales as a power law with model size, dataset size and compute. These scaling laws hold when training is not bottlenecked by the other two resources. Specifically, the model size power law applics when models are trained to convergence, while the compute scaling law is valid when training optimal-sized models. By combining these laws power-law scaling of optimal model size with compute is obtained, with an exponent derived from the loss scaling exponents. These exponents, later recalculated by Hoffmann et al. 2022), tend to fall in the range [0, 1].
3.2 POWER LAW SCALING AND ELO RATING
The Elo rating system Elo 1978) is a popular standard for henchmarking of MARL algorithms, as well as for rating human players in zero sum games. The ratings r; are calculated fit the expected score of player i in games against player j:
E, 1 10(r,-r.)/100
where possible game scores are {0, 0.5, 1} for loss/draw/win respectively. This rating system is built on the assumption that game statistics adhere to the Bradley- Terry model (Bradley & Terry) 1952), for which each player i is assigned a number % representing their strength. The player- specific strength determines the expected game outcomes according to:
E
(2)
7
Elo rating is log-scale representation of player strengths (Coulom 2007
400 log (7:).
An observed logarithmic scaling of the Elo score is hence equivalent to a power-law scaling of the individual strengths: If there exists constant such that т; log1o (X,) for some variable X; (e.g. number of parameters), then the playing strength of player scales as power of X₃:
Y x X."
(3)
where a = e/400. Note that multiplying 76 by constant is equivalent to adding a constant to the Elo score rs, which would not change the predicted game outcomes. This power law produces simple expression for the expected result of a game between and j:
(x,
4 EXPERIMENTAL SETTING
We train agents with the AlphaZero algorithm using Monte Carlo tree search (MCTS) (Coulom 2006 Browne et al. 2012) guided by a multilayer perceptron. Kaplan et al. (2020) have observed
Table 1: Summary of scaling exponents. a describes the scaling of performance with model size, ac of performance with compute, and ac and Pentago have nearly identical values.
opt of the optimal model size with compute. Connect Four
Exponents Connect Four
Pentago
ON 0.88
0.87
ac
0.55
0.55
a
opt
0.62
0.63
3

---

that autoregressive Transformer models display the same size scaling trend regardless of shape de- tails, provided the number of layers is at least two. We therefore use a neural network architecture where the policy and value heads, each containing folne-connected hidden layer, are mounted on a torso of two fully-connected hidden layers. All hidden layers are of equal width, which is varied between 4 and 256 neurons. Training is done using the AlphaZero Python implementation available in OpenSpiel (Lanctot et al. 2019)).
In order make our analysis as general as possible we specifically avoid hyperparameter tuning whenever possible, fixing most hyperparameters to the ones suggested in OpenSpiel's AlphaZero example code, see Appendix [A] The only parameter tailored each board game is the temperature drop, as we find that it has substantial influence on agents' ability to learn effectively within a span of 10a training steps, the number used our simulations.
We focus on two different games, Connect Four and Pentago, which are both popular two-player zero-sum open-information games. In Connect Four, players drop tokens in turn into a vertical board, trying to win by connecting four of their tokens in a line. In Pentago, players place tokens on a board an attempt to connect line of five, with each turn ending with a rotation of one of the four quadrants of the board. These two games are non-trivial to learn and light enough to allow for training a larger number of agents with a reasonable amount of resources. Furthermore, they allow for the benchmarking of the trained agents against solvers that are either perfect (for Connect Four) or near-to-pertee -e--effect (for Pentago). Our analysis starts with Connect Four, for which we train agents with varying model sizes. Elo scores evaluated both within the group of trained agents and with respect to an open source game solver (Pons) 2015). We follow up the Connect Four results with agents trained on Pentago, which has much larger branching factor and shorter games (on average, see Table 2
For both games we train AlphaZero agents with different neural network sizes and/or distinct com- pute budgets, repeating each training six times with different random seeds to ensure reproducibility (Henderson al., 2018). We run matches with all trained agents, in order to calculate the Elo rat- ings of the total pools of 1326 and 714 agents for Connect Four and Pentago, respectively. Elo score calculation done using BayesElo (Coulom 2008).
RESULTS
5.1 NEURAL NETWORK SIZE SCALING
Fig. (top) shows the improvement of Connect Four Elo scores when increasing neural network sizes across several orders of magnitude. We look at the infinite compute limit, in the sense that performance is not bottlenecked by compute-time limitations, by training all agents exactly 1 optimization steps. One observes that Elo scores follow a clear logarithmic trend which breaks only when approaching the hard boundary of perfect play. In order to verify that the observed plateau in Elo is indeed the maximal rating achievable, we plot the Elo difference between each agent and an optimal player guided by a game solver (Pons) 2015) using alpha-beta pruning (Knuth & Moore, 1975). A vertical line in both plots marks the point beyond which we exclude data from the fit. This is also the point where agents get within 10 Elo points difference from the solver.
Combining the logarithmic fit with Eq. yields simple relation between the expected game score E, for player against player j and the ratio N./N,, of the respective numbers of neural network
Table 2: Game details. Connect Four games last longer on average, but the branching factor of
Pentago is substantially larger. Game lengths are averaged over all training runs, citing the maximal allowed number of turns as well.
Game
Branching Factor
Average Game Length
Maximal Game Length
Connect Four
< 7
25.4
42
Pentago < 288
16.8
36
4

---

Connect Four
Elo relative to solver
1000
900
-200
800
Relative Elo
-400
700
600
Mean Elo
800
400
Agent
-1000
0.88 400logio(N)
695
1200
10
10a
10a
10a
10 Parameters
10a
Parameters
Pentago
Elo relative to solver
1100
1000
-200
900
Relative Elo
-100
-600
400
Agent Elo
-800
300
0.87 400logoe(M)
669
1000
10a
10
10a
10a
10
10a
Parameters
Parameters
Figure 2: AlphaZero performance scaling with neural-network parameters. Top left: Agents trained on Connect Four. Elo scores follow clear linear trend in log scale that breaks off only when approaching the optimal playing strategy. This can can be seen using game solver (top right). Top right: Elo scores obtained by only considering matches against game solver (Pons) 2015). Agents to the right of the vertical line are within ten Elo-points distance from the perfect solver. Increased data scattering is due to Elo scores being calculated against single benchmark player. Bottom left: The Elo scores of Pentago
optimal player, created with
function of model size follow the same trend as Connect Four. Improvement flattens for large parameter counts when performance approaches the level of benchmark agent aided by optimal play database (Irving) 2014) (bottom right).
parameters:
E, 1+ (N,/N,)
(5)
where the power o. is a constant determined by the slope of the fit. In the context of the Bradley- Terry model, Eq. (2) this means that player strength follows a power law model size, y x Nox
Results for the game of Pentago presented at the bottom of Fig. 2 Pentago similarly exhibits log scaling of average agent Elo scores as function of model size that flattens out only with a high number of neural network parameters. The scaling exponent is strikingly close to that of Connect Four (see Table [ suggesting this exponent might be characteristic for a certain, yet unknown class of games. Again, in order verify that the observed plateau is due to approaching the boundary of optimal play, we plot the Elo difference between each agent and a close to optimal benchmark player aided by game solver. The bottom right of Fig. 2 shows that large agents are much closer to the level of the benchmark player. The benchmark player is based on the best performing Pentago agent trained, given 10 times more MCTS steps per move. We further improve by constraining all moves before ply 18 to be optimal, using 4TB dataset created by a Pentago solver program (Irving) 2014). A vertical line in both plots marks the point the log trend breaks, where agents approach the level of the benchmark player. Similar to Connect four exclude the largest agent size beyond that point from the fit where adding more parameters no longer increases performance.
The law scaling of the playing strength of MARL agents can be considered analogous to the scaling laws observed for supervised learning. A noteworthy difference is the type of performance parameter being scaled. Scaling laws for language models tend to scale cross-entropy test loss with resources, which has been correlated to improved performance on downstream tasks (Hoffmann) et al. 2022). In comparison, Eq. 5 shows power law scaling of player strength rather than test loss,
5

---

Connect Four
Pentago
800
800
10a
10a
steps
600
steps
10a
102
200
200
10a
10a Parameters
10a
10a
10a
Parameters
Figure 3: Elo scaling with parameters during training. For both games, log-linear scaling appears already at early stages training, albeit with lower slopes. Elo curves converge for large numbers of training steps when agents approach their maximal potential.
which is directly related the Elo score, common measure of task performance in MARL. In fact, neural network loss on test set generated by a game solver are observed to be poor predictor of performance scaling both for the value head and the policy head, Appendix
5.2 EFFECT OF TRAINING TIME ON SIZE SCALING
Similar to language model scaling laws, the scaling exponent of Eq. 5 onnly applies at the limit of infinite compute when training is not bottlenecked by the number of training steps. Fig. 3 shows how performance scaling is influenced by the amount of available compute in terms of training steps. One observes that the Elo score scales logarithmically with model size for all training regimes when performance is not affected, as before, by the perfect play barrier. The slope increases with the number of training steps, but with diminishing returns, see Appendix [D] For both games one observes that Elo scores converge to limiting curve when the compute budget becomes large. There is hence a limiting value for the scaling exponent o,, as defined by Eq. (5).
5.3 COMPUTE SCALING
To examine compute scaling plot Elo scores against training self-play compute, defined here by S..T.F. D, where S is the number of optimization steps, 7 is the number of MCTS simulations per move, F the compute cost of single neural network forward pass, and D) is the amount of new datapoints needed to trigger an optimization step. We do not count the compute spent during optimization steps, which is much smaller than the compute needed generate training data. In our case we have approximately 2000 forward passes for each backward pass. We also approximate the number MCTS simulations maximum is reached at all game positions except late-game positions, where the remaining game tree is already fully mapped.
the maximum number of simulations allowed, since the
Fig. 4 shows the Elo scores of agents with different training compute measured in units of floating- point operations (FL.Oss). The Pareto front, which is the group of all agents with maximum Elo among agents with similar or less compute, follows log-linear trend up the point of optimal play. Together with Eq. this yields an expected game score of:
E, = 1+ (C,/C,)a'
(6)
for agents with training computes C. Cj and optimal neural network sizes. Player strength scales with the same exponent, x Cae This scaling behaviour also holds for the average agent performance when plot the Pareto front of compute curves averaged over multiple training runs with different random seeds (see Appendix D). Eq. 6 therefore applies not only to the best case training scenario. but also when training single agent instance.
As with the size scaling exponent, Connect Four and Pentago have similar compute scaling expo- nents (see Table T). We note that training compute exponents are significantly smaller than the

---

Connect Four
Pentago
10a
1000
800
10
Parameters
800
Parameters
600
104
400
400
10
200
Parato
200
front
r= 0.55 400 (qu(C) 2419
100
r= 0.55 4000logs(C)
2338
10
1012
1013
10
1015
10th
10
10
1034
10
10th
FLOPs
FLOPs
Connect Four training curves
Optimal data consumption
Elo relative Pareto front
Connect Four Pentago
200
10a
Data (game states)
Parameters
107
10+
-400
10
-600
10a
10a
10
10
10
10
10a
10a
10a
10*
FLOPs Pareto front FLOPS
Parameters
Figure 4: Top: Performance scaling with training compute. The Pareto front follows a log-linear trend in compute right up the optimal play plateau, both for Connect Four and Pentago (left/right panel). Bottom left: Average Connect Four training curves relative to the point at which they reach the Pareto front. Agents continue improve significantly after passing through the Pareto optimal point, meaning optimal training should stop long before convergence (compare to the training curves Silver al. (2017b a)). Bottom right: The amount data required reach the compute Pareto front increases as the parameter count increases. Large dots are the geometric mean of data usage on the Pareto front.
compute exponent obtained when applying Eq. 6 the data obtained by Jones (2021) for small- board Hex, which would be roughly ac 1.3. The reason for this difference is at present unclear. Possible explanations could be the different game types, or the choice of neural net architecture; while we keep the depth constant, Jones (221)) increase both depth and width simultaneously.
5.4 ESTIMATING THE COMPUTE OPTIMAL MODEL SIZE
We now turn to our central result, efficiency scaling. Scaling laws are attracting considerable at- tention as they provide simple rules for the optimal balancing between network complexity and computing resources. The question is, to be precise, whether we predict how many neural net- work parameters one should use when constrained by certain compute budget in order to get an AlphaZero agent with maximal Elo score.
The agents on the Pareto front shown in Fig. have the highest Elo score for all agents with the same lower compute budget. In order to find the relationship between optimal network size and the given compute budget, we examine the network size of the Pareto agents and the amount of FLOPSs they consumed in training. This is shown in Fig. which contains aggregate of all compute-optimal agents trained on Connect Four and Pentago. We see clear power law trend common both games of the form:
a
N Nopt (C) =
Co
where Nopt is the optimal neural network size and a Co are constants.
Next we point out that the optimality scaling exponent opt can be calculated analytically given the scaling laws demonstrated for parameter count and compute. For this we use Eq. (5) and Eq. (6)

---

Connect Four
Pentago
800
10
10
Parameters
800
Parammterrs
104
600
-10
200
200
10
0
10a
10
10
10a
10
10
Data (game states)
Data (game states)
Figure 5: Performance improvement with data (game states) generated. Model sizes are color coded. Larger models are more sample efficient, achieving better Elo scores than smaller models when training on the same amount of game states.
Based on the fact that the Bradley Terry playing strength y scales as limc (NN, x Nox at the limit of infinite compute, and as ([Vopt. C) C along the curve of optimal sizes, we can combine the two to obtain the constraint:
opt a
ON
This relationship obtained when Nopt and C' - co, but holds for every choice of N or C' since the exponents constants. The scale constant Co is caleulated empirically. We use the scaling exponents oN and ac obtained by fitting the Elo ratings for Connect Four and Pentago to opt via Eq. (8). The predicted scaling law is included Fig. fitting only Co. The predicted exponent fits the data remarkably well.
get ac
5.4.1 EXTRAPOLATING TO STATE OF THE ART MODELS
For comparison, Fig. right shows where AlphaGo Zero, trained by Silver al. (2017b), and AlphaZero, trained by Silver et al. (2)17)), would stand relative to the power law specified by Eq. (8). We base estimates for the number of parameters from the respective papers, and use the training compute values estimated by [Sevilla et al. (2022) and Amodei et al. (2018) for these models. We find that these models fall well below the optimal curve predicted from scaling MLP agents playing Connect Four and Pentago.
The scaling law plotted in Fig. miight accurately describe AlphaGo Zero and AlphaZero trained on Go, chess and shogi due to the differences between the games: these games are much larger in observation size and branching factor than Connect Four and Pentago, as well having different win conditions and game mechanics. The use of ResNets rather than MLPs can also be significant. remains to future study to determine if these models exhibit related scaling trends. In the case that Eq. (8) does apply to agents using ResNets and trained on Go, chess, or shogi, then better Elo score could have been achieved with the same training time by training larger neural nets by up to two orders of magnitude. A possible indication that these models were too small for the amount of compute spent training them can be seen from the training figures of [Silver et al. 2017bbaa a); the AlphaGo Zero and AlphaZero training curves both have long tails with minimal improvement at late stages of training. Our analysis shows that optimal training for fixed compute budget will stop long before performance has converged, in agreement with language model scaling laws (Kaplan al.) 2020), compare Fig. bottom left. That being said, a reason to favour smaller models is their lower inference-time compute cost, which can be critical in the of time limit move selection. We elaborate on inference compute scaling in Appendix
5.5 SAMPLE EFFICIENCY
Unlike supervised learning models where dataset size an important metric due to the cost of collecting it, AlphaZero not constrained by data limitations since it generates its data while training. It is however important to understand how data requirements change as it provides us an insight into other sub-fields of reinforcement learning where data generation is costly, for example
8

---

in robotics models, which need real life interactions with the environment in order to gather training data (Dulac-Amon Amold et al. 2021).
For this reason we look at performance scaling with data for different sized models, as shown in Fig. We observe that agents improve at different rates with the amount of training data, which is defined here by the number of game states seen during self-play. Specifically, agents with larger neu- ral nets consistently outperform smaller agents trained on the same amount of data, a phenomenon of data efficiency that has been observed in language models (Kaplan et al. 2020). However, when training is not bottlenecked by the amount of training data, an opposite trend appears: the amount of data required to reach optimal performance under fixed compute budget increases with model size, as is visible in the bottom right panel of Fig.
DISCUSSION
In this work we provided evidence to the existence of power-law scaling for the performance of agents using the AlphaZero algorithm two games, Connect Four and Pentago. For both games one observes scaling laws qualitatively paralleling the ones found neural language models. We pointed out that log-scaling of Elo scores implies power law in playing strengths, a relation that is captured by Eq. 4 and that can be used to predict game results. We showed that AlphaZero agents that are trained to convergence, with no limit available compute, achieve Elo scores that are pro- portional to the logarithm of the respective number of neural network parameters. We demonstrated this for two games with significantly different branching parameters and tyical game lengths, find- ing that they share similar scaling exponents for playing strength. This result indicates the possible existence of universality classes for performance scaling in two-player open information games.
We then demonstrated that Elo scores scale logarithmically with available compute when training optimally sized agents, again finding similar scaling exponents for the two games studied. We showed that the optimal number of neural network parameters scales as power of compute with an exponent that can be derived from the other two scaling laws. This scaling model is especially im- portant as it allows one to achieve maximal performance within the limits of given training budget by using an optimally sized neural net. We showed optimal training stops long before convergence, in contrast to what was done with state-of-the-art models. Finally we observed that agents with larger neural nets are always more data efficient than smaller models, which could be important in particular for reinforcement learning studies with expensive data generation.
We find it note worthy that scaling laws that are common to language and other supervised learning models are also present in one of the most important MARL models. This scaling behavior could be common to other reinforcement learning algorithms, which would provide an opportunity to optimize their resource allocation. We note that the main significance of the scaling exponents found in this study is not their reported value, but the evidence of power-law scaling. Consider the scaling exponents found by Kaplan al. (2020), which were later corrected by optimizing the learning rate schedule (Hoffmann et al.] (022). The tuning of AlphaZero's 's hyperparameters could affect the observed exponents in a similar manner. Particularly important hyperparameters are the number of states explored by the MCTS, the frequency of optimization steps, and the update frequency of the data generating model. We leave the investigation of hyperparameter impact to future work.
As a side remark, we would like to comment on the relevance of scaling laws for studies without massive computing budgets. A rising concern since the discovery of performance scaling laws regards the resulting push towards substantially increased model sizes. This decreases the impact of research utilizing smaller-sized models, the bread-and-butter approach of smaller research groups. (Strubell et al. 2019). On the other hand, the existence of scaling laws implies that it is possible to gain relevant insights into the behaviour of large models by training medium- sized models, the case of the present work. Interestingly, our initial test were performed
toy problem that involved training agents on a desktop computer with just four CPU cores. Even with such a minimal compute budget a clear scaling trend emerged (Neumann & Gros 2021). We believe our work shows that significant insights can be obtained at times also with small-scale analysis.
ACKNOWLEDGMENTS
We would like to thank Ivo FD Oliveira and Geoffrey Irving for helpful discussions and feedback.

---

REFERENCES
Jean-Baph San-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. /4/98, 2022.
Dario Amodei, Danny Hernandez. Girish SastryJack, ClarkGreg Brockman, and Ilya Sutskever. Ai and compute, 2018.
Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas. Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. Advances in Neural Information Processing Systems, 34:22614-22622 2021.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-344 34523444444224 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 33:1877-1901, 2020.
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter I Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1 113, 2012.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoff- mann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, pp. 4057- 4086. PMLR, 2022.
Rémi Coulom. Efficient selectivity and backup operators monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
Rémi Coulom. Computing "clo ratings" of move patterns in the game of go. ICGA journal, 30(4): 198-208, 2007.
Rémi Coulom. Whole-history rating: A bayesian rating system for players of time-varying strength. In International Conference on Computers and Games, pp. 1 113-124. Springer, 2008.
Gabriel Dulac-Amold, Nir Levine. Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. Challenges of real-world reinforcement learning: definitions, benchmarks and analysis. Machine Learning, (1009)2241-24661 2021.
Arpad E Elo. The rating of chessplayers, past and present. Arco Pub., 1978.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5915-5922. 2021.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010. 14701, 2020.
Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01220 2021
Joel Hestness, Sharan Narang, Newsha Ardalani. Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Patwary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint 1rXiv 2017.
10

---

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Train- ing compute-optimal large language models. arXiv preprint arXtv:2203.15550 2022.
Geoffrey Irving. Pentago is first player win: Strongly solving game using parallel in-core retro- grade analysis. arXiv preprint arXiv:1404.000 0743, 2014.
Andy Jones. Scaling scaling laws with board games. arXiv preprint arXiv:2104.03/13, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint wrXiv:2001.08360 2020.
Donald E Knuth and Ronald W Moore. An analysis of alpha-beta pruning, Artificial intelligence, (4):293-326, 1975.
Mare Lanetot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay. Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Sacta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schrittwieser, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. Open- Spiel: A framework for reinforcement learning in games. CoRR, abs/1908 094533 2019. URL httt:://arxiv.org//2e
09453
Kuang- Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang. Henryk Michalewski, et al. Multi-game decision transform- ers. arXiv preprint arXiv: 2,21,,,0
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver. and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.0290 2015.
Siqi Liu, Guy Lever, Zhe Wang, Josh Merel, SM Eslami, Daniel Hennes, Wojciech M Czarnecki, Yuval Tassa, Shayegan Omidshafiei, Abbas Abdolmaleki, et al. From motor control to team play in simulated humanoid football. arXiv preprint arXiv:2105.12190 2021.
Oren Neumann and Claudius Gros. Investment vs. reward in a competitive knapsack problem. arXiv preprint ar 1rXiv:2101.19%6 2021.
Oren Neumann and Claudius Gros. Size scaling in self-play reinforcement learning. In ESANN, 2022.
Ivo FD Oliveira, Nir Ailon, and Ori Davidov. A new and flexible approach to the analysis of paired comparison data. The Journal of Machine Learning Research, (9(1):2458-243) 2018.
Shayegan Omidshafiei, Christos Papadimitriou, Georgios Piliouras, Karl Tuyls, Mark Rowland, Jean-Baptiste Lespiau, Wojciech Czarnecki, Marc Lanctot, Julien Perolat, and Remi Munos. a-rank: Multi-agent evaluation by evolution. Scientific reports, 9(1):1-29, 2019.
Pascal Pons. Connect four solver, 2015. URL www.saaver..oo.oooooo...
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.06177 2022.
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXiv:1909.133 12673, 2019.
Julian Schritwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with learned model. Nature, (88((88)))604 2020.
Julian Schrittwieser, Thomas Hubert. Amol Mandhane, Mohammadamin Barckatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with a learned model. Advances in Neural Information Processing Systems, 34:27580-27591, 2021.

---

Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos. Compute trends across three eras of machine learning. arXiv preprint arXiv:2202.0592. 2022.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprini arXiv: 1712.0181... 2017a.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-335 2017b.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint /906.09990.22200 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 12104-12113, 2022.
A HYPERPARAMETERS
In order to keep results general possible, try avoid hyperparameter tuning and choose to train all agents with the values suggested by OpenSpiel's Python AlphaZero example. Table contains summary of all hyperparameters used and their meaning.
Table 3: Training hyperparameters used for Connect Four, Pentago and Oware.
Hyperparameter
Value
Description
Cuct
2
MCTS exploration constant
Max simulations
300
Number of MCTS steps to per move
Batch size
200
Batch size during optimization steps
Replay-buffer size
216
Number of game states stored in the replay buffer
Replay-buffer reuse
10
Number of optimization steps a game state stays the buffer before it is erased
Learning rate
0.001
Optimization learning rate
Weight decay
0.0001
L2 regularization strength Policy noise
Policy
0.25
Policy a
Dirichlet noise variable
Temperature
Temperature applied to the policy for final move selection
Temperature drop
varied
Number of turns before temperature to 0
Temperature drop is the only parameter with different value in each game, set to 15 and 5 for Connect Four and Pentago, respectively. We do this since we find that this term, which should correlate with game length, can drastically affect the performance of agents when we train them for 10a optimization steps.
Replay-buffer reuse is 10 rather than 3, the value suggested in the original example, to speed up training by leaving states for longer in the replay buffer (Lillicrap et al. 2015).
During matches, all noise is set drop, meaning temperature stays constant throughout the game. We do this since without a random element all games between two agents will be identical. A high temperature would distort the data
but temperature is set 0.25, with infinite temperature
12

---

900
107
107
800
Inference FLOPs
10a
Inference-dime FLOPs
MCTS steps
600
10
10a
10
500
400
10a
0.60
400
221
10a
10
10a
10a
10
10a
Parameters
Parameters
Figure 6: Left: Connect Four performance at fixed values of inference-time compute, enforced by scaling MCTS search size inversely with forward-pass compute costs. Elo scores steadily im- prove with increasing model size for smaller models, but the trend breaks for large models. Right: The number MCTS steps used by different sized models with different inference-time compute budgets. Agents with less than MCTS steps per turn excluded from both plots.
by reducing the Elo difference between players, which is why we use a low temperature value of 0.25 that produces roughly 5% less repeating games. The number MCTS simulations per move is the same for all agents at test time, equal to the number used during training.
B INFERENCE- TIME COMPUTE SCALING
Reinforcement learning models like AlphaZero are often required to perform fast at test time, ei- ther in order to finish their calculation within a thinking time limit react quickly rapidly changing environment. The AlphaZero algorithm provides the ability to tailor the move-selection inference time to given time limit by changing the number MCTS steps, which does have to match the number of steps used during training. Since our main results all use a fixed number of 300 MCTS steps both at train and test time, we add here the scaling behavior different sized Connect Four agents using varying numbers of MCTS steps. We look at the case where training compute is abundant but performance is bottlenecked by inference-time compute. To do so we match fully con- verged agents against each other, all receiving fixed amount of inference- time compute enforced by scaling MCTS search size inversely with forward-pas compute costs.
Fig. 6 shows the Elo scaling of different sized agents under inference-tine time compute constraints. Agents steadily improve with size at smaller sizes for all amounts of available inference- com- pute. Larger agents break off from this trend at different points which depend the amount of compute given, such that the the trend seems to hold longer for larger compute values. Interest- ingly, power law appears at the limit of large inference-time compute which breaks off only for the largest agents. The scaling exponent is significantly smaller than that of Fig. 2 which is not surprising given that small agents have access to up to two orders of magnitude more MCTS steps than the largest agents, see Fig. right. This scaling law may be related to the power law scaling of inference-time compute with training compute found by Jones (2021) when fixing the Elo score of AlphaZero agents.
C MEASURING ELO AGAINST OTHER PLAYERS
Elo rating is system that generates predictions of game outcomes that are based on ground truth values. As such, it is vulnerable to biases in the underlying dataset game outcomes. One example of potential bias is when all players will not necessarily represent the interactions of these players with agents utilizing different sets of strategies. Here provide evidence that the scaling laws presented in this paper not the result of such biases.
restricted set of strategies, resulting in Elo scores that
Although Fig. 2 right presents Elo scores calculated solely from matches between agents and perfect Connect Four game solver, these scores are only accurate close to the solver score and are unreliable for agents far from optimal play. For example, the worst agent is almost 1 1,200 Elo points
13

---

AlphaZero agents
Solver agents
1000
900
800
800
700
600
600
400
Mean Elo
Agent Elo
200
400
r= 0.89 400llg1o (N)
699
lemperature-solver Hio
True solver (T-0)
True solver (T-0)
10a
10a Parameters
10a
10
10
10
Temperature
Figure 7: Left: AlphaZero size scaling with Elo scores calculated by only using matches between solver agents and AlphaZero agents, or matches between two solvers. The scaling behavior of Fig. 2 holds here well, with an almost identical scaling exponent. Since the true solver, with temperature set to zero, part the agent pool, has a well defined Elo score (green line). Right: the prior produced by the Connect
Solver-agent scaling with temperature. Each solver agent Four game solver (Pons) 2013). applying a temperature transformation to according to Eq. 9 Low temperature agents asymptotically approach the solver Elo temperature agents converge to the score of a random player. The range of Elo scores covers that of AlphaZero agents, allowing accurate measurement of their Elo scores.
(green line), while high
away from the solver, meaning it should win about one game in 1,000. This requires a large amount of matches in order to model accurately, and indeed several agents are missing from this figure since they received an Elo score of minus infinity, never winning single match.
Here we solve this issue by producing a set solver agents of varying playing strengths, matching them with each other and with AlphaZero agents trained Connect Four produce Fig. 7 All solver agents base their prior on the vector q provided by the game solver, which contains informa- tion about the quality of all legal moves. Taking action in game state s, the number (s,, a) equals to 42 minus the minimal number of turns until either player can force a victory under perfect play, with minus sign if the current player would lose. See [Pons] (2015) for detailed explanation of how is generated. The final prior æ is created by applying a softmax to g divided by temperature value T:
els.a)/T
TT8, 0) = Eg clla.iiTT
This provides a smooth interpolation between the optimal prior and a uniform prior. At T = O the prior leads to the fastest victory if possible, otherwise to draw if possible, and if neither then the slowest defeat. At the limit T' + oo the prior is uniform, generating completely random agent. Note that is not the temperature mentioned in Table 3 which is set to zero for the solver agents and to 0.25 for AlphaZero agents, as it is done throughout this paper, see Appendix A
Fig. 7 shows the Elo scaling calculated with solver agents. Unlike in Fig. 2 Elo scores are calculated exclusively from data on matches between AlphaZero agents and solver agents well as matches between pairs of solver agents. No game between two AlphaZero agents was included in this cal- culation. The resulting Elo scores scale as log of model size with almost identical exponent to that found in Table I providing evidence that this scaling behavior is universal and not the product of bias in the choice of players.
We note that while this way of measuring Elo strengthens our results by reproducing the scaling exponents, it is probably a less accurate measure than the one used in Section 5.1 This is because all solver agents use the same strategy, which introduces bias into the match outcome dataset. This could be the reason why the spread of agent Elo scores in Fig. is slightly wider than that of Fig. for smaller agents. Despite this bias, the model size scaling law seems robust enough to appear here well.
14

---

D SUPPLEMENTAL FIGURES
D.1 AVERAGE COMPUTE SCALING
The scaling law presented Fig. Ais obtained by finding the Pareto optimal agents among all agents trained. It does not make clear whether the same scaling law would apply when training single agent, rather than training with several random seeds and picking the best one. We therefore plot Fig. g which displays Elo scaling with compute, but with scores averaged over each neural-net size group. The scaling exponents deviate only slightly from the exponents calculated with individual agent data. This implies
expect to find compute log-scaling of Elo also when training single agents, as opposed to training many seeds and picking the best performing one.
Connect Four
Pentago
10
1000
10a
Parameters
Parameters
600
104
600
400
400
10
200
Parato front
Pareta front
r= 0.54 400logus(C; 2417
10a
0.58
4000lggwlC 2568
0-
10
10
10ah
10
10
10
10
10
10t4
103s
10tH
FLOPs
FLOPs
Figure 8: Averaged performance scaling with training compute. Re-plotting figure 4 with data averaged over different seeds for each number of neural-net parameters produces Pareto front with similar scaling exponent.
D.2 SIZE SCALING CONVERGENCE
The parameter-count scaling law presented in section 5.1 only holds in the limit of infinite compute, when agents are allowed to train to convergence. A way of measuring the proximity to this limit is by looking the evolution of the log-linear fit to the data. We have shown in Fig that log- log-linear scaling appears already at short training lengths, but with lower slopes. In Fig FigBin we plot the exponent values derived from these slopes. One can see that the exponent value slowly converges its final value at 10 training steps.
Connect Four
Pentago
0.9
Exponent fit
Exponent
0.7
0.6
0.5
0.4
0.3
0.88
0.3
Op 0.87
0.2
10a
10a
10a
102
10a
10
Training steps
Training steps
Figure 9: Convergence of the scaling exponent ON- Fitting the curves in Fig 3
we obtain interme-
diate values for the scaling exponent during training. The exponent fits grow larger with training length, slowing down towards final value for both games.
E OWARE SCALING
We measure the scaling laws of another popular board game, Oware. We choose this game because of its length; unlike Connect Four and Pentago, Oware games not bound end after fixed
15

---

number of turns. This also means the average number of turns per game is much larger compared to the other games, see Table 4 In practice, infinite games are handled in OpenSpiel by calling them off after 1,000 steps have been played. These games, although rare, excluded from statisties. Another difference between Oware and the other games is the shape of the observation tensor. For Oware, neural network input is in the shape of 14-elements vector of integers, counting the number of seeds in each pit. This is unlike the observation tensor of Connect Four and Pentago as well as games like Go, where board state is presented in binary encoding.
We find that Oware exhibits power-law scaling in playing strength in neural network size and com- pute, but encounter two serious issues during training. One issue is the lack of sufficient training time for agents to converge, not allowing to gauge correctly the true value of the size scaling ex- ponent. The other issue that the largest models not manage to reach their full potential, getting Elo scores significantly lower than smaller models. We give possible explanation to this in the last section. This issue mostly affects the correct estimation of the compute exponent.
Table 4: Game details including Oware. Oware games are not bounded to a maximal number of moves, As result, they last much longer than Connect four Pentago games on average. Additionaly, the neural network input comes in the form of integer numbers in Oware, rather than zeros and ones.
Game
Branching Factor Average Game Length
Observation Shape
Connect Four
7
25.4 (max. 42)
3 6 7 Boolean tensor
Pentago
S 288
16.8 (max. 36)
3 x 6 Boolean tensor
Oware
< 6
100 (unbounded)
14-long integer vector
E. SIZE SCALING
We repeat the analysis of neural network parameter scaling for Oware agents. Fig. 10 presents the scaling of Elo scores with model size. For Oware observes, interestingly, that agents with large numbers of parameters do not train to optimality. Small- and medium-sized agents show however evidence of power-law scaling. is clear that large agents could theory reach better performance, since the range of strategies MLP network could implement is always contained within that of larger model. We speculate about the cause this in section
Oware
/00
600
500
400
Mean
Agent Elo
300
0.75 (00log: 407
10a
10a
Parameters
Figure 10: Oware size scaling. Oware shows a logarithmic scaling trend of Elo with neural net size that breaks abruptly for larger networks. Since the strategy space available to smaller agents is contained within that available to larger ones, large agents should at least reach an equal Elo score to smaller agents. It is therefore clear that the large agents did not converge to optimal performance.
Medium-sized agents performance follows log scaling with model size, with smaller scaling exponent than that found with Connect Four and Pentago. However when we plot the changing of the exponent fit during training in Fig. 11 it seems very likely that it will keep increasing with
16

---

longer training. Unfortunately could not extend training length by another order of magnitude in the time given for this analysis.
Oware
700
10
0.6
500
Exponent fit
10a
400
steps
300
200
10?
100
0.2
10a
10
10a
10*
Parameters
Training steps
Figure 11: Left: Elo scaling with parameters during training. Log-linear scaling already appears at early stages of training, converging to fixed curve. Right: convergence. The exponent fit keeps increasing with training time for Oware, suggesting the final exponent should be larger.
E.2 COMPUTE SCALING
Fig. 12 shows the Elo scores of Oware agents with different training compute. Despite the failure of large agents learn effectively, Oware matches log-lineat fit over several orders of magnitude with a smaller exponent than that found Connect Four and Pentago.
Oware
700
600
1104
Parameters
500
400
300
10a
200
100
Pareto front
0.37
403
1434
10
10
1013
1014
10.5
10
FLOPs
Figure 12: Oware performance scaling with training compute. The Pareto front follows a log-linear trend in compute. Large models, which failed train well, do not make it the Pareto front.
E.3 ESTIMATING THE COMPUTE OPTIMAL MODEL SIZE
We use the exponents an and ac found for Oware to new scaling law for the optimal model size, Nopt- Since Oware has smaller compute-scaling exponent than the other games, its optimality exponent ag opt also smaller. Overlaying the resulting power law data from other games in Fig. [3 we find that does not fit Connect Four and Pentago well. Even though the Oware exponent significantly smaller, it still predicts that AlphaGo Zero and AlphaZero would benefit from an increase in model size.
E.4 TRAINING ISSUES
Reinforcement learning algorithms can exhibit convergence issues, as evident in the case of Oware. The largest agents trained this game clearly failed to converge to their optimal playing ability, since they ander-performett smaller agents, which they should at least match in performance. We believe likely cause to that is the format of the input tensor, which is composed of vector of
17

---

Optimal (parameters)
10a
Connect
Optimal size (parameters)
My: - 16/10-1590s
Pentago
10
1C/10
Oware
Move 1C/100
AlphaGo Zero AlphaZere
100
10
107
Nop = (0211044554
10
10a
10
10
10
10
10
10
10
1013
1015
1017 FLOPs
10th
10a:
10
FLOPs
Figure 13: Optimal number of neural network parameters as in Fig. dicted with Oware in green. Even this lower exponent scaling law predicts previous state-of-the-art models were undersized, although to much lesser degree.
with the scaling trend pre-
integers rather than a boolean tensor as used Connect Four and Pentago and games like Go and chess (see Table 4. We observe the same trend of large models consistently failing train well in another game, a toy model problem not presented here with floating- point inputs rather than binary ones. We find that large models trained the toy problem with no issues when we modified version of AlphaZero created by us, which imports a key feature from AlphaGo Zero Specifically, the modified version tests each new candidate for the data generating model against the current model, and only updates it if the new candidate is proved to perform better. OpenSpiel currently does not contain this feature, but we believe could be useful in the case of non-boolean input.
Table 5: Scaling exponent for all three games. Oware shows smaller size and compute exponents, and larger optimal size scaling exponent.
Exponents
Connect Four
Pentago
Oware
ON
0.88
0.87
0.75
ac ac
0.55
0.55
0.37
0.62
0.63
0.49
F TEST SET LOSS
Most language model scaling laws revolve around scaling test-set loss as power law of some quantity. In contrast, scaling laws for AlphaZero appear in the form of playing strength power-laws. When we look at the predictive ability of trained agents test dataset, we find that test loss does not scale as power law.
To measure test loss scaling, create dataset of game states from random Connect Four games generated by taking random moves. We then annotate these states with a perfect solver (Pons) 2015). The solver provides us with the ground-truth state value (1, 0, -1 for win, draw loss from the current position, respectively) and a policy prior which a uniform probability distribution over all optimal moves. Optimal moves defined as those which lead to either the fastest victory, draw otherwise, or the slowest loss otherwise. All sub-optimal moves are given probability zero.
We then compare the value estimations and policy priors created by the solver to those predicted by fully trained Connect Four agents, by calculating the value and policy losses. These are defined in the full AlphaZero loss function (Silver al. 2017b):
= (2 - 0
logp c110
(10)
where is the recorded game outcome from the current position and is the improved policy generated by the MCTS using the visit count of each node. We replace & and π with predictions made by the solver.
18

---

Value head
Policy head
0.90
Lass Mean loss
2.0
0.85
Los inreduc cible
1.9
Uniform prior
test.
0.80
1.8
1.7
0.75
Mean loss
1.5
0.70
Parameters
Parameters
Figure 14: Test loss of Connect Four agents dataset created by perfect solver. Left: Value head test loss. Loss changes very little with model size, large models have similar loss to small models. For comparison,
add baseline loss received when always predicting draw (v 0). Right: Policy head test loss minus irreducible loss. Loss does decrease with size, indicating larger models produce policy prior that is closer the perfect policy. However the distance the perfect prior is still large across all scales. For comparison, we add baseline achieved when always predicting uniform prior over all legal moves. Smaller agents achieve worse loss than the uniform baseline.
The resulting scaling trends appear in Fig. 14 For the policy head, plot the test loss minus the irreducible loss (equal 0.336), the minimum loss achieved by matching the solver's policy prior exactly.
While policy loss does seem to decrease with model size, is contained to a small range far from the minimal possible loss. Value loss decreases and then increases with model size. Both quantities do not seem to be good indicators of performance scaling.
We note that even though Elo score turns out be better measure of performance than loss, there may exist even better measures. Elo rating fails to correctly model player dynamics when they are -on-transitive, for example (Omidshafiei al. 2019). Some alternatives exist (Omidshatiet al., 2019 Oliveira et 2018), and it is possible they could capture elements of the agent dynamics that Elo rating would miss.
19


Reviews:
[{"review_id": "xREvkT7O7s", "reviewer": "Reviewer_6fgP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors claim to find a relationship between compute, model size and performance (here measured by ELO). Studies are conducted over 2 games (Connect Four and Pentago) and the AlphaZero algorithm. Finally the authors combine these relationships to make recommendations based on optimal model size given the amount of compute available. \n\nThe authors also have a third game (Oware) over which they observe a scaling law, which different scaling parameters.\n", "summary_of_the_review": "\nInteresting results but the claim is way too broad for the lack of empirical work done to verify it.\n", "strength_and_weaknesses": "### Strengths\n\nInteresting results.\nProof of existence is good\n\n### Weaknesses \n\nI think this paper misses the point of the LM scaling laws. Here Kaplan’s results were only significant because we found that the log_loss score for language modeling was very indicative of success for  downstream LM tasks. Here we don’t have this justification for why we should care about ELO. \n\nELO does relate to performance on the task but i’d expect that to change with compute and model size - the claims for this being significant are only:\n\n* It looks like a scaling law\n* This is true for multiple games (sample size 2)\n\nFor this result to be compelling I'd either like to be shown over multiple methods (PPO, DQN)  or many many more games.  Simple games such as pong or gridworlds would also be welcomed. \n\nhe state space argument really doesn’t make sense to me - I would try discrete the space or form some 1 hot encoding (as done by most PPO implementations for OpenAI gyms' continuous action problems).\n\nGenuinely i can’t tell if this rule generalises to all 2player zero-sum games or a specific niche currently.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gnJ5p-AWZX", "reviewer": "Reviewer_T3hx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Inspired by the recent interest in power-law scaling of performances in large transformer-based models, this work investigates that in two-player zero-sum games with reinforcement learning (RL). By conducting experiments in two domains, Connect Four and Pentago games, authors show that the playing strengths of agents scale as a power law with the neural network size when models are trained until convergence. They also investigate the trade-off between model size and compute. Lastly, they utilise the two scaling laws to find a scaling law for the optimal model size given the amount of compute available.", "summary_of_the_review": "Overall, this is an interesting paper that tackles an important problem, the scaling laws in a subclass of multi-agent RL methods. The evaluation protocol is novel and the paper includes a comprehensive set of experiments to support its main claims. That said, I still have some concerns about the generality of the results achieved here. The paper could have been improved by adding more games in the experimental setup. Furthermore, there are some concerns regarding the ELO-ranking population and scaling exponent values.", "strength_and_weaknesses": "## Strengths\n\n- The paper takes one of the most important issues of deep learning: how the changes in model size, compute, and data affect the performance of the model. This question has been studied extensively for NLP methods. However, not a lot of analysis exists for RL, particularly multi-agent domains. For me, the problem itself is real and practical.\n- The idea of using ELO as the y-axis for RL scaling laws is interesting.\n- This paper provides a comprehensive set of experiments, including ELO performances relative to the population of agents as well as solvers.\n\n## Weaknesses\n\n- While I appreciate that the authors have restrictions for running large-scale experiments, I do still think that the choice of games for this work is limited.\n    - The authors use their results from Connect Four and Pentago to extrapolate insights about AlphaZero (Silver et al, 2017a) and AlphaGoZero (Silver et al, 2017b). This is an Apples to Oranges comparison involving different games and model architectures. I’d expect at least using the game dynamics (perhaps with smaller board sizes) before being able to comfortably generalise claims based on limited data.\n    - For example, 8x8 checkers or smaller-sized Go would have strengthened the paper.\n    - This would also allow authors to analyse bigger models that do not saturate as quickly.\n    - With current results, it is unclear if such phenomena will be observed in more challenging games, such as Go and Chess.\n- The ELO ranking results depend heavily on the population of agents being used for comparison. The authors use the ELO between the population of trained agents to estimate the power-law scaling in MARL. It is unclear if these insights can hold more generally. For example, the Relative ELO relative solver in Pentago does not seem to be as neat as the ELO relative to the agent population.\n- The results suggest that the two games share exactly similar scaling exponents for playing strengths ($\\alpha_N$, $\\alpha_C$, and $\\alpha_C^{opt}$). I find this quite surprising and unintuitive. This is also among the reasons why I’d like to see more games investigated in this game to assess the universality of these constants.\n- The authors factor the compute by the forward pass costs and MCTC simulations. This is sensible, considering the model architecture used. Considering that the games are relatively easy, I’d be curious to see if such a power law emerges with model-free RL approaches.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9RkA9N3CkB7", "reviewer": "Reviewer_537d", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the scaling law of AlphaZero algorithm in the MARL setting. Specifically, it uses the Elo rating as the performance criteria, and demonstrates that there is power-law scaling between the Elo rating and the model size (also the computation budget).", "summary_of_the_review": "So far I am fairly concerned with the novelty and significance of the current submission. I am willing to raise my score if the authors can address my concerns.", "strength_and_weaknesses": "# Strength\n\n- This paper is clearly written.\n-  Most of the experiments are repeated with several different random seeds, and the results displayed align well with authors' interpretation.\n\n\n# Weakness\n\n- Since there is neither new algorithm proposed nor insightful phenomenon observed throughout the experiments, I am little bit concerned with the technical novelty of this work.\n\n- It is likely that the discovered scaling law only holds under the very specific training methods, the model architecture and the game environment. Once you changed any of these factors, the scaling law might crash or hold with a very different exponent parameter. The experiments in the current submission cannot rule out such possibility. \n\n- I am also not convinced by the motivation of studying the scaling law for this very specific algorithm in these two moderate-size boardgames. It is unclear to me what insight these results can provide for both practitioners and theorists. \n\n- The authors claimed \"This scaling law implies that previously published state-of-the-art game-playing models are significantly smaller than their optimal size, given the respective compute budgets\" but to me it is more likely that (1) the previous works might have used different training (engineering) techniques from this paper, and (2) the scaling law discovered in this paper does not generalize to more complex games of different styles (like Go) and even if it can generalize, it may hold with a different exponent parameter. Therefore, I find such claim scientifically misleading.\n\n- The Elo score highly depends on the strength and the playing style of other players. In this paper, other players are generated from AlphaZero with different training configurations. It is unclear to me whether the same scaling law still holds or not if I generate the opponent players in a different way. \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0952yurWyJ_", "reviewer": "Reviewer_kPYj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this paper ran a large amount of AlphaZero-based training runs, with many different sizes of neural networks, for two different board games: Connect Four and Pentago. Based on these experiments, the paper demonstrates for both games, very similar power laws exist that predict how the playing strength (measured in Elo) scales with respect to various aspects such as compute, num NN params, and generated self-play data.", "summary_of_the_review": "Overall a good paper, but some of its limitations need to be more explicitly acknowledged, and some conclusions / statements about related work be made with more nuance.", "strength_and_weaknesses": "**Strengths**:\n- Very clear writing.\n- Good, extensive experiments (albeit restricted to two games, plus one more in supplementary material).\n- Interesting and likely-impactful results and conclusions.\n\n**Weaknesses**:\n- I think that a few conclusions / remarks about related work need to be toned down, due to the use of only two games, and many similarities between those two games.\n\n**Detailed Comments**\n- On page 2, some motivation is provided for the selection of the 2 games is provided, **stating that they are sufficiently different** with respect to branching factors and game length. I agree that they are quite different in terms of branching factor, but in terms of game lengths they are really quite similar: they are both relatively short (25.4 vs 16.8 according to Table 2). There are many other board games out there with substantially greater durations (e.g., Chess, Shogi, Go, Amazons, Hex, Havannah, ...). These two aspects (branching factor and game duration) are also only two, relatively abstract, aspects. We can think of many other properties of board games: What is the board size? What kind of shape/tiling does the board use? What kinds of win conditions are used? What kinds of movement mechanisms are used? And so on. In almost all of these aspects, the two selected games are very similar. They have similar board sizes (and hence input shapes). They both use tilings of squares (although Pentago does at least have the thing where parts of the board get rotated). They both have line-completion goals (no checkmate or capturing or connection or racing or any other kinds of win conditions). Neither game features movement of pieces (as in Chess/Shogi/Amazons/etc.), only placement of pieces (and indirectly some movement in Pentago due to partial board rotations). Neither game features any capturing/removal of pieces. Both games naturally converge towards a terminal position (because pieces never get removed). **Considering all of these aspects together, I would argue that the two selected games are in fact highly similar, having only a few differences.**\n- Given my point above, I feel that some conclusions need to be weakened and limitations more explicitly acknowledge. **Firstly, while I do think the attempt at extrapolating to AlphaZero/AlphaGo Zero is interesting to see, everything related to this needs to be presented with a lot more nuance and acknowledgement of the major differences.** Go is a very different game from Connect Four and Pentago in many important aspects: significantly longer games, much bigger board, very different style of win condition, involves capturing of pieces, does not necessarily naturally converge to a terminal position (I guess it does under expert play when the board gets filled up, but not under non-expert play where giant portions of the board might be freed up during gameplay). The same holds for Chess and Shogi (*are these games actually included when you say \"AlphaZero\" in e.g. Figure 1?*). **Secondly, in general the main conclusions of the paper about power-law scaling being likely to exist for AlphaZero in general**, even if the exact coefficients may be different, **should again be stated with much more nuance and acknowledgement of the fact that the two evaluated games are extremely similar in many aspects**. Demonstrating this behaviour in two highly-similar board games is not representative of the overall space of board games.\n- In principle, I am fine with having used only these 2 games. I understand that computational limitations are an issue, and I also think there are very good reasons to select 2 relatively small games like this (for example because it allows for comparisons against near-perfect players). While more games, bigger games, and more variety in games could've made the work even stronger, I do not see it as a reason to reject. However, I do see it as something that needs to be more explicitly acknowledged as a limitation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster