AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Laws
Authors: Oren Neumann
Claudius Gros
Venue: NeurIPS 2025 spotlight ()
Keywords: Scaling Laws
Reinforcement Learning
Primary area: reinforcement_learning
TL;DR: We show evidence that LLM scaling theory, connecting Zipf's law to scaling laws, applies to AlphaZero scaling.

Abstract: Neural scaling laws are observed in a range of domains, to date with no universal understanding of why they occur. Recent theories suggest that loss power laws arise from Zipf's law, a power law observed in domains like natural language. One theory suggests that language scaling laws emerge when Zipf-distributed task quanta are learned in descending order of frequency. In this paper we examine power-law scaling in AlphaZero, a reinforcement learning algorithm, using a model of language-model scaling. We find that game states in training and inference data scale with Zipf's law, which is known to arise from the tree structure of the environment, and examine the correlation between scaling-law  and Zipf's-law exponents. In agreement with the quanta scaling model, we find that agents optimize state loss in descending order of frequency, even though this order scales inversely with modelling complexity. We also find that inverse scaling, the failure of models to improve with size, is correlated with unusual Zipf curves where end-game states are among the most frequent states. We show evidence that larger models shift their focus to these less-important states, sacrificing their understanding of important early-game states.

AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Laws
Oren Neumann, Claudius Gros
Institute for Theoretical Physics Goethe University Frankfurt
oneumann, papers@gmail.cmmmmmmmmmmmmmmmmmm com, gros@itp tp. wni-ffannffurt.dde
Abstract
Neural scaling laws are observed in range of domains, to date with no universal understanding of why they occur. Recent theories suggest that loss power laws arise from Zipf"s law, a power law observed in domains like natural language. One theory suggests that language scaling laws emerge when Zist-distributed task quanta are learned in descending order of frequency. In this paper we examine power-law scaling in AlphaZero. reinforcement learning algorithm, using model of language- model scaling. We find that game states in training and inference data scale with Zipf's law, which known to arise from the tree structure of the environment, and examine the correlation between scaling llw and Zipf's- law exponents. In agreement with the quanta scaling model, we find that agents optimize state loss in descending order of frequency, even though this order scales inversely with modelling complexity. We also find that inverse scaling, the failure models to improve with size, correlated with unusual Zipf curves where end same states are among the most frequent states. We show evidence that larger models shift their focus to these less- less-important states, sacrificing their understanding of important early-game states.
1
Introduction
Neural scaling laws, describing the scaling of model performance with training resources, have been documented across many architectures and use cases 2 4 5). The performance of these models, typically measured by test loss, follows a power law in either compute, model size or training data. The robustness of these power laws and their usefulness in training large models, especially large language models (LLMs) [6], prompted a range of attempts to find a model explaining the mechanism behind neural power-law scaling 7 10 IT] Several models connect performance power laws to Zipf's s law, a universal power law appearing in natural language [12 [3] 14] [5] [6].
far relatively rare compared to supervised learning AlphaZero, a multi-agent RI. algorithm that beat human champions in games like chess and Go 20 exhibits power laws of Elo score with training resources, as well power law relation between compute and optimal model size 21 (22)
Scaling laws in reinforcement learning (RL)
[7 19
In this paper, explore the mechanism behind known RL scaling laws, comparing this mechanism to the quantization model LLM scaling [12] Our main findings are:
1. We find that AlphaZero agents that exhibit scaling laws, such as those trained in Neumann & Gros (2022) [S. produce train and test data that follows Zipf's 's law, suggesting that learned tasks also scale with Zipf's ' law.
2. We show how these smooth Zipf laws form when agent policies are combined with the tree-structure of board games, which is known to create a Zipf-like state distribution [3]
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We find correlation between the Zipf curve and scaling exponents, exposed by modulating policy temperature at inference.
3. In agreement with the quanta scaling model, we show that agents minimize loss on game states in descending order of frequency, counter-intuitivcly achieving better performance harder tasks.
4. Inverse scaling, the abrupt failure of scaling laws at large sizes, coincides with an unusual state- frequecyyyn distribution. We show that large models scale negatively on games with an unusual tree structure that increases the frequency of panttegcclll-unimporre states. Larger agents achieve better loss on these late-game states, sacrificing performance on more important early-game states. We propose an explanation of inverse scaling in AlphaZero using a combination the quantization model and the selective non-stationarity of training targets.
late- -ame
All code and data required to reproduce our results publicly available!
2 Background
short description of the AlphaZero algorithm and AlphaZero scaling laws is given in ap- Ae]dix[A]
Zipf's law Zipf's law is an empirical law de- scribing the distribution of values when sorted in decreasing order, typically with respect to frequency 224] When sorting elements in de- creasing order of frequency, e.g. words in natural language texts, one regularly finds that the fre- quency distribution S(n) follows a power law in element rank
10
10
Connect Four 0
Frequencc
10
10
Pantago: 8 0.86
10F
102
S(n) x
State rank
na
Figure Zipf's law AlphaZero games. Board- state frequency follows power law in state rank, here for Connect Four and Pentago. Similar expo- nents appear for agents of various sizes trained on different games, see appendix C.T
often with a = 1 for natural language datasets 25 26
Quantization model of LLM scaling Michaud et al. (2023) [12 proposed that "quantization model' may explain the origin of power-law scaling in LLMs, connecting neural-network training language Zipf's laws. The model assumes that LLMS learn discrete task quanta and that the frequency of these tasks follows Zipf's law. The quantization model suggests that language models learn tasks descending order of frequency, which leads to power-law scaling laws. If the loss on each task is reduced by fixed amount AL after it learned, then a model that learned the first n tasks will have an expected loss of
AL
n 1-a + Loo = (L- - Loo) x
=
(c((++))
where Ç is the Riemann zeta function. It is assumed that a fixed neural capacity is needed to fit each of the independent task quanta. At the limit of infinite compute and data, the number of learned task quanta n = N is determined by the number of parameters N. Eq. then becomes power-law scaling law:
(L(N) Loo) x
Na-1
and the Zipf exponent a determines the size-scaling exponent oN = a 1. Similar reasoning relates compute- and data-scaling laws to Zipf"s law.
'Code and data available at ggitubbcwwOCOennneunann//hhee zipf'_ law

---

Other supervised learning scaling models A large body of work exists on models of neu- ral scaling laws on static datasets, several of which making the connection between Zipf's law
and scaling laws. Hutter (2021) [9] argues that data scaling laws emerge when training Zipf-
distributed data under the assumption of infinite memory. Bordelon et al. (2020) [13] develop a
model of data scaling laws for kernels, which applies to neural nets at the infinite width limit. Other
works expand this line of work on kernel models, including results on real datasets [14 27]. Mal-
oney et al. (2022) [15] develop a model that explains joint scaling of model size and dataset size. Cabannes et al. (2023) 16 assume Zipf-distributed data and develop a model for scaling of attention models. Schaeffer et al. (2025) 28 propose that exponential scaling individual tasks can aggregate to power law when the distribution of success probabilities is heavy tailed, in the case of multiple attempts per task. Dohmatob et al. (2024) [29] explore model collapse through the effect of synthetic data on heavy-tailed data distributions.
We frequently compare our results to Michaud et al. (2023) 12 throughout this paper, since their model develops the full set of scaling laws for neural nets (compute, parameters, data) by assuming a Zipf's law of the data. Our comparisons also hold for the above-mentioned works, depending on the assumptions they made. To our knowledge, no complete model exists that explains scaling laws in any RL setting.
3 Related work
Finding Zipl"s law in board games To our knowledge, no study has been made on Zipf's 's law in Al games. Blasius & Tönjes (2009) [23] and Georgeot & Giraud (2012) [30] show that the popularity of openings in human game datasets follows Zipf's law in chess and Go, respectively. Blasius & Tönjes are the first to propose a connection to the branching- ttree structure of chess, without proof. We provide evidence supporting such a connection in section
Analyzing concept learning in AlphaZero In sections 7 and we present results on the scaling of AlphaZero loss on individual states. A similar analysis is done by McGrath et al. (2022) [31] and Lovering et al. (2022) (32), who both analyze how human concepts are learned by AlphaZero by measuring accuracy. McGrath et al. (2022) [3 map how human chess concepts are learned, both as function of training length and neural network depth. They measure the accuracy of few hand-crafted concepts, in contrast to our analysis that measures loss on up to 10 unique states. Similar to our inverse-scaling results in section 8 Lovering et al(2022) [32 observe differences between how short-term and long-term concepts are learned in the game of Hex.
Effects of state frequency on training Our work discusses the effects of training-data state frequency on AlphaZero agent performance. Another observation regarding the relation of data frequency with performance can be found in Ruoss al. (2024), {} where Transformers are trained with supervised learning on annotated human chess games. The authors perform an ablation study where they change the frequencies of board states in the dataset, either sampling states uniformly or sampling them according to their frequency in human games, which follows Zipf's s law. Testing on chess puzzle accuracy, they find that changing the state frequency has a strong effect on model performance, favoring uniform sampling over sampling from the Zipf distribution.
4 Methods
We use AlphaZero agents trained on four board games: Connect Four, Pentago, Oware and Checkers. All neural nets are either open-source models trained in Neumann & Gros (2022) (5) or newly trained using the same open-source OpenSpiel code (34). We use the open-source model weights for Connect Four and Pentago, and train new models with the same hyperparameters on Oware and Checkers. All of these models exhibit plwer-law scaling laws, either partially or fully: Connect Four and Pentago scaling laws are discussed at length at Neumann & Gros [5], here we present size-scaling curves for Oware and Checkers, see Fig. A The power laws are for the Bradley-Terry playing strength 7, which determines the Elo score: Elo x log (7) 335 36. One finds that scales as a power of
neural- net parameters and training compute [5]:
? x Noo
C9c
Most results in this paper involve the frequency of states 8 E in a Markov decision process M (S, A. R) (37] In practice, we identify each state by its observation tensor, which is the
3

---

A. Toy-moder distribution
B. Random game distribution
100
Gonnect Four
Perragg
0.67
Froquency
Frequency
10
10
10 State
10
State
Zipf power laws emerging from the tree structure of board games. A: In a toy-model game where board states branch out symmetrically, state frequency is organized series of plateaus centered around a linear line. B: Adding game rules, but still playing random moves, the platcaus smoothed out, producing a power law with slightly different exponent. This is in particular evident for Connect Four, which has a much lower branching factor than Pentago. The tail plateaus caused by the finite amount of data.
Figure 2:
input to the neural net. In all four games, this observation is the position of each piece on the board, and does not include the history of previous moves (unlike the chess analysis of Blasius & Tönjes (2009) (23)). All AlphaZero agents use MLP nets with 3 hidden layers at exponentially-incrasiin widths, with fixed amount of 300 MCTS steps per move during both training and inference.
Zipf's law in AlphaZero
AlphaZero board state distribution We find Zipf distribution in games played by AlphaZero agents that exhibit scaling laws. Fig. shows the frequency distribution of board states played by Connect Four and Pentago AlphaZero agents of varying sizes during training. The number of times each state was visited follows a clear Zipf power law when sorted in descending order. The exact value of the exponent defined in Eq. fluctuates in the range
0.8 - 1.0 for different models and games, close to the value observed in human Go games [30] and to the exponent of random games, discussed in the next section. Appendix [C. contains plots of individual-agent Zipf curves, and discusses the variation of the power-law exponent a. The plateaus at the tail-end of each distribution due to finite-size effects, and smooth-out when
state-frequency data is available.
This result strengthens the assumption that Zipf's law is not caused by human decision-making but rather derived from the game rules. The existence of Zipf's law in human games has long since been established, but its cause not entirely clear. The frequency of popular opening moves in games like Chess and Go follows descending power law in rank, both for expert and amateur human players The fact that an RL model trained with no human knowledge produces Zipf's law similar to humans implies that Zipf's 's law is not caused by human strategy preference.
23 30
Visualizing the origin of Zipf's law A first intuition regarding the source of Zipf's law is presented in Fig. 2 The relation of Zipf's law to the underlying branching process of board games was already pointed out by Blasius & Tönjes (2009) 23 It easy to show that an ideal game, with constant branching factor and a tree structure free of loops, would produce a Zipf-like distribution. Specifically, one can show that the frequency distribution in such game is series of plateaus around the line S(n) x n,, see appendix [ The same reasoning explains the appearance of Zipf's law in randomly- generratd texts (38). In Fig. 2 this ideal-game Zipf's law is visualized by plotting the state-frequency distribution for a non-looping, constant branching game with fixed length. Increasing the branching factor widens the plateaus, spreading them out. We use the initial branching factor of Connect Four.
In Fig. show that adding game rules molds the plateaus into smoother curve. Using uniformly random policy to play Connect Four and Pentago smooths the pewer-law-centere plateaus, roughly merging them into straight line. The toy-model exponent changes slightly after adding game rules, but the general power-law trend remains intact.
Appendix C.2 contains additional results on the emergence of Zipf's 's law, and shows that non-uniform policies can also smooth the distribution into a power law. As we see Fig. I a learned policy smooths the distribution further, keeping the power-law shape while slightly changing the exponent.

---

A. Inference Zipf's law
B. Size scaling law
C. Exponent correlation
Froquancy
1
Eib oxganant
0.00
Temperalure
10
400
0.95
200
0.00
10
10
10'
10"
107
107
10
10'
12*
10
State rank
Neurak-nll parameters
Zpll exporent (tai)
Figure 3: Measuring correlation between Zipf's law and scaling exponents. By changing the move-selection temperature at inference time, both the state- distrriistiitttion Zipf's law and the size- scaling law are augmented, plotted here for Connect Four. A: As T + O, the Zipf curve starts to bend, following a steeper power law at high-ranks. B: T" also changes the Connect Four size scaling lawe) either directly by lowering policy quality high T, or indirectly by changing the game state distribution. C: By modulating 7 at low values, we plot the dependence of the scaling power law on Zipf's law, using the tail exponent.
6 Temperature and correlation with scaling laws
Although measuring the correlation between Zipf"s law and scaling laws is not straightforward, we present how this can be done using temperature. Natural language Zipf exponents are impossible to change in real-world data. In contrast, RL Zipf laws can be augmented by changing the policy temperature T. AlphaZero uses Monte-Carlo tree search (MCTS) to generate policy π::
[(s0, a)1/T
π (so, a) = E. Ns0. b)1/T
(5)
where (ss, a) is the number of times the state-action pair (s, a) was probed, see appendix F for detailed explanation. The temperature interpolates between a deterministic policy choosing the best move at T and a uniform policy at T oc. At inference time, c.g. when calculating Elo ratings, T' is either set to zero or reduced to low value, the latter to avoid deterministic policies.
Temperature has strong influence the state-frequee state-frequency curve, as we show for Connect Four games in Fig.BAr As 7 approaches zero, the Zipf curve starts to bend due to the formation of plateaus of popular games that played repeatedly when temperature is low. Deviations from these games make up the post-bend tail. The Zipf exponent of the tail is not well-defined for T + 0 for a finite number of agents and finite numbers of games played. The bending of the frequency curve with decreasing T is somewhat intuitive, considering how temperature affects games. At low T, fewer unique states are played, meaning the curve must end at lower rank. This pushes up the rest of the frequencies, since the total number of states played remains constant.
Exponent correlation In Fig, 3C tune the inference temperature T to analyze the correlation between the size scaling law (Fig. B) and the Zipf exponent (Fig. 3 The two exponents change together, indicating that the inference-tine state distribution does affect the scaling law, predicted by the quantization model. Unlike the linear correlation predicted by the quantization model for I.Ms,, we observe non-linear relation. The non-linearity is expected when considers the T 0 limit, in which RI. scaling laws converge to a finite exponent while the Zipf tail exponent diverges to infinity.
Additional experiments confirm that the relation between the state distribution and the scaling law is causal, meaning the scaling law changes only because of the changing Zipf distribution. A possible non-causal effect could be Elo scores changing due to the degradation of the policy л at high T. We measure the effect of T on policy quality in appendix Dld find that the probability the policy л (T) assigns to optimal moves only starts decreasing at 7 0.5. The correlation in Fig. [3 is strictly causal, since it falls in the temperature range where policy quality is preserved. Interestingly, the Zipf exponent only changes in the temperature where policy quality is optimal, and is fixed for T > 0.5.
Limitations Augmenting policy temperature is not perfect comparison method, since is not clear how much influence low-rank plateaus have on agent performance. This is especially true at the
'Where we plot Elo scores (Figs, 5
we set the Elo of the lowest agent to 100, Each curve in Fig. B is to
scale, but the positions of the curves are not to scale.
5

---

A. Value loss (train set)
B. Value loss (ground truth)
C. Time required, a-3 pruning
0.6
0.6
GPU 11me
Farametorr
10P
10-
rank
State rank
rank
Figure 4: Connect Four Saaite-lossssalinnn with rank. A: Average loss on each agent's own training data. Loss steadily increases with rank. Larger agents achieve better loss. B: Loss on dataset annotated by game solver. Larger models are closer the ground truth values of optimal play. The overall trend of loss increasing with rank is surprising, when one considers that the complexity of states decreases with rank. See appendix E for detail on error bars. C: The time needed evaluate the same states using alpha-beta pruning 39 drops on an exponential scale with rank (mean and standard deviation are geometric).
lowest temperatures (T 5 1), where the plateaus most dominant, accounting for 75 90% of all states visited. In this temperature range, it is hard to tell how influential is the tail distribution compared to the plateaus.
Temperature can augment the Zipf curve since affects exploration, which in turn affects the state distribution, as we also show in appendix C.2] This implies that changing the MCTS parameters could also have an effect on state distribution, but we did not verify this relation.
7 Analyzing how loss scales with Zipf's law
agreement with the quantization model 12]. we find that AlphaZero models reduce their loss on game states in descending order of frequency. In Fig. fwe plot the value-head loss Connect Four training data in the limit of abundant compute, tracking the loss of each agent on its self-play games
Comparison the quantization model The quantization hypothesis assumes models learn tasks in binary way, sharply reducing the loss each task in descending order of frequency. As function of neural net size N, this assumed loss function is step function that changes from low to high loss at rank N 112. In comparison, we find that AlphaZero loss increases smoothly with rank rather than resembling step function, likely because board positions are not independent task quanta.
In Fig. 4B we plot the value loss Lyaaue = (z u)² on ground-truth labels 2, calculated by an alpha-beta pruning solver [39]. We see that larger agents have significantly better ground-truth loss over smaller agents, confirming that increasing model size improves the absolute value loss. Ground-truth value loss is directly related to better performance, similar to test loss LLMs that known to correlate with accuracy downstream tasks 40 6].
States are optimized descending order of frequency & complexity It is surprising that absolute value loss (Fig. 4B) tends to increase with state rank, since higher rank states are easier to model, meaning loss increases as state complexity decreases. Low-loss states are the hardest to model using conventional methods, as we show in Fig. 4C for alpha-beta pruning [22.. The most frequent states have a larger state tree branching out from them, making it harder for search-based algorithms like MCTS to tackle them. As result, the CPU time required to calculate the ground truth value of state decreases steadily with rank on an exponential scale.
'The largest nets approach optimal play limit [5). causing loss to stagnate model size increases to largest values.
Nott that AlphaZero's value head is not equivalent the value of actor-critic methods, appendix

---

A. Scaling curves
B. Oware Zipf's law
C. Checkers Zipf's law
Checkers
ho lite game states Al asama
10"
No late game St385
Owaree (1-dss--00) 50)
10"
All satee
Oware (Tdiopp11))
Frequency
Risquency
0
40"
10
10P
100
10
10
105
10"
Neural-net
Figure 5: Inverse scaling in AlphaZero. A: Agents playing Checkers and Oware follow a size scaling law that abruptly changes direction, when large models fail to utilize their capacity. The scaling curve does not flip due to approaching the perfect-play ceiling: training a suite Oware agents with different hyperparameters (light green), they extend to higher Elo scores. Error bars are one standard deviation. B: Oware games played by AlphaZero follow Zipf's law, interrupted by small plateau. This is caused by high-frequency late-game states, present due to the unusual tree-structure of Oware, see Fig. 6 C: Checkers shares the same tree structure, but the effect on the Zipf curve is less visible.
A. Turn distribution
B,
Late-turn fraction
1.0
Connect Four
Pentago
Oware
Turn numbar
Checkers
Fraction of late turns
0.6
10F
10a
10
10
10
10
10
10
M
102
number
Oware
Checkers
0.2
10'
10F
12 107
10
123
10"
10
13'
122
10'
10a
10
State rank
State rank
rank
Figure 6: Turn distribution anomaly coincides with inverse scaling, A: When the game tree spreads exponentially, e.g. in Connect Four and Pentago, turn number correlates with state frequency. However, in games with inverse scaling (Oware and Checkers) the game tree converges to compara- tively small set of endings, resulting in frggffreueecy late- sate-game states. B: The fraction of late sate-game states (above dashed line in Oware and Checkers) rises either sharply or gradually with rank.
8 Connecting inverse scaling to the game structure
Inverse scaling, i.e. performance scaling negatively with training resources, is known to oceur in LLM training 443 RL models can similarly fail to scale reliably, sometimes decreasing in performance as neural-network size and compute budget are increased. We clear example of inverse scaling for AlphaZero agents trained on Checkers and Oware, as shown in Fig. 5 These agents follow a steady scaling curve that abruptly breaks down beyond a certain neural network size, where Elo starts to scale negatively with size.
8.1 Inverse scaling coincides with frequency-distribue anomaly
As we show in Fig. 5B, the state- frequenyy distribution of Oware games is visibly different from the Zipf's law present in other games. The Oware Zipf curve is interrupted by bump starting after rank 100, caused by a group of states with roughly constant frequency. The Zipf power-law exponent remains unchanged before and after this interruption. In this section, we show that the anomalous Zipf curve is caused by a property of the Oware game tree. The same anomaly occurs in Checkers, but is harder detect solely by looking at frequency curves.
7

---

Turn distribution anomaly The distribution of state turn numbers exposes a distinct difference between games with clean scaling laws and games with inverse scaling. In Fig. O we plot the average number of moves played in an episode before encountering each state. Connect Four and Pentago turn numbers are strongly correlated with state rank, due to the number of possible states diverging exponentially with time. In contrast, Oware and Checkers display forked turn distributions where substantial fraction of the most frequent states appear late in the game, In Oware, the sharp change in the fraction of late states (Fig. (B) happens at the same rank where the Zipf curve bump appears.
States above and below the dashed line Fig. 6A are clearly distinguishable when visualized, see examples in appendix G Late-game states above the dashed line are positions close to the end of a game, when the board is mostly empty, while early-gam game states below the line are opening moves, i.e. small variations of the imitial board state.
Anomaly is caused by game rules The prevalence of high-frequency late-game states in Oware and Checkers data is a direct result of the game rules, which shape the game-tree structure. In both games, pieces are successively removed from the board, ending the game when one of the players has no pieces left (infinite-loop draws are rare). This win condition produces game trees that expand and then contract: starting from a predetermined position, expanding exponentially every move, but then contracting again towards the end of the game as the number of pieces decreases and with it the number of possible board-state permutations. The probability that a game ends in frequently-visiter board state high.
In contrast, Connect Four and Pentago have a different tree structure. These are line-completion games where pieces are repeatedly added to empty board, and the number of possible full-board combinations is still high. Go and Chess also do not have converging game trees; a Go game typically ends with many pieces on the board, with large combinatorial complexity. Chess is a piece-elimination game similar to Checkers, but winning is not conditioned on capturing all the opponent's ' pieces and checkmates can occur while the board is relatively full.
8.2 Relation to inverse scaling
The occurrence of inverse scaling at large model sizes as well as an anomaly in the train-set distribu- tions suggests that the two phenomena might be linked. A possible connection is that an increased neural capacity leads to higher-rank states having more influence on model training. Here we present evidence for such connection between state distribution and inverse scaling.
Oware loss spikes at transition point. As we show in Fig. 6 late-game states (above the dashed line) dominate the Oware distribution at higher ranks, in sharp transition from 0% to > 80% of all states. This transition is drastic enough to visibly skew the frequency distribution seen in Fig. 5B, bending the Zipf power law. The distribution shift causes Oware value loss spike, as we show in Fig. 77A. The loss of all but the largest models jumps by up to an order of magnitude at the transition point, showing that models that fit early-game states well have hard time fitting late-game states. This counter-intuitive behavior of getting worse loss on easier states, which we discussed in section 7 is even more surprising here: Oware states after the transition point are predominantly end-game configurations and should be significantly casier to model than carly-game states using search algorithms.
The quantization model provides an explanation for the sharp rise in loss. As models fit the most frequent states, smaller models do not have enough capacity to additionally model less frequent states accurately. The large qualitative difference between early- and late-game states makes the difference in loss especially pronounced. Interestingly, larger agents converge on a smooth loss curve similar to that of Connect Four, at the cost of higher loss on early-game states.
Large models overfit on late-game states. Splitting the loss curve to early- and late-game states paints clearer picture. Loss on early-game positions scales smoothly for all models (Fig. B)) with smaller models achieving lower loss than larger models. Surprisingly. large models have better loss on late game states (Fig. (C) following reversed order compared to early-gam game states. We note that the data collected from games at late stages of training. when game quality is high and relatively constant.
Inverse scaling is likely caused by large model overfitting on late-game states. Larger models fit end-game states better, for the price of worse performance on game openings. Opening moves
8

---

A. Oware value loss
B. Early-turn loss
C. Late-turn loss
10
10
10
10
Parameeto memeererr
10"
State
State rank
State
Figure 7: Increasing neural capacity leads to overfitting on end-game states in Oware. A: Value training loss sharply increases at the point where late-game moves dominate the state distribution (dashed line). Late-game states have much higher loss despite being much casier to model. B&C: Looking at the loss on late- and early-gama states separately, we see that larger models predict late-game states progressively better, but predict less-frequent, parly-game states progressively worse. Increasing the capacity allows larger models optimize intermediate- states, shifting their focus to frequently appearing late-game data. Forgetting crucial early-game information could explain the onset inverse scaling Oware.
Large agent value
Early-turn
Late-turn
Treining time Sammalzze (aralizze)
10
10-4
10
10
10
rark
Figure 8: Large models forget knowledge on early-game states. A: The loss of the largest Oware agent improves (on average) during training. B: On early-game game states, loss initially goes down, then stops improving and eventually worsens towards the end of training. C: Loss on late-game states steadily improves during training. The fact that large nets fit early-game states but then forget them suggests that target non-stationarity might be the cause of inverse scaling.
crucial for good performance, while end-game states are mostly insignificant; the branching factor of end-game positions is small, meaning even an unguided MCTS agent could find the optimal move through brute-force search. Sacrificing important strategic knowledge for the ability to fit numerous trivial positions accurately directly leads to the inverse scaling of Elo seen in Fig.
8.3
Why do large models overfit?
The degradation of large model performance on early-game states is an example where RI. scaling diverges from the quantization model. The quantization model explains how larger models could fit more tasks, without forgetting higher-frequency tasks. In RI., we see that fitting less- aas-ffrquent states coincides with lower performance on the most frequent states, when states split into two distinct groups. The difference between supervised learning and RL scaling might be caused by shift of the target distribution during training, which is known to cause training failure, e.g. through dormant neurons 44
Large models selectively forget. During training, large models seem to forget what they Iearn about sarly-game states, but not about sate-game. In Fig. g we plot the loss of the largest Oware model at different stages of training. We see a clear difference between the learning dynamics of early- and late-states: late-game loss improves gradually throughout training, while early-game loss improves, then degrades at late training.
A possible explanation of this selective forgetting is a selective shift in the distribution of the target labels, i.e. the value of each state. Sokar al. (2023) (44) have shown that changes to the target distribution in RL can harm performance by creating dormant neurons. Unlike early-game states, late-game states have relatively fixed label distributions since the outcome is already decided by that point in the game. A hypothetical mechanism of inverse scaling then follows: 1) Models fit states by

---

order of frequency. gradually expanding the fraction of late- lite-game states in the pool of learned states. 2) As new states are optimized, higher-frequency states learned earlier can be forgotten. 3) In Oware, late-game states are immune to forgetting due to their strongly fixed labels. 4) Large Oware models are trapped in local optima, fitting many end- ind-game states well but forgetting smaller fraction of crucial early-game states. See appendix H for more in-depth explanation.
Comparison inverse scaling in supervised learning We note a certain similarity between this inverse scaling phenomenon and one of the causes for inverse scaling identified by McKenzie et al. (2023) 43 namely distractor tasks. Some LLM benchmarks require mastering a set of tasks T, but as model scale increases, models first learn a set of easier tasks 1 that harm performance, leading to inverse scaling. Another similar concept is competing tasks, which can cause inverse scaling in a supervised learning setting 45 While the concept of competing/distractn tasks bears resemblance to the Oware case, the mechanism behind these phenomena different. These examples of inverse scaling in supervised learning are driven by the similarity of tasks, while in our case early- and late-game states are far from similar, which is why we believe inverse scaling is caused by indirect interaction between tasks through forgetting.
Limitations. We note two limitations of our analysis of inverse scaling. First, unlike Oware, the influence of late game configurations is harder to visualize in Checkers. Since the transition to late-game states smoother Checkers (Fig. 6B), the Zipf curve only visibly skews at very high rank numbers (Fig. (C). As a result, one would need exceedingly large amounts of data to accurately plot the rank at which late-states dominate the distribution.
The second limitation is the use of game states as proxies for independent task quanta. Oware inverse scaling cannot be solved simply by removing all late turns from the training distribution, possibly because the group of tasks that models tend to overfit only partially correlates with late- states. Fig. | miisses some of the dataset due to states that only appear once in training. These states are at the tail-end of the frequency curve, given their low individual frequencies, but they constitute the majority of states encountered in training. In appendix[]w visualize data frequency over a different axis and show that the tail-end of the state distribution, which we removed for our above analysis, is crucial for accurately calculating frequency scaling along other axes. A better grouping of task quanta might allow one to know which states to exclude from training in order to prevent inverse scaling.
Discussion
We presented evidence connecting the quantization model of LLM neural scaling AlphaZero power laws. In particular, we found that AlphaZero data follows Zipf's 's law and that agents optimize states according to their frequency rank, counter- inntter-ittuitively achieving worse loss easier states. We also shed light on the mechanism behind inverse scaling, showing that inverse scaling correlates with an abnormal frequency distribution. This in turn can cause agents with larger capacity to focus on end-game states, getting better loss on these states but worse loss on crucial opening moves.
While this work touches several aspects of scaling law theory, we still lack full model of RL scaling that can fully connect state distribution to Elo scaling laws. We provided clear evidence the effect of Zipf's law on loss, but loss is only a proxy for performance and the multi-agent Elo metric is connected to it non-trivial way. significant problem with applying the quantization model to explain scaling laws is the difficulty of identifying task quanta, both for LLMs and RL, which is why we use game states as a proxy of tasks. Identifying the tasks or concepts that AlphaZero learns is a challenging interpretability problem, mostly limited to human game knowledge 31 46).
We hope the analysis presented here will help advance the understanding of RL scaling. RL methods are often challenging to scale, and understanding why is a prominent topic in the field today [47] 48
44 49 Our results hint at a possibility for improved RL algorithms using curriculum informed by the frequency distribution, similar to recent attempts in supervised learning (33
Acknowledgments and Disclosure of Funding
We thank Eric J. Michaud and David J. Wu for insightful feedback and discussions.
10

---

References
[1] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, arXiv preprint K1iv:2001.08366 2020.
[2] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint wrXiv:2203.15555 2022.
[3] Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint 10Xx::2010.14001,, 2020.
[4] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12104-12113, 2022.
[5] Oren Neumann and Claudius Gros. Scaling laws for multi reinforcement learning model. In The Eleventh International Conference on Learning Representations, 2022.
[6] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sehastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0768 07682, 2022.
[7] Utkarsh Sharma and Jared Kaplan. A neural scaling law from the dimension of the data manifold. arXiv preprint arXiv:2004. 10802, 2020.
[8] Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint arXiv:2102.0677 2021.
[9] Mareus Hutter. Learning curve theory. arXiv preprint arXiv:2102.04074. 2021.
[10] Jinyeop Song, Ziming Liu, Max Tegmark, and Jeff Gore. A resource model for neural scaling law. arXiv preprint arXiv:2402.0s 164, 2024.
[11] Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ 3 phases of compute-optimal neural scaling laws. Advances in Neural Information Processing Systems, 11116499-1657,, 2024.
[12] Eric J Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling, arXiv preprint arXiv:2303. 13506, 2023.
[13] Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pages 1024-1034. PMLR, 2020.
[14] Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 1013-1111,,,,, 2021.
[15] Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.16859, 2022.
[16] Vivien Cabannes, Elvis Dohmatob, and Alberto Bietti. Scaling laws for associative memories. 2023.
arXiv preprint
[17] Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning. arXiv preprint arXiv:2301.13442 2023.
[18] Jens Tuyls, Dhruv Madeka, Kari Torkkola, Dean Foster, Karthik Narasimhan, and Sham Kakade. Scaling laws for imitation learning in nethack. arXiv preprint arXiv:2307.09423 2023.
11

---

[19] Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe. Philemon Brakel, Sarah Bechtle, Steven Kapturowski, Roland Hafner, al. Of- fline actor-critic reinforcement learning scales to large models. arXiv preprint arXiv:2402.05546, 2024.
[20] David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. Mastering chess and shogi by self-play with a general reinforcement learning algorithm. arXiv preprint arXiv:1712.01815, 2017.
[21] Andy I. Jones. Sealing scaling laws with board games. arXiv preprint urXiv:2104.031 13, 2021.
[22] Oren Neumann and Claudius Gros. Investment vs. reward in a competitive knapsack problem. arXiv preprint arXiv:2101.10964, 2021.
[23] Bernd Blasius and Ralf Tönjes. Zipf's law in the popularity distribution of chess openings. Physical Review Letters, 103(21):218701, 2009.
[24] George Kingsley Zipf. The psycho-biology of language: An introduction dynamic philology. Routledge, 2013.
[25] Isabel Moreno-Sánchez, Francese Font-Clos, and Álvaro Corral. Large-scale analysis of zipf's law in english texts. PloS one, 11())=e147073, 2016.
[26] Steven Piantadosi. Zipf's word frequency law in natural language: A critical review and future directions. Psychonomic bulletin & review,
2014.
[27] Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Error scaling laws for kernel classification under source and capacity conditions. Machine Learning: Science and Technology, 4(3):035033, 2023.
[28] Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, and Sanmi Koyejo. How do large language monkeys get their power (laws)? arXiv preprint arXiv:2502.1 /7578, 2025.
[29] Elvis Dohmatob, Yunzhen Feng. Pu Yang, Francois Charton, and Julia Kempe. A tale of tails: Model collapse change of scaling laws. arXiv preprint -XXix2402.07744: 2024.
[30] Bertrand Georgeot and Olivier Giraud. The game of go as a complex network. Europhysics Letters, 9766),68002 2012.
[31] Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. Acquisition of chess knowl- edge in alphazero. Proceedings of the National Academy of Sciences, :19645))e220662111 19, 2022.
[32] Charles Lovering, Jessica Forde, George Konidaris, Ellie Pavlick, and Michael Littman. Evalu- ation beyond task performance: analyzing concepts in alphazero in hex. Advances in Neural Information Processing Systems, 35:25992-26006 2022.
[33] Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya Li Kevin Wenliang, Elliot Catt, John Reid, and Tim Genewein. Grandmaster-leved chess without search. arXiv preprint arXiv:2402. 044444 2024.
[34] Mare Lanctot, Edward Lockhart, Jean- Bann-stt Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, Daniel Hennes, Dustin Morrill, Paul Muller, Timo Ewalds, Ryan Faulkner, János Kramár, Bart De Vylder, Brennan Saeta, James Bradbury, David Ding, Sebastian Borgeaud, Matthew Lai, Julian Schritwieser,, Thomas Anthony, Edward Hughes, Ivo Danihelka, and Jonah Ryan-Davis. OpenSpiel: A framework for reinforcement learning in games. CoRR, abs/1908.094 908409453, 2019.
[35] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 394344)324-345, 1952.
12

---

[36] Arpad E Elo. The rating of chessplayers, past and present. Arco Pub., 1978.
[37] Martin L. Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
[38] Wentian Li. Random texts exhibit zipf's-law-like word frequency distribution. IEEE Transac- lions on information theory, 38(6):1842-1888 1992.
[39]
Pascal Pons. Connect 4 solver, 2015. https://connect4. gww.roove..g............oo org/
[40] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 33:1877-1901. 2020.
[41] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Farriga-Alonss al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint urXiv:2206.0460 04.15, 2022.
[42] Donald E. Knuth and Ronald W Moore. An analysis of alpha-heta pruning. Artificial intelligence, 6(4):293-326, 1975.
[43] Ian McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. Inverse scaling: When bigger isn't better. arXiv preprint arXiv:2306.09477) 2023.
[44] Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evei. The dormant neuron phenomenon in deep reinforcement learning. In International Conference on Machine Learning. pages 32145-333 021553-218... PMLR, 2023.
[45] Muhammed E Ildiz, Zhe Zhao, and Samet Oymak. Understanding inverse scaling and emergence in multitask representation learning. In International Conference on Artificial Intelligence and Statistics, pages 1726-47734 PMLR, 2024.
[46] Lisa Schut, Nenad Tomasev, Tom McGrath, Demis Hassabis, Ulrich Paquet, and Been Kim. Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero. arXiv preprint arXiv:2310. 164 2023.
[47] Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under- parameterization inhibits data-efficient deep reinforcement learning. 10X1::2010.14498, 2020.
arXiv preprint
[48] Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:2204.0955 2022.
[49] Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. arXiv preprint arXiv:2403.033950 2024.
[50] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[51] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
[52] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint 10XXi::8/2.05000 2018.
[53] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, al. Mastering the game of go without human knowledge. nature, 550(7676):354-35,, 2017.
[54] Christopher D Rosin. Multi-armed bandits with episode context, Annals of Mathematics and Artificial Intelligence, 2133)220,- 2011.
13

---

A Additional Background
AlphaZero AlphaZero is an RL algorithm for learning two-player rero-sum games, learning solely from self play [20]. The algorithm performs a Monte-Carlo tree search on the tree of future moves, guided by a neural network. The neural net receives the current board state as input, providing an estimate of its value, together with policy prior, p. The loss function is composed of separate elements for y and p:
== - u)² x log
During training, the agent visits board states s, having ground truth labels 2, for which it generates estimates and for the state value and target prior, respectively. The MSE value-loss term (= - v)² pushes fit the observed game outcome = E {1, -1}, representing win, draw and loss respectively. The cross-entropy policy loss - log p pushes p to fit the actual policy π generated by the tree search.
AlphaZero scaling laws AlphaZero scaling is one of the earliest examples of neural power-law scaling in RL [). Jones [2I] showed that AlphaZero agents trained to play Hex improve their Elo score r with the log of training compute C. log(C), with coefficient that remains constant across board sizes. As they point out, this predictable scaling behavior allows one to plan the compute budget ahead of training compute scaling in agents trained on Connect Four and Pentago, as well as scaling with neural network size, and pointed out that these scaling curves are power laws of the Bradley-Terry playing strength [35]. since xx lognu(?) Similar to LLM scaling laws, these scaling laws of AlphaZero allow one to know in advance the optimal model size that would achieve maximal Elo score, given the available training compute budget. An interesting result from Jones shows that training compute can also be traded off predictably for inference compute 21
fit a desired performance level. Neumann & Gros [5] found similar
B Board game descriptions
We describe here the rules of all four games examined in this paper. All games are two-player, zero-sum, open information games, where players alternate turns between them moving pieces on a board.
Connect Four The players alternate placing disk of their color into one of seven vertical columns on a 6x7 grid, and the disk falls to the lowest unoccupied position in that column. player wins by arranging four of their disks in a row, either vertically, horizontally, or diagonally. If all cells are filled without such an alignment, the game is drawn.
Pentago Players alternate placing marble of their color any empty cell of a 6x6 board, which divided into four 3x3 rotating quadrants. After placing, the player must rotate one quadrant by 90F either clockwise or counter-clockwise. A player wins immediately upon obtaining five marbles in a row (horizontal, vertical, or diagonal) either before or after rotation. If the board becomes full without five in row, the game is drawn.
Oware This game is played on a board with two rows of six houses and optionally store pits. Each of the twelve small houses starts with four seeds. Each player controls the six houses their side. On their turn, a player selects one of their houses with seeds, removes all seeds, and sows them counter-clockwise, placing one seed per subsequent house (excluding stores and skipping the original house when it's revisited). If the last seed lands in an opponent's 's house and brings its count to exactly two three, those seeds are captured into the player's store. Capture continues backward from that house as long as the immediately preceding opponent's 's houses also contain exactly two or three seeds. Capturing all of an opponent's seeds is forbidden; in that case no capture is made. The game ends when a player captures 25 or more seeds, or both players capture 24 each (draw), or if no legal move exists; remaining seeds are then collected by their owners. In our analyses, draw is also declared the game lasts 1000 turns.
The players each control 12 pieces on opposite dark squares of an 8x8 chess board. Normal pieces move diagonally forward one square to unoccupied dark square. If an opponent's
Checkers
14

---

Connect Four
10"h parsmwts
10ah permmeter
10 parsmrtten
10" permmtten
10
0.00
1..0.
0.
10
10
13
10F
Frqqencyy
10"
10"
100
120
10P
10P
100
10
10
10
10
104
Pontago
10"h
10"h parsrreeess
10"h parsrreeess
10"h parsmeters
10P
0.00
10
139
105
100
Frquerncy
10"
12"
10"
13P
122
10P
10P
10P
120
50P
10
10
10
10
pararneeess
12"
parsreters
10"
1..0.
0..0
100
1.001.00
1.04
11- 1.05
10
10
12P
Freqaany
107
13F
130
10
10
122
10
10
107
137
10'
10
10
10'
10
10
12
12
107
10
12
10
10
101
parmeees
parmmeless
parsmeters
parsmelers
10
-=00
0.70
...00.7
+.70.
0.008
10P
10"
104
10"
Requency
101
120
10P
100
10F
107
10
10'
10
10 10F
10F
10P
10P
122
107
10
Figure Zipf's law curves for AlphaZero games. On the left column, we plot the frequency distribution of states collected from different-sized agents training each of the four board games discussed in this paper. In each row we plot curves describing the distribution of games played by single agents. We find small variations of the power-law exponent between agents, overall staying close to the general exponent each game. The averaged exponents vary in a range that in agreement with human-gan exponents.
piece is diagonally adjacent and the following square is empty, the adjacent piece may be jumped and removed; multiple jumps allowed in sequence if available. When piece reaches the farthest row on the opponent's side, it is crowned as king and gains the ability to move and jump both forward and backward. Play continues until one player has no legal move (draw) player lost all their pieces. We also draw the game if it lasts too long.
C Zipf's law variation
C.1 Zipf-exponent variation among agents
The different strategies learned by individual agents lead to small differences their frequency- distribution curves. Here we show examples of Zipf curves obtained from games played by individual agents during training. Agents follow slightly different Zipf laws, but the deviation between agents smaller than the deviation between environments.
15

---

A. Lightly-skawed policy, (0.7.0.3)
B. Heavily- policy, p (0.95. 0.05)
10"1
10-2
10-3
Game length
Frequenoy
Frequency
10
15
10-3
10-4
10-7
10
10-9
10
10
107
10
10
10
10"
10P
10'
10P 10
10
10'
10"
10
10
State rank
State rank
Figure 10: Zipf's law in random games with non-uniform policy, A: In toy-model game with branching factor of policy that prefers one action over the other will smooth-out the plateaus of Fig. 2 into Zipf's law with exponent a 1. The power law dies out near the end of the distribution, which depends on game length. B: If the policy heavily prefers one action over the other, it produces series of plateaus again. These plateaus quickly diverge from the a = Zipf"s ' law in short games, but converge to Zipf"s law as game length increases.
Total Zipf distribution In Fig. plot the state-frequency distributions of different games, plotting the combined distribution of many agents the left-most column. We do this by counting the total frequency of states across games played by several different-sized agents. Each environment has its own Zipf exponent, but all exponents lie the range [0.75, 1.05]. For Oware, we the part of the curve that comes after the bump caused by late-game states, appendix G Checkers also exhibits a bump in the distribution, albeit harder to see since starts at a much higher rank.
Single-agent Zipf distribution Another set of Zipf curves can be obtained by counting state frequencies only in games played by single agent. In each column after the left-most one, we plot the distribution generated by an agent with a different number of neural-net parameters. The standard deviation of the exponent among agents is low in all environments, standing at 0.01 or lower for Connect Four, Pentago and Checkers, and at 0.02 for Oware. The uniformity of the exponent across different agents agrees with prior observations on human games: the opening frequencies of human Go games also follow Zipf's law, with the same exponent fitting both amateur games and games played at prestigious Go tournaments [30]
C.2 Zipf's law in non-uniformly-rannn
games
Zipf's law in games arises from their branching-trea structure. In section we discuss the origin of Zipf's law, showing that random toy-model games create frequency distribution of plateaus centered on a power-law with exponent = 1. We show that this distribution can be smoothed into a Zipf's law by adding complicated game rules, but keeping the games random.
Here we show another mechanism that can smooth the plateau distribution into a Zipf's law. the policy generating the games is not uniformly random, it generates smooth power laws. However, if the policy approaches a deterministic policy, the frequency plateaus reappear.
Policy bias smooths the distribution In Fig. 10 we plot the state- fretuefre distribution of toy- model games under different agent policies. These games have constant branching factor of 2, and last fixed number of turns. We look at games generated by random policies, that are skewed towards preferring one action over the other at every level of the game tree.
Shifting away from the uniform policy p (0.5, 0.5), become shorter and denser, eventually becoming a smooth curve. The distribution follows the a = power law, diverging only at the very end of the distribution. The power law stretches over more orders of magnitude as game length is increased.
that the plateaus of the uniform
Extreme bias reintroduces plateaus Surprisingly, changing the policy too far from uniform eventually leads to frequency plateaus again. In Fig, L0B we plot games played by a policy that is
16

---

Decrease of policy quality with temperature
Fig. 3 cutoff
10
Probability to play an optimal move
Parameters
10
Policy quality
Policy quality decreasing
unaffected
103
10
10
10
Temperature
Figure 11: The effect of temperature on agent performance. We plot the probability that an agent will play optimal move, against policy temperature T,. As T is increased, the probability to take the best action remains constant, starting to drop only above T xs 0.5. The region where performance is unaffected is the same region plotted in Fig. BB confirming that the relation between Zipf- and scaling-law-exponentt is causal, rather than the result degrading performance due to high T.
close to being deterministic. The strong preference one action creates plateaus of common games: the first plateau corresponds to an entire game where the same action was played every turn, the second plateau is made of games where the less-preferred action was played once, and so on. The plateau distribution still follows the linear = trend, but diverges from it at a much earlier point than more-uniform policies. One must increase game length significantly to see that this distribution converges to a linear trend at the mamnite-gam-leentth limit.
D Policy degradation with temperature
In section we plot the correlation between the scaling-law and Slpw"s-law exponents modulating the temperature of the action-selection policy, defined in Eq. We claim that this correlation, plotted in Fig. 33, is caused solely by the shifting Zipf distribution that affects agent performance, and by temperature causing agents to play suboptimal moves.
Here back up this claim by demonstrating that agent performance individual states does not change in the temperature range used to plot Fig. 3B. To estimate performance on states, we look at the policy л (T) produced by Monte-Carlo tree search and plot the probability it assigns optimal actions. For each state s, we calculate the probability the agent plays an optimal action:
p(oplimal)
Σ (T(
(EA(s)
The set of optimal actions A(s) is calculated with Connect Four solver using alpha-beta beta pruning. An optimal action is defined as any action that, under optimal play, will lead to victory, or to a draw if victory possible. States where every move leads to loss do not have any optimal actions, and are ignored in this analysis. The probability to "blunder". i.e. lose the lead a game, is therefore cquivalent p(optimal).
Temperature does not harm performance in the examined range In Fig. 11 we plot p(optimal) against policy temperature 7' for several Connect Four agent with different sizes For temperatures below the vertical dashed line, marking the region where exponents were calculated for Fig. 3 the
Error bars the standard error of the mean.
17

---

0.9
10P
Elo expoment
Temperature
0.7
0.5
10
Fig. 3B cutoff
0.5
0.4
10-2
Zipt exponent (tail)
Figure 12: A zoomed-out version of Fig. 3 that includes higher temperatures (= smaller exponents). FC only contains data to the right of the vertical cutoff line, which marks the temperature where T stops affecting policy quality (equivalent to the line in Fig. 11.. This line happens to coincide with the point where the Zipf exponent starts changing.
probability to take an optimal action is constant and does not change with T. This confirms that the change of the scaling-law exponent in this region only due the effect T'' has the Zipf curve.
is also possible to notice the effect of temperature on the scaling laws in Fig. 33. The scaling law exponent changes rapidly for T 0.5, but slows down near T = 0.5. It then starts change more rapidly only when the Zipf exponent starts to change significantly.
3
Zipf's law only changes when policy quality is constant The full effect of temperature
exponent correlation is seen in Fig. 12 where we plot a zoomed-out version of Fig. 3 The dashed line marks the border of Fig. C, which is the same point where temperature stops affecting policy
quality in Fig. II
Interestingly, the vertical line in Fig. 12 marking the T value where policy stops affecting policy quality coincides with the point where the Zipf curve starts to bend, as can be seen when looking the change of the x-axis values. Policy quality changes significantly above T 0.5, but the Zipf distribution changes significantly only below T 0.5.
E
Ground-truth loss scaling on different distributions
We present here another plot of ground-truth value-loss scaling, in addition to Fig. 43. In Fig. 4B, plot the loss of agents on a collective dataset, aggregated from the training runs of several agents of varying sizes. Here in Fig. 13A plot the same loss, this time by calculating each agent's loss their own dataset. The only difference between the two plots is the order of states; the ground-truth labels of each state remain the same. The error bars of Fig. 4 and Fig. 13 are the of agent loss across agents, either geometric when the y-axis is in logscale (Fig. 4A) or arithmetic when the y-axis is linear.
F The difference between AlphaZero value and actor-critic value
In sections] 7 and S we present results on AlphaZero loss scaling, focusing on the loss the network's value head. We chose the value head since it directly dictates the agent's policy both at training and inference time. This appendix, aimed for readers unfamiliar with AlphaZero, clarifies how AlphaZero's 's value head differs from the value net of actor-critic methods, which has effect policy at inference time.
Actor-critic methods are family of popular policy-gradient algorithms, such as PPO and SAC 50 51 52 These models use an architecture where two neural network outputs are used, namely a value estimation (or alternatively Q value) and a policy vector p. Actions are sampled from the
18

---

A.
Value loss on own train set
B. Value loss on all train sets
108
Parameters
Loss
0.4
0.4
10
0.2
0.2
10a
0.0
0.0
10
10'
10
10
10*
10a
109
10'
10"
10*
10a
State rank
State rank
Figure 13: Connect Four ground-truth value-loss on different datasets. A: Average loss on each agent's own training data. B: Fig. 4B, presented here again for comparison, displaying average loss on common dataset collected by different agents.
policy vector, which in turn is trained on the output the value net v. While the loss of the value net on test data should be low for better agents, it has effect the actions played at test time.
In contrast, AlphaZero's value head the main parameter used for choosing actions, while the policy head plays a secondary role as static prior for Monte-Carle tree search (MCTS). The AlphaZero policy not the output of the policy head. Rather, æ is calculated using Eq, | repeated here for clarity:
π S0, a) N(so, a) b)1/T
(8)
The visit count N (s, a) is the number of times action a was probed during MCTS, sampled this equation for each action at the root node, meaning at the current game state The visit count is indirectly determined by the neural net's value and policy outputs (u. p), since these parameters are used by MCTS to decide which future actions to probe when performing rollouts. The decision what action to probe at each junction in the game tree is done by maximizing the following quantity [53]
MCTS action argmax (Q(s, U(s,a))
The exploitation term Q(s, a) is the average value v(s') of all states s' stemming from the state-action pair (s, a). It is approximated by averaging the halue-head output over all daughter nodes probed by MCTS so far:
Σ u(s')
daughter nodes s¹ N(s, a)
((s, a)
(10)
The exploration term (ss,a) a) is a variation of the PUCT algorithm visit counts N:
54 favoring actions with low
N(s.b)
U U(s a) = Cpuct p(s, a) 1+N(s,
(11)
This term is weighted by the policy- head output which serves as a constant prior that favors exploring some actions others.
Low value head loss strong predictor of good policy æ, since means the value head produces an accurate exploitation term Q(s, a). The most frequently probed actions will be those with high
19

---

Checkers Board states
State
State
State #100
State #1000
A
A
4
Late-game board states
State
State #141
State
State
Figure 14: Checkers board positions. Low-rank (= high-frequency) states are mostly opening moves, visited shortly after the game begins. These states are mixed with high-frequency end-game positions, where only about 2-3 pieces still remain the board, all promoted from to king. The time it takes to end the game after reaching such an empty-board state is long; about half of all games end in a draw after no capture occurred for 40 turns.
(ss, a) value. contrast, the policy prior controls an exploration term that only significant at early stages of search, and vanishes at the limit of large search:
U(s, a)
0
(12)
Intuitive explanation We illustrate the difference between AlphaZero and actor-critic algorithms with simple example. A trained actor-critic agent will be unaffected we replace its value net with random function at test time. In contrast, replacing the value head of AlphaZero with random function at test time will generate an almost random policy; replacing the policy head with a uniform distribution will only somewhat degrade the policy.
G Board position visualization
We provide here visual examples of the two groups of high-freque gigh-freuuency states found in Oware and Checkers. As show in Fig. [6] aute-game states get mixed with carly-game states in the set of high-frequency states. This happens because by late game most pieces are captured, and the number of possible board states narrows down small selection.
In Figs. 14 and 15 we plot randomly-same sampled game states from the state- cuate-frequncyy distribution, for both Checkers and Oware games. High-frequene states split into two clusters, namely opening moves and end-game moves. Early-game openings, plotted the top row each figure, are all small deviations from the initial board state. In the initial state of Checkers, each players' man pieces occupy their 3 closest rows; in the Oware initial state, there are 4 seeds in each pit.
The bottom row of each plot shows examples of late- taee states. In these states, only a few pieces are still left on the board, after most pieces have been captured by the players. In Checkers, these states contain only 2-3 pieces that have already been promoted to king: in Oware, only few seeds remain uncaptured. It is easy to see why these states have such high frequencies: most games end one of those configurations, and the number of such possible configurations is very small. For example,
20

---

Oware Board states
State
State #10
State #100
State #1005
Score: 0
Score: 0
Score: 0
Score: D
Score:
Score:
Late-game board states
State #164
State #335
State #1420
State #1 1649
Score: 24
Score: 22
Score: 21
Score: 19
Score: 21
Score: 20
Score: 20
Score: 24
Figure 15: Oware board positions. States below rank 100 are all openings, deviating only slightly from the initial position of 4 seeds in each pit. At rank 164, end-game states start to dominate the distribution, making up > 80% of all states. Most seeds have been captured in these late-gat late-game states, and the game ends shortly after they are played, when the active player has no seeds left on their row.
the number of possible Checkers configurations with two kings is 992, and the number of Oware configurations with three seeds is 1728 (ignoring player scores). Moreover, these configurations are not played with equal probability, leading to some states having even higher frequency.
H Inverse scaling mechanism
In section
8.3 we mention a hypothetical mechanism that could cause the observed inverse scaling Here we discuss this mechanism in more detail.
phenomenon.
We make the following main assumptions:
Agents fit states in descending order of frequency. An agent with a neural network of size N can perfectly fit the values of N/c states, where is the neural capacity needed fit a single state [1].
As the agent improves during training, game outcomes (i.c. state values) change. The values of early-game states change significantly during training, while the values of late-game states change only slightly, at all.
The shift of state values causes neural plasticity loss due to dormant neurons
44
The first assumption is based on the quantization model of neural scaling 12
The second assumption is based the fact that, in most board games, player skill greatly influences the value of early-game states. In the games discussed in this paper, a match between random agents will usually end in draw
0) while match between optimal agents will end in first-player victory (v = 1), except for checkers. In contrast, late game states that are played a few steps before the game ends have values that largely independent of player skill, due to power imbalance. Very late states have fixed values if MCTS can probe the entire game tree stemming from them.
The third assumption is based on the work of Sokar al. (2023) 44 the labels of dataset during training causes some neuron activations to die out. These so-called "dormant neurons" effectively reduce the neural capacity of the model.
who showed that changing
Assume we train a small agent with 10"h neurons and a large agent with 10t neurons, > m. In our model, the small agent roughly fits 10"h /c s Tearly states, where Nearly the number of carly-game states with higher frequency than late game states. In Oware and Checkers, Rearty falls in the range [100, 200), see Fig. The large agent fits 10 /c >>> Hearly and mostly memorizes the values of late-game states.
21

---

For simplicity, let us treat the change of state values as a single, abrupt event during training. After this event, both models have high loss on early-game states, but maintain the same loss as before on late-game states, since the latter group kept their value labels. Due to plasticity loss, both models can now fit fewer states than before. The small agent now has high loss, since could only fit the first Pearly states before values changed, meaning most of the states it memorized now have new labels. The large model, on the other hand, maintains a lower loss, since it could fit orders of magnitude more states than the first flerry states, most of which are late-game states.
It is likely that the small agent, which now has loss close to that of random agent, will continue to train and re-learn as many states as it can, starting from the most frequent ones, Eventually it will regain low loss on most all of the first Rearly states. contrast, the large agent maintains low loss, and could be close to local minimum of the loss landscape that fits late-game states well but assigns wrong labels to early-gan game states. To leave this minimum and re-learn the first Rearty, states, the large agent will have to sacrifice knowledge of some late-game states, since it lost fraction of its neural capacity. In this scenario, the large model could remain stuck in the local loss minimum and never regain performance on early-game states.
In reality, it is more likely that value labels shift gradually rather than abruptly, meaning that loss increases gradually during training rather than spiking abruptly. We observe such a loss degradation in Fig. where large agent value loss on early-game game states starts to degrade mid-traine mid-training.
I Frequency scaling on different axis: capture difference
The quantization model of neural scaling laws states that models will learn independent task quanta by descending order of frequency. In language modalities, it is assumed that the known Zipf's law of word frequencies will lead to a Zipf's law of tasks. Similarly, we show that board states follow Zipf's law, suggesting that AlphaZero task quanta will also scale with a frequency power law. The states themselves are not independent quanta, since many of them share similarities, and strategic knowledge of one state can often transfer to another.
Here show different way to visualize training data frequency, illustrate that more than one dimension of the data can have clear frequency scaling. When models fit tasks by frequency, some most-frequent tasks will not be represented at all on the state-rank curve, because states do not correspond to tasks directly.
To illustrate the digensiinnsionlltty of the training data, we plot in Fig. the the frequency of capture- differences in Oware and Checkers board states. Capture difference is defined as the difference between the number of pieces captured by each player at certain point in the game, in absolute value. We see that frequency drops smoothly in log-scale with capture difference, as states with a higher score- ccore-difference between players are rarer. According the quantization model, one would expect agents to fit states according to the exponential distribution of -ifuurr-dfffeencsss giving aioooentiallyydecraasinn importance to states with higher differences.
The distribution of capture-differente cannot be represented correctly in our state-frequency analysis. We demonstrate this in Fig. [6] where we plot in yellow the same distribution, but omitting the tail of the Zipf's 's law stateffrequency distribution. The tail is mostly composed of one-time states, i.c. states that appeared only once in training. In fact, the majority of unique states in the dataset have frequency of 1. By discarding these states, as we do in our main results, we lose information about the exponential decay of capture-differencc frequencies. Our main results cannot take these states into account, because exponentially-mon games must be played in order to correctly measure the frequencies of one-time states.
Capture difference an arbitrary measure, but it showcases the difficulty of visualizing data frequency. By visualizing one dimension of the data, we lose useful information about the frequency of other properties. If task quanta are indeed learned by the models, then these tasks would likely have a non-trivial representation, and visualizing their frequency could be difficult.
22

---

Oware capture distribution
B. Checkers capture distribution
10
A1stars
Cnty high desuernyy 93Ms5
Onty reequeccy d909
Requency
Reqquany
10
10"
20
Copture difference
Captura difference
Figure 16: State frequency distribution by capture difference. Measuring the score difference between players provides another natural axis along which frequency drops smoothly. Agents could potentially prioritize optimizing mmall-difference states over augg-difference, modelling more states with increasing capacity. Prioritizing states in this way is not equivalent to prioritizing states by their frequency-rank, i.e. by the Zipf's 's law of Fig. 7 since a large portion of the states come from the tail-end of the Zipf distribution which is discarded in the main section results. If we ignore the tail of Fig. bby plotting only high-frequency states, the frequency distribution changes significantly (yellow). It is likely impossible to visualize the true order in which data is learned with a simple metric.
J Distribution of ideal games
In section [ we claim that random games played in toy-model setting will result in a frequency distribution shaped series of plateaus, centered around power law of exponent 1. We verify this claim here.
In short, in a simple branching game the number of possible states is exponential in the turn number t, and all states turn have an equal probability to appear a game, hence the series of frequency plateaus observed in Fig. 2A.
Frequency distribution function Consider random game setting following the rules presented in section namely:
Each game lasts K turns.
Each turn t, a move o is sampled uniformly from b options.
Each board position played at turn t(s) only be reached by single, unique sequence of moves a, (ai, .2,........................... de(=i). This means that one can define a state only by the sequence of moves played to reach it.
The position turn t effectively sampled uniformly from b' possible positions, since the moves a,, i E {l.....t] f that lead to it are each sampled uniformly from b options. Since K turns are played each game, when sampling a board position S randomly from all games played the frequency of sampling board state is:
1
P(S s) P(t=t(sss)) P(S = s/t t(s)) =
(13)
K b
where t(s) is the turn number of board state s and P(t = =s(s)) is the probability to sample from turn 1(s).
Sorting the board positions descending order of frequency results in a series of plateaus, each corresponding to a turn number. The number of board states in plateau number is b². Plateau number t starts after board position number netars (t), defined by:
6 b
Tastart (1)
(14)
1=1
23

---

since each preceding plateau contains b' states. Board state number belongs to plateau number t for the integer t N that satisfies:
Dattrt (t) I n < Dstart (t 1):
(15)
or:
b' b
b+ - b
b S n2
(16)
1
Shifting terms in the inequality and taking the log get:
b' b s
(b 1)m < b(+1 -
(17)
6º <
(6 1)n + b b
(18)
t llog(b) - log [(b 1)n + bl < (t + 1) log(b)
(19)
log [b 1)n + b]
<t+1
(20)
log
since E N, this is equivalent to using the floor operator:
log
[[b 1)n + b
t(n)=
(21)
log b
Using Eq. 13 one sees that the frequency of the n-th most common board state, sn, is proportional to:
1
P(S sn,)
(22)
b((n)
where t(n) defined in Eq. 21 This is the series of plateaus seen in Fig. 2
Distribution bounded around power law Let us define f((n) E as t(n) without the floor operator:
log [(b 1)n + b]
i(n)
(23)
log
Since x - [x] S v, P(S= = sa) is bounded by:
K bé(m) PPSS sn) b((m)-1
(24)
Expanding the bounding functions we get:
1
(25)
bk(m)
Ingi(h- (intt) lox
elog((b-1)nn ()]((b--)+++)))))
(b - 1)n +
leaving us with:
- (b-1))++ P(SS= K (b 1b-11)++b
(26)
We therefore see that the frequency distribution P(S = 8n)) is bounded by two straight lines (power laws with power 1) with different coefficients, which appear two parallel lines in log-log scale, as we observe in Fig. A
24

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims accurately match our results and supplementary experiments. We provide list of claims in the introduction, section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of every hypothesis suggested in this paper. We also provide dedicated subsections to discuss limitations in some sections (sections appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed explanations of how data was gathered from open-source models. We also provide the full code to reproduce our results, see answer to next question.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide a link to an anonymized GitHub repository (here) containing all the code used to produce the results of this paper, together with links to external open source code used to train the models and detailed instructions how to reproduce each experiment. Since most experiments required scraping large amounts of game state data from numerous games, it is infeasible to provide the full raw dataset; to circumvent this, we provide both the code used to generate the raw game state data, as well as the full dataset curated from it through statistical analysis. The latter smaller dataset is available in the repository and can be used with the provided code and instructions to reproduce the figures in this paper. We will gladly provide the full raw state dataset, or parts of it, upon request.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (kttpss//nips.cem
poplic/guides.. SonoeddSumissiioPoooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all details needed understand our results. Models were trained with the hyperparameters specified in cited papers (e.g. (5). and all deviations from these hyperparameters are stated the paper, see section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
27

---

Justification: Error bars are plotted where applicable, namely in figures results are based on aggregating frequency distributions of game states played by numerous agents, and we dedicate appendix [C. to report and discuss the statistical significance of these distributions with respect to model architecture and training environment.
Our main
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative eтror rates).
If eгтor bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information on the resources needed to run all new experiments in this paper in the supplementary code, available in the GitHub repository. We use open source AlphaZero models trained by [5] who provided information on the resources used to train them.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We conform with the Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
28

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: No societal impact is applicable. While the field of neural scaling does have potential impact through guiding the creation of large models, this work does not directly lead to societal impact not discussed by previous literature.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., degloyment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: no risk of misuse is applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
29

---

Answer: [Yes]
Justification: We cite all data sources and specify their license in the supplementary material provided the GitHub repository.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are introduced in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
No crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
30

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No crowdsourcing human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: No LLMs were used in this research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM for what should or should not be described.
31


Reviews:
[{"review_id": "NX8JVGgWxn", "reviewer": "Reviewer_c7VB", "summary": "The authors study how neural‐scaling phenomena occur in AlphaZero agents on four board games (Connect Four, Pentago, Oware, Checkers). They show that the frequency of board states follows Zipf's law, which arises from the tree structure of legal moves in these games.\nThe agents reduce value-head loss on states in descending frequency order, which aligns with the “quantization” theory of LLM scaling.\nTwo of the games (Oware, Checkers) exhibit inverse scaling, beyond a size threshold the Elo falls because high-capacity nets overfit to strategically trivial late-game positions, which gives an anomalous bump in the Zipf curve.", "questions": "- How large is \"exceedingly large amounts of of data\" to visualise Zipf curve skewness in checkers?\n- What does \"strongly fixed labels\" mean at the end of page 8?\n- Why is it surprising that the large models have better loss on late game states? Wouldn't we expect this given late game states have lower variance?\n\nMinor:\nPage 2 Line 72: change end of equation (2) to be a comma, and Where to where.", "limitations": "This is discussed in the paper clearly, in page 9.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Stengths:\n- This paper highlights an interesting pattern for four games with branching states, and draws the connection with Zipf's law.\n- The paper is in general well-written and clear.\n- There is a good range of empirical analyses on the four games.\n\nWeaknesses:\n- The authors make an assertion \"It is easy to show that an ideal game, with a constant branching factor and a tree structure free of loops, would produce a Zipf-like distribution\", I think it would improve the paper to actually show this, e.g. in an Appendix.\n- Some of the discussion of results does not quite align. For example, it is claimed that ``larger agents achieve better loss'' in the caption of Figure 3. However, we see that the yellow curve, which has the highest parameters, is actually in the middle of A (train set) and not the lowest for B (ground truth).\n- The connection between RL games and LLMs seem a bit tenous. E.g. the worse performance for larger models is just part of the convergence issues of nonlinear function approximators (c.f. deadly triad), there is no need to mention LLM training (page 5).\n- The validation of this theory is on fairly small 2-D games, it is uncertain how this would extend for more complicated games or those having a stochastic transition.\n- The error bars only appear occasionally, e.g for the time taken in Figure 3C. How might these look for 3A or B?", "was_revised": true, "final_justification": "I have increased my score to 4, I think this paper explores interesting aspects of games and provides useful analysis, therefore can be accepted at the conference. I still have reservations that the exploration is only done on small 2D deterministic games, so the significance may be limited.", "rebuttal": "Thank you for your review. We are glad that you find our paper interesting and well-written, with a good range of empirical analyses. We address the points you brought up in order:\n\n\n- We thank you and reviewer cnDZ for this suggestion, we agree that showing explicitly why this claim is true would improve the paper. We added a new appendix section verifying our claim, appended at the end of this comment.\n\n- Thank you for pointing this out, we now added a footnote to the paper clarifying why this happens. The reason why the largest agent seems a bit worse than the second-largest agent on ground-truth Connect Four is due to a combination of noise and approaching the optimal-play limit.\nThe agents we use were taken from Neumann & Gros's work [1]. If you look at their figure 2, you will see that the largest Connect Four agents all have roughly the same performance, because they are already close to the level of an optimal player. For completeness, we decided not to exclude the largest agents in our analysis, even though their performance already stagnates at the largest model sizes.\nOne would therefore expect that the largest agents have similar loss values, which is the case. The small difference in loss between the largest agents is likely due to noise.\nTo clarify this issue, we added the following footnote to figure 3:\n\"The largest nets approach the optimal play limit [5], causing loss to stagnate as model size increases to the largest values.\"\n\n- We mention the quantization model of Michaud et al. [2] in page 5 since our paper discusses its potential application to AlphaZero training. We mentioned LLMs in page 5 even though the quantization model applies more generally, since Michaud et al. assume Zipf's law originates from the modality, i.e. natural language. However, we now changed the wording of the paragraph header (\"Comparison to LLMs\" --> \"Comparison to the quantization model\") and changed the text to dicuss this model in general, rather than LLMs.\nRegarding your other point on worse performance of larger models, discussed in section 7.3: this is indeed an RL phenomenon not seen in LLMs, and we state that explicitly in the text.\n\n- Indeed, similar to prior works (Jones [3], Neumann & Gros [1]), our paper explores deterministic 2D games that are easier to tackle than games like chess with a larger state space. While additional experiments on a wider range of problems could strengthen our results, we feel the paper is already packed with results in its current form, and leave such experiments for future work.\n\n- Figures 3A and 3B are already a bit cluttered, which is why we did not add error margins that could reduce visibility. However, we do see their necessity, especially since they can help clarify the issue you brought up earlier with figure 3. We will add error margins to figures 3A and 3B in the camera-ready version.\n\n\n### Questions:\n\n- We say \"exceedingly large amounts of data\" since this is a log-log plot, and extending it requires increasing the dataset size by orders of magnitude. We cannot know inadvance the exact dataset size that's sufficient to visualize the checkers distribution skewness. The required dataset size should be larger by some unknown number of orders of magnitude. Notice that although figure 4C goes up to rank $10^7$, the dataset used to generate it is much larger. First, because we cut off the majority of states from the graph (roughly 90% of states) since they only appeared once or twice during training. Second, because each state with frequency $n$ appears in the dataset $n$ times.\n\n- \"Strongly fixed labels\" means labels that change very little, if at all, during training. End-game states tend to have such labels, since by that point in the game the outcome is almost determined and even a random agent is almost guaranteed to produce the same outcome as a perfect player. We added a new appendix section expanding on this, which you can find at the end of our reply to reviewer hdGQ.\n\n- What's surprising is that larger agents have a clear preference to reducing loss on end-game states, considering that they have worse performance on early-game states. As you suggest, one would expect all agents to have lower loss on end-game states due to low variance, but that applies to small agents as well.\n\n- Thank you for noticing the typo, we fixed it.\n\n\nOther than the new addition below, we encourage you to look at other new sections we added to the paper, which you can find in our replies to reviewers hdGQ, a847 and Vr29.\nWe hope we have addressed all your concerns. If we did, we hope you would consider updating your score. Otherwise, we would be glad to answer any remaining questions.\n\n### References:\n\n[1] Neumann and Gros. \"Scaling laws for a multi-agent reinforcement learning model.\"  (2022).\n\n[2] Michaud et al. \"The quantization model of neural scaling.\" (2023).\n\n[3] Jones \"Scaling scaling laws with board games.\" (2021).\n\n\n\n\n\n\n# New appendix section: Distribution of ideal games\n\nIn section 5 we claim that random games played in a toy-model setting will result in a frequency distribution shaped as a series of plateaus, centered around a power law of exponent $\\alpha = 1$. We verify this claim here.\n\n## Frequency distribution function\nConsider a random game setting following the rules presented in section 5, namely:\n\n- Each game lasts $K$ turns.\n- Each turn $t$, a move $a_t$ is sampled uniformly from $b$ options.\n- Each board position $s$ can only be reached by a single sequence of moves $\\{ a_1, ... a_t\\}_s$.\n\nThe position at turn $t$ is effectively sampled uniformly from $b^t$ possible positions, since the $t$ moves $a_i$, $i \\in \\{1, \\dots, t\\}$  that lead to it are each sampled uniformly from $b$ options.\nSince $K$ turns are played each game, when sampling a board position randomly from all games played the frequency of sampling board state $s$ is:\n\n$P(s) = P(t(s)) \\cdot P( s | t=t(s))=\\frac{1}{K} \\cdot \\frac{1}{b^t} $ ,\n\nwhere $t(s)$ is the turn number of board state $s$ and $P(t(s))$ is the probability to sample from turn $t(s)$.\n\nSorting the board positions in descending order of frequency results in a series of plateaus, each corresponding to a turn number. The number of board states in plateau number $t$ is $b^t$. Plateau number $t$ starts after board position number $n_{start}(t)$, defined by:\n\n$n_{start}(t) = \\sum_{i=1}^{t-1} b^i = \\frac{b^{t} - b}{b - 1} $,\n\nsince each preceding plateau $i$ contains $b^i$ states.\nBoard state number $n$ belongs to plateau number $t$ for the integer $t\\in \\mathbb{N}$ that satisfies:\n\n$n_{start}(t) \\leq n < n_{start}(t+1) $,\n\nor:\n\n$\\frac{b^{t} - b}{b - 1} \\leq n < \\frac{b^{t+1} - b}{b - 1} $.\n\nShifting terms in the inequality and taking the log we get:\n\n$b^t - b \\leq (b-1)n <  b^{t+1} - b $\n\n$ b^t  \\leq (b-1)n +b <  b^{t+1}  $\n\n$t \\cdot log(b)  \\leq log [ (b-1)n +b ] <  (t+1)log(b)  $\n\n$t  \\leq \\frac{log [ (b-1)n +b ]}{log b}  <  t+1$\n    \nsince $t \\in \\mathbb{N}$, this is equivalent to using the floor operator:\n\n$    t(n) = \\left\\lfloor \\frac{\\log \\left[ (b-1)n +b \\right]}{\\log b} \\right\\rfloor $. (21)\n\nUsing Eq. 13 one sees that the frequency of the $n$'th most common position, $s_n$, is proportional to:\n\n$    P(s_n) = \\frac{1}{K} \\cdot \\frac{1}{b^{t(n)}} $,\n\nwhere $t(n)$ is defined in Eq. 21. This is the series of plateaus seen in Fig. 2A.\n\n\n## Distribution is bounded around power law\nLet us define $\\widetilde{t}(n) \\in \\mathbb{R}$ as $t(n)$ without the floor operator:\n\n$   \\widetilde{t}(n) =  \\frac{\\log \\left[ (b-1)n +b \\right]}{\\log b}  $ .\n\nSince $x - 1 < \\left\\lfloor x\\right\\rfloor \\leq x$, $P(s_n)$ is bounded by:\n\n$    \\frac{1}{K} \\frac{1}{b^{\\widetilde{t}(n)}} \\leq P(s_n) < \\frac{1}{K} \\frac{1}{b^{\\widetilde{t}(n)- 1}} $.\n\nExpanding the bounding functions we get:\n\n$    \\frac{1}{b^{\\widetilde{t}(n)}} = \\frac{1}{b^{\\frac{\\log \\left[ (b-1)n +b \\right]}{\\log b}} } = \\frac{1}{e^{\\log \\left[ (b-1)n +b \\right]}} = \\frac{1}{ (b-1)n +b} $,\n\nleaving us with:\n\n$    \\frac{1}{K} \\frac{1}{ (b-1)n +b} \\leq P(s_n) < \\frac{1}{K} \\frac{b}{ (b-1)n +b} $ .\n\nWe therefore see that the frequency distribution $P(s_n)$ is bounded by two straight lines (power laws with power 1) with different coefficients, which appear as two parallel lines in log-log scale, as we observe in Fig. 2A."}, {"review_id": "Y5tHue3nLx", "reviewer": "Reviewer_hdGQ", "summary": "This paper studies scaling laws in AlphaZero for board games like Checkers, Oware, etc.. The authors show that due to the tree-structure, the game states follow Zipf's law wherein certain positions appear much more frequently than others and that the model learns frequent states first. The work also explains \"inverse scaling\" phenomena where larger models perform worse in games like Oware because they focus too much on frequent but unimportant late-game positions.", "questions": "1. What exactly is the surprising or unexpected finding here? Since Michaud et al.'s theory is modality-agnostic and requires only multi-task setting with Zipfian data, isn't finding Zipf's law in board games somewhat expected? It would be nice to have a much clearer distinction between: (a) novel empirical discoveries, (b) expected theory confirmations, and (c) new theoretical insights.\n\n2. I am also curious if there are any games or text-based environments where “task quanta” can be defined much more clearly. \n\n3. Another curious question I had is related to the last line of Conclusion section: \n\n> “Our results hint at a possibility for improved RL algorithms using a curriculum informed by the frequency distribution, similar to recent attempts in supervised learning [31].”\n\nCould the preference of large models for late-game states be due to confounding factors other than frequency? For e.g., late-game states might have inherently lower variance in their true values (since fewer future moves remain) or more stable training targets (since outcomes are nearly determined), making them objectively easier optimization targets regardless of how often they appear? Have you controlled for such intrinsic difficulty differences to isolate the frequency effect?\n\n4. Typo: Line 114 “distirbution” → “distribution”", "limitations": "Authors have adequately discussed limitations of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is well-written with clear figures. \n2. The authors have done a comprehensive analysis, exploring multiple aspects like temperature effects, loss scaling patterns, and game-specific variations.\n3. I particularly find the inverse-scaling analysis to be very interesting and the overall analysis carried out in Section 7.\n4. I also appreciate that authors have released their code for reproducibility of their experiments.\n\n## Weaknesses\n\n1. To me, overall it feels like:\n- Jones (2021): Already showed scaling laws exist in board games\n- Michaud et al. (2023): Already provided general theory connecting Zipf's law to scaling laws\n- This paper: Applies Michaud's theory to Jones's domain. \n\nTherefore, the novelty feels limited. The authors essentially took two existing pieces (Jones's empirical observation + Michaud's theory) and combined them. But since Michaud's theory was already modality-agnostic, this feels more like validation than discovery.\n\n2. While the empirical observations are interesting, it's unclear how these findings can improve RL algorithms in general. The empirical validation is very narrow - only discrete, tree-structured board games. There's no evidence this applies to partial observability, or other RL environments. \n\n3. The inverse scaling explanation, while interesting, remains largely speculative despite the experiments in Section 7.3. I feel in order to make this a strong accept, I would want more in-depth analysis of “selective forgetting” explanation provided on lines 275-284.", "was_revised": true, "final_justification": "I had initially given 4: borderline accept. In their rebuttal, the authors were able to answer my queries and also further expanded on the inverse scaling mechanism. I still have slight apprehensions about the novelty of this work, but I think the results presented so far in this paper are useful and interesting. Therefore, I now recommend this paper as an \"5: Accept\" to this conference.", "rebuttal": "Thank you for your review. We appreciate that you find our paper clear and well-written with a comprehensive analysis of the problem. We address the points you brought up in order:\n\n1) While we do build on the works of Jones [1] and Michaud et al. [2], we disagree that our paper only amounts to validation. Michaud's quantization model is only modality-agnostic within supervised learning, and does not apply to RL directly (our paper would have been much shorter if it did). The various experiments we perform probe how far does the quantization model apply to AlphaZero training, and where this model fails to apply, e.g. going from loss scaling to Elo scaling.\nAdditionally, section 7 (inverse scaling) does not deal with the quantization model, though it is inspired by it.\n\n2) We are glad you find our observations interesting. The scope of our experiments is indeed limited to fully-observable board games, similar to Jones [1]. One could probably expand this analysis by using algorithms tailored to games with hidden information and stochastic dynamics [3,4]. However, we feel that the amount of experiments already makes this paper quite packed in its current form.\nSecondly, we note that RL training of autoregressive LLMs, a highly active research area, can also be seen as a discreet, tree-structured game with open information, albeit a non-MARL setting.\n\n3) We are glad you liked our explanation on inverse scaling. Following your suggestion, we added a new appendix section explaining in detail the selective forgetting mechanism we suggest in section 7. We appended the new section to the end of this comment.\n\n### Questions:\n\n1) The surprising findings are the loss scaling (section 6) and inverse scaling results (section 7), which already amount to half the paper. Finding Zipf's law in AlphaZero board games is novel, but not surprising, given the results of Blasius & Tönjes [5], as we clearly state in section 5.\n\n2) That's an interesting point. It's impossible to guess in advance what game will have clearly-identifieable learned task quanta, and as we discuss in the paper, finding even a few such quanta in games like chess is quite hard. We believe the best candidate would be some artificial game built similarly to Michaud et al.'s artificial \"multitask sparse parity\" dataset. Perhaps a game that involves solving instances of the multitask sparse parity problem in order to gain advantage in the game.\n\n3) We believe part of the reason why large models overfit on late-game states is exactly what you suggest, i.e. that these states have lower variance in their outcome. That is the point we were trying to make in lines 275-284. We hope the new appendix section added below will answer your question.\n\n4) Thanks for noticing, we fixed the typo.\n\nWe hope we have addressed all your concerns, and encourage you to look at other new sections we added to the paper found in our replies to reviewers c7VB, a847 and Vr29. If we did, we hope you would consider updating your score. Otherwise, we would be glad to answer any remaining questions.\n\n### References\n\n[1] Jones \"Scaling scaling laws with board games.\" (2021).\n\n[2] Michaud et al. \"The quantization model of neural scaling.\" (2023).\n\n[3] Perolat et al. \"Mastering the game of Stratego with model-free multiagent reinforcement learning.\" (2022).\n\n[4] Cui et al. \"Adversarial diversity in hanabi.\" (2023).\n\n[5] Blasius and Tönjes. \"Zipf’s law in the popularity distribution of chess openings.\" (2009).\n\n\n\n# New appendix section: Inverse scaling mechanism\n\nIn section 7.3, we mention a hypothetical mechanism that could cause the observed inverse scaling phenomenon. Here we discuss this mechanism in more detail.\n\nWe make the following main assumptions:\n- Agents fit states in descending order of frequency. An agent with a neural network of size $N$ can perfectly fit the values of $n=N/c$ states, where $c$ is the neural capacity needed to fit a single state (Michaud et al. 2023).\n- As the agent improves during training, game outcomes (i.e. state values) change. The values of early-game states change significantly during training, while the values of late-game states change only slightly, if at all.\n- The shift of state values causes neural plasticity loss due to dormant neurons (Sokar et al. 2023).\n\n\nThe first assumption is based on the quantization model of neural scaling (Michaud et al. 2023). \n\nThe second assumption is based on the fact that, in most board games, player skill greatly influences the value of early-game states. In the games discussed in this paper, a match between random agents will usually end in a draw ($v=0$) while a match between optimal agents will end in a first-player victory ($v=\\pm1$), except for checkers. In contrast, late game states that are played a few steps before the game ends have values that are largely independent of player skill, due to power imbalance. Very late states have fixed values if MCTS can probe the entire game tree stemming from them.\n\nThe third assumption is based on the work of Sokar et al. (2023), who showed that changing the labels of a dataset during training causes some neuron activations to die out. These so-called \"dormant neurons\" effectively reduce the neural capacity of the model.\n\nAssume we train a small agent with $10^m$ neurons and a large agent with $10^k$ neurons, $k>m$. In our case, the small agent roughly fits $10^m/c \\approx n_{early}$ states, where $n_{early}$ are the number of early-game states with higher frequency than late game states. In Oware and Checkers, $n_{early}$ falls in the range $[100,200]$, see Fig. 5. The large agent fits $10^k/c \\gg n_{early}$ and mostly memorizes the values of late-game states.\n\nFor simplicity, let us treat the change of state values as a single, abrupt event during training. After this event, both models have high loss on early-game states, but maintain the same loss as before on late-game states, since the latter group kept their value labels. Due to plasticity loss, both models can now fit fewer states than before. The small agent now has high loss, since it could only fit the first $n_{early}$ states before values changed, meaning most of the states it memorized now have new labels. The large model, on the other hand, maintains a lower loss, since it could fit orders of magnitude more states than the first $n_{early}$ states, most of which are late-game states. \n\nIt is likely that the small agent, which now has loss close to that of a random agent, will continue to train and re-learn as many states as it can, starting from the most frequent ones. Eventually it will regain low loss on most or all of the first $n_{early}$ states. In contrast, the large agent maintains low loss, and could be close to a local minimum of the loss landscape that fits late-game states well but assigns wrong labels to early-game states. To leave this minimum and re-learn the first $n_{early}$ states, the large agent will have to sacrifice knowledge of some late-game states, since it lost a fraction of its neural capacity. In this scenario, the large model could remain stuck in the local loss minimum and never regain performance on early-game states.\n\nIn reality, value labels shift gradually rather than abruptly, meaning that loss will increase gradually during training rather than spike abruptly. We observe this loss degradation in Fig. 7, where large agent value loss on early-game states starts to degrade mid-training."}, {"review_id": "Z9K8Ni2amR", "reviewer": "Reviewer_a847", "summary": "This paper applies the quantization model of language model scaling to analyze the scaling behavior of AlphaZero in board games. The authors first empirically show that AlphaZero-generated data follows Zipf’s law and offer intuitive explanations for how this distribution arises from the tree-like structure of board game state spaces. Building on this, they investigate the scaling properties of AlphaZero across the state spectrum and reveal a surprising pattern: the models appear to optimize state prediction loss in descending order of state frequency. The paper also observes instances of inverse scaling in certain games and provides empirical insights linking these behaviours to the unique tree structures of those games.", "questions": "I have a few questions on the empirical design and analyses that need further clarifications:\n* Regarding the loss vs. state rank plots used throughout the paper — since the state spectrum can vary across training runs or over the course of training, the specific state corresponding to a given rank may also change. Could this variation confound the analyses when comparing different curves? In particular, in Figure 7, while we observe an increase in loss for \"early turns\" as training progresses, could this trend simply be due to the change of the underlying set of states being measured, rather than a true degradation in performance on those states?\n* In the scaling law analyses, the value loss is used as the primary metric. Is there a specific reason for choosing value loss? How would the results differ if policy loss or Elo score were used instead?\n* Regarding the inverse scaling, do you think some double decent phenomenon could be observed if we further increase model sizes?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n* The paper provides a very first empirical study on the scaling behaviours of AlphaZero, a problem that may become increasingly important as similar algorithms may be applied in the training of frontier models\n* The paper offers many insightful empirical analyses that may shed light on how AlphaZero models learn and scale, such as how the Zipf’s distribution emerges and how it connects to inverse scaling.\n* The empirical analyses in the paper are extensive and sufficiently support the major claims in the paper.\n\nWeaknesses\n* The paper lacks a dedicated section that provides readers with sufficient background on the AlphaZero algorithm and the various board game configurations used in the study.\n* The empirical analyses in the paper are somewhat limited to board game environments and specific design choices. However, I think this is acceptable for an initial exploration aimed at understanding the scaling behavior of AlphaZero. I detail some questions on the specific empirical design in the followup section.", "was_revised": true, "final_justification": "Overall, I think the paper provides a meaningful contribution, and the author response clarifies my questions regarding experiment setups. I will keep my positive rating for an acceptance.", "rebuttal": "Thank you for your review. We are glad that you see the relevance of this paper to frontier model RL training, and that you find our analyses extensive and sufficient to support our claims.\nWe address the points you brought up in order: \n\n- We do have a short appendix section (appendix A) describing the basics of the AlphaZero algorithm, but we agree the paper does not provide an overview of the board games we test. Following your suggestion, we now added a new appendix section describing all four board games. We appended the new section to the end of this comment.\n\n- Thank you for acknowledging the scope of our experiments is sufficient for the paper's purpose. We answer your questions below.\n\n### Questions\n\n- This is a great question. We answer regarding figure 7 and in general:\n\t- About figure 7, we specifically used the same state distribution for all training-time curves in order to avoid the confounding factor of a shifting rank distribution. We aggregated the state distribution over the entire training period (excluding early training), and then calculated the loss of these states for each agent checkpoint.\n\t- In general, the state distribution does change between training runs and during training, which poses a problem. To reduce the variance of state rank ordering, we aggregated states only after 30% of training time has passed, to reduce noise from early-training exploration. When possible, we aggregated over different random seeds as well.\n\n- We plot loss rather than Elo score since there is no straightforward way to assign an Elo score to a single state. Between value loss and policy loss, we decided that value loss is a more useful metric for two reasons:\n\t1) Policy loss does not necessarily tend to zero as policy improves. For example, two maximally different policies can both be optimal when one than one action is optimal. For example, if $a=1$ and $a=3$ are both optimal actions, both policy $p=(1,0,0)$ and policy $p=(0,0,1)$ will be optimal, as well as any combination of the two. When using an optimal policy as reference for calculating loss, higher loss doesn't mean that the policy is far from optimal play.\n\t2) The value has more influence on performance than the policy in AlphaZero. This is a bit counter-intuitive for people more familiar with actor-critic models. Appendix section D explains this point in detail.\n\n- That could be possible, although we haven't found any evidence to support it. We suggest in section 7 that AlphaZero inverse scaling might be caused by loss of neural capacity through dormant neurons; at the infinite size limit, the model should have enough capacity to re-learn the values of early-game states after the labels shift.\n\nWe would also encourage you to check out other new additions to the paper, found in our replies to reviewers c7VB, hdGQ and Vr29.\n\n# New appendix section: Board game descriptions\n\nWe describe here the rules of all four games examined in this paper. All games are two-player, zero-sum, open information games, where players alternate turns between them moving pieces on a board.\n\n### Connect Four\n\nThe players alternate placing a disk of their color into one of seven vertical columns on a 6x7 grid, and the disk falls to the lowest unoccupied position in that column. A player wins by arranging four of their disks in a row, either vertically, horizontally, or diagonally. If all cells are filled without such an alignment, the game is drawn.\n\n### Pentago\n\nPlayers alternate placing one marble of their color on any empty cell of a 6x6 board, which is divided into four 3x3 rotating quadrants. After placing, the player must rotate one quadrant by 90° either clockwise or counter-clockwise. A player wins immediately upon obtaining five marbles in a row (horizontal, vertical, or diagonal) either before or after rotation. If the board becomes full without five in a row, the game is drawn.\n\n### Oware\n\nThis game is played on a board with two rows of six houses and optionally store pits. Each of the twelve small houses starts with four seeds. Each player controls the six houses on their side. On their turn, a player selects one of their houses with seeds, removes all seeds, and sows them counter-clockwise, placing one seed per subsequent house (excluding stores and skipping the original house when it's revisited). If the last seed lands in an opponent's house and brings its count to exactly two or three, those seeds are captured into the player's store. Capture continues backward from that house as long as the immediately preceding opponent's houses also contain exactly two or three seeds. Capturing all of an opponent's seeds is forbidden; in that case no capture is made. The game ends when a player captures 25 or more seeds, or both players capture 24 each (draw), or if no legal move exists; remaining seeds are then collected by their owners. In our analyses, a draw is also declared if the game lasts 1000 turns.\n\n### Checkers\n\nThe players each control 12 pieces on opposite dark squares of an 8x8 chess board.  Normal pieces move diagonally forward one square to an unoccupied dark square. If an opponent's piece is diagonally adjacent and the following square is empty, the adjacent piece may be jumped and removed; multiple jumps are allowed in sequence if available. When a piece reaches the farthest row on the opponent's side, it is crowned as a king and gains the ability to move and jump both forward and backward. Play continues until one player has no legal move (draw) or a player lost all their pieces."}, {"review_id": "p3eXRX1npC", "reviewer": "Reviewer_cnDZ", "summary": "- Observes that AlphaZero agents produce power law distributions for game states\n- Compares this result with the quantization hypothesis proposed by Michaud et al. 2023\n- Provide evidence from Pentago and Connect Four supporting a prior theory by Blasius & Tönjes (2009)\n- Shows that models exhibit lower loss on more frequent states\n- Studies inverse scaling and provides evidence that it can be caused by end-game states that are disproportionately prevalent", "questions": "Please see \"Strengths and Weaknesses\"", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Abstract\n\n- “Neural scaling laws are observed in a range of domains, to date with no clear understanding of why they occur.” This seems objectively false. I would strongly urge the authors to strike this sentence or explain in more detail why they think the numerous theoretical works on neural scaling laws are inadequate. Also, some somewhat recent theoretical work on neural scaling laws goes uncited: https://arxiv.org/abs/2405.15074 and https://arxiv.org/abs/2502.17578; these papers might have more theoretical citations worth including.\n\n- Related to the above point, perhaps the authors are trying to suggest that the origin of neural scaling laws in RL are much less well understood theoretically than in other toy models? If so, that seems like a more defensible claim.\n\n- Suggestion: The first 3-4 sentences are a bit rough. It might be simpler to just state, “In this paper we examine power-law scaling in AlphaZero, a reinforcement learning algorithm, using a theory of\nlanguage-model scaling. Specifically, we leverage the so-called quanta scaling theory, which posits “\n\n- I’m not sure whether the “quanta scaling theory” should be called a theory. Is it a hypothesis or a theory? I learn more towards the former but I’m open to being persuaded.\n\n## Introduction\n\n## Background\n\n- Figure 1: Regarding the claim “Board-state frequency follows a power law…”, one detail I’m confused by is that surely this is policy-dependent? Like, if my Connect Four policy is to always play in the left-most available column, one would find a different state-frequency distribution. Is this claim indeed policy dependent? If not, why not? If yes, then under what conditions does the policy yield a power law board-state frequency?\n\n- Line 70: I know that Michaud’s paper is not your paper, but the condition “ If the loss on each task is reduced by a fixed amount ∆L after it is learned” seems like a strong condition. Similarly, “It is assumed that a fixed neural capacity c is needed to fit each of the independent task quanta” seems like a strong assumption again. \n\n## Methods\n\n- Line 121 “we present similar size-scaling curves for Oware and Checkers in Fig. 4A”. As best as I can tell, Figure 4A does not show power laws. Perhaps Fig 4B and 4C were intended?\n\n- “unlike the chess analysis of Blasius & Tönjes (2009)”. Could the authors please clarify in what way Blasius & Tönjes assumed the history of previous moves?\n\n## Zipf’s Law in AlphaZero\n\n- Line 143 “This result strengthens the assumption that Zipf’s law is not caused by human decision-making, but is rather derived from the game rules.” This seems like an incorrect inference to draw. Maybe the power laws are attributable to AlphaZero as an algorithm. Or maybe the power laws are attributable to being trained for optimal play. At this point in the paper, I don’t think any evidence has been provided that justifies a statement like “Zipf’s law is caused by the game rules.”\n\n- Line 142-147: After reading citation 27 (Blasius and Tonjes 2009), it becomes clear that (1) the cause of the power law scaling is clear, and that (2) its cause is not attributable to humans but to the structure of hierarchical fragmentation: “We propose a simple stochastic process that is able to capture the observed playing statistics and show that the Zipf law arises from the self-similar nature of the game tree of chess. Thus, in the case of hierarchical fragmentation the scaling is truly universal and independent of a particular generating mechanism.” Consequently, I feel that this paragraph does not accurately represent prior work and paints a misleading picture of a “puzzle” which has already been solved for over a decade.\n\n- Lines 148 to 156 claim to offer *intuition* of the source of Zipf’s law, but I can find no intuition in either that paragraph or Figure 2. If the authors aim to offer intuition, please improve this exposition; otherwise, strike this language. Statements like “ is easy to show that an ideal game, with a constant branching\nfactor and a tree structure free of loops, would produce a Zipf-like distribution.” are not offering intuition; they’re just restating the claim.\n\n- I’m not sure I understand how Figure 2A supports the claim of Zipf’s law. Rather, the plateaus suggest that the frequency distribution does not depend on the element rank n, but on whether n falls into some bucket. To maybe express this idea differently, when we say something follows Zipf’s law, how precisely do we mean?\n\n- I’m not sure I understand Figure 2B. The power laws that are displayed for Connect Four and Pentago do not go through the plateaus, as shown in Figure 2A. Rather, the power laws appear to be an envelope over the plateaus.\n\n## Analyzing how loss scales with Zipf’s Law\n\n- Figure 3 C: “The time needed to evaluate the same states using alpha-beta pruning [37] drops exponentially”. I would not classify this as exponential. It looks pretty clearly linear in log-log space, at least from below 1e2 to 1e5. Same with line 190. I would not call Figure 3 C exponentially decreasing.\n\n- Line 171 “we find that AlphaZero models reduce their loss on game states in descending order of frequency”. -> This is unusual phrasing. I think it is clear that models show lower loss on higher ranked states. But “reduce their loss” is ambiguously and possibly suggestive of learning dynamics. I think it’d be more clear to say “Models exhibit lower loss on more frequent states” or something like this.\n\n- Line 174: “Comparison to LLMs” I think that this phrasing is misleading. The quantization hypothesis is a hypothesis that to my knowledge has never been confirmed/verified in language models. The paragraph is fine, but the leading phrase should say “Comparison to Quantization Hypothesis”\n\n- Line 184 “It is surprising that absolute value loss (Fig. 3B) tends to increase with state rank,” -> I don’t find this surprising at all. Using an example of classification with imbalanced classes, we expect models to perform better on majority classes rather than minority classes because doing so minimizes the loss more; for an extreme example, if a class has exactly 0 samples, then there’s no reason for the model to learn about this class. Returning to this AlphaZero setting, the first few states appear in (nearly) every game, whereas the 1e5 state occurs much much much more rarely.\n\n## Connecting Inverse Scaling to the Game Structure\n\n- Line 264: Please stop calling the quantization hypothesis “LLM scaling theory”", "was_revised": true, "final_justification": "I think I overanchored on some details in the paper that either I misunderstood or perhaps were expressed suboptimally. Based on my discussion with the authors, I think the paper is quite solid.", "rebuttal": "Thank you for your thorough review. We address the points you brought up in order:\n\n\n### Abstract\n\n1) (first two points) Thank you for the recent citations, we have added them into the text.\nIndeed, as you observed, our intention in the sentence “Neural scaling laws are observed in a range of domains, to date with no clear understanding of why they occur” was that no model of neural scaling can explain this scaling in all domains, including the RL domain. There are indeed good models for explaining power law scaling in supervised learning settings. To avoid confusion, we changed the wording of that sentence, from \"no clear understanding\" to \"no universal understanding\".\n\n3) Thank you for the suggestion. Since the other reviewers did not comment on the abstract wording, we feel that the current phrasing is a good fit (including the change mentioned above). If you still feel strongly about the wording we are open to finding a good replacement.\n\n4) We see the problem with the wording \"theory\". We now refer to it throughout the paper as the \"quantization model\", as is done in Michaud et al. [1]\n\n### Background\n\n1) We discuss this point thoroughly in section 5 (\"Zipf’s law in AlphaZero\"). It seems  that you have answered your questions yourself in your later comments on section 5 and Blasius & Tönjes [2], but if something is still unclear we will gladly clarify it.\n\n2) As you point out, Michaud et al. [1] is not our paper. We do agree with you that the assumptions they make are somewhat strong, since they discuss a toy model.\n\n### Methods\n\n1) It seems our use of the word \"similar\" in line 121 lead to this confusion. We meant it in the sense that we calculate size-scaling curves in the same manner they were calculated by Neumann & Gros [3]. We removed the word \"similar\" to avoid confusion.\n\n2) Blasius & Tönjes [2] discuss the frequency of chess openings, where \"opening\" means a sequence, or history, of specific moves.\n\n### Zipf’s Law in AlphaZero\n\n1) It seems you answered this question in your next point. We state the justification of line 143 in the rest of the paragraph, which we feel is adequate, namely that seeing Zipf's law in AI games strengthens the assumption that Zipf's law is not unique to humans, but is rather a general phenomenon for (optimized) play.\n\n2) We explicitly state in the next paragraph (line 149) that Blasius & Tönjes already suggested (without proof) the connection of game trees to Zipf's law. Nowhere in the paper do we claim that we are the first to make that observation.\n\n3) We thank you and reviewer c7VB for pointing this out. We now added an appendix section verifying our statement on Zipf's law in ideal games (\"it is easy to show...\"). Since it doesn't fit in this comment, we urge you to look at our comment to reviewer c7VB where we posted the content of the new appendix.\n\n4) We never claim figure 2A is a Zipf's law, indeed calling it that would be wrong. It is a series of plateaus around a straight line, which is a \"power law\" with power 1. We use the term \"Zipf-like\" since this distribution is centered around a power law.\n\n5) Figure 2B shows the Frequency distributions of random games. Indeed, these are not clean Zipf's laws, and not plateaus, but rather a transition point between the two. The power law fit we added there fits the mid-section of the Connect-Four distribution, ignoring the noise at low ranks and the finite-size effects at high ranks. Notice the plateaus at high ranks are just an artifact of our limited calculation, and will disappear at the limit of infinite data (and never-ending games). In contrast, the plateaus of figure 2A are not artificial and will not disappear at this limit.\n\n### Analyzing how loss scales with Zipf's Law\n\n1) We agree with you on this point, it looks like a power law would be the best fit to this trend. We meant \"drops exponentially\" as in \"drops on an exponential scale\", we now changed the wording to avoid confusion. \n\n2) Thank you for the suggestion, we agree it is better worded. We incorporated it into the text.\n\n3) We also incorporated this naming suggestion.\n\n4) You touch here exactly on the point we make in this section: our results _are_ surprising _despite_ the fact that they would be trivial for a simple classification model. In your example, the model learns static labels through supervised learning. In our paper, labels (=values) are not static. The hardest labels to model correctly through self play are the labels of the most frequent states, as we show in figure 4C.\nTo use your example, the first few steps do appear in nearly every game, but at the start of training they all have wrong labels, i.e. they have a label of roughly 0 (draw) instead of the ground-truth label, which can be 1 or -1. That is because two inexperienced players will usually draw, compared to optimal players who will always end the game with a first player win (excluding checkers). Compare this to end-game states, where the label is almost guaranteed to be accurate since MCTS will probe all possible future moves.\nTo summarize, “It is surprising that absolute value loss tends to increase with state rank” since the model has to improve, then unlearn and re-learn the labels of the most frequent (early-game) states, before absolute loss will decrease.\n\n### Connecting Inverse Scaling to the Game Structure\n\n1) We acknowledge this in our reply to the last point in \"Abstract\".\n\nWe hope we have addressed all your concerns, and encourage you to look at other new sections we added to the paper. Other than the one found in our reply to reviewer c7VB following your question, we also posted new material in our replies to reviewers hdGQ, a847 and Vr29. If we did address all your concerns, we hope you would consider updating your score. Otherwise, we would be glad to answer any remaining questions.\n\n\n### References:\n\n[1] Michaud et al. \"The quantization model of neural scaling.\" (2023).\n\n[2] Blasius and Tönjes. \"Zipf’s law in the popularity distribution of chess openings.\" \n\n[3] Neumann and Gros. \"Scaling laws for a multi-agent reinforcement learning model.\"  (2022)."}, {"review_id": "f3DJvyJHX1", "reviewer": "Reviewer_Vr29", "summary": "This paper presents an empirical study of neural scaling in AlphaZero across four board games, showing that the frequency of self-play states follows a Zipf power law and that model performance (Elo and value-head loss) scales predictably with network size in accordance with a quantization framework. By varying MCTS temperature, the authors demonstrate a direct link between the Zipf exponent of state distributions and the size-scaling exponent of performance. They further reveal an inverse scaling phenomenon in certain games and show that larger models learn rarer states earlier. Together, these findings extend scaling-law theory from supervised learning to reinforcement learning and suggest that distribution-aware training curricula could improve RL scalability.", "questions": "1. How robust are your Zipf distributions and scaling links to changes in MCTS rollout count or exploration parameters?\n2. Could you discuss the key challenges you anticipate and how you might adapt your analysis to handle games with imperfect information?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\nI really enjoyed how thorough the authors were—testing across very different board games, scaling up network sizes, and tweaking MCTS temperature to show that Zipf’s law in state frequencies isn’t just a curiosity but actually drives the familiar power‐law scaling (and even the surprising inverse scaling) we see in performance. I also found the diagnosis of overfitting to those “end‐game” bumps especially interesting. Besides, the paper is well-written and easy to follow.\n\n**Weaknesses**:\n1. My main concern is the insufficient discussion about the related work. The paper cites Jones et al.’s “Scaling scaling laws with board games” only in the bibliography and never contextualizes how its Zipf-based findings compare conceptually or empirically.\n2. While the authors note that loss is only a proxy for true playing strength, there is no formal model or empirical ablation connecting loss-rank curves to the multi-agent Elo metric.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We are glad you enjoyed how thorough the paper is, and found our results interesting. We address the points you brought up in order:\n\n- We would like to point out that we do cite Jones [1] in the main section, as well as the appendix. Our paper is a direct extension of Jones' work. That being said, Jones only discusses the scaling form of AlphaZero and not the origin of scaling. Neither do Neumann & Gros [2]. We therefore find both [1] and [2] to be worthy of discussion as background rather than related work. We agree with you that the paper would benefit from discussing this background more thoroughly, which is why we added a new part to appendix A that covers this. You can read the new addition at the end of this comment. Ideally we would put this in the \"Background\" section, but due to page limit constraints we refer the reader to the appendix, the same way we cover AlphaZero (Silver et al. [3]).\n\n- Indeed we do not present a model connecting loss and Elo, as we point out in the discussion. Elo is a multi-agent metric that is not easy to connect to single-agent attributes like loss, let alone to performance on individual states.\n\n\n### Questions\n\n- This is an interesting question, considering MCTS scaling is one of the main finds of Jones (2021).\nWe did not directly perform an ablation study on MCTS rollouts, but we did measure the effect of exploration on Zipf's law in appendix sections B.2 and C.\nIn appendix C we change the temperature and see that it significantly changes both the Zipf distribution and the scaling law, see figure 10.\nIn appendix B.2 we look at random games and show that the state frequency distribution shifts significantly between a uniform policy (exploration) and a skewed policy (exploitation).\n\n- It should be possible to perform a similar analysis on imperfect information games like Stratego [4], Diplomacy [5] or Hanabi [6], although an extensive analysis might be hard since each of these games is solved by a different algorithm. Other than focusing on AlphaZero, our paper is agnostic to perfect vs. imperfect information, except for the state definition. In an imperfect information setting it might be reasonable to count the frequency of the observable states rather than look at the full hidden state. \nIf one still wants to calculate value loss on the hidden states, and not the observable states, then one would have to account for irreducible loss since the state is effectively stochastic from the agent's point of view.\n\n\nWe hope we have addressed all your concerns, and encourage you to look at other new sections we added to the paper found in our replies to reviewers c7VB, a847  and hdGQ. If we did, we hope you would consider updating your score. Otherwise, we would be glad to answer any remaining questions.\n\n### References\n\n[1] Jones \"Scaling scaling laws with board games.\" (2021).\n\n[2] Neumann and Gros. \"Scaling laws for a multi-agent reinforcement learning model.\"  (2022).\n\n[3] Silver et al. \"A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.\" (2018).\n\n[4] Perolat et al. \"Mastering the game of Stratego with model-free multiagent reinforcement learning.\" (2022).\n\n[5] Bakhtin et al. \"Mastering the game of no-press diplomacy via human-regularized reinforcement learning and planning.\" (2022).\n\n[6] Cui et al. \"Adversarial diversity in hanabi.\" (2023).\n\n\n# New addition to appendix A: AlphaZero scaling laws\n\nAlphaZero scaling is one of the earliest examples of neural power-law scaling in RL [20,5]. Jones [20] showed that AlphaZero agents trained to play Hex improve their Elo score $r$ with the log of training compute $C$, $r \\propto log(C)$, with a coefficient that remains constant across board sizes. As they point out, this predictable scaling behavior allows one to plan the compute budget ahead of training to fit a desired performance level.\nNeumann \\& Gros [5] found similar compute scaling in agents trained on Connect Four and Pentago, as well as scaling with neural network size, and pointed out that these scaling curves are power laws of the Bradley-Terry playing strength $\\gamma$ [33], since $r \\propto log_{10}(\\gamma)$.\nSimilar to LLM scaling laws, these scaling laws of AlphaZero allow one to know in advance the optimal model size that would achieve maximal Elo score, given the available training compute budget. An interesting result from Jones shows that training compute can also be traded off predictably for inference compute [5]."}]

Author rebuttal:

Meta-review: The paper investigates neural scaling in AlphaZero agents across four board games, Pentago, Connect Four, Oware and Checkers. They observe that the frequency of board states follows Zipf's law due to the tree structure of legal moves. They make a connection to the quantization theory of LLM scaling, and they show agents reduce loss on states in descending frequency order. The analysis includes a good range of empirical studies, exploring aspects like temperature effects and loss scaling patterns. A particularly interesting finding is the inverse scaling observed in Oware and Checkers, where high-capacity nets overfit to strategically trivial late-game positions, leading to an anomalous bump in the Zipf curve and a drop in Elo score beyond a certain size threshold. The authors have released their code for reproducibility.

This is a strong paper due to its novel application of Zipf's law to game states, its empirical rigor, and its connection to broader theories of neural network scaling. The identification of inverse scaling in specific game contexts is a significant contribution, highlighting complexities in scaling laws for RL.

The reviewers have highlighted some important minor directions of improvement, and the authors have suggested many concrete changes, including multiple new appendices. I’m sure it goes without saying, but please implement these changes for the camera ready version.
Decision: Accept (spotlight)