Title: An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem
Venue: NeurIPS 2024
Accepted: yes
Keywords: science of deep learning

emergence

skills

scaling laws
Abstract: Deep learning models can exhibit what appears to be a sudden ability to solve a new problem as training time, training data, or model size increases, a phenomenon known as emergence. In this paper, we present a framework where each new ability (a skill) is represented as a basis function.  We solve a simple multi-linear model in this skill-basis, finding analytic expressions for the emergence of new skills, as well as for scaling laws of the loss with training time, data size, model size, and optimal compute.  We compare our detailed calculations to direct simulations of a two-layer neural network trained on multitask sparse parity, where the tasks in the dataset are distributed according to a power-law.  Our simple model captures, using a single fit parameter,  the sigmoidal emergence of multiple new skills as training time, data size or model size increases in the neural network.

An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem

Abstract

Deep learning models can exhibit what appears to be a sudden ability to solve a new problem as training time, training data, or model size increases, a phenomenon known as emergence. In this paper, we present a framework where each new ability (a skill) is represented as a basis function. We solve a simple multi-linear model in this skill-basis, finding analytic expressions for the emergence of new skills, as well as for scaling laws of the loss with training time, data size, model size, and optimal compute. We compare our detailed calculations to direct simulations of a two-layer neural network trained on multitask sparse parity, where the tasks in the dataset are distributed according to a power-law. Our simple model captures, using a single fit parameter, the sigmoidal emergence of multiple new skills as training time, data size or model size increases in the neural network.

1 Introduction

Emergence in large language models (LLMs) has attracted a lot of recent attention [1-4]. It motivates the costly drive to train ever larger models on ever larger datasets, in the hope that new skills will emerge. While the concept of emergence has been critiqued on the grounds that the sharpness of the transition to acquiring a new skill may be sensitive to the measure being used [5], the observation that important new skills are learned for larger models raises many challenging questions: when the skills emerge and what drives the emergence. These questions are complicated by difficulties in formally defining skills or capabilities [6], and by our general limited understanding of the internal representations of deep neural networks [7].

Another widely observed property of deep learning models is that the loss improves predictably as a power-law in the number of data points or the number of model parameters or simply in the amount of compute thrown at a problem. These neural scaling laws [8, 9] have been widely observed across different architectures and datasets [10-16]. While the scaling exponents can depend on these factors, the general phenomena of scaling appear to be remarkably robust. This raises many interesting questions such as: What causes the near-universal scaling behavior? How does the continuous scaling of the loss relate to the discontinuous emergence of new skills?

Achallenge in answering the questions raised by the phenomena of emergence and scaling laws arises from the enormous scale and expense of training cutting-edge modern LLMs, which are optimized for commercial applications, and not for answering scientific questions about how they work. One way that progress can be made is to study simpler dataset/architecture combinations that are more tractable. The current paper is inspired in part by recent work in this direction that proposed studying emergence in learning the sparse parity problem [17, 18], which is easy to define, but known to be

computationally hard. In particular, Michaud et al. [18] introduce the multiple unique sparse parity problem - where tasks are distributed in the data through a power-law distribution of frequencies - as a proxy for studying emergence and neural scaling in LLMs. For this data set, the authors empirically measure the scaling laws of a 2-layer multilayer perceptron (MLP) as a function of training steps ( T ), parameters ( N ), and training samples ( D ). Based on their quanta model of abrupt skill acquisition, they schematically derive neural scaling laws as a sum of emergences of new skills. However, no link was established between the neural network dynamics and the quanta model.

In this paper, we introduce an analytically tractable model by defining a basis of orthogonal functions for the multitask sparse parity problem. Each basis function corresponds to a skill that can be learned, and their respective frequencies are distributed following a power-law with exponent α + 1 . We then propose a simple multilinear expansion in these orthogonal functions that introduces a layered structure reminiscent of neural networks (NNs) and gives rise to the stage-like training dynamics [19]. With our simple model, we can analytically calculate full scaling laws, including pre-factors, as a function of data exponents α , T, D, N , and optimal compute C . Our simple model can, with just one parameter calibrated to the emergence of the first skill, predict the ordered emergence of multiple skills in a 2-layer MLP. We summarize our contributions as follows:

Skills as basis functions. We establish a framework for investigating emergence by representing skills as orthogonal functions that form a basis in function space (Section 2). We apply our methods to controlled experiments on the multitask sparse parity dataset.

Multilinear model. We propose an analytically tractable model that is expanded in the basis of skill functions, and is multilinear with respect to its parameters so that it possesses a layerwise structure (Section 3). The multilinear nature of the model produces non-linear dynamics, and the orthogonal basis decouples the dynamics of each skill.

Scaling laws. We derive scaling laws for our multilinear model, including the prefactor constants, which relate the model's performance to training time ( T ), dataset size ( D ), number of parameters ( N ), and optimal compute ( C = N × T ), see Section 4. We show that the scaling exponents for these factors are -α/ ( α +1) , -α/ ( α +1) , -α , -α/ ( α +2) , respectively, where α +1 is the exponent of the power-law input data.

Predicting emergence. We demonstrate that our multilinear model captures the skill emergence of an MLP with 2 layers for varying training time, dataset size, and number of trainable parameters. Our results show that the multilinear model, calibrated only on the first skill, can predict the emergence of subsequent skills in the 2-layer MLP, see Fig. 1 and Section 5. We obtain an equivalent result on the time emergence for a transformer architecture (Fig. 4).

10 3

10 4

T

0.0

0.2

0.4

0.6

0.8

1.0

Skill

strength

R

k/S

(a)

Time

emergence

10 3

10 4

D

0.0

0.2

0.4

0.6

0.8

1.0

Skill

strength

R

k/S

(b)

Data

emergence

10 0

10 1

N

0.0

0.2

0.4

0.6

0.8

1.0

Skill

strength

R

k/S

(c)

Parameter

emergence

k

=1

k

=2

k

=3

k

=4

k

=5

2 Setup

In this section, we define the multitask sparse parity problem under the mean-squared error (MSE) loss. We represent skills as orthogonal functions and measure their strength in a model by calculating the linear correlation between the model output and the skill basis functions. For a comprehensive list of notations, refer to the glossary in Appendix A. Our code is also available online. 1

Multitask sparse parity problem. In the sparse parity problem, n b skill bits are presented to the model. The target function is a parity function applied to a fixed subset of the input bits. The model must detect the relevant m < n b sparse bits and return the parity function on this subset ( M ( i, x ) , see Table 1). Michaud et al. [18] introduced the multitask sparse parity problem by introducing n s unique sparse parity variants - or skills - with different sparse bits (for a representation, see Table 1). Each skill is represented in the n s control bits as a one-hot string, and the model must solve the specific sparse parity task indicated by the control bits (for more details, see Appendix B.1).

The n s skills (random variable I ∈ { 1 , 2 , . . . , n s } ) follow a power law distribution P s , and the skill bits (random variable X ∈ { 0 , 1 } n b ) are uniformly distributed. Because P s and P b are independent, the input distribution P ( I, X ) follows a product of two distributions:

̸

Skill basis functions. We represent the k th skill as a function g k : { 0 , 1 } n s + n b →{-1 , 0 , 1 } that returns the parity ( {-1 , 1 } ) on the k th skill's sparse bits if i = k , but returns 0 if the control bit mismatches that of the k th skill ( i = k ):

where M : { 0 , 1 } n s + n b → { 0 , 1 } m is the map that selects the relevant sparse bits for the i th skill (Table 1) and M j ( i, x ) is the j th entry of M ( i, x ) . Note that different skill functions have 0 correlation as the supports of skills functions are mutually exclusive :

The target function. The target function is a sum over n s skill functions multiplied by a target scale S :

The target scale S is the norm of the target function ( E I,X [ f ∗ ( I, X ) f ∗ ( I, X )] = S 2 ). Note that the skill functions serve as 'features' or countable basis for describing the target function as in Hutter [20].

MSE loss. We use MSE loss for analytic tractability:

where f is the function expressed by a given model. We define the skill loss L k as the loss when only the k th skill is given, which can be weighted by their skill frequencies to express the total loss:

Skill strength. The skill strength or the linear correlation between the k th skill ( g k ) and a function expressed by the model at time T ( f T ) is

The skill strength R k is the k th coefficient if a model is expanded in the basis of the skill functions ( g k ). The skill strength, like the test loss, can be accurately approximated by a sum (see Appendix K.3). The skill loss L k (Eq. (6)) can be expressed by the skill strength and the norm of the learned function for I = k :

The skill loss becomes 0 if and only if f T ( I = k, X ) = Sg k ( I = k, X ) .

Experimental setting. We use a 2-layer MLP that receives the n s + n b bits as inputs and outputs a scalar ( { 0 , 1 } n s + n b → R ). In most of the experiments, the NN is trained with stochastic gradient descent (SGD) with width 1000 , using n s = 5 , m = 3 , and n b = 32 , unless otherwise stated. A decoder transformer is also used for the time emergent experiments. See Appendix K for details.

3 Multilinear model

We propose a simple multilinear model - multilinear with respect to the parameters - with the first N most frequent skill functions g k ( i, x ) as the basis functions (features):

where a, b ∈ R N are the parameters. The model has built-in skill functions g k - which transform control bits and skill bits into the parity outputs of each skill - so the model only needs to scale the parameters to a k b k = S .

The multilinear structure (product of a k , b k ) is analogous to the layered structure of NNs and results in emergent dynamics (Fig. 1(a)) different from a linear model with the same basis functions (Appendix H). A similar model has been studied by Saxe et al. [19] in the context of linear neural networks (Appendix B.2).

For the multilinear model, note that a k ( T ) b k ( T ) is the skill strength R k (Eq. (7)) and the skill loss (Eq. (6)) is a function of S and R k only:

Assuming that we are training the model on D samples from P ( I, X ) , the empirical loss decomposes into a sum of empirical skill losses because g k 's supports are mutually exclusive. This decouples the dynamics of each skill ( R k ( T ) ), which is analytically solvable under gradient flow (Appendix C.1).

̸

where d k is the number of samples of the k th skill (i.e., number of samples ( i, x ) with g k ( i, x ) = 0 ), η is the learning rate, and 0 < R k (0) < S is the skill strength at initialization.

4 Scaling laws

Recent literature has extensively explored scaling laws; see Section 7 for an overview. In this section, we derive the scaling laws of our multilinear model (Section 3) for time ( T ), data ( D ), parameters ( N ) and optimal compute ( C ). We define compute as C := T × N [21].

Table 2 shows our analytical scaling laws including their prefactor constants (Appendix J) and Fig. 2 compares the simulation of our model with our scaling law predictions. For the scaling law exponents, we achieve the same exponent as in Hutter [20] for D and in Michaud et al. [18] for T, D, and N . Assuming 0 < α < 1 , the exponents are consistent with the small power-law exponents reported in large-scale experiments, see, e.g., [9, 14, 22].

Using Eqs. (6), (10) and (11), we derive the loss as a function of time ( T ), data ( D ), parameters ( N ), and the number of observations for each skill [ d 1 , · · · , d n s ] :

Under suitable assumptions (e.g., for the T scaling law, we take D,N →∞ and d k /D →P s ( k ) ), we can use Eq. (12) to derive the scaling laws. For T, D , and N , we used Eq. (11) - decoupled dynamics induced the basis functions g k - to decouple the evolution of each skill loss:

For the time scaling law, each L k shares the same dynamics with T scaled by P s ( k ) .

For the data scaling law, each L k depends only on the observation the k th skill ( d k > 0 ).

For the parameter scaling law, each L k depends on whether the model has g k as a basis function.

For the optimal compute scaling law, we show in Corollary 4 (Appendix J) that the optimal tradeoff between T and N for given C is when T is large enough to fit the N th skill (Fig. 3). In Appendix J, we show rigorous derivations of all scaling laws, including the prefactors, error bounds, and conditions (e.g., how large N must be compared to T to be treated as infinity). For simplified derivations for the exponents only, see Appendix E. For an intuitive derivation (stage-like training) and connection to Michaud et al. [18], see Appendix D.

5 Predicting emergence

The literature on emergence has rapidly expanded lately; for a review of these developments, see Section 7. In this section, we analyze the emergence of a 2-layer NN (Section 2) and discuss to what degree the emergence in NNs can be described with our model. At initialization, NNs lack the information about the data and must 'discover' each g k . To take this effect into account in our model, we add an extra parameter which we calibrate (fit) on an NN trained on one skill ( n s = 1 ) system and use it to predict the emergence of subsequent skills for the n s = 5 setup (Fig. 1).

5.1 Time emergence

In our multilinear model, the layerwise structure - the product of parameters a k b k - leads to a sigmoidal saturation where an update of one layer hastens the update of the other layer. Feature

10 0

10 1

10 2

10 3

10 4

T

10

2

10

1

L

(a)

Time

scaling

10 0

10 1

10 2

10 3

D

10

1

L

(b)

Data

scaling

10 0

10 1

10 2

N

10

2

10

1

L

(c)

Parameter

scaling

α

=

0.3

α

=

0.6

α

=

0.9

L

=

A

TT

-

α/

(

α

+1)

D,N

→∞

L

=

A

DD

-

α/

(

α

+1)

N,T

→∞

L

=

A

NN

-

α

T, D

→∞

α

=

0.3

α

=

0.6

α

=

0.9

10 2

10 4

10 6

10 8

C

10 0

L

10 2

10 4

10 6

10 8

C

10

1

10 0

L

10 2

10 4

10 6

10 8

C

10

2

10

1

10 0

L

α

=

0.3

α

=

0.6

α

=

0.9

learning dynamics in a 2-layer MLP shares the positive feedback between the layers but require a non-trivial update of parameters to express g k .

Extended model. Given that feature learning, though nonlinear, involves parameter updates, we compensate for the additional delay in feature-learning by multiplying g k by a calibration constant 0 < B < 1 :

The calibration constant B rescales the dynamics in T (Eq. (11)):

where d k /D →P s ( k ) because we assume D →∞ . We observe that B 2 = 1 / 22 fits the NN trained on one skill (see Fig. 11 in Appendix I), and the calibrated model predicts emergence in the n s = 5 system (Fig. 1(a)): suggesting that the dynamics of feature-learning g k in 2-layers NNs is similar to that of parameter learning ( a k b k ) in a simple multilinear model. For further intuition of the extended model, see an example of time emergence in an NN in Appendix G.

5.2 Data point emergence

Our multilinear model can learn the k th skill with a single observation of the skill because the skill functions g k are built in (see Corollary 1 in Appendix C.2). NNs, without the fixed basis functions, must 'discover' each g k , which requires multiple samples from the k th skill.

Extended model. To make our model a D c -shot learner, we extend it by replacing g k with the e k,l basis:

where the matrix B ∈ R N × D c is an extension of b ∈ R N in Eq. (9), D c is a fixed scalar, and e k,l ( i, x ) : { 0 , 1 } n s + n b → R are functions with the following properties:

̸

The first property states that e k 's, when I = k , are orthonormal in X . The second property asserts that, similar to g k (Eq. (2)), e k,l is non-zero only when I = k , and fitting of the k th skill only occurs among e k,l 's, keeping the skills decoupled . The third property states that g k can be expressed using e k,l .

For the k th skill, the extended model overfits g k when there are fewer observations ( d k ) than the dimension of the e k,l basis ( D c ), and fits g k when d k ≥ D c , making our model a D c shot learner.

D c shot learner. If we initialize the extended model in Eq. (15) with sufficiently small initialization and if the conditions in Eq. (16) are satisfied, then the skill strength after training ( T →∞ ) on D datapoints is

̸

The number d k is the number of samples in the training set for the k th skill (i.e., datapoints with g k ( i, x ) = 0 ).

■

Using Eq. (17), we can calculate the emergence of R k /S as a function of D . Note that Eq. (17) is similar to the model in Michaud et al. [18] in that, to learn a skill, the model requires a certain number of samples from the skill.

The derivation of Eq. (17) follows trivially from the dynamics of the extended model (Eq. (15)) and well-known results in linear/kernel regression [23-27]. To be more specific, the model finds the minimum norm solution as if we performed ridgeless regression on g k with basis functions [ e k, 1 , · · · e k,D c ] . See Appendix F.3 for details.

We observe that D c = 800 approximates the data emergence for the n s = 1 system (see Fig. 11 in Appendix I) and also the emergence for n s = 5 system (Fig. 1(b)), suggesting that the NN discovers g k when it observes D c samples from the k th skill.

5.3 Parameter emergence

Since our multilinear model has g k 's as the basis functions, it requires only one basis function ( 2 parameters) to express a skill (see Corollary 2 in Appendix C.3). A 2-layer NN cannot express a skill with a single hidden node (i.e., a hidden layer with width 1 ); it requires multiple hidden nodes to express a single skill.

Extended model. To compensate for the need for multiple hidden nodes in expressing one skill, we extend our model similarly to Eq. (15). Because the number of parameters is now a bottleneck, we ensure the model has N basis functions ( e k,l 's):

where N c is the number of basis functions needed to express a skill, quotient q is ⌊ ( N -1) /N c ⌋ +1 and remainder r is such that ( q -1) N c + r = N . In short, the N basis functions are

Similar to Eq. (16), the basis functions satisfy the following properties

̸

N c basis functions for a skill. For the extended model in Eq. (18) , the skill strength at T, D →∞ for a given N becomes

■

Proof See Appendix F.4.

The model can express the k th skill based on the number of available basis functions for the given skill (Eq. (21)). For example, skills with k < q have all N c basis functions [ e k, 1 , · · · , e k,N c ] to express the k th skill (Eq. (20)), while for k = q , only r of the N c basis functions are available.

We observe that N c = 4 fits the parameter emergence for the n s = 1 system (see Fig. 11 in Appendix I) and also the emergence for the n s = 5 system (Fig. 1(c)), suggesting that the NN requires 4 nodes in expressing g k . The results also suggest that an NN, while lacking the ordering of basis functions (Eq. (19)), prefers to use the hidden neuron in fitting more frequent skills. The 'preference' toward frequent skills agrees with Fig. 1(a) where the NN learns more frequent skills first. Note that for the parameter emergence experiment, Adam [28] was used, instead of SGD, to increase the chance of escaping the near-flat saddle points induced by an insufficient number of parameters.

5.4 Time emergence in a transformer

To test whether our conceptual framework extends to other architectures, we perform a time emergence experiment with a transformer (Fig. 4). Note that the emergent time τ emerge - when the skill strength is sufficiently larger than 0 - follows the same power-law relationship as Eq. (11): τ emerge ( k ) ∝ k α +1 (see Fig. 6 in Appendix D for a discussion on emergent time). This suggests that, in the multitask sparse parity setup, other architectures may follow similar decoupled dynamics (Eq. (11)) and the consequent scaling laws (Section 4) and emergence (Section 5). An in-depth study of these findings across different architectures is left for future work.

10 3

10 4

10 5

10 6

T

0.0

0.5

1.0

R

k/S

k

=1

k

=2

k

=3

k

=4

k

=5

1

2

3

4

5

Skill index

k

10 4

10 5

τemerge

(

k

)

τemerge

(

k

)

∝

k α

+1

5.5 Limitations of the multilinear model

The strength of our extended multilinear model comes from the decoupled dynamics for each skill: leading to the prediction of the time, data, and parameter emergence with a single calibration. The weakness of our model is that it simplifies the more complex dynamics of NNs.

Time emergence. We note that the NN and the multilinear model emerge at similar instances, but the NN takes longer to saturate fully. This is because, for a given skill, the dynamics of the NN is not one sigmoidal saturation but a sum of multiple sigmoidal dynamics with different saturation times. To express the parity function, the NN must use multiple hidden neurons, and the skill strength can be divided into the skill strength from each neuron whose dynamics follow a sigmoidal saturation. Because of the non-linearity and the function it expresses, each neuron is updated at different rates, and the slowly saturating neurons result in a longer tail compared to our multilinear model. For an example, see Fig. 8 in Appendix G.

Data point emergence. Our extended model (Eq. (17)) deviates from NNs when d k ≪ D c and NNs show a more abrupt change in R k as a function of D . This is because our model asserts strict decoupling among the skills: even a few d k will contribute to learning g k from e k,l . This differs from the NN, which lacks strict decoupling among the samples from different skills. We speculate that because NNs can perform benign [29] or tempered [30] overfitting, they treat a few data points from less frequent skills as 'noise' from more frequent skills: requiring more samples to learn the infrequent skills.

Parameter emergence. Note that Fig. 1(c) has high variance compared to other emergence plots in Fig. 1; this is because the NN sparsely, over many repeated trials, uses the hidden neurons to learn less frequent skills over more frequent ones (see Table 5 in Appendix I for an example of such outliers). Because NNs are less strictly biased toward frequent skills than our model, we speculate that initial conditions favoring less frequent skills may contribute to the outliers.

6 Discussion and conclusion

This work demonstrated scaling laws and predicted emergence in a 2-layer MLP using a tractable multilinear model. We found that representing skills as mutually exclusive functions leads to the decoupled dynamics, resulting in the scaling laws observed in a 2-layer MLP. The layerwise structure leads to emergent (sigmoidal) saturation of the skill strength, similar to what is observed in 2-layer MLPs.

Despite lacking explicit skill functions, NNs exhibit similar emergence patterns. We speculate that the model's layerwise structure and power-law frequencies of the skills induce stage-like dynamics

(Appendix D) in NNs. The parameters relevant for expressing more frequent skills are updated significantly faster than those for less frequent skills. When skill 'discovery' operates on different time scales with minimal interaction, the skill dynamics effectively become decoupled , justifying our model setup.

Our results suggest a link between feature learning and emergence [6] driven by decoupled, stage-like dynamics. The layerwise dynamics leading to sigmoidal saturation may also disentangle the problem into skills (features) of varying importance (frequencies). Then feature learning, or discovering the basis functions that describe the target function [31, 32] (for recent studies, see [33-38]), likely occurs in stages. Investigating this connection through layerwise dynamics is left for future work.

Similar to many prior works (see, e.g., [20, 18]), we studied a simple model on an idealized powerlaw distributed dataset. Also, our model cannot capture the complex non-linear interactions among multiple skills but can express any linear superposition of skills. In future work, we will explore 'complex skills' in language as a superposition of linearly independent skills. By validating our findings in language tasks, we aim to contribute to a broader understanding of how neural networks acquire and exhibit complex behaviors.

7 Related works

In this section, we review the literature on scaling laws and emergence in NNs. Focusing on data scaling, Hutter [20] develops a model with a discrete set of features. Under the assumption of a power-law distribution of features, this model demonstrates that the error decreases as a power law with increasing data size. In a related vein, Michaud et al. [18] propose a model of neural scaling laws in which the loss is decomposed into a sum over 'quanta'. Their model aims to reconcile the apparent discrepancy between loss metrics' regular power-law scaling and the abrupt development of novel capabilities in large-scale models. Various other models for neural scaling laws have been proposed in recent research, including connecting neural scaling exponents to the data manifold's dimension [39] and their relation with kernels [40], proposing solvable random-feature models [41, 21], and developing data scaling models using kernel methods [42, 43, 25].

Closely related to the study of neural scaling laws is the understanding of emergent abilities in large language models. Several studies [1-4] document examples of such emergent abilities. Arora and Goyal [44] propose a framework for the emergence of tuples of skills in language models, in which the task of predicting text requires combining different skills from an underlying set of language abilities. Okawa et al. [45] demonstrate that a capability composed of smoothly scaling skills will exhibit emergent scaling due to the multiplicative effect of the underlying skills' performance. Other works related to the skill acquisition include Yu et al. [46], who introduce a new evaluation to measure the ability to combine skills and develop a methodology for grading such evaluations, and Chen et al. [47], who formalize the notion of skills and their natural acquisition order in language models.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1747-1764, 2022.

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint:2206.04615 , 2022.

Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint: 2206.07682 , 2022.

Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? Advances in Neural Information Processing Systems , 36, 2023.

Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hÉigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Yoshua Bengio, Danqi Chen, Samuel Albanie, Tegan Maharaj, Jakob Foerster, Florian Tramer, He He, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint: 2404.09932 , 2024.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread , 2023. https://transformercircuits.pub/2023/monosemantic-features/index.html.

Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint:1712.00409 , 2017.

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint:2001.08361 , 2020.

Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. A constructive prediction of the generalization error across scales. arXiv preprint: 1909.12673 , 2019.

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint:2010.14701 , 2020.

Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 5915-5922, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12104-12113, 2022.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint:2203.15556 , 2022.

Vivien Cabannes, Elvis Dohmatob, and Alberto Bietti. Scaling laws for associative memories. arXiv preprint arXiv:2310.02984 , 2023.

Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann. Scaling mlps: A tale of inductive bias. Advances in Neural Information Processing Systems , 36, 2024.

Boaz Barak, Benjamin Edelman, Surbhi Goel, Sham Kakade, Eran Malach, and Cyril Zhang. Hidden progress in deep learning: Sgd learns parities near the computational limit. Advances in

Neural Information Processing Systems , 35:21750-21764, 2022.

Eric Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The quantization model of neural scaling. Advances in Neural Information Processing Systems , 36, 2023.

Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. Proceedings of the International Conference on Learning Representations 2014 , 2014. arXiv:1312.6120.

Marcus Hutter. Learning curve theory. arXiv preprint:2102.04074 , 2021.

Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint:2402.01092 , 2024.

Tamay Besiroglu, Ege Erdil, Matthew Barnett, and Josh You. Chinchilla scaling: A replication attempt. arXiv preprint:2404.10102 , 2024.

Abdulkadir Canatar, Blake Bordelon, and Cengiz Pehlevan. Spectral bias and task-model alignment explain generalization in kernel regression and infinitely wide neural networks. Nature communications , 12(1):2914, 2021.

Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Kernel alignment risk estimator: Risk prediction from training data. Advances in Neural Information Processing Systems , 33:15568-15578, 2020.

Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems , 34:10131-10143, 2021.

Ouns El Harzli, Bernardo Cuenca Grau, Guillermo Valle-Pérez, and Ard A Louis. Doubledescent curves in neural networks: a new perspective using gaussian processes. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 11856-11864, 2024.

James B Simon, Madeline Dickens, Dhruva Karkada, and Michael R DeWeese. The eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks. Transactions on Machine Learning Research , 2023. arXiv:2110.03922.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint:1412.6980 , 2014.

Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences , 117(48):30063-30070, 2020.

Neil Mallinar, James Simon, Amirhesam Abedsoltan, Parthe Pandit, Misha Belkin, and Preetum Nakkiran. Benign, tempered, or catastrophic: Toward a refined taxonomy of overfitting. Advances in Neural Information Processing Systems , 35:1182-1195, 2022.

Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):17981828, 2013.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436, 2015.

Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning , pages 11727-11737. PMLR, 2021.

Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. arXiv preprint arXiv:2111.00034 , 2021.

Arthur Jacot, Eugene Golikov, Clément Hongler, and Franck Gabriel. Feature learning in l _ 2 -regularized dnns: Attraction/repulsion and sparsity. Advances in Neural Information Processing Systems , 35:6763-6774, 2022.

Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. Advances in Neural Information Processing Systems , 35:32240-32256, 2022.

Inbar Seroussi, Gadi Naveh, and Zohar Ringel. Separation of scales and a thermodynamic description of feature learning in some cnns. Nature Communications , 14(1):908, 2023.

Hugo Cui, Luca Pesce, Yatin Dandi, Florent Krzakala, Yue M Lu, Lenka Zdeborová, and Bruno Loureiro. Asymptotics of feature learning in two-layer networks after one gradient-step. arXiv preprint arXiv:2402.04980 , 2024.

Utkarsh Sharma and Jared Kaplan. Scaling laws from the data manifold dimension. Journal of Machine Learning Research , 23(9):1-34, 2022. arXiv:2004.10802.

Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint:2102.06701 , 2021.

Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint:2210.16859 , 2022.

Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher-student paradigm. Journal of Statistical Mechanics: Theory and Experiment , 2020(12):124001, 2020.

Blake Bordelon, Abdulkadir Canatar, and Cengiz Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning , pages 1024-1034. PMLR, 2020.

Sanjeev Arora and Anirudh Goyal. A theory for emergence of complex skills in language models. arXiv preprint:2307.15936 , 2023.

Maya Okawa, Ekdeep S Lubana, Robert Dick, and Hidenori Tanaka. Compositional abilities emerge multiplicatively: Exploring diffusion models on a synthetic task. Advances in Neural Information Processing Systems , 36, 2024.

Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-mix: A flexible and expandable family of evaluations for ai models. arXiv preprint:2310.17567 , 2023.

Mayee Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models. Advances in Neural Information Processing Systems , 36, 2023.

Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv:2201.02177 , 2022.

Irina Gennad'evna Shevtsova. Sharpening of the upper bound of the absolute constant in the berry-esseen inequality. Theory of Probability and Its Applications , 51(3):549-553, 2007.

Hugh L Montgomery and Robert C Vaughan. Multiplicative Number Theory I: Classical Theory . Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2007.

A Glossary

A Normalization constant for P s such that P s ( k ) = Ak -( α +1)

T Time or step

D Number of data points

N Number of parameters (skill basis functions in the model for the multilinear model; the width of hidden layer for MLP)

C The computation cost T × N

n s The number of skills in the multitask sparse parity problem

I Random variable of the control bits

X Random variable of the skill bits

P s Probability of skills (control bits)

P b Probability of skill bits

S The target scale or the norm of the target function

R k Skill strength of the k th skill (Eq. (7))

L Total (generalization) loss

L ( D ) Empirical loss for D samples

L k Skill loss of the k th skill (Eq. (6))

̸

d k Number of observation of the k th skill (i.e. number of training points ( i, x ) with g k ( i, x ) = 0 )

f ∗ Target function f ∗ : { 0 , 1 } n s + n b →{-S, S } (Eq. (4))

g k The k th skill basis function g k : { 0 , 1 } n s + n b →{-1 , 0 , 1 } (Eq. (2))

B Background

In this section, we review the multitask sparse parity dataset, as described by Michaud et al. [18] and discuss the nonlinear dynamics of two-layer linear networks, following the work of Saxe et al. [19].

B.1 Multitask sparse parity

The sparse parity task can be stated as follows: for a bit string of length n b , the goal is to determine the parity (sum mod 2) of a predetermined subset of m bits within that string. The multitask sparse parity [18] extends this problem by introducing n s unique sparse parity variants in the dataset. The input bit strings have a length of n s + n b . The first n s bits function as indicators by assigning a specific task. The frequency of the distinct parity tasks follows a rank-frequency distribution with an inverse power law relation (power-law distribution). The last n b bits are uniformly distributed. This sets a binary classification problem { 0 , 1 } n s + n b →{ 0 , 1 } where only a single bit of the initial n s bits is nonzero. In Table 3, the many distinct parity tasks represent different skills. 2

The proposal in [18] aims to reconcile the regularity of scaling laws with the emergence of abilities with scale using three key hypotheses: (i) skills, represented as a finite set of computations, are distinct and separate; (ii) these skills differ in their effectiveness, leading to a ranking based on their utility to reduce the loss; and (iii) the pattern of how frequently these skills are used in prediction follows a power-law distribution. Interestingly, the multitask problem has a consistent pattern across scaling curves: each parity displays a distinct transition, characterized by a sharp decrease in loss at a specific scale of parameters, data, or training step. Such a sudden shift occurs after an initial phase of no noticeable improvement, leading to reverse sigmoid-shaped learning curves. Michaud et al. [18] empirically show that for a one-hidden-layer neural network with ReLU activation, trained using cross-entropy loss and the Adam optimizer, these transitions happen at different scales for distinct tasks. This results in a smooth decrease in the overall loss as the number of skill levels increases.

B.2 Nonlinear dynamics of linear neural network

Saxe et al. [19] have solved the exact dynamics for two-layer linear neural networks with gradient descent under MSE loss (Fig. 5(a)). 3 The dynamics decompose into independent modes that show sigmoidal growth at different timescales (Fig. 5(c)). The setup assumes orthogonal input features X ∈ R d 1 and input-output correlation matrix Σ ∈ R d 1 × d 3 for target output f ∗ ( X ) ∈ R d 3 :

...

...

(a) Linear neural network

...

...

(b) Independent modes

0

25

50

75

100

T

0.0

0.2

0.4

0.6

0.8

1.0

akbk/λk

λ

1 =

1.0

λ

2 =

0.5

λ

3 =

0.25

(c) Dynamics of modes

By performing SVD (singular value decomposition) on input-output correlation matrix Σ = U Λ V , the target function f ∗ : R d 1 → R d 3 becomes:

where u k ∈ R d 1 , v k ∈ R d 3 are the row vectors of U, V and λ k ∈ R are the singular values of Λ .

Saxe et al. [19] have shown that the dynamics of a two-layer (one-hidden-layer) undercomplete (the width of the hidden layer is smaller than the width of the input and output) linear neural network decomposes into that of the following 'modes':

where a k , b k ∈ R are the parameters. Note that Eq. (24) are d 2 decoupled functions v T k f ( x ) : R d 1 → R (Fig. 5(b)). Assuming small and positive initialization ( 0 < a k (0) b k (0) ≪ λ k ), the dynamics of Eq. (24) under gradient descent with learning rate η can be solved analytically; the product of parameters a k b k grows sigmoidally with saturation time proportional to λ -1 k (Fig. 5(c)):

Using the analytic equation of the multilinear model, Saxe et al. [19] have empirically demonstrated that the dynamics of both linear and nonlinear neural networks closely resemble that of the multilinear model (Eq. (25)).

C Derivation of the multilinear model

In this section, we provide derivations of how the skill loss of our multilinear model evolves with a given resource: time (Lemma 1), data (Corollary 1), and parameters (Corollary 2). Note that two corollaries for data and parameters (Corollaries 1 and 2) follow from the decoupled dynamics (Lemma 1).

C.1 Decoupled dynamics of the multilinear model

Lemma 1. Let the multilinear model Eq. (9) be trained with gradient flow on D i.i.d samples for the setup in Section 2 (input distribution: Eq. (1) , target function: Eq. (4) , and MSE loss: Eq. (5) ). Let k ≤ N be a skill index in the multilinear model and the input distribution ( k ≤ n s ). Then assuming the following initialization a k (0) = b k (0) and 0 < a k (0) b k (0) < S , the dynamics of the k th skill strength ( R k ) is

̸

where η is the learning rate and d k is the number of observations with g k ( I = k, x ( j k ) ) = 0 .

Proof For j = 1 , · · · , D , denote ( i ( j ) , x ( j ) ) be the j th data point in the training set. Then the empirical loss for D datapoints is given as

and the skill loss is

We note that

̸

as g i ( i, j ) ∈ { 1 , -1 } and g k ( i, j ) = 0 for i = k . So if we denote d k the number of data points with i ( j ) = k , then we can conclude

which is the decoupled loss in the main text (Eq. (11)). Using the gradient descent equation and Eq. (29), we obtain

Likewise, we can obtain the equation for b k as

Because of symmetry between a and b (See Appendix B.2 or [19]), assuming a k (0) = b k (0) , and a k (0) b k (0) > 0 results in a k ( T ) = b k ( T ) for all T . The equation for R k = a k b k is

Assuming a k (0) b k (0) < S , we can solve the differential equation to obtain

The equation for L k follows from Eq. (10).

C.2 One-shot learner

Corollary 1. For the setup in Lemma 1, the k th skill loss ( L k ) at T, N →∞ is

where d k is the number of k th skill's observations.

Proof The corollary follows directly from Lemma 1. By taking T, N →∞ ,

We obtain the result by using the relationship between R k and L k in Eq. (10). ■

C.3 Equivalence between a basis function and a skill

Corollary 2. Let the multilinear model Eq. (9) be trained with gradient flow on D i.i.d samples for the setup in Section 3 (input distribution: Eq. (1) , target function: Eq. (4) , and MSE loss: Eq. (5) ). Assume a k (0) = b k (0) , 0 < a k (0) b k (0) < S , and that the model has the N most frequent skills as basis functions. Then R k for the k th ≤ n s skill at T, D →∞ is

Proof The corollary follows directly from Lemma 1. By taking T, D →∞ ,

We obtain the result by using the relationship between R k and L k in Eq. (10) and R k (0) ≪ S . ■

■

D Stage-like training: intuitive derivation of the scaling laws

Even though we provide more detailed (Appendix E) and rigorous (Appendix J) derivation of the scaling laws, a less general yet more intuitive solution aids in understanding the scaling laws of our model and NNs. In this section, we define stage-like training - one skill is completely learned before the next skill initiates learning (Fig. 6(a)) - and state the conditions for it to occur. We provide an example of how stage-like training results in the time scaling law and explain how the model in Michaud et al. [18] may arise from the NN dynamics. Finally, we discuss the stage-like training's role in emergence in NNs.

0

50

100

150

200

T

0.0

0.2

0.4

0.6

0.8

1.0

R

k

(

T

)

/S

/epsilon1

1

-

/epsilon1

τ

(

e

)

1

(

/epsilon1

)

τ

(

s

)

1

(

/epsilon1

)

τ

(

e

)

2

(

/epsilon1

)

k

=1

k

=2

(a) Emergent and saturation time

0

50

100

150

200

T

10.5

11.0

11.5

12.0

12.5

13.0

L

1

2

S

2

N

∑

k

=1

P

s

(

I

=

k

)

∆

τ

(

e

)

(

/epsilon1

)

1

2

S

2

P

s

(1) +

O

(

/epsilon1

)

(b) Loss change between emergences

D.1 Stage-like training

When a model exhibits an emergence behavior - when saturation of skill occurs abruptly after a delay - and the intervals between each emergence are sufficiently large, the model admits stage-like training. The multilinear model (sigmoidal saturation of skills strength, Eq. (11)) in the multitask sparse parity dataset (power-law decay of skill frequencies, Eq. (1)) can satisfy such conditions: In Fig. 6(a), we observe the stage-like training in time in which one skill saturates (reaches R k /S ≈ 1 ) before the next skill initiates its emergence. To quantify this behavior, we define two intervals for each skill (see Fig. 6(a)):

The emergent time τ ( e ) k ( ϵ ) : the time for R k /S to reach ϵ ;

The saturation time τ ( s ) k ( ϵ ) : the time for R k /S to saturate from ϵ to 1 -ϵ .

Using the dynamics equation (Eq. (11)) and that d k /D →P s ( k ) , the emergent time and saturation time of the k th skill becomes

For sufficiently small initialization ( R k (0) ≪ S ), we get a stage-like training:

where the model finishes learning (saturating) the k th skill before starting to learn (emerging) the next skill.

D.2 Time scaling law from stage-like training

Assuming our model satisfies the stage-like training for all k of interest, we can derive the time scaling law from the stage-like training.

At τ ( e ) k ( ϵ ) , because of stage-like training, all skills with index up to but not including k have saturated ( R i<k ≈ S ), or equivalently L i<k ≈ 0 (Eq. (10)). The total loss, the sum of L j weighted by P s ( j ) ∝ j -( α +1) (Eq. (6)), becomes ∑ ∞ j = k P s ( I = j ) S 2 / 2 (Fig. 6(b)). The saturation of the k th skill results in a loss difference of P s ( I = k ) S 2 / 2 . Thus, we obtain

Accordingly, the emergent interval between the k and k +1 skills relative to the τ ( e ) k ( ϵ ) is

Assuming k ≫ 1 and combining Eq. (43) and Eq. (45) to the largest order, we have the equation for the power-law with exponent -α/ ( α +1) in Fig. 2(a):

If the stage-like training holds for any resource (e.g., time, data, or parameters), the scaling law can be derived using the ratio of change in loss per skill (Eq. (43)) and the ratio of change with respect to the resource (given by the emergent time in Eq. (45)). The quanta model in Michaud et al. [18] is an example where the stage-like training holds for all resources.

D.3 Discussion on the effective decoupling of skills in neural networks

In Section 5, we have empirically demonstrated that the multilinear model predicts the emergence of a 2-layer NN (Fig. 1). In Section 6, we briefly discussed why NNs, despite their lack of the decoupling among the skills, behave similarly to the decoupled model with g k s as fixed basis functions: the stage-like training in NNs - induced by the model's layerwise structure and power-law frequencies of the skills - effectively decouples the skills. In this subsection, we extend the discussion in more detail.

In NNs, even though g k s are 'discovered' (feature learned) by non-tractable dynamics, we speculate that similar stage-like dynamics also hold in 'discovering' (feature learning) g k s: parameters 'useful' for expressing more frequent skills will be updated significantly faster than parameters useful for expressing less frequent skills.

If skill discovery and saturation dynamics operate at different time scales (stages), with negligible interaction among the skills, the skill dynamics become effectively decoupled . Because the dynamics are decoupled in stages, NNs repeat the feature learning process - using the limited resource (time, data, parameters) to express the skill - for all skills with each iteration varying only in the scale of the resource (e.g. training time, number of observations, and number of hidden layer neurons): resulting in a similar emergence to our multilinear model.

A more concrete understanding of our speculation that feature learning also occurs in stages due to a layerwise structure is left for future work.

E Derivation of the scaling law exponents

This section provides a detailed derivation of the scaling laws up to a rigor common in physics and engineering. For example, we approximate the Riemann sum as integral or treat k , the number of

skills, as a differentiable parameter. For more general and rigorous derivations including the prefactor constants, see Appendix J. Instead, for more intuition and the relationship to the quanta model in Michaud et al. [18], see Appendix D.

E.1 Time scaling law exponent

To derive the time scaling law exponent, we assume the time as the bottleneck and take N,D →∞ . By using the decoupled dynamics of each skill loss (Lemma 1),

Noting that d k /D →P s ( k ) as D →∞ , where P s ( k ) = Ak -( α +1) , we have

This is a function of k -( α +1) T only, suggesting the decoupling dynamics for each skill. Thus,

Using Eq. (6) and taking N,n s →∞ at the same rate, 4 we can approximate the loss as an integral instead of a sum over k :

where A is the normalization constant for P s . We can differentiate the loss and use Eq. (49) to express the equation in terms of k :

Integrating by parts, we obtain

The first term goes to 0 as N →∞ and the second term goes to 0 exponentially faster compared to the last term for T ≫ 1 , which leads to the scaling law with exponent -α/ ( α +1) :

Finite N correction for small α . In Fig. 7, we observe that our model with α = 0 . 1 deviates from the expected power-law with exponent -α/ ( α +1) . The deviation can be explained by the antiderivative term in Eq. (52):

10 0

10 1

10 2

10 3

10 0

10 1

L

simulation

power law

(

corrected

)

power law

α

=

0.1

α

=

0.3

α

=

0.5

α

=

0.7

T

The second term ( k = 1 ) goes to 0 faster than O ( T -1 ) for sufficiently larger T but the first term ( k = N ) may not decay fast enough for finite N and sufficiently small α . For example, N = 50 , 000 and α = 0 . 1 leads to N -α ≈ 0 . 3 , which is not negligibly small.

Assuming finite N and small α such that the first term in Eq. (55) is non-negligible, we can rewrite Eq. (52) as

where we assumed a small initialization S/ R k (0) ≫ 1 and sufficiently large number of parameters N α +1 ≫ T to approximate L C . Because the total loss at initialization is L (0) = S 2 / 2 , L C is non-negligible compared to the loss for sufficiently small α . Thus considering L C , we obtain the corrected power-law which better approximates the time scaling law (dashed lines in Fig. 7). For a rigorous and comprehensive analysis of the time scaling law, see Theorem 2 and Theorem 3 in Appendix J.

E.2 Data scaling law exponent

In this section, we derive the data scaling law exponent. The data scaling law assumes T → ∞ and N →∞ with data as the bottleneck. From the decoupled dynamics of the multilinear model (Lemma 1), we can show that our model is a one-shot learner (Corollary 1):

̸

One shot learner. Given that N > k , T →∞ , and d k is the number of samples from the training set with g k ( i, x ) = 0 , the k th skill loss after training is

Proof See Appendix C.2.

■

Our model requires only one sample from the k th skill to learn such a skill, similar to how language models are few-shot learners at inference. 5 The model can one-shot learn a skill since it has g k as the basis functions, and the dynamics among different skills are decoupled. A similar one-shot learner has been studied in Hutter [20] where the error depends on a single 'observation' of a feature.

Because the k th skill loss only depends on d k (number of observations for the k th skill), we can calculate the expectation of the skill loss for D data points from P observed ( k | D ) or the probability that d k > 0 :

Using the one-shot learning property (Eq. (57)), the probability of observing the k th skill (Eq. (58)), and the decomposition of the loss into skill losses (Eq. (6)), the expected loss for D datapoints is

where the expectation E D is over all possible training sets of size D , and A is the normalization constant such that P ( k ) = Ak -( α +1) . The difference in the loss ∆ L = E D +1 [ L ] -E D [ L ] is

We can integrate ∆ L by parts.

In the second line, the first term goes to 0 for D ≫ 1 . In the last line, we used the expression for ∆ L (Eq. (62)) and E D [ L ] (Eq. (59)). Rearranging the equation above and using that D ≫ 1 , we obtain the scaling law with exponent -α/ ( α +1) :

where in the last line, ∆ D/D = 1 /D as the change in the number of data points relative to D is one.

E.3 Parameter scaling law exponent

The parameter scaling law assumes T → ∞ and D → ∞ , with the parameters N < n s as the bottleneck. Because our model is a one-shot learner (Eq. (57)), learning of the k th skill only depends on the existence of g k in the model; the model with [ g 1 , · · · , g N ] will learn all k ≤ N skills with L k = 0 .

The L k dependence on g k is formalized in Corollary 2, which we repeat here.

Equivalence between a basis function and a skill. Given T, D →∞ and if the multilinear model has the N most frequent skill functions as a basis,

■

Using Eq. (66) and Eq. (6), we can express the total loss as function of N :

By approximating N ≈ N +1 for N ≫ 1 , we obtain the power-law with exponent -α .

E.4 Optimal compute scaling law

For analytical tractability, we define compute as C := T × N . We start from Eq. (12) with D →∞

We can use Eq. (56) to calculate the first term and integrate the last term to get

where we used that L (0) ≫ L C and S 2 A/ (2 α ) -L C > 0 . Intuitively, the approximation shows the tradeoff between T - when increased, decreases the loss of the first N skills - and N - when increased, decreases the loss at sufficiently large T - for fixed compute C . For a comprehensive analysis of the approximation above, see Appendix J.

Removing the irrelevant constant terms,

We can use the method of Lagrangian multiplier to obtain

where λ is the Lagrange multiplier and C is compute. We can solve the above set of equations to obtain T α +1 ∝ N or equivalently

We can plug it in Eq. (71) to get

This derivation is similar to that of Bordelon et al. [21] (see Appendix N: Compute Optimal Scaling from Sum of Power-Laws in [21]). For a rigorous derivation of the optimal compute scaling law, see Corollary 4 and Appendix J.

F Derivation of the extended multilinear model

In this section, we show the derivation for the extended multilinear model.

F.1 Gradient flow in the extended multilinear model

Lemma 2. Let the extended multilinear model Eq. (15) be trained with gradient flow on D i.i.d samples for the setup in Section 2 (input distribution: Eq. (1) , target function: Eq. (4) , and MSE loss: Eq. (5) ). For the skill index k ≤ N be a skill index in the multilinear model, let the feature matrix Φ ∈ R D c × d k for the k th skill be

and SVD on Φ = USV . Assuming that the system is overparametrized ( d k < D c ), the gradient on ⃗ B k ∈ R D c ( [ B k, 1 , · · · , B k,D c ] ) is contained in the column space of semi-orthogonal matrix U ∈ R D c × d k :

Proof Similar to Lemma 1, the total loss can be decomposed into each skill such that the dynamics of B k,l relies only on d k observations of the k th skill:

̸

In the second line, we used Eq. (16) that e k,l ( I = k, x ) = 0 and the orthogonality of g k (Eq. (3)). In the last line, we used Eq. (16) that g k = D c -1 / 2 ∑ l e k,l . We can find the gradient descent equation of B k,l from Eq. (81):

which in the matrix form is

where D c dimensional vectors ⃗ B k and ⃗ S are [ B k, 1 , · · · , B k,D c ] and [ S, · · · , S ] respectively. It illustrates that dB k dt is contained in im(Φ) , which is contained in im( U ) (immediate from Φ = USV ). As UU T ( Uz ) = U ( U T U ) z = Uz , UU T acts as identity on image of U , showing that UU T d ⃗ B k dt = d ⃗ B k dt .

■

F.2 Conserved quantity of extended multilinear model

Lemma 3. In the setup of Lemma 2, a 2 k -| ⃗ B k | 2 is conserved over time.

Proof We can use Eq. (81) to find the equation for a k :

which in the matrix form is

Then

where we used Eq. (83) in the last line. Thus, a 2 k -| ⃗ B k | 2 is conserved during the dynamics. ■

F.3 D c shot learner

Proposition 1. Let the setup be as that in Lemma 2. Suppose that a k ( T ) is eventually bounded away from zero, i.e. there exists δ > 0 and M > 0 such that T > M ⇒| a k ( T ) | ≥ δ . Also assume that U ⊥ -component of ⃗ B k (0) a k (0) and ⃗ B k (0) S is negligible. Then the skill strength R k is

Proof First, we show that d L k dt ≤ 0 with equality only holding when the gradient is 0.

The equality holds only when

We show that both a k and ⃗ B k are bounded throughout whole dynamics. As

for σ 2 the smallest nonzero eigenvalue of ΦΦ T , where Φ = USV . This shows that

is bounded, so UU T ⃗ B k a k is bounded. Meanwhile, in Lemma 2, we showed that (1 -UU T ) d ⃗ B k dt = 0 , so (1 -UU T ) ⃗ B k a k is bounded. This shows that ⃗ B k a k is bounded. As a 2 k - | ⃗ B k | 2 is constant (Lemma 3) and | ⃗ B k a k | = | a k || ⃗ B k | is bounded, this shows that both a k and | ⃗ B k | are bounded.

The dynamics moving in some bounded region always has at least one accumulation point, which we denote as p . We will show that d L k dt = 0 at p . The function L k ( t ) in t is a decreasing differential function which is positive. We also note that d 2 L k ( t ) dt 2 is globally bounded, as it can be expressed in polynomial expression in ( a k , ⃗ B k ) and we showed that ( a k ( t ) , ⃗ B k ( t )) is bounded. From Taylor's theorem, one can obtain

for M = sup | d 2 L k ( t ) dt 2 | . Choosing t 2 = -d L k dt ( t 1 ) M -1 shows that

and letting t 1 →∞ here gives

so d L k dt → 0 as t →∞ . Meanwhile, as p is accumulation point of ( a k , B k ) , d L k dt ( p ) is accumulation point of d L k dt ( a k ( t ) , ⃗ B k ( t )) . As lim t →∞ d L k dt ( t ) = 0 , the only accumulation point of d L k dt ( t ) is zero, which shows that d L k dt ( p ) = 0 .

We have seen that a 2 k -| ⃗ B k | 2 and ( I -UU T ) ⃗ B k are conserved in our dynamics. A quantity conserved in dynamics should also be conserved at p , so p = ( a, ⃗ B ) should satisfy the following conditions:

( I -UU T ) ⃗ B = ( I -UU T ) ⃗ B k (0) (Lemma 2);

d L k dt ( a, ⃗ B ) = 0 , or equivalently the gradient is 0 at p .

We will solve for p satisfying those three conditions. The third condition is equivalent to that

̸

As a k ( T ) is eventually bounded away from zero, we have a = 0 , so

It follows that

and substituting to first condition gives

This is equivalent to a quadratic equation in a 2 , and has a following solution of

This shows that there are two candidates for p , with a given as two square roots of Eq. (101) and B determined from a by Eq. (99). It is impossible for L k ( t ) to have accumulation points both in regions a > 0 and a < 0 , as it would imply a k ( t ) = 0 happens infinitely many often, contradicting that a k is eventually bounded away from zero. Thus it follows that L k ( t ) can only have one accumulation point. As dynamics having unique accumulation point should converge, it follows that

One can check that the U ⊥ -component of ⃗ B k ( ∞ ) a k ( ∞ ) is given as

and this is bounded by | (1 -UU T ) B k (0) | ( S + a k (0)) , so by our assumption this is negligible. Thus, we find that ⃗ B k ( ∞ ) a k ( ∞ ) is the pseudo-inverse solution, which is also found by the linear

model with e k,l as basis functions. We can calculate L k ( ∞ ) using the result from kernel (linear) regression [23-27] (for a summary, see tables 1 and 2 in appendix A of [27]). Using the terminology in table 1 of [27], the sample size is d k ; the number of parameters is D c ; ridge and noise are absent; the eigenfunctions are [ e k, 1 , · · · , e k,D c ] ; the eigen coefficients are E X [ e k,i ( x ) Sg k ( x )] = SD -1 / 2 c (Eq. (16)); eigenvalues are uniform; the learnability is d k /D c for all i ; and the overfitting coefficient is (1 -d k /D c ) -1 . Taking into account that we have halved the MSE loss (Eq. (5)), the test loss is

We obtain the result by using Eq. (10).

F.4 N c basis functions for a skill

Proposition 2. Let the extended multilinear model Eq. (18) be trained with gradient flow on D →∞ i.i.d samples for the setup in Section 3 with n s → ∞ (input distribution: Eq. (1) , target function: Eq. (4) , and MSE loss: Eq. (5) , initialization: that of Proposition 1). For a model with the following finite N basis functions

where quotient q = ⌊ ( N -1) /N c ⌋ +1 and remainder r is such that ( q -1) N c + r = N . The skill strength at T →∞ becomes

Proof Because we have D →∞ and because [ e k, 1 , · · · e k,N c ] can express g k (Eq. (20)), it is trivial to show that R k ( ∞ ) = S for k < q . For k = q , the gradient descent dynamics (Eq. (83)) leads to

where the matrix Φ ∈ R r × d k and vector ⃗ B k ∈ R r are the feature matrix(Eq. (77)) and parameters for the k th skill respectively. As D →∞ , the matrix ΦΦ T becomes a rank r identity matrix scaled by the frequency of the skill:

Plugging in ΦΦ T ,

Assuming the initialization in Proposition 1, we can show that a k ( ∞ ) B k,l ( ∞ ) = S/ √ N c for l ≤ r . From Eq. (7), the skill strength R k ( ∞ ) is

where we used Eq. (20) for the linear correlation between e k,l and g k . ■

■

G Time emergence example in NN

In this section, we discuss an example for the time emergence case (Fig. 1(a)) in which the saturation of skill in an NN consists of multiple saturating 'modes' as in Fig. 8.

(a) neuron modes for a parity function

0

250

500

750

1000

T

0.0

0.2

0.4

0.6

0.8

1.0

R

/S

mode

1

mode

2

mode

3

NN

MulLin

(b) mode/skill strength

Task. We assume an input X ∈ R 3 × 8 (note that we are not using X as a random variable) that is all 8 possible inputs for bits with dimension 3 . The target Y is the parity function scaled by S .

NN. We assume a 2-layer width 3 NN with ReLU activation with the input dimension 3 (Fig. 8(a)). The NN has 16 parameters, but to simplify the argument, we use weight sharing so NN has only 4 parameters:

where σ is the ReLU activation and W,b,w are

Modes. It is easy to see that α = β = γ = √ 2 S and c = -S leads to the target parity function. We note that one parameter except c (i.e. α, β, γ ) maps to one neuron or a mode (colors in Fig. 8(a)). We define the first mode f (1) as

where w 1 , b 1 are the first entry of w,b respectively and W 1 is the first row of W . Note that f (1) ( x ) takes a form similar to the multilinear model (Eq. (9)) but with h 1 as the respective basis. We define f (2) , f (3) similarly, and the sum of modes becomes the NN:

which resembles the multilinear model with different skills.

Mode strength. Analogous to the skill strength in Eq. (7), we define mode q 's strength R ( q ) as

where f ( q ) ( X ) = [ f ( q ) ( X 1 ) , · · · , f ( q ) ( X 8 )] and X j are the j th column of X . By the linearity of the expectation,

Note that constant c always has zero correlation (inner product) to the target ( Y ).

Analysis. The dynamics of each mode R ( q ) ( x ) differs from that of the multilinear model (Eq. (11)) because h q ( x ) often depends on the parameter, and the dynamics are no longer decoupled among each mode. Nevertheless, each mode follows a sigmoid-like growth (Fig. 8(b)). We note that each mode has a different saturation time scale or is updated at different frequencies. A mode with a longer time scale leads to a longer 'tail' of saturation as discussed in the main text.

Update frequency. Because of the non-linearity, each mode differs in the gradients it receives. We can explicitly calculate the gradient for each parameter as:

We immediately notice that c will grow the fastest for small initialization ( α, β, γ, c ≪ 1 ) because it saturates exponentially while other parameters saturate sigmoidally. Considering that S is always the largest term and c saturate to S quickly, we notice that the saturation is in the order of α 2 ( ≈ 2 S +2 c ≈ 4 S ), β 2 ( ≈ -S +5 c ≈ 4 S ), and γ 2 ( ≈ 2 S ). We observe that our crude approximation holds in Fig. 8(b): the first ( α ) and the second ( β ) modes saturate at similar timescale, while the third mode ( γ ) requires approximately twice the time for saturation.

H Details of the multilinear model

The multilinear model (Fig. 9(a)) has two identifying properties: 1) the layerwise structure and 2) g k as the basis functions. In this section, we discuss the role of each property in more detail.

...

(a) Multilinear model illustration

0

25

50

75

100

T

0.0

0.2

0.4

0.6

0.8

1.0

R

k/S

d

1

/D

=0

.

5

d

2

/D

=0

.

2

d

3

/D

=0

.

1

(b) Decoupled dynamics

Multilinearity. The product of two parameters ( a k b k ) creates the layerwise structure (Fig. 9(a)) that gives rise to the emerging dynamics (sudden saturation or sigmoidal growth) in Fig. 9(b). The time emergence of NN is well-described by the sigmoidal dynamics (Fig. 1(a)); a non-sigmoidal saturation dynamics, for example, that of linear models (Fig. 10(a)), would inadequately describe the time emergence. Such dynamics have first been studied by Saxe et al. [19] (See Appendix B.2 for an overview).

Assuming a sufficiently fast decay of d k for the skills, the sigmoidal growth results in a stage-like training (Appendix D) where one skill fully saturates before the next skill emerges. In Appendix D, we discuss how the stage-like training can describe the quanta model [18] and how NNs, without explicit g k s, decouple each skill.

Finally, note that even though sigmoidal saturation has a resemblance to the test accuracy in grokking [48], our model is irrelevant to grokking because R k - which is defined over the expectation over the k th skill (Eq. (7)) - appears both in the empirical loss (Eq. (11)) and the test loss: failing to describe the discrepancy between train and test accuracy in grokking.

Connection to linear models. In Section 4 and Appendix E, we have shown how the scaling laws follow from the basis functions g k that decouples the loss. To analyze the role of g k , we can ask whether a simpler linear model with g k as basis functions (Eq. (124)) also recovers the scaling laws. The answer is yes and we outline how a linear model can recover all scaling laws. In addition, we also outline how extended linear models - extended similar to Section 5 such that skills are decoupled - can recover all emergence behaviors shown in Appendix F except the time emergence.

By replacing a k b k with w k , we obtain the linear model with skill basis functions:

The dynamics of the linear model under gradient flow is

where we assumed w k (0) = 0 . The linear model follows an exponential saturation of the skill strength in contrast to the sigmoidal saturation of the multilinear model (Fig. 10).

0

50

100

T

0.0

0.5

1.0

R

k/S

(a)

Linear

0

50

100

T

0.0

0.5

1.0

R

k/S

(b)

Multi

Linear

d

1

/D

=0

.

5

d

2

/D

=0

.

2

d

3

/D

=0

.

1

Nevertheless, the linear model Eq. (125) results in the same scaling laws in Section 4. For the time scaling law, we recover the relationship between d L k /dT and d L k /dk in Appendix E.1 because R k ( T ) is a function of d k D T only (where d k /D = P s ( k ) for D → ∞ ). For the data scaling law, we recover Corollary 1 because each w k (i.e. R k ) is decoupled. For the parameter scaling law, we recover Corollary 2 trivially as the linear model shares the same basis functions.

The data and parameter emergence in Section 5 can be obtained from the linear model in Eq. (124) if we extend the model analogous to Eqs. (15) and (18). For example, we can extend the model for data emergence as

where the matrix W ∈ R N × D c is an extension of w ∈ R N in Eq. (124), D c is a fixed scalar, and e k,l ( i, x ) : { 0 , 1 } n s + n b → R are functions with the following properties:

̸

The equivalence can be shown by Lemma 2 which states that the multilinear model finds the minimum norm solution: the solution that the linear model finds in a ridgeless regression setup.

Thus, for our setup, the basis functions play a critical role in the scaling laws and data/parameter emergences. The choice of basis functions, also known as the task-model alignment (see [23, 27]), determines the linear model's scaling laws and emergence behaviors. See Bordelon et al. [21] for a study of the scaling laws in linear models.

I Additional plots and tables

0

1000

2000

T

0.0

0.2

0.4

0.6

0.8

1.0

Skill

strength

R

k/S

(a)

Time

calibration

1000

2000

D

0.0

0.2

0.4

0.6

0.8

1.0

Skill

strength

R

k/S

(b)

Data

calibration

0

5

10

N

0.0

0.2

0.4

0.6

0.8

1.0

Skill

strength

R

k/S

(c)

Parameter

calibration

NN

extended model

NN

extended model

k

= 1

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

0.98

k

= 2

4.5

0.95

0.95

0.95

0.96

0.96

0.04

0.96

0.96

0.95

k

= 3

0.6

0.0

0.72

0.90

0.92

0.64

0.88

0.8

0.58

0.52

k

= 4

0.0

0.78

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

k

= 5

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

1.0

0.8

S

0.6

S

S

S

S

R

0.4

R

0.2

0.0

2400

2600

2800

0000

25000

26000

27000

28000

44000

45000

46000

47000

48000

00008

82000

120000

125000

130000

135000

140000

T

T

T

T

T

J Rigorous derivation of the scaling laws

In Appendix E, we discussed the scaling laws in simplified settings, favoring intuition over mathematical rigor. Building upon the intuitive understanding developed in Appendix E, we now turn our attention to a rigorous analysis of the scaling laws. In this section, we will derive general scaling laws by considering a comprehensive set of parameters and variables. Our goal is to establish the conditions under which these scaling laws hold and to quantify the associated error terms. By explicitly analyzing the error terms, this section aims to provide a rigorous assessment of the validity and limitations of our scaling law estimates.

J.1 General set up, repeated

We go back to the most general settings possible. Our starting point is Eq. (27), which describes the dynamics of R k and L k valid for k ≤ N :

We do not use skills for indices k > N in our model, but we can still denote

For P s ( k ) = Ak -α -1 , the total loss is given as

When n s , N, T are all set, their dependency with the data is only determined by the statistics d k , the number of data with i ( j ) = k . We assumed that ( i, x ) ∈ I ×{ 0 , 1 } n d was collected as random samples with i following the Zipfian distribution of size n s and exponent α + 1 , or equivalently P ( i = k ) = P s ( k ) = Ak -α -1 for 1 ≤ k ≤ n s . Then ( d 1 , · · · , d n s ) is a vector denoting the number of occurrences in D independent sampling from that distribution. It follows that d i follows binomial distribution B ( D, P s ( k )) .

In this complete perspective, our loss is dependent on all of those parameters and variables

where R init = ( R 1 (0) , · · · , R N (0)) denotes the vector representing initial condition. We will also simply denote r k = R k (0) . We will not assume much on r k , but we absolutely need 0 < r k < S for dynamics to hold, and we also should have

We will not impose any particular distribution on R init . Instead, we will try to identify sufficient conditions on r k for our desired result to hold, and those conditions will differ by the situation we are considering. For example, in Theorems 2 and 3 where we prove time scaling law L = Θ( T -α/ ( α +1) ) for large enough D and bottleneck T , we only require ϵ < r k < S/ 2 for some ϵ > 0 . However, the exact constant depends on the distribution of r k , and figuring out the explicit constant seems to be only feasible when we fix r k = r as in Theorem 4.

J.2 Estimates for large D

We will first consider the situation where D becomes the 'large resource' so that its effect on the loss function is negligible. The number of data d k follows binomial distribution B ( D, P s ( k )) , so d k /D converges to P s ( k ) for large enough D . So taking the limit of L when we let D →∞ has the effect of replacing d k /D by P s ( k ) in the expression of L . We will establish an explicit inequality comparing the difference between L and this limit.

Lemma 4. For a function F : R → R with its total variation V ( F ) bounded, we have

where N ( µ, σ 2 ) denotes normal distribution of mean µ and variance σ 2 .

Proof This is just an application of the Berry-Esseen inequality (with constant 1 , see [49] for modern treatment) applied to d k following binomial distribution B ( D, P s ( k )) . ■

Lemma 5. Let F : R → R be a C 2 function such that F ′′ is bounded. Then we have

Proof First, we apply Taylor's theorem to show that

Taking expectation when z follows normal distribution N ( P s ( k ) , P s ( k )(1 -P s ( k )) D ) gives

■

Proposition 3. We have

Proof Consider the function F : R → R given as

This function is monotone decreasing and C 2 on the whole domain, and its supremum and infimum are given as

This implies that where

The constant on the O term only depends on α .

Also, we will show that F ′′ is globally bounded. We first calculate

We consider the following inequalities

to show that

for all z . Thus we can apply both Lemma 4 and Lemma 5 to this function F and we have

where the last line follows from that we always have

■

̸

Lemma 6. For any integer N and σ ≥ 1 / 2 and σ = 1 , we have

where ζ is the Riemann zeta function (defined over the whole complex plane except 1 via analytic continuation). In addition,

where γ = 0 . 5772156649 ... is Euler's constant.

Proof

See Corollary 1.15 of [50], or other analytic number theory textbooks. ■

Proposition 4. (Large D approximation) We have

Proof From the description of L in Eq. (129), we have

We apply Proposition 3 to give

Each of these sum involving P s ( k ) is bounded as

which follows from Lemma 6. Combining those two gives

■

While Proposition 4 holds for any D , it becomes only meaningful if the resulting error terms are less than the main term we desire. We will revisit this when the exact main term is found, and determine the sufficient size of D for error terms to become small enough.

J.3 Estimates for not too small n s

We next discuss the effect of n s . When n s →∞ heuristically, then intuitively we have P s ( k ) → k -( α +1) /ζ ( α +1) . We will discuss the difference between when we regard n s as ∞ and when we do not.

Proposition 5. The following equations hold:

All implied constants on O only depend on α .

and

Proof The first statement Eq. (160) follows from substituting σ = α +1 in Lemma 6. As P s ( k ) = Ak -( α +1) , the second statement Eq. (161) immediately follows. If we substitute n s = N into Eq. (160) and calculate differences between them, we obtain

Thus we have

Regardless of the size of n s , We always have

so the third statement Eq. (162) follows.

We go back to the description of total loss given in Eq. (129) as

and we take its expectation in D . Proposition 4 suggests that its limit when D →∞ is given as

Denote

■

We discuss the effect of n s in L 1 and L 2 , by comparing limit of L 1 and L 2 when n s →∞ and their original values.

For the term L 1 , the change of letting n s as finite value from n s → ∞ has effect of multiplying T by 1+ n -α s / ( αζ ( α +1)) , and multiplying whole L 1 by 1+ n -α s / ( αζ ( α +1)) . It can be equivalently put as

We always have n s > N and N → ∞ eventually, so if dependency of L 1 with respect to T is at most polynomial order, then change of main term of L 1 is negligible. We can't establish exact statements yet without the descriptions of size of L 1 .

The term L 2 only depends on N and n s , not on T . Applying Proposition 5 (especially Eq. (162)) gives

When n s grows faster than N then n -α s part is totally negligible, and when n s has same order as N then n -α s affects the constant for main term of L 2 . Things might get little complicated when n s = N + o ( N ) , where N -α -n -α s = o ( N -α ) can happen then.

where and

Comparing size of L 1 and L 2 mainly depends on time. The term L 2 is fixed, and L 1 decreases as T increases. For T = ∞ we have L 1 = 0 , so L 2 having order N -α dominates (this proves scaling law for N of exponent α ), so restriction on n s becomes quite substantial. For small T and large N where the size of L 2 is small, we can expect the restriction on n s to be less substantial. For example, in the extreme case N = ∞ , we have L 2 = 0 , and n s does not matter at all (except that, of course, it should satisfy n s ≥ N ).

For such reasons, it is hard to quantify exact conditions for n s such that error terms are controlled, unless we specify relative growth of ( N,T ) . However, n s = ω ( N ) suffices to assure that setting n s = ∞ has zero effect on the main term. We will not worry about n s in this setting anymore too, and come back to this at the very end to determine enough n s .

J.4 Estimating main terms

We assume D = ∞ and n s = ∞ - virtually implying that d k /D = P s ( k ) and P s ( k ) = k -α -1 /ζ ( α +1) (calculated by rule of n s = ∞ ). We decomposed our main term into

By Proposition 5, L 2 is determined almost completely as

Now focus on L 1 . For

(note: it really depends on r k so it is correct to write F k , but for convenience we will keep using F .) one can express L 1 as

Lemma 7. Let F ( z ) be defined as Eq. (175) .

(Estimate for large z ) We have

(Estimate for small z ) For z ≥ 0 , we have

The left side is obvious. For the right side, F ( z ) ≤ ( S -r k ) 2 / 2 follows from noting that F (0) = ( S -r k ) 2 2 and proving F ′ ( z ) ≤ 0 , and F ( z ) ≤ ( S -r k ) 2 2 S 2 r 2 k e -4 ηSTz follows from just replacing 1 + ( S r k -1 ) -1 e 2 ηSTz in the denominator of F by ( S r k -1 ) -1 e 2 ηSTz .

Proof

1.

2. For the left side, it suffices to show -F ′ ( z ) ≤ 8 ηS 3 T 27 . One can calculate

and

so F has unique inflection point at

and this point is where -F ′ ( z ) obtains maximum. Substituting this to the expression of F ′ ( z ) gives -F ′ ( z ) = 8 ηS 3 T 27 .

■

Our threshold for distinguishing two approximation methods will be set as z = z 0 = ( ζ ( α + 1) ηST ) -1 , where both two error terms are bounded by O ( S 2 ) . The constant ζ ( α +1) is set to make later calculations much easier. Applying Lemma 7 gives

Denote

Proposition 6. Suppose that there exists 0 < r < √ S such that r ≤ r k < S/ 2 for all k . In the decomposition of

given as above, we have the following bound.

2. If ( ηST ) 1 / ( α +1) < N , then

Here all constants in O and Θ terms are absolute with respect to η, S, T, N . (They may depend on α .)

Proof We first note that the condition P s ( k ) < z 0 = ( ζ ( α +1) ηST ) -1 is equivalent to

Thus we can rephrase the descriptions of terms as

Applying Proposition 5 easily shows that

For M and E 1 , we will consider them by dividing two cases depending on whether ( ηST ) 1 / ( α +1) > N or ( ηST ) 1 / ( α +1) < N . If ( ηST ) 1 / ( α +1) > N , then the condition ( ηST ) 1 / ( α +1) < k ≤ N is never satisfied, so M = E 1 = 0 . Now suppose ( ηST ) 1 / ( α +1) < N . We first note that

As ( S -r k ) 2 = Θ( S 2 ) , we can let

and using Proposition 5 gives the desired estimate L 2 + M = Θ( S 2 ( ηST ) -α/ ( α +1) ) . For E 1 , estimating sum of P s ( k ) 2 using Lemma 6 gives

For E 2 we always have

regardless of the size of N , so it suffices to bound this sum. If we denote l = ( ηST ) 1 / ( α +1) and define

it suffices to show the bound

We will approximate this sum as

to obtain the form of Riemann sum approximation for the integral of

at P s ( l ) < P s ( l -1) < · · · < P s (1) . As F 2 ( z ) is decreasing function, this Riemann sum is always less than the integral, so we obtain

We note that P s ( l ) = ( ζ ( α +1) ηST ) -1 . The threshold for F 2 ( z ) to become 1 is given at

As r < √ S , this value is always greater than P s ( l ) . Thus we can divide our integral as

The first part is bounded by

which can be shown to be O ( (log( S/r )) α/ ( α +1) ( ηST ) -α/ ( α +1) ) . For the second part, we apply substitution of w = 4 ηSTz to show

and applying the asymptotic Γ( s, x ) = O ( x s -1 e -x ) suggests that this is bounded by

■

Theorem 1. (Parameter scaling law) Assume the following conditions: n s > N with lim( N/n s ) = γ < 1 ( γ can be zero), and there exists 0 < r < √ S such that r < R k (0) < S/ 2 for all k . If N,T → ∞ while satisfying N α +1 = o ( T ) , the expected loss E D [ L ] for all datasets D of size D satisfies

where

The constant on the O term only depends on α . When D ≫ T 3 , then all the error terms involving D are negligible.

Proof In the situation n s = ∞ and D = ∞ , Proposition 6 shows that

We consider the effect of n s first. As L 1 becomes an error term in this estimation, letting n s as a finite value has no effect on overall estimation. The term L 2 accounts for the main term, and letting n s as finite value changes it to

This accounts for the factor (1 -γ α ) on the main term and O ( N -min( α +1 , 2 α ) S 2 ) added to the error term. The effect of D is exactly described in Proposition 4, contributing the error term of O ( S 2 D -1 / 2 f α ( N ) + S 4 η 2 T 2 D -1 ) . Regarding the sufficient condition for D , if D ≫ T 3 then we have

so all error terms involving D are less than O ( T -α/ ( α +1) ) .

■

For the situation T = O ( N α +1 ) however, the error terms E 1 and E 2 are of same size, so we can only say that the main term is of O ( S 2 ( ηST ) -α/ ( α +1) ) .

Theorem 2. (Upper bound for the time scaling law) Assume the following conditions: n s > N , and there exists there exists 0 < r < √ S such that r < R k (0) < S/ 2 for all k . If N,T → ∞ while satisfying ηST = O ( N α +1 ) , the expected loss E D [ L ] is

with constant on O only depending on α and lim sup(( ηST ) 1 / ( α +1) /N ) , with f α defined as in Theorem 1. If D ≫ NT 2 and D ≫ T 3 , then all the error terms involving D are negligible.

Proof The error term regarding D can be obtained in the same way as Theorem 1, so we will let D = ∞ for the rest of the proof. Also, we can let n s = ∞ , as we observed that it contributes at most to the constant factor of the upper bound and does not change the scaling.

In the decomposition of Proposition 6, we always have

and holding regardless of N , so it only remains to consider L 2 + M . If ( ηST ) 1 / ( α +1) < N , then L 2 + M is of size O ( S 2 ( ηST ) -α/ ( α +1) ) . If ( ηST ) 1 / ( α +1) ≥ N , then N and ( ηST ) 1 / ( α +1) has same order, so L 2 + M = L 2 = Θ( S 2 N -α ) is O ( S 2 ( ηST ) -α/ ( α +1) ) . Thus in either cases we have the desired bound. ■

Theorem 3. (Lower bound for the time scaling law) Assume the following conditions: n s > N and 0 < R k (0) < S/ 2 . If N,T →∞ while satisfying (8 ζ ( α +1) -1 ηST ) 1 / ( α +1) < N , the expected loss E D [ L ] is

for κ and constant on O only depending on α , with f α defined as in Theorem 1. If D ≫ NT 2 and D ≫ T 3 , then all the error terms involving D are negligible.

Proof The error term regarding D can be obtained in the same way as Theorem 1, so we will let D = ∞ for the rest of the proof. We only show the lower bound for L 1 , holding regardless of N and n s . In Lemma 7 (Eq. (178)) we have

for z ≥ 0 , so if z ≤ (4 ηST ) -1 then F ( z ) ≥ S 2 / 8 -2 S 2 / 27 > S 2 / 20 . The condition P s ( k ) ≤ (4 ηST ) -1 is equivalent to that k ≥ (4 ζ ( α + 1) -1 ηST ) 1 / ( α +1) . In evaluating L 1 = ∑ N k =1 P s ( k ) F ( P s ( k )) , we will only add over k in range of

From the assumption, this interval sits inside 1 < k < N . For such k we use upper bound of F ( P s ( k )) > S 2 / 20 . Then by using Proposition 5 we can obtain

The possible effect of n s on the main term is to multiply both the main term by and T by (1 + n -α s ) , so it increases the bound. ■

The condition (8 ζ ( α +1) -1 ηST ) 1 / ( α +1) < N is not absolutely necessary for lower bound. The condition ( ηST ) 1 / ( α +1) = Θ( N ) and n s ≥ 2 N would suffice and one can formulate a similar theorem, although the constant of lower bound might be much smaller if ( ηST ) 1 / ( α +1) /N is small.

Lastly, we provide a simpler version of those results combined and discuss the special case where the optimal compute C = NT , or the given engineering budget, is specified.

Corollary 3. (Summary of the large data estimation) Assuming D ≫ NT 2 , T 3 and n s ≫ N 1+ ϵ such that effects of n s and D are negligible, then for N,T →∞ we have

where Θ η,S,r denotes that the implied constant depends on η, S, α and r = min R k (0) > 0 . In particular, we have

and

Proof Apply Theorem 1 if N α +1 = o ( T ) and Theorem 2 and Theorem 3 if N α +1 ≫ T . ■ Corollary 4. (The 'computationally optimal' case) Denote C = NT and assume the conditions in Corollary 3. Then we have

When N = Θ( C 1 / ( α +2) ) and T = Θ( C ( α +1) / ( α +2) ) , we achieve E D [ L ] = Θ( C -α/ ( α +2) ) . (Its implied constant may depend on implied constant for growth of N and T .)

Proof The first part follows from

and

The second part can be checked by substituting ( N,T ) = ( C 1 / ( α +2) , C ( α +1) / ( α +2) ) (or their constant multiples) to Corollary 3. ■

J.5 Computing the constant for time scaling law

While we have found the time scaling law E [ L ] = O ( T -α/ ( α +1) ) holding for T = O ( N α +1 ) , bounds in Theorem 2 and Theorem 3 were chosen rather lazily and do not depict the correct picture. We will find the constant using a more refined estimation, but we require additional assumptions on parameters. We will focus on the setting where D and n s are large enough to be negligible, R k (0) = r is fixed, and T = O ( N α +1 ) with fixed constant such that time scaling law holds.

Theorem 4. (Constant for time scaling law) Denote L ∞ as the loss when D,n s →∞ so that their effect is negligible:

When T, N →∞ and lim N/ ( ηST ) 1 / ( α +1) = λ for a fixed constant λ ∈ (0 , ∞ ] , the following limit exists:

The prefactor constant A as the a function of λ (when λ = ∞ then let λ -α = λ -( α +1) = 0 ) is

where

Proof We first observe

We will seek to convert it into Riemann sum form of certain integral. We start by noting that

Denote u k = ηST P s ( k ) , then the sum can be approximated to

if we ignore small k . As Φ S,r is decreasing, this corresponds to Riemann sum taking minimum in the interval [ u k +1 , u k ] . So integral provides an upper bound for this sum. Similarly, we can approximate it with Riemann sum taking maximum in [ u k , u k -1 ] if we use

instead. As Φ S,r shows exponential decay, we can ignore values at small k , so this shows

and from that

we obtain our desired result.

Theorem 4 basically tells that for N = λ ( ηST ) 1 / ( α +1) and D,n s large enough, we have

with A ( λ ) given as Eq. (238), thus specifying the constant for time scaling law. For finite λ , this theorem covers the computationally optimal case of ( N,T ) = ( λ 1 C 1 / ( α +2) , λ 2 C ( α +1) / ( α +2) ) for some nonzero constant λ 1 , λ 2 . For λ = ∞ , it describes the case T = o ( N α +1 ) where effect of N is negligible.

Corollary 5. Denote L ∞ as L ∞ as the loss when D,n s →∞ same as Eq. (236) . Denote C = NT and suppose that

for a fixed constant 0 < λ < ∞ . Then as C →∞ , we have

where A is given as Eq. (238) of Theorem 4.

Proof As lim N/ ( ηST ) 1 / ( α +1) = λ ( α +2) / ( α +1) under above conditions, we can apply Theorem 4 and substituting Eq. (250) into Eq. (249) gives the desired result. ■

Technically we can optimize L ∞ for a given fixed value of C = NT by letting λ as argument of minimum of A ( λ ( α +2) / ( α +1) ) λ -α/ ( α +1) , although it seems almost impossible to obtain any form of formula for such λ .

Lastly, we provide the following estimate for the time scale constant ( A ( λ ) ) when r is small, especially the first term in Eq. (238).

Proposition 7. As r → 0 , we have ( Λ > 0 fixed)

Proof Denote M = ( S r -1) , and replace u by (log M ) v . Then we have

As M →∞ , the integrand converges to

The integrand is bounded by v -1 / ( α +1) if v ≤ 1 / 2 and v -1 / ( α +1) e -2(2 v -1) if v > 1 / 2 , those of which are all integrable. So we can apply Lebesgue's dominated convergence theorem to show

Thus we have

which can be observed to be equivalent to the desired expression of Eq. (252).

■

J.6 Estimates for large T and threshold between data/parameter scaling

The estimates for small D require different techniques from estimates for large D . We will consider the situation T grows much faster than D and N , and discuss when data scaling law of L = Θ( D -α/ ( α +1) ) happens. We will consider a simpler setting of ' n s = ∞ ' or equivalently that effects of n s are negligible ( n s = ω ( N ) seems to suffice) and R k (0) = r < S is fixed, although it won't be impossible to discuss their subtle effects.

First we single out effect of T by comparing L ( T ) and L ( ∞ ) . We remind

and its limit when T →∞ is given as

Proposition 8. Suppose that R k (0) = r < S is fixed. For large T , we have

Proof As L k ( T ) is decreasing in T , we always have L k ( T ) ≥ L k ( ∞ ) so therefore

So we only need to establish an upper bound for L k ( T ) -L k ( ∞ ) . We note that L k ( T ) -L k ( ∞ ) when d k = 0 , so one can write

where 1 d k > 0 denotes the characteristic function

We use simple bound of

As d k follows binomial distribution B ( D, P s ( k )) , considering its moment generating function gives

so thus

Meanwhile, for 0 ≤ u, v ≤ 1 real numbers, we have

so, applying this inequality to above gives

Thus, we can deduce

and thus

■

This provides an almost complete account for the effect of very large T . We will let T = ∞ from this point. We have

Applying Lemma 6 gives

Meanwhile, by noting that

so it suffices to focus on the first sum. We will divide the range of k into two 1 ≤ k ≤ M and M < k ≤ N . For the sum over 1 ≤ k ≤ M , we will apply the following simple bound (in the last part, we used 1 -x ≤ e -x )

For the sum over M < k ≤ N , we will approximate the sum into some integral, which happens to be incomplete gamma function.

Proposition 9. For 2 < M < N integers, we have

Here Γ denotes the incomplete gamma function

Proof Consider the interval [ P s ( N ) , P s ( M )] and its partition P = {P s ( N ) < P s ( N -1) < · · · < P s ( M ) } . For a function f ( x ) = x -1 / ( α +1) (1 -x ) D , we will consider its upper and lower Darboux sums with respect to P . As f is decreasing in (0 , 1] , its upper and lower Darboux sums are given respectively as

and those give bound of the integral of f as

one can show

Applying a similar argument for upper Darboux sum gives

and from Eq. (283) it follows

From now we will estimate the integral

We replace x = y/D in the integral inside, then it becomes

We want to approximate ( 1 -y D ) D by e -y , so we will estimate difference between them. We have

so if D > 2 y then

so

where we used the inequality 1 -x ≤ e -x . As P s ( M ) < 1 / 2 if M > 2 (obvious from P s ( M ) < ( P s (1) + P s (2)) / 2 < 1 / 2 ), any y in the interval [ D P s ( N ) , D P s ( M )] satisfies D > 2 y . So, we can apply this approximation in every y . It follows that

Combining this with Eq. (289) and Eq. (291) gives the desired result.

We combine Proposition 8 and Proposition 9 together to obtain this final estimation result.

■

Theorem 5. (Scaling laws for large time estimation) Suppose that N,D →∞ and n s ≫ N 1+ ϵ for some ϵ > 0 so that effect of n s is negligible. Suppose that R k (0) = r for all 1 ≤ k ≤ N .

1. (Parameter scaling law) If N = o ( D 1 / ( α +1) ) , then we have

2. (Data scaling law) If D = O ( N α +1 ) and µ = lim( D/N α +1 ) exists (it can be zero), then

Here Γ denotes the incomplete gamma function

In particular, if D = o ( N α +1 ) such that µ = 0 , we have

1+ ϵ

In either case, T ≫ D (log D ) for some ϵ > 0 implies that error terms involving T are negligible. Proof Proposition 8 states

and we showed

and

For the sum of P s ( k )(1 -P s ( k )) D over 1 ≤ k ≤ N , we use the estimate (see Eq. (276)) of

and the estimate of Proposition 9. Combining all those gives

We will prove our main statement by choosing appropriate M depending on size comparison between D and N .

and

Proof Just summarize the results of Theorem 5.

■

If N = o ( D 1 / ( α +1) ) , then we let M = 3 , and also regard all incomplete gamma function values as O (1) . Then it follows

and thus obtaining the parameter scaling law.

Suppose D = O ( N α +1 ) and µ = lim( D/N α +1 ) exists. We want

to be our main term, and set M < N such that the term

and error terms not depending on T given as

are all bounded by O ( D -(2 α +1) / (2 α +2) ) . Set M = D 1 / (2 α +2) . Then P s ( M ) = D -1 / 2 /ζ ( α +1) , so applying the asymptotic Γ( s, x ) = O ( x s -1 e -x ) gives

This term and e -P s ( M ) D = e - √ D/ζ ( α +1) are less than D -α/ ( α +1) M -1 = O ( D -(2 α +1) / (2 α +2) ) , and obviously D -(2 α +1) / ( α +1) is less than D -(2 α +1) / (2 α +2) . Thus it follows that

Regarding the final statement regarding sufficient condition for large T , T ≫ D (log D ) 1+ ϵ implies

for any K > 0 , showing that the error term O ( S 4 r -2 De -4 ηST/D ) is negligible compared to all other error terms of Eq. (299) and Eq. (300). ■

We also provide a summary of all large time estimation results.

Corollary 6. (Summary of large time estimation) Assuming T ≫ D (log D ) 1+ ϵ and n s ≫ N 1+ ϵ such that effects of n s and T are negligible, and R k (0) = r for all 1 ≤ k ≤ N . Then for D,N →∞ , we have

where Θ η,S,r denotes that the implied constant depends on η, S, r and α . In particular, we have

K Methods

In this section, we present the methods used in our experiments.

K.1 2-layer MLP

We trained a 2-layer fully connected neural network (MLP) with ReLU activations. All parameters of the MLP were initialized with a Gaussian distribution with a standard deviation of 0 . 001 . The input dimension of the model was n s + n b = 5 + 32 where n s is the length of control bits (number of skills) and n b is the length of the skill bits. Each skill has m = 3 mutually exclusive sparse bits that are used to express the skill function. The target scale was S = 5 . The model was trained with SGD without momentum and no weight decay (the exception is the parameter emergence experiment where Adam with learning rate 0 . 001 and weight decay of 5 × 10 -5 was used to escape the local minima). 6 For the data emergence experiment, the learning rate was halved every 50 , 000 step.

The skill strength R k ( T ) (Eq. (7)) was measured using 20 , 000 i.i.d samples from the k th skill. 7 For the time emergence, the skill strengths were measured every 50 steps, while for other experiments, they were measured after training. To mimic the infinite parameter N →∞ , we used the model of width 1000 (for the hidden layer). To mimic the infinite time T →∞ , we trained for 5 × 10 5 steps ( 3 × 10 4 steps for time emergence) where each step had the batch size of 4000 ( 2000 for the data emergence experiment). To mimic D →∞ , we sampled new data points for every batch. The details are given in the following table.

K.2 Transformer

This section outlines the transformer architecture used in Fig. 4. Data is encoded as for the 2-layer MLP, but with one-hot positional encoding appended to the data. We use a basic decoder transformer with 1 block, an initial embedding layer with output dimension 512, and a final linear layer. For the attention mechanism, we used 4 attention heads. For non-linearity, we used ReLU. A batch size of 5000 was used with a target scale S = 1 and default Pytorch initialization. The model was trained with SGD with a learning rate of 5 × 10 -5 , weight decay of 10 -5 , and momentum with β = 0 . 9 . At every 100 steps, the skill strength R k ( T ) (Eq. (7)) was measured using 20 , 000 i.i.d samples from the k th skill.

K.3 Measurement of skill strength

The skill strength R k is a simple linear correlation between the learned function f - function expressed by NN - and g k for P b given I = k . We approximate the expectation over X by taking the mean over 20 , 000 i.i.d samples from P b for the k th skill:

where the notation x ( j ) denotes the j th sample.

K.4 Details of the scaling law experiment

For the loss of the model (solid lines) in Fig. 2, we used the analytic equation for the model (Eq. (12)) under suitable assumptions such as sufficiently large n s (Table 2). For the scaling laws (dotted lines) in Fig. 2, we used the exponents from Appendix E or Appendix J and the prefactor constants from Theorem 4 (time scaling law), Theorem 5 (data scaling law), and Theorem 1 (parameter scaling law). For the hyperparameters of the simulation, we used n s = 10 5 such that n s is large compared to other resources; S = 1 and R k (0) = 0 . 01 such that S -R k (0) ≈ S ; and η = 1 .

Time scaling law. The total loss as a function of T for D,N →∞ (Fig. 2(a), solid) is

which follows by taking D →∞ and N = n s on Eq. (12). The scaling law (Fig. 2(a), dotted) is

where the exponent is derived in Appendix E.1 or Theorems 2 and 3. The prefactor constant is

which we obtained by taking D →∞ on Eq. (238).

Data scaling law. The total loss as a function of D when N,T →∞ (Fig. 2(b), solid) is

which follows from Eq. (58). The scaling law (Fig. 2(b), dotted) is

where the exponent follows from Appendix E.2 or Theorem 5. The prefactor constant is

which we obtained by taking N →∞ in Eq. (302).

Parameter scaling law. The total loss as a function of N when T, D →∞ (Fig. 2(c), solid) is

which follows from taking T, D →∞ on Eq. (12). The scaling law (Fig. 2(c), dotted) is

where the exponent follows from Theorem 1. The prefactor constant is

which we obtained by taking D,T →∞ , N/n s → 0 , and ζ ( α +1) ≈ α -1 in Eq. (217).

Compute scaling law. The total loss as a function of T and N for D →∞ (Fig. 3, solid) is

which follows by taking D → ∞ in Eq. (12). In Fig. 3, we plotted for N ∈ { 10 , 20 , 50 , 70 , 100 , 200 , 500 , 700 , 1000 , 2000 , 5000 , 10000 } and T ∈ [1 , 1000] as examples of different tradeoff between T and N for fixed C .

The scaling law (Fig. 3, dotted) is

where the exponent is derived in Appendix E.4 or Corollary 4. Using Corollary 5, the prefactor constant is

where A : R → R is defined in Eq. (238). We used the minimum value of A c for λ ∈ (0 , ∞ ] .

K.5 Estimates of the compute use

On CPU, our emergence experiments on the 2-layer MLP (Fig. 1) take 2 ∼ 5 hours for a single run of time emergence experiments and 20 ∼ 40 hours for a single run of other experiments depending on the CPU. All experiments were repeated 10 times (except for parameter emergence where we repeated the experiment 50 times). Each experiment requires memory of at most 5 GB. The CPU cluster in which we experimented contained the following CPUs: Intel(R) Core(TM) i5-7500, i79700K, i7-8700; and Intel(R) Xeon(R) Silver 4214R, Gold 5220R, Silver 4310, Gold 6226R, E5-2650 v2, E5-2660 v3, E5-2640 v4, Gold 5120, Gold 6132. The transformer experiment (Fig. 4) takes 48 ∼ 72 hours for each run; we used an RTX4090 with 24GB RAM, with 1 CPU from the list above.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims in the abstract and introduction accurately reflect the paper's contributions, as evidenced by the contribution list in the introduction section, which references the sections presenting each result.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The paper discusses the limitations of the multilinear model in Section 5.5 and the general limitations regarding the assumptions about the framework in Section 6 (Discussion and Conclusion). These sections address the robustness of the results and the scope of the claims made.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All theoretical results are accompanied by intuitive explanations in the main text, with detailed derivations (Appendices C and E) and rigorous proofs (Appendix J) in the supplemental material. In addition, an alternative derivation of the scaling laws via stage-like training is given in Appendix D.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided details - architecture, initialization, learning setup, and measurements - of our experiments in Appendix K including the details of NN for emergence experiment (Fig. 1) in Appendix K.1, the details of scaling law experiment (Figs. 2 and 3) in Appendix K.4, the details of the transformer experiment (Fig. 4) in Appendix K.2.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide a link to our source code in the main text.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We specify the details in Appendix K.1 (2-layer NN) and Fig. 4 (transformer).

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Yes, all our figures in the main text, which are not simulations (Figs. 1 and 4), have 1-standard deviation error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the compute resource details in Appendix K.5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in this paper adheres to the principles outlined in the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This research aims to deepen the fundamental understanding of emergence phenomena and scaling laws in deep learning. As our theoretical and empirical investigations are conducted in a carefully controlled, idealized environment, we do not anticipate any immediate societal consequences arising directly from the findings of this particular study.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The research presented in this paper does not involve the release of data or models that pose a risk for misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use any existing assets from other sources.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve any crowdsourcing experiments or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No IRB approvals or equivalent reviews were required.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a model capable of predicting the appearance of emergent abilities, using only information on the emergence of the first ability. The model successfully predicts emergence in a 2-layer MLP solving the multitask sparse parity problem, a toy-model problem constructed with a power-law skill distribution, to mimic the theorized distribution of language datasets. The model also exhibits scaling laws known to exist in this problem.\n\nSTRENGTHS:\nThe paper expands on recent works, in a significant and active field of research. The authors present convincing results when evaluating the proposed emergence model against MLP training data. Extensive derivations are provided in the appendix, although I did not carefully check them. The paper is written in a clear manner and results are visualized in clear formats.\n\nWEAKNESSES:\n- The scope of the experiments is too small in my opinion, specifically Figure 1. It would be nice to see results for a lot more than 5 skills, since it seems possible that predicting the emergence of less-frequent skills becomes harder at higher orders of magnitude. My impression is that the experimental setup is light enough that scaling up will not require unreasonable resources.\n\n- The general scope of the paper is a bit small, focusing only on the multitask sparse parity problem. It would have been nice to see results on more natural settings. That said, the current results on the chosen problem setting are still interesting on their own.\n\n- Regarding the scaling laws presented in Table 2, glossing over the paper gives the impression that these are new scaling laws found by the authors, when in fact they are reproductions of the scaling laws calculated by Michaud et. al. It might help to clarify the relation to Michaud et. al. in the main section of the paper.\n\nQUESTIONS:\nThe main section of the paper does not explain why the input data is distributed as a power law (Zipf's law), making it hard to understand the justification for readers unfamiliar with previous literature, i.e. Michaud et. al. I would suggest adding a short explanation when introducing the problem.\n\nLIMITATIONS:\nLimitations are adequately discussed, including scope limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nI’ll write two summaries: one to state my “moral” understanding of the work and another to state the specific contributions \n\n**“Moral” Summary:** The authors propose an analytically solvable model to study scaling laws and emergent abilities by combining the problem of Barak et al 2022 + the data distributional assumptions of Michaud et al. 2023/2024 + the model of Saxe et al. 2024 + their own innovations.\n\n**Specific Summary:**\n\n- The authors study the multi-task sparse parity problem studied by Barak & Michaud\n- The authors propose a multilinear model that identifies each sparse parity problem as a “skill” and then define the learned function as a “multilinear” (i.e. two independent linear parameters) function of the “skill” basis functions\n- In this model, one can then exactly compute scaling and emergent abilities as a function of the typical scaling parameters (compute, data, parameters)\n- The authors also train 2 layer MLPs and transformers to test how closely their maths match empirical results\n\nSTRENGTHS:\n- Overall, I think this is a really well done paper (although I’m concerned about Figure 1 - see weaknesses). What it does, it does thoroughly.\n- Table 1 is useful in helping explain the multi-task parity problem\n- Table 2 is a great way to summarize and organize both the results as well as the conditions of each result\n\nWEAKNESSES:\n- Emergence is a phenomenon studied at scale, and while I strongly support trying to find simplified models of large-scale phenomena, I feel that there needs to be an attempt to connect back to the original phenomena of interest. For instance, Michaud et al. 2023/2024 (Citation 17 in this paper) - on which this work most closely connects - at least attempts to look into real language model pretraining data. In contrast, this paper makes no such attempt, which I think strongly limits it. This is the #1 reason I don't feel comfortable giving a higher score.\n- Following the above point, I consequently think a more focused title (e.g., “An exactly solvable model for emergence and scaling laws in the multitask sparse parity problem”) would better represent this paper.\n- For an exactly solvable model, Figure 1 shows that the predictions only roughly match the experimental results. This becomes especially pronounced for higher k. For instance, look at Figure 1(c) orange. The prediction is that there should be a rapid leap from 0 to 1, but instead, there is a sigmoid-like transition with long tapering tails. Green, red and purple are all similar. The inability to predict 2-layer MLPs makes me think that this analytically solvable model is already only an approximation of incredibly simple networks\n\nQUESTIONS:\nFigure 4: What is the timescale of each skill’s emergence for the transformer? I can’t tell if the higher k lines are step functions or sigmoidal functions that are compressed by the log scaling of the x axis. To be clear, I’m not asking for when the skills emerge, but how long it takes for each skill to emerge.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides an in-depth study of a toy dataset and model, both in terms of scaling laws and emergent skills. They study the ‘multitask sparse parity’ synthetic problem introduced by Michaud et al. (target is the parity function of a string of random bits, each task indicates a different subset of bit ids, task frequency follows a power law). New in this work, they consider a ‘multilinear’ model (i.e. $y=ab x$ rather than simply $y=cx$) to incorporate the dynamics of a two layer neural net (as per Saxe et al. in a different setting).\n\nThe paper recovers scaling laws (which agree with Michaud et al.’s coefficients for T, D & N, and additionally adds for C=TN). The model also gives rise to emergence of each skill with a sigmoidal shape. Rarer skills are learned later on.\n\nSTRENGTHS:\n- From a technical perspective, the paper is a very strong and complete piece of work. It exhaustively sweeps through results and surrounding analysis of the setup studied in the paper (even, for example, having proofs of the scaling laws in increasing resolution).\n\n- I have confidence in the claims and analysis. I spot checked several parts in depth, however, I have not been able to interrogate all claims and analysis in the paper (the appendix pushes the paper to 54 pages).\n\n- Providing a model that unites sigmoidal-shaped skill emergence with scaling laws, is a tantalizing prospect, as these are two major properties of LLMs. A theoretical model combining the two aspects would be of high interest to several parts of the community.\n\nWEAKNESSES:\n- The paper contains a huge amount of material. A lot of the good stuff is buried in the appendix (related work is important, the contrast with a linear $y=cx$ model, the scaling law proof sketches, the stage-like training discussion). The reading experience suffers from this, and a conference paper struggles to do it justice. It's possible it would be better suited to a long-form journal format (and would allow reviewers more time to comb through the details).\n\n- The scaling laws derived by the paper match the coefficients found in Michaud et al. (with the addition of $C$). There is a slight difference in that the setup now uses $y=abx$, but I generally felt that given the repeated outcome, providing scaling laws derived with varying resolutions need not be such a major focus of the paper and appendix (maybe some nuance has been lost on me?). Reducing this would free up bandwidth to allow readers to absorb the other more interesting parts.\n\n- A concern I have is in how realistic the setup of the dataset and model is. The emergence and scaling laws come from a lightweight two-parameter linear regressor which receives ideal task indicators as input. The closest interpretation in a real setting I can think of, is as a two-layer linear MLP head, placed on top of a deep pretrained model that is frozen with powerful representations for each task already learned. This paper is still valuable, but I might suggest a more scoped title because of this – ‘an exactly solvable model for emergence and scaling laws’ is not inaccurate, but might be a little broad.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nFine.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to use a certain generalization of the well-known sparse parity problem, as a theoretical framework for neural scaling laws.\n\nThe theoretical setup doesn't seem to make sense (I'm open to changing my mind). Indeed, equations (2) and (4) taken together give\n\\begin{equation*}\n    f^*(i,x) = S\\sum_{k=1}^{n_s} g_k(i,x) = S\\sum_{k=1} \\delta_{ki}g_i(i,x) = S g_i(i,x).\n\\end{equation*}\nThis is definitely not \"a sum of $n_s$ skills\" (as claimed by the authors); it is a single skill. The same issue repeats itself in (9) when the authors introduced their so-called multi-linear model. Indeed, that model simplifies to (again thanks to (2))\n\\begin{equation*}\n    f_T(i,x) = \\sum_{k=1}^{n_s} a_k(T)b_k(T)g_k(i,x) = \\ldots = a_i(T)b_i(T)g_i(i,x).\n\\end{equation*}\nFrom this point onward, it is not clear what the paper is trying to do. \n\nSecond, it is not clear what the paper is trying to achieve beyond what was already done in Michaud et al's \"Quantization Hypothesis\" paper (for the record, the paper proposed the multi-task sparse parity problem as a example exhibiting scaling laws w.r.t sample size and model size, within the framework of their quantization hypothesis).\n\nFinally, the paper seems to be missing some important literature, for example Cabannes et al. \"Scaling Laws for Associative Memories\" (ICLR 2024), which proposes finite-capacity extension of Hutter's model, and establishes an array of different scaling laws for different learning algorithms.\n\nSTRENGTHS:\nAs explained already, my low score is because I think the theoretical setup of the paper is unclear. I'm open to changing my mind.\n\nWEAKNESSES:\nAs explained already, my low score is because I think the theoretical setup of the paper is unclear. I'm open to changing my mind.\n\nQUESTIONS:\nIn what way do (3) and (9) represent \"a sum of skills\" ?\n\nLIMITATIONS:\nYes (in Section 5.4)", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2024-07-16"}