How Feature Learning Can Improve Neural Scaling Laws
Authors: Blake Bordelon
Alexander Atanasov
Cengiz Pehlevan
Venue: ICLR 2025 Spotlight (2025)
Keywords: neural scaling laws
feature learning
kernel methods
linear networks
mean field theory
Primary area: learning theory
TL;DR: A solveable model of neural scaling laws beyond the lazy training regime.

Abstract: We develop a simple solvable model of neural scaling laws beyond the kernel limit. Theoretical analysis of this model predicts the performance scaling predictions with model size, training time and total amount of available data. From the scaling analysis we identify three relevant regimes: hard tasks, easy tasks, and super easy tasks. For easy and super-easy target functions, which are in the Hilbert space (RKHS) of the initial infinite-width neural tangent kernel (NTK), there is no change in the scaling exponents between feature learning models and models in the kernel regime. For hard tasks, which we define as tasks outside of the RKHS of the initial NTK, we show analytically and empirically that feature learning can improve the scaling with training time and compute, approximately doubling the exponent for very hard tasks. This leads to a new compute optimal scaling law for hard tasks in the feature learning regime. We support our finding that feature learning improves the scaling law for hard tasks with experiments of nonlinear MLPs fitting functions with power-law Fourier spectra on the circle and CNNs learning vision tasks.

Published as a conference paper at ICLR 2025
How FEATURE LEARNING CAN IMPROVE NEURAL SCALING LAWS
Blake Bordelon', Alexander Atanasov", Cengiz Pehlevan
ABSTRACT
We develop solvable model of neural scaling laws beyond the kernel limit. The- oretical analysis of this model shows how performance scales with model size, training time, and the total amount of available data. We identify three scaling regimes corresponding to varying task difficulties: hard, easy, and super easy tasks. For easy and super-easy target functions, which lie in the reproducing kernel Hilbert space (RKHS) defined by the initial infinite-widih Neural Tangent Kernel (NTK), the scaling exponents remain unchanged between feature learning and kernel regime models. For hard tasks, defined as those outside the RKHS the initial NTK, we demonstrate both analytically and empirically that feature learning improve scaling with training time and compute, nearly doubling the exponent for hard tasks. This leads to a different compute optimal strategy to scale parameters and training time in the feature learning regime. We support our find- ing that feature learning improves the scaling law for hard tasks but not for easy and super-easy tasks with experiments of nonlinear MLPs fitting functions with power-law Fourier spectra on the circle and CNNs learning vision tasks.
INTRODUCTION
Deep learning models tend to improve in performance with model size, training time and total avail-
able data. The dependence of performance on the available statistical and computational resources
often regular and well-captured by power-law (Hestness al. [017 Kaplan et al.) 2020). For example, the Chinchilla scaling law (Hoffmann et al., 2022) for the loss N), of a N N-parameter model trained online for t steps (or tokens) follows
(t,, cet N-NN + Loo.
(1)
where the constants Ca. eN and exponents Ti, TN dataset and architecture dependent and Loo represents the lowest achievable loss for this architecture and dataset. These scaling laws enable in- telligent strategies to achieve performance under limited compute budgets (Hoffmann et al., (022) or limited data budgets (Muennighoff et al.] 2023). A better understanding of what properties of neural network architectures, parameterizations and data distributions give rise these neural scaling laws could be useful to select better initialization schemes, parameterizations, and optimizers (Yang et al.)
2021 Achiam al. 2023 Everett al. 2024) and develop better curricula and sampling strategies (Sorscher et al. 2022).
Despite significant empirical research, a predictive theory of scaling laws for deep neural network models is currently lacking. Several works have recovered data-dependent scaling laws from the
analysis of linear models (Spigler et al. 2020 Bordelon al., 2020 Bahri al.) 2021 Maloney
et al. 2022; (Simon et 20.1 Bordelon et al. 2024a]] Zavatone- Veth & Pehlevan 2023 Paquette
eet al. 2024 Lin et al. 2024). However these models are fundamentally limited to describing the
kernel or lazy learning regime of neural networks (Chizat et al., 2019). Several works have found
that this fails to capture the scaling laws of deep networks in the feature learning regime (Fort et al.)
2020 Vyas al. 2022 2023al Bordelon al.) (2024)). A theory of scaling laws that capture consistent feature learning even in an infinite parameter N - oo limit is especially pressing given the success of mean field and u-pparameerizzatios which generate constant scale feature updates across model widths and depths (Mei et al. 2019: Geiger et al.) 2020 Yang & Hul 2021 [Brdelon
& PPellevan 2022
Yang al. 2022
Bordelon et al.) 2023 2024b). The training dynamics of the
"Equal Contribution

---

Published as a conference paper at ICLR 2025
Dynamics, -
Dynamice, y
RcT Regine
12
Lazy
0.6
2110
0.0
Easy
(a) limx L(t, N) ~ x(P))
(b) Lazy Limit
(c) Rich Regime
Figure Our model changes its scaling law exponents for hard tasks, where the source is suf- ficiently small 33 < 1. (a) The exponent x(3) which appears the loss scaling L(t) ~ (-x(3) of our model. (b)-(c) Phase plots in the a, 3 plane of the observed scalings that give rise to the compute-optimal trade-off. Arrows (-> represent transition from one scaling behavior to another as + co, where the balancing of these terms at fixed compute C' Nt gives the compute optimal scaling law, In the lazy limit 7 - O, we recover the phase plot for a
of Paquette et al. (2024). that the set of "hard tasks", given by 8 exhibits improved scaling exponent. The compute optimal curves for the easy tasks with B > 1 are unchanged.
At nonzero y, however,
infinite width/depth limits in such models can significantly differ from the lazy training regime. Infinite limits which preserve feature learning are better descriptors practical networks Vyas et al. (003a). Motivated by this, we ask the following:
Under what conditions can feature learning improve the scaling law exponents of neural networks compared to lazy training regime?
Question:
1.1 OUR CONTRIBUTIONS
In this work, we develop theoretical model of neural scaling laws that allows for improved scaling exponents compared to lazy training under certain settings. Our contributions are
1. We propose simple two-layer linear neural network model trained with a form projected gradient descent. We show that this model reproduces power law scalings in training time, model size and training set size. The predicted scaling law exponents are summarized in terms of two parameters related to the data and architecture (0,
2. We identify condition the difficulty of the learning task, measured by the source ponent 3, under which feature learning can improve the scaling of the loss with time and with compute. For easy tasks, which define tasks with , >>>>>>>>> 1 where the RKHS norm of the target is finite, there is no improvement in the power-law exponent while for hard tasks (3 < that outside the RKHS of the initial limiting kernel, there can be im- provement. For super-easy tasks 3 > 2 L. which have very low RKHS norm, variance from stochastic gradient descent (SGD) can alter the scaling law large time. Figure summarizes these results.
3. We provide an approximate prediction of the compute optimal scaling laws for hard, easy tasks and super-easy tasks. Each of these regimes has different exponent for the compute optimal neural scaling law. Table summarizes these results.
4. We test our predicted feature learning scalings by training deep nonlinear neural networks fitting nonlinear functions. In many cases, our predictions from the initial kernel spectra accurately capture the test loss the network in the feature learning regime.
Overall our results suggest that feature learning may improve scaling law exponents by changing the optimization trajectory for tasks that are hard for the initial kernel.
1.2 RELATED WORKS
Our work builds the recent results of Bordelon et al. (202a)) and Paquette et al. 2024) which analyzed the SGD dynamics of a structured random feature model. Statics of this model have been
analyzed by many prior works (Atanasov al. 2023 Simon et al. 2023 Zavatone- Nethh & Pehlevan,
2

---

Published as a conference paper at ICLR 2025
2023 Bahri al. 2021 Maloney et al. 2022 These kinds of models can accurately describe networks in the lazy learning regime. However, the empirical study of Vaas et al. (2022) and some experiments in Bordelon et al.] (2024a) indicate that the predicted compute optimal exponents were smaller than those measured in networks that learn features on real data. These latter works observed that networks train faster in the rich regime compared to lazy training. We directly address this gap in performance between lazy and feature learning neural networks by allowing the kernel features to adapt to the data. We revisit the computer vision settings of Bordelon et al. (2024a) and show that our new exponents more accurately capture the scaling law in the feature learning regime.
Other work has investigated when neural networks outperform kernlls (Ghorbani et al., 2020). Ba et al.] (2022) and Abbbe et al. (2023) have shown how feature learning neural networks can learn low rank spikes in the hidden layer weights/kernels to help with sparse tasks while lazy networks cannot. Target functions with staircase properties, where learning simpler components aid learning of more complex components also exhibit significant improvements (with respect to a large input dimension) due to feature learning (Abbe et al. 2021 Dandi et al. 2023 Bardone & Goldt 2024). Here, we consider a different setting. We ask whether feature learning can lead to improvements in power law exponents for the neural scaling law. The work of Paccolat al. (2021) asks similar question in the case of a simple stripe model. Here we investigate whether the power law scaling exponent can be improved with feature learning in model that only depends on properties of the initial kernel and the target function spectra. Recent works have examined the dynamics of linear networks, contrasting the dynamics in lazy and feature learning regime, including analysis of infinite width linear networks Chizat et al. (2024), and linear networks with varying and unbalanced initialization and learning rates Kunin et al. (2024); Tu al. 2024). Our model can be interpreted as two-layer linear network which captures finite width effects (with task aask-dependent scaling laws) from random initialization. Like these related works, our model also has unbalanced learning rates between hidden and readout weights set by parameter that recovers a lazy limit as 7 - O.
SOLVABLE MODEL OF SCALING LAWS WITH FEATURE LEARNING
We start by motivating and defining our model. Our goal is to build simple model that exhibits fea- ture learning the infinite-widte limit but also captures finite network size, finite batch SGD effects, and sample size effects that can significantly alter scaling behavior. In this work, our operational definition of feature learning is evolution of the neural tangent kernel (NTK) of the model [
Following the notation of [Bordelon et al. (2024a), we introduce our model from the perspec- tive of kernel regression. We first assume a randomly initialized neural network in an infinite- width limit where NTK concentrates, We then diagonalize the initial infinite- width NTK. The re- sulting eigenfunctions (x)) E R.M have an inner product that define the infinite-width NTK Koc (z. x²) xx)) (xx). These cigenfunctions are orthogonal under the probability distri- bution of the data p(x) with
(v (x) (x)
/a>p(w)) ) A diag (A1...., AMr).
We will often consider the case where M + DC first so that these functions tho (x) form a complete basis for the space of square integrable functions. We next consider finite sized model with N parameters. We assume this model's 's initial parameters are sampled from the same distribution as the infinite model and that the N N N-parameter model, initialization regime, in the feature learning regime, these features will evolve during training.
limit recovers the same kernel Koe (2, The finite O. has N eigenfeatures (1 0) E RV Unlike the lazy
The finite network's learned function f is expressed in terms of the lower dimensional features, while the target function v(z) can be decomposed in terms of the limiting (and static) features 2/ (z: with coefficients w². The instantaneous finite width features (0. L) can also be expanded as a linear combination of the basis functions ahoo (x) with coefficient matrix A(t) E NxM We can therefore view our model the following student-teacher setup
f(x, () N (n) 1) ((x, 1) (1)
(3)
)( = w 2 (x)
'Other definitions are possible, but NTK evolution is least necessary condition for feature learning.
3

---

Published as a conference paper at ICLR 2025
If the matrix A(t) is random and static then gradient descent this random feature model recovers the lazy network analyzed by Bordelon et al. (2024a); Paqueete et al. (2024). In this work, we extend the analysis to cases where the matrix ((t is also updated, to allow for the evolution of the kernel.
We consider online training in the main text but discuss and analyze the case where samples are reused in Appendix [D]
We allow (t) to evolve with stochastic gradient descent (SGD) and A(t) evolve by projected SGD on mean square error with batch size B. Letting oo (t) RBxM represent a randomly sampled batch of B points evaluated on the limiting features {1 (x and r) to be the learning rate, our updates take the form
wttt + 1) - w(t) (t( W v vo0 (t)( o(()), e²()) = w.
1 A(t)
N
1 ( ( N A(L + 1) - A(1) 1/7 w0()) w(())((1)
A(O) A(O)
(4)
The fixed random projection ( AO) A(O)) present in A())ss dynamics ensure that the features cannot have complete access to the infinite width features &h but only access to the initial N- dimensional features x0) If this term were not present then there would be no finite parameter bottlenecks in the model and even a model with N = could fully fit the target function, leading to trivial parameter scaling law In this sense, the vector space spanned by the features vp does not change over the course of training, but the finite-width kernel Hilbert space does change its kernel: (br, I)- <b(r,, l). Feature learning in this space amounts to reweighing the norms of the existing features. We have chosen At) to have dynamics similar to the first layer weight matrix of linear neural network. As we will see, this enough to lead an improved scaling exponent.
The hyperparameter y sets the speed of A's dynamics and thus controls the rate of feature evolution. The Y + 0 limit represents the lazy learning limit Chizat et al. (2019) where features are static and coincides with random feature model dynamics of Bordelon et al. (2024a); Paquette et al. 22024)
The test error after t steps on a N parameter model with batch size
C(t, N. B,T) = ([Voc(e) - w² (xx,t) w((t]
- (t)
(5)
In the next sections we will work out a theoretical description of this model as a function of the spectrum A and the target coefficients wt. We will then specialize to power-law spectra and target weights and study the resulting scaling laws.
3 DYNAMICAL MEAN FIELD THEORY OF THE MODEL
We can consider the dynamics for random A() and random draws of data during SGD in the limit of M - oo and N, B > D This dimension-free theory is especially appropriate for realistic trace class kernels where (K (x, x²)), Ex Define wt. (t) to be respectively the components of w", x²(()) in the keth eigenspace of A. The error variables (t) are given by a stochastic process, and yield deterministic prediction for the loss C(1, N, B, )),, analogous to the results of Bordelon et al. (2024a).
x d < so (equivalent to a > 1), which is our focus.
Since the resulting dynamies for (1) at 7 > 0 are nonlinear and cannot be expressed terms of matrix resolvent, we utilize dynamical mean field theory (DMFT), flexible approach for han- dling nonlincar dynamical systems driven by random matrices (Sompolinsky & Zippelius! 1981
Helias & Dahmen 2020 Mannelli et al. 2019 MMgnacco al.) 2020) Gerbelot et al. 2022
Bordelon et al.] 2024a). Most importantly, the theory gives closed analytical predictions for L(1, B). We defer the derivation and full DMFT equations the Appendix C The full set of closed DMFT equations are given in Equation equation 26 for online SGD and Equation
We could also solve this problem by training a model of the form f w()) B(t)Au where w(t) and
(t) are dynamical with initial condition B(0) I and A frozen and the matrix B(t following gradient descent. We show that these two models are actually exhibit equivalent dynamics in Appendix [
'There are finite size fluctuations around the mean-field at small N, B which are visible in errorbars in can operate in proportional limit with M/M, B/ M approaching constants, which is exact with no finite size fluctuations.
Figure (c))re[(c)-(dd),, (c)- which could also be extracted from the theory. Alternatively,
4

---

Published as a conference paper at ICLR 2025
equation 41 for offline training with data repetition. Informally. this DMFT computes closed set of equations for the correlation and response functions for a collection of time-varying vectors V {t) w (t), v²()), o (t), o (e) including Co(t. a (t) As* which di- rectly gives the test loss [(t) (t, t). This theory is derived generally for any spectrum A and any target w.. In the coming sections we will examine approximate scaling behavior of the loss when the spectrum follows power law. In the figures, will plot the predictions from the full
DMFT equations as dashed black lines.
4 POWER LAW SCALINGS FROM POWER LAW FEATURES
We consider initial kernels that satisfy source and capacity conditions as in (Caponnetto Vito 2005 Pillaud- Vivien et al.) 2018; Cui et al.] 2021; 2023). These conditions measure the rate of decay of the spectrum of the initial infinite width kernel K (I, x²) and target function y(z) in that basis. Concretely, we consider settings with the following power law scalings:
d ~ k
Exe(wt) E>
~ k8
(6)
The exponent is called the capacity and measures the rate of decay of the initial kernel eigenvalues. We will assume this exponent is greater than unity a > since the limiting N - oo kernel should be trace class. The exponent / is called the source and quantifies the difficulty of the task under kernel regression with Kx
The RKHS norm | 12x of the target function is given by:
3
lu = E(E))² = Σ k- aa(---1)-1 2
o(3-1)
8
B 1
While the case of finite RKHS norm (3 >> 1) is often assumed in analyses of kernel methods that rely on norm-based bounds, such as (Bartlett & Mendelson] 2002 Bach 2024). the 3 < 1 case is actually more representative of real datasets. This was pointed out in ( (Wei et al. 2022). This can be seen by spectral diagonalizations performed on real datasets in (Bahri et al.] [021]] Bordelon al.)
2024a) as well as in experiments in Section 5.2 We stress this point since the behavior of feature learning with /3 and < will be strikingly different in our model.
General Scaling Law in the Lazy Limit For the purposes of deriving compute optimal scaling laws, the works of Bordelon et al. (2024a) and Paqueete et al, (2024) derived precise asymptotics for the loss curves under SGD. For the purposes of deriving compute optimal scaling laws, these asymptotics can be approximated as the following of power laws at large N
lim [(t, N, B,7) 2 7-+0
+ a
a (2,8)
+ (2-22-) 2)
(8)
N
B
Limiting Gradient Flow
Model Bottleneck
Finite N Transient
SGD Transient
where we neglect prefactor constants that are independent of N, B though these can be extracted from the full theory). The first terms represent Dottieneck/reollutionniiiin sent the loss obtained by taking all but one of the scaling quantities infinity (Bahri et al.] I) The first term gives the loss dynamics of population gradient flow (N, B + oo) while the second (model bottleneck) term describes t - oc limit of the loss which depends on N. The third and fourth terms are mixed transients that arise from the perturbative finite model and batch size effects. While Bordelon et al.] (2024a) focused on hard tasks where 3 where the first two terms domi- nate when considering compute optimal scaling laws, Paquette et al. (2024) also discussed two other phases of the easy task regime 1 < B < 1/a where the first and third term dominate and the super casy regime 3 2 1/a where the final two terms dominate the compute optimal scaling.
scalings which repre-
General Scaling Law in the Feature Learning Regime For y 0. approximations to precise DMFT equations under power law spectra give the following sum of power laws
L(t, N, B,,)
t- max 22
N (in-mmin
t (1- #1 max + +
t 1-(12-1 + max (1
B
Limiting Gradient Flow
Model Bottleneck
Finite N' Transient
SGD Transient
"The source exponent used in Pillaud- al. 2018 and other works is given by 2r
5

---

Published as a conference paper at ICLR 2025
10a
10
10
10
y=
y=
10-1
y
((e)
L(2)
10
2
y
y=
y 2
10
y
2
20
y=
10a5
y 21
10"2
t-P
8
2011+8
10
10a
10a
10a
201
102
10a
(a) Easy Task B = 1.2 with N, +
(b) Hard Task
0.4 with N, + 000
10a
10
2=
N=
N-2"
N=
10-1
N-25 2
N 2
c(t)
2
((t)
N
N 28
10
N 2
10-2
N 2
N 2
N
= 2
2
[-|
[-7A11 +
10-3
Theory
Theory
10-1
10a
10
10a
10a
10a
10a
(c) Easy Task 1.2 with finite N
(d) Hard Task 3 0.4 with finite
Figure 2: The learning dynamics of our model under power law features exhibits power law scaling with an exponent that depends on task difficulty. Dashed black lines represent solutions to the dynamical mean field theory (DMFT) while colored lines and shaded regions represent means and errorbars over 32 random experiments. (a) For easy tasks with source exponent > 1, the loss improved with feature learning but the exponent of the power law is unchanged. We plot the approximation L ~ -ー in blue. (b) For hard tasks where 3 < 1, the power law scaling exponent t T#. which
improves. An approximation of our learning curves predicts new exponent matches the exact N, B - co equations. (c)-(d) The mean field theory accurately captures the finite N effects both the easy and hard task regimes, As N + oo the curve approaches t- max 2 1233)
where neglect prefactor constants that are independent of N, B. We that in the rich regime, all exponents except for the model bottleneck are either the same or are improved. For easy tasks and super-easy tasks where / > 1, recover the same approximate scaling laws those computed in the linear model of Bordelon et al. (2024a) and Paquette et al. (2024). For hard tasks, 3 < 1, all exponents except for the model bottleneck term are improved. Below will explain why each of these terms can experience an improvement in the 3 < case. We exhibit a phase diagram all of the cases highlighted in equation 8, equation 9 in Figure I
Accelerated Training in Rich Regime The key distinction between model and the random feature model (y 0) is the limiting gradient flow dynamics, which allow for acceleration due feature learning. For nonzero feature learning 7 > O, our theory predicts that in the N - oo limit, the loss scales power law L(t) t x(i)) where the exponent x(8) satisfies the following celf-consistent equation
(ww)) exp (->e 7x22xx] max [
2
x(0)
lim t- In!
In
1,
(10)
B

---

Published as a conference paper at ICLR 2025
10r
10a2
16
30-1
32
64
54
Theory
10-3
Treory
10
10-1
(a) SGD Transients Hard 8 0.5
(b) SGD Transients Super-Easy 3.6
Figure 3: SGD Transients in feature learning regime. (a) In the hard regime, the SGD noise does not significantly alter the scaling behavior, but does add some additional variance to the predictor. As B - oo, the loss converges the t- 28/(1+8) scaling. (b) In the super-easy regime, the model transitions from gradient flow scaling -ー SGD noise limited scaling 1 - and finally to finite transient scaling 1-111/a.
We derive this equation in Appendix E
We see that if B > 1 then have the same scaling law
lazy learning model C(t) ~ 5_ However, if the task is sufficiently hard (3 < 1), then the exponent
increased to X
2,9
3. The time it takes transition to the new scaling is y x-xtaj as
1+.3
we discuss in Appendix E.2.1
This acceleration is caused by the fact that the effective dynamical kernel K(t) defined by the dynamics of our features (xx t) diverges as powerlaw ((t) ---x when 3 < 1 (see Appendix
This is due to the fact that the kernel approximation at finite y is not stable when training on tasks out of the RKHS. As a consequence, time t, the model is learning mode k (t) x(2-x)/o
which gives loss
L(1) {w ~ 7 8(2-xxx = 8-- B max (1.
(11)
kDk.
While our model predicts that the scaling exponent only changes for hard tasks where 8 < 1, it also predicts an overall decrease training loss as y increases for either easy or hard tasks (Appendix 1. In Figure 2 (a)-(b) we show the the N, B + oe limit of our theory at varying values of For easy tasks B > 1, the models will always follow C ~ 1-8 at late time, but with a potentially reduced
constant when y is large. For hard tasks (Fig. 2 (b)) the scaling exponent improves L ~ T-3 for
1-3
y > 0. The full DMFT predictions in Figure 2 (c) are plotted as dashed black lines.
Model Bottleneck Scalings Our theory can be used to compute finite N effects in the rich regime during SGD training. In this case, the dynamics smoothly transition between following the gradient descent trajectory at carly time to an asymptote that depends on N as t + co. In Figure 2 (c)-(d) we illustrate these learning curves from our theory and from finite N simulations, showing good match of the theory to experiment.
We derive the asymptotic scaling of N min (2.9) Appendix E.3 Intuitively, at finite N, the dynamics only depend on the filtered signal (+ A(O) w.. Thus the algorithm can only estimate, at best, the top N components of w., resulting in the following ( - oo loss
((N) Σ 1-8--1 -000~~-" ~
(12)
RSN
SGD Noise Effects The variance in the learned model predictions due to random sampling of minibatches during SGD also alters the mean field prediction of the loss. In Figure we show SGD noise effects from finite batch size B for hard 3 < 1 and super easy 3
1/a tasks.
Compute Optimal Scaling Laws in Feature Learning Regime At a fixed compute budget C = Nt, one determine how allocate compute towards training time 1. and model size N

---

Published as a conference paper at ICLR 2025
Task Difficulty
Hard <
Easy 1 < p < 1/a
Super-Hasy B B> 1/a
Lazy (7 = 0)
o8
o88
o+1
a3+1
2a
Rich (y > 0)
2a.5
0.
+1++8)+2
a3++1
2a
Table 1: Compute optimal scaling exponents To for the loss (C) C-re for tasks of varying
difficulty in the feature learning regime. For 3 > 1, the exponents coincide with the lazy model
analyzed by Bordelon al. (2024a); Paquette et al. (2024), while for hard tasks they are improved.
using our derived exponents from the previous sections. Choosing N,t E optimally, we derive the following compute optimal scaling laws (C) in the feature learning regime > 0. These are also summarized in Figure
1. Hard task regime (3 < 1): the compute optimum balances the population gradient flow
20 term 1 I- and the model bottleneck N-a._
2. Easy tasks (1 < B 2 4): the compute optimum compares gradient flow term tー to finite N transient term 3/ -1+1//a
3. Super easy tasks (3 > 2 ): compute optimum balances the finite N transient 1 -+11/ N
and SGD transient terms # 2+
We work out the complete compute optimal scaling laws for these three settings by imposing the constraint = Nt, identifying the optimal choice of N and at fixed t and verifying the assumed dominant balance. We summarize the three possible compute scaling exponents in Table
In Figure R we compare the compute optimal scaling laws in the hard and easy regimes. We show that the predicted exponents are accurate. In Figure 3 we illustrate the influence of SGD noise on the learning curve in the super easy regime and demonstrate that the large C' compute optimal scaling law is given by C 1+ 20.
10'
6-
Crft
Theor
Theary
C.t+g Theary
10-1
12-4
10
12-2
10-4
13a9
13-3
134
13-4
30P
10a
10t
1.2
122
122
10"
1.0
124
201
102
20f
20"
Compute C
Compute C
Compute C
(a) Hard Task Scaling 3 = 0.5 (b) Easy Task B 1.3 (c) Super Easy 8 1.75
Figure 4: Compute optimal scalings in the feature learning regime (7 = 0.75). Dashed black lines are the full DMFT predictions. (a) In the 3 1 regime the compute optimal scaling law is determined by a trade-off between the bottleneck scalings for training time t and model size N, giving (C) C .X where X = 2 is the time-exponent for hard tasks in the rich-regime.
1+8
(b) In the easy task regime 1 < 3 < 2 - 1, the large C' scaling is determined by competition between the bottleneck scaling in time t and the leading order I/N correction to the dynamics (C) C a+I, (c) In the super-easy regime, the scaling exponent at large compute is derived by balancing the SGD noise effects with the 1 1/N transients.
5 EXPERIMENTS WITH DEEP NONLINEAR NEURAL NETWORKS
While our theory accurately describes simulations of our solvable model, now aim to test if these new exponents are predictive when training deep nonlinear neural networks. Apriori, there is no reason for our toy model's predicted exponents match those observed in deep nonlinear networks, yet in many cases they are descriptive.
'The three regimes of interest correspond to Phases I,II,III in Paquette et al. (024). These are the only relevant regimes for trace-class (K (æ. x)) = Σ. As < (finite variance) kemelss (equivalent 1).
8

---

Published as a conference paper at ICLR 2025
5.1 SOBOLEV SPACES ON THE CIRCLE
We first consider training multilayer nonlinear MLPs with nonlinear activation function ((h)
[R) in the mean field parameterizationluuP (Geiger et al. 2020, Yang & Hu 2021 Borde- lon & Pehlevan 2022) with dimensionless sridth-indeeeeddent) feature learning parameter "/o We consider fitting target functions y(z) with æ = [cos(0), (), (i)( on the circle 0 E [0, 2m]. The eigen- functions for randomly initialized infinite width networks are the Fourier harmonics. We consider target functions (() with pawer-law Fourier spectra while the kernels at initialization K (0, 8') also
admit Fourier eigenexpansion
v(0)
cos(k0), K(0,0') dE cos(k(0-000)
(13)
We show that the eigenvalues of the kernel at initialization decay A 24 in the Appendix A The capacity and source exponents a, B required for our theory can be computed from q and qs
2q 2q-1
a = 2q 3
(14)
Thus task difficulty can be manipulated by altering the target function or the nonlinear activation function of the neural network. We show in Figure S examples of online training in this kind of network on tasks and architectures of varying B. In all cases, our theoretical prediction of
t. max (1 ) provides very accurate prediction of the scaling law.
10a
10-1
10-4
10ah
8- 0.10
c(0
8-025
10-3
8- 0,50
B- 0.75
$-0.50
10-4
1:00
10-3
B- 1.25
6= 1.00
10-a
B=30
B=22
(-Amax:3.202
10-6
10a
10
10:
10a
10a
10c
(a) ReLU (gs = 1.0) with varying 8
2
(b) Varying ga with fixed target (q 1.4)
24
Figure 5: Changing the target function's Fourier spectrum or the neural network can change the scaling law in nonlinear networks trained online. These MLPs are depth and width 512. (a) Our predicted exponents are compared to SGD training in a ReLU network. The exponent 3 is varied by changing g. the decay rate for the target function's Fourier spectrum. The scaling laws are well predicted by our toy model t. -8 max ) can also be manipulated by changing properties of the model such as the activation function Jø-
(b) The learning exponent for a fixed target function
5.2 COMPUTER VISION TASKS (MNIST AND CIFAR)
We next study networks trained on MNIST and CIFAR image recognition tasks. Our motivation to study networks training in the online setting over several orders of magnitude in time. To this end, we adopt larger versions of these datasets: "MNIST-IMA IM" and CIAFR-5M. We generate MNIST- MM using the denoising diffusion model (Ho et al. 2020) in Pearce (022). We use CIFAR- 5M from Nakkiran et al. (2021). Earlier results in Refineti et al. (2023) show that networks trained on CIFAR-5M have very similar trajectories to those trained on CIFAR-10 without repetition. The resulting scaling plots are provided in Figure 6 MNIST-IM scaling is very well captured by the our theoretical scaling exponents. The CIFAR-3M scaling law exponent at large To first follows our predictions, but later enters a regime with exponent larger than what our theoretical model predicts.
DISCUSSION
We proposed a simple model of learning curves in the rich regime where the original features can evolve as linear combination of the initial features. While the theory can give a quantitatively
9

---

Published as a conference paper at ICLR 2025
10"3
1e-02
Test Loss
==
Test Loss
11-1e:01
Ys 36-01
Linearized
1e+00
Ve m 0.25
0.33
10-2
2.12.181
10-1
Vo 0.50
2-2011-
10a
10a
10a
10a
10
10a
10a
10a
Steps
(a) CNNS on MNIST- 1M, 0.30
(b) CNNs CIFAR-5M 8 = 0.075
The improved scaling law with training time gives better predictions for training deep networks on real data, but still slightly underestimate improvements to the scaling law for Residual CNNs trained on CIFAR-5M, especially at large richness to (experimental details in Appendix A). (a)-(b) Training on MNIST-IM is well described by the new power law exponent from our theory. (c) CNN training on CIFAR-5M is initially well described by our new exponent, but eventually achieves a better power law.
Figure 6:
accurate for the online learning scaling exponent in the rich regime for hard tasks, the CIFAR-5M experiment suggests that additional effects in nonlinear networks can occur after sufficient training. However, there are many weaker predictions of our theory that we suspect to hold in wider set of settings, which we enumerate below.
Source Hypothesis: Feature Learning Only Improves Scaling For 3 < 1. Our model makes general prediction that feature learning does not improve the scaling laws for tasks within the RKHS of the initial infinite width kernel. Our experiments with ReLU networks fitting functions in different Sobolev spaces with 3 > 1 support this hypothesis. Since many tasks using real data appear to fall outside the RKHS of the initial infinite width kernels, this hypothesis suggests that lazy learning would not be adequate to describe neural scaling laws real data, consistent with empirical findings (Vyas et al. 2022).
Insignificance SGD for Hard Tasks Recent empirical work has found that SGD noise has little impact in online training of deep learning models (Vyas et al., 2023b Zhao al. 2024). Our theory suggests this may be due to the fact that SGD transients are always suppressed for realistic tasks which are often outside the RKHS of the initial kernel. The regime in which feature learning can improve the scaling law in our model is precisely the regime where SGD transients have no impact on the scaling behavior.
Ordering of Models in Lazy Limit Preserved in Feature Learning Regime An additional in- teresting prediction of our theory is that the ordering of models by performance in the lazy regime preserved is the same as the ordering of models in the feature learning regime. If model A outper- forms model B on task in the lazy limit (B > BB), then model A will also perform better in the rich regime A > x(3)) (see Figure l]. This suggests using kernel limits of neural architectures for fast initial architecture search may be viable, despite failing to capture feature learning (Park
al. 2020). This prediction deserves greater degree of stress testing.
Limitations and Future Directions There are many limitations to the current theory. First, we study mean square error loss with SGD updates, while most modern models are trained on cross- entropy loss with adaptive optimizers (Everett et al., 2024). Understanding the effect of adaptive optimizers or preconditioned updates on the scaling laws represents important future direction. In addition, our model treats the learned features as linear combinations the initial features, an assumption which may be violated in finite width neural networks. Lastly, while our theory is very descriptive of nonlinear networks on several tasks, we did identify a noticeable disagreement on CIFAR-5M after sufficient amounts of training. Versions of our model where the learned features are not within the span of the initial features or where the matrix undergoes different dynamics may provide promising avenue of future research derive effective models of neural scaling laws.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank Jacob Zavatone- Veth, Jascha Sohl-Dickstein, Courtney Paquette, Bruno Loureiro, and Yasaman Bahri for useful discussions. B.B. is supported by Google PhD Fellowship. A.A. is supported by Fannie and John Hertz Fellowship. C.P. is supported by NSF grant DMS- 2134157, NSF CAREER Award IIS-2239780, and a Sloan Research Fellowship. This work has been made possible in part by a gift from the Chan Zuckerberg Initiative Foundation to establish the Kempner Institute for the Study of Natural and Artificial Intelligence.
REFERENCES
Emmanuel Abbe, Enric Boix-Adsera, Matthew S Brennan, Guy Bresler, and Dheeraj Nagaraj. The staircase property: How hierarchical structure can guide deep learning. Advances in Neural In- formation Processing Systems, 34:26989-27002, 2021.
Emmanuel Abbe, Enric Boix Adsera, and Theodor Misiakiewicz. Sgd learning on neural networks: leap complexity and saddle-to-sadeee dynamics. In Thee Thirty Sixth Annual Conference Learn- ing Theory, pp. 2552-2623 2623. PMLR, 2023.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 08774, 2023.
Alexander Atanasov, Blake Bordelon, and Cengiz Pehlevan. Neural networks as kernel learners: The silent alignment effect. In International Conference on Learning Representations, 2022.
net URL http:t:://pppeeeeepeeea wttp://wwwweeeeeeeeeeiieeeea ftr.miirmmmi/////// INvflqAdoom
Alexander Atanasov, Blake Bordelon, Sabarish Sainathan, and Cengiz Pehlevan. The onset of variance-limited behavior for networks in the lazy and rich regimes. In The Eleventh Interna- fional Conference Learning Representations, 2023. URL http:///epeeeeeieeeeeeeee net/
forum? (II..
Alexander B Atanasov, Jacob A Zavatone- Veth, and Cengiz Pehlevan. Scaling and renormalization in digh--imensional regression. arXiv preprint arXiv:2405.00592 2024.
Jimmy Ba, Murat Erdogdu, Taiji Suzuki, Zhichao Wang, Denny Wu, and Greg Yang. High- dimensional asymptotics of feature learning: How one gradient step improves the representation. Advances in Neural Information Processing Systems, -5:37932-37946, 2022.
Francis Bach. Learning theory from first principles. MIT press, 2024.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jaehoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint wwX.vv2102.067000 2021.
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vin- cent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 2279-2277. PMIR, 2021.
Lorenzo Bardone and Sebastian Goldt. Sliding down the stairs: how correlated latent variables accelerate learning with neural networks. arXiv preprint 2xXi::2404008622. 2024.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-482 2002.
Blake Bordelon and Cengiz Pehlevan. Self-consistent dynamical field theory of kernel evolution in wide neural networks. arXiv preprint arXiv:2205.0940 09653, 2022.
Blake Bordelon, Abdulkadir Canatar, and Cengiz. Pehlevan. Spectrum dependent learning curves in kernel regression and wide neural networks. In International Conference on Machine Learning, pp. 1024-1034. PMLR, 2020.
Blake Bordelon, Lorenzo Noci, Mufan Bill Li, Boris Hanin, and Cengiz Pehlevan. Depthwise hyperparameter transfer in residual networks: Dynamics and scaling limit, 2023.
11

---

Published as a conference paper at ICLR 2025
Blake Bordelon, Alexander Atanasov, and Cengiz Pehlevan. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092, 2024a.
Blake Bordelon, Hamza Tahir Chaudhry, and Cengiz Pehlevan. Infinite limits of multi-head trans- former dynamics. arXiv preprint arXiv:2405.15712. 2024b.
Abdulkadir Canatar and Cengiz. Pehlevan. A kernel analysis of feature learning deep neural networks. In 2022 58th Annual Allerion Conference Communication, Control, and Computing (Allerton), pp. 1-8. IEEE, 2022.
Andrea Caponnetto and Ernesto De Vito. Fast rates for regularized least-squares algorithm. 2005.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Lénaic Chizat, Maria Colombo, Xavier Fernández-Real and Alessio Figalli. Infinite-width limit of deep linear neural networks. Communications on Pure and Applied Mathematics, 77(10)39958- 4007, 2024.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Generalization error rates in kernel regression: The crossover from the noiseless to noisy regime. Advances in Neural Information Processing Systems, 34:10131-10144, 2021.
Hugo Cui, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborová. Error scaling laws for kernel classification under source and capacity conditions. Machine Learning: Science and Technology, 4(3):035033, 2023.
Yatin Dandi, Florent Krzakala, Bruno Loureiro, Luca Pesce, and Ludovic Stephan. How two-layer neural networks learn, one (giant) step at a time. In NeurIPS 2023 Workshop on Mathemat- ics of Modern Machine Learning, 2023. URL https:: http::ss//peezeview..co http:/wwweeeeoooo.ooeeeeeee www.mfooofmmii
iRDcaBLhz2
Simon Du, Wei Hu, and Jason Lee. Algorithmic regularization in learning deep homogeneous models: Layers automatically balanced. Advances in neural information processing systems, 31, 2018.
Katie Everett, Lechao Xiao, Mitchell Wortsman, Alexander A Alemi, Roman Novak, Peter Liu, Izzeddin Gur, Jascha Sohl-Dickstein, Leslie Pack Kaelbling, Jaehoon Lee, et al. Scaling expo- nents across parameterizations and optimizers. arXiv preprint (xXi:22407.05872. 2024.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.), Advances in Neural In- formation Processing Systems 33: Annual Conference on Neural Information P Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthieu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2020 111):113301, 2020.
Cedric Gerbelot, Emanuele Troiani, Francesca Mignacco, Florent Krzakala, and Lenka Zdeborova. Rigorous dynamical mean field theory for stochastic gradient descent methods. arXiv preprint arXiv:2210.06591, 2022.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? Advances in Neural Information Processing Systems, 33: 14820-14830, 2020.
Moritz Helias and David Dahmen. Statistical theory for neural networks, volume 970. Springer, 2020.
12

---

Published as a conference paper at ICLR 2025
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:
2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-66 685000668111-0 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Train- ing compute-optimal large language models. arXiv preprint arXiv:2203. 15556, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Simon Korbllihh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference on machine learning, pp. 3519- 3529. PMLR, 2019.
Daniel Kunin, Allan Raventós, Clémentine Dominé, Feng Chen, David Klindt, Andrew Saxe, and Surya Ganguli. Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning. arXiv preprint arXiv:2406.066 066.0618.8 2024.
Licong Lin, Jingfeng Wu, Sham M Kakade, Peter L Bartlett, and Jason D Lee. Scaling laws in linear regression: Compute, parameters, and data, arXiv preprint arXiv:2406. 08466666666 2024.
Alexander Maloney, Daniel A Roberts, and James Sully. A solvable model of neural scaling laws. arXiv preprint arXiv:2210.1663 16859, 2022.
Stefano Sarao Mannelli, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborova. Passed & spurious: Descent algorithms and local minima in spiked matrix-tensor models. In international conference on machine learning, pp. 4333 4342. PMLR, 2019.
Song Mei, Theodor Misiakiewiez, and Andrea Montanari. Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit. In Conference on Learning Theory, pp. 2388- 2464. PMLR, 2019.
Francesca Mignacco, Florent Krzakala, Pierfrancesco Urbani, and Lenka Zdeborová. Dynamical mean-field theory for stochastic gradient descent in gaussian mixture classification. Advances in Neural Information Processing Systems, 33:9540-9550, 2020.
Niklas Muennighoff, Alexander Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Noua- mane Tazi, Sampo Pyysalo. Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. arXiv preprint arXiv:2305. 16264, 2023.
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. International Conference on Learning Represen- tations. 2021.
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2021(4):044001 2021.
Courtney Paquette, Kiwon Lee, Fabian Pedregosa, and Elliot Paquette. Sgd in the large: Average- case analysis, asymptotics, and stepsize criticality. Conference on Learning Theory, pp. 3548- 3626. PMLR, 2021.
Elliot Paquette, Courtney Paquette, Lechao Xiao, and Jeffrey Pennington. 4+ phases of compute- optimal neural scaling laws. arXiv preprint arXiv:2405. /5074, 2024.
Daniel S Park. Jachoon Lee, Daiyi Peng. Yuan Cao, and Jascha Sohl-Dickstein. Towards nngp- guided neural architecture search. arXiv preprint arXiv:201 200:::001.00000 1.06006, 2020.
13

---

Published as a conference paper at ICLR 2025
Tim Pearce. Conditional diffusion mnist.
Conditional Diffusion, MNIST, 2022. Accessed: 2024-05-14.
Loucas Pillaud- Vivien, Alessandro Rudi, and Francis Bach. Statistical optimality of stochastic gra- dient descent on hard learning problems through multiple passes. Advances in Neural Information Processing Systems, 31, 2018.
Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with sgd learn distributions of increasing complexity. In International Conference Machine Learning, pp. 28843-28863. PMLR, 2023.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynam- ics of learning in deep linear neural networks. arXiv preprint
2013.
James B Simon, Madeline Dickens, Dhruva Karkada, and Michael R DeWeese. The eigenlearning framework: A conservation law perspective on kernel regression and wide neural networks. arXiv preprint arXiv:2 10. 03022, 2021.
James Simon, Dhruva Karkada, Nikhil Ghosh, and Mikhail Belkin. More is better in modern machine learning: when infinite overparameterization is optimal and overfitting is obligatory. arXiv preprint arXiv:2311.1466 2023.
Haim Sompolinsky and Annette Zippelius. Dynamic theory of the spin-glass phase. Physical Review Letters, 47(5):359, 1981.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neu- ral scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523-19536, 2022.
Stefano Spigler, Mario Geiger, and Matthieu Wyart. Asymptotic learning curves of kernel methods: empirical data versus teacher-student paradigm. Journal of Statistical Mechanics: Theory and Experiment, 2020(12):1 2020.
Zhenfeng Tu, Santiago Aranguri, and Arthur Jacot. Mixed dynamics in linear networks: Unifying the lazy and active regimes. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL (tRt:tts::/ hwwp:///weeeeeeeeeeeeeeeeeeeeeee wwwp://wtteooooooteteeeea/taa 9=22117mmu
Nikhil Vyas, Yamini Bansal, and Preetum Nakkiran. Limitations of the ntk for understanding gen- eralization in deep learning. arXiv preprint arXiv:2206. N.XX::2206.10022, 2022.
Nikhil Vyas, Alexander Atanasov, Blake Bordelon, Depen Morwani, Sabarish Sainathan, and Cen- giz Pehlevan. Feature-learning networks are consistent across widths at realistic scales. arXiv preprint ar arXiv:2305.184111 2023a.
Nikhil Vyas, Depen Morwani, Rosie Zhao, Gal Kaplun, Sham Kakade, and Boaz Barak. Beyond implicit bias: The insignificance of sgd noise online learning. arXiv preprint arXiv:2306.0855 030500 2023b.
Alexander Wei, Wei Hu, and Jacob Steinhardt. More than toy: Random matrix models predict how real-world neural representations generalize. In International Conference on Machine Learning, pp. 23549-23588. PMLR, 2022.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pp. 3633-3673. PMLR, 2020.
Greg Yang and Edward Hu. Tensor programs iv: Feature learning in infinite-width neural networks. In International Conference on Machine Learning, pp. 1 11727-11737. PMLR, 2021.
Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tuning large neural networks via zero-shot hyperparameter transfer. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. //openrevis ew. net.fforu wttp:rummmmmxxxxxxxxxxxeee d=
URL https:
14

---

Published as a conference paper at ICLR 2025
Greg Yang. Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ry- der, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXi::203.03466, 2022. 2022.
Jacob A. Zavatone-Vers Veth and Cengiz Pehlevan. Learning curves for deep structured gaussian feature models, 2023.
Rosie Zhao, Depen Morwani, David Brandfonbrener, Nikhil Vyas, and Sham Kakade. Deconstruct- ing what makes a good optimizer for language models. arXiv preprint arXiv:2407.07972. 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A ADDITIONAL EXPERIMENTS AND EXPERIMENTAL DETAIL
A.I MLPS SOBOLEV TASKS
The MLPs in Figure 5 were depth L = 4 with nonlinearities Φ(h) ReLU(())=, giving the follow- ing forward pass
W ((h") (1,2)) h 1 VD
[(x)) Nr w o(h"), h
W.. x.
where D = 2 is the input dimension. The data are sampled randomly with 0 Unif 2n| and are preprocesed as a = (cos(O), sin(() E RR2 We diagonalize neural tangent kernels (NTKs) for architectures with varying 9s in Figure
showing change in the power law spectra.
10"
10"
10
10-1
10
0p=0.. 0.75
10-2
4
B-10
10
1.5
$ 1.5
1.75
k-24
10
107
101
10a
10a
10
10a
10a
10t
k
(a) Kernel eigenspectra a 2ga
(b) Easy Sobolev Tasks (ReLU de 1)
Figure 7: Additional experiments in the setting with data drawn from the circle. (a) The spectra of kernels across different nonlinearities ((h) ReLU(h)): which scale Å k --2g, (b) More experiments in the easy task regime, show that feature learning does not alter the long time scaling behavior for 3 >>
A.2 CNNS ON VISION TASKS
The CNN experiment on MNIST uses a depth L architecture with two convolutional layers and two Dense layers. The predicted exponent in the lazy regime from the spectra is B 0.3.
For the CIFAR-5M experiment, we use the same deep residual architecture of Bordelon et al.
2024a) We reproduce the diagonalization of the kernel on CIFAR-5M in Figure
A.3 LANGUAGE MODELING TASK
We also tried an initial test of our theory in a deep transformer trained on next token prediction. In Figure we plot cross entropy loss as function of training time. Despite our theory heing derived under mean square error minimization, the loss dynamics at large are roughly twice the exponent as the loss dynamics at small 7.
B FURTHER DISCUSSION OF MODELS
We seek a model that incorporates both the bottle-necking effects of finite width observed recent linear random feature models of scaling laws while still allowing for notion of feature learning. Exactly solvable models feature learning networks are relatively rare. Here, we take inspiration from the linear neural network literature Saxe al. (2013). Linear neural networks exhibit both lazy and rich regimes of learning Woodworth et al. (2020). in which they can learn useful task-relevant features in way that can be analytically studied Atanasov al. (2022). In work, go beyond
16

---

Published as a conference paper at ICLR 2025
N=8
Unexplained Task Fraction
t=0
10
10a
N
t=2e+03
10
N 64
t-5e103
10
N 256
k-20
C(k)
1-20:04
5e4400
10
10
1-26105
5e
10-9
t=2e006
b=4c006
10-11
P-222
10a
104
107
10
10
Figure 8: The spectra of ReLU residual CNNs on CIFAR-5M for varying width N. (a) The eigen- values fall approximately
dB k-2 which means 2. (b) The cumulative power spectrum 1- ((k) = =i> (wz) - ~ -0015 is estimated at l = which implies 3 R 0.075. The eigenvec- tors of the kernel change over time and align to the task direction, evidenced by the larger fraction of variance captured by the top eigenmode.
Cross Entropy Loss
8 10P
7 x 10
0.0001
0.01
109
0.05
0.1
10
10a
107
10a
10"
Tokens
Figure 9: A depth L = 4 decoder-only transformer (16 heads with daeud 128) trained on next- word prediction with SGD on the C4 dataset tokenized with the SentencePiece tokenizer. plot cross entropy loss and fit powerlaw -ー to lazy learning curve y 10 over the interval from 106 to 3 x 10 tokens. We then compute the new predicted exponent (-3(2-3) and compare to simulation at y = 0.1. Though our theoretical prediction of a doubling of the scaling exponent was derived in the context of MSE, the new scaling exponent fits the data somewhat well for this setting at early times.
these models of linear neural networks and show that linear neural networks trained on data under source and capacity conditions improve the convergence compared that predicted by kernel theory.
The model introduced in section is give by two-layer linear network acting on the (x)::
f(x)
N w (x).
(15)
There, we constrained it to update weights by a form of projected gradient descent as given by Equation 4 Here, we show that running this projected gradient descent is equivalent to running or- dinary gradient descent on a two layer linear network after passing &o through random projections. Define Ao A()) and (tt) = Aft) where + denotes the Moore-Penross psuedoinverse. Then assuming N < and Ag is rank N, we have
B()) A(1, B(0) INXN-
(16)
17

---

Published as a conference paper at ICLR 2025
fox
f(x)
f(x)
N
N
Ag
w(O
A(O)
w(O
w(t)
(a)
(b)
(c)
Figure 10: Three different models studied in this work and prior work. Black weights are frozen while orange weights are trainable. a) A linear random feature model with only the readout weights trainable. This model was studied Maloney et al. (2022); Bordelon et al. (2024a): Paquette et al. (2224) as solvable model of neural scaling laws. b) A two layer linear network with both weights trainable. This model does not incur bottleneck due to finite width but undergoes feature learning, which improves the scaling of the loss with time. We study pure linear neural networks in Appendix G In the main text, we train this model with projected version gradient descent. This is equiv- bottlenecks as well improvements scaling due to
alent to and gives both feature learning.
Now consider taking dh and passing it through Ao. and then training B, w with ordinary gradient descent. We have update equations:
1
wl + - 1))(((() BB())A
TT
v0(1)
B
(17)
B 1) B(t) - mtt
1
N A
B
Multiplying the second equation by Ao on the right recovers equation 4 Here, 7 acts as a rescaling of the learning rate for the B update equations. We illustrate this model, as well as the linear random feature and linear neural network models in Figure TO
Several papers Maloney et al. (2022)) Atanasov et al. (2023) Bordelon et al. (2024a); Atanasov al. (2024) have studied the model given in equation with frozen under the following interpretation. The samples va E RD correspond to the dataset expressed in the space of an infinitely wide NTK at initialization. These are passed through a set of frozen random weights W1. which are thought of the projection from the infinite width network to the finite-width empirical NTK, corresponding to a lazy network. From there, the final layer weights are not frozen and perform the analog of regression with the finite-wide NTK. In Bordelon al. (2024a), this model was shown reproduce many of the compute optimal scaling laws observed in practice. It was also however shown there that the scaling laws for lazy networks are very different from those observed for feature-learning networks.
Our motivation is to develop a simple and solvable model of how the finite-width network features <xx;t) Att)&b (x) might evolve learn useful features. The projected linear model defined above states that the 4p recombine themselves in such a way so that empirical neural tangent better aligned to the task. The simple model of linear neural network rich enough to yield improved power law, while still being analytically tractable.
kernel (x;; t) <b(x''; ; 1)
B. COMPARISON TO VERY RELEVANT PRIOR WORKS
We are using identical notation for our model's projection weights A, and the error vectors (v0) wt ,..,, ...,,'} to Bordelon et al. (002a) and our dynamics match theirs in the limit of 7 -> 0.
18

---

Published as a conference paper at ICLR 2025
We also work with the DMFT correlation and response as in those works. Indeed, the same tech- niques (path integral or cavity methods) used their prior work can be used to derive the mean field equations. Their DMFT equations could be solved exactly in Fourier space since the system was linear and tamme-traslationninvaiian (TTT). This Fourier representation is also very close to the results of Paquette et al. (2024). However, tracking the evolution of (t) and the resulting dynamics become non-TTI. However, we can still close the equations in terms of time time matrices.
results significantly harder since they require
C DERIVATION OF THE MEAN FIELD EQUATIONS
In this setting, we derive the mean field equations for the typical test loss L(t, BB as function of training time. To accomplish this, we have to perform disorder averages over the random matrices (O) and (1))) We start by defining the following collection of fields
1 N
(1) = w
A(L) T (())
(1) (1)) ( (1) x')))
B
1
v (1) A (t), v 0²(1)
N A(0) (()())
v" (1 N A()) m(()
(18)
From these primitive fields, we can simplify the dynamics of A, w(t)
A(t = A() + m/? Σ (s))
w(t + 1) = w(t) + (t)))((
=w(() + 7 + +1? Σ w(s) (s)
w(t + no 1 w(s)Ca(tts)
(19)
where we introduced the correlation function Cs(t, = (²) (3)) = =t²(t) . (s). Similarly for (t) and 20 (1) we have
(1) w² -v" 177 Σ (s)C
s<t
Tthh vt + 1)= v" + m7o + 12> Σ (t)(()
(20)
where we introduced C.. (1, = (l) w)))) We see that, conditional on the correlation function (t, s), the vector (t)(() can be interpreted as a linear filtered version of (er((s) and is thus redundant. In addition, projections of this matrix on vectors of interest. Since all dynamics only depend on the random variables V = f v 2 3), therefore only need to characterize the joint distribution of these variables.
no longer have to work with the random matrix A(t) but can rather track
Disorder Averages We now consider the averages over the random matrices which appear in the dynamics (t)) and A(0). This can be performed with either path integral or a cavity derivation following the techniques Bordeee al. (2024a). After averaging over { (t) one obtains the following process for u' (t) and (t)
(1) w (t), n (t) ~ N(0, (t)
<e(()) =f((t) Atol(t), u²(()) ~ N (-. (t)dc))t)))
(21)
19

---

Published as a conference paper at ICLR 2025
where the correlation functions Co and C₁ have the forms
Colt, 5) - Σ A (ol()))(s)) - (u"
(22)
The average over the matrix (() couples the dynamics for (t), (t) resulting in the following
(1) = (t) +
R2, (8) (²(1) ~ NO, 8))
N
S
(t) = (t) + Σ
4 (O Ra s))((s)
(23)
s<t
where
C2(t,s)= = (oR(0)))()) =a(t,s)= (t)
(24)
Lastly, we have the following single site equation for w(t) which can be used to compute (t, s)
wit + 1) wtt) you (t) r²a E )a((ss))oss).
(25)
Final DMFT Equations for our Model for Online SGD The complete governing equations for the test loss evolution after averaging over the random matrices can be obtained from the following stochastic processes which are driven by Gaussian noise sources {1) w² ((), (1)). Letting (.)
represent averages over these sources of noise, the equations close as
<2(() - - - 17 Σ Ct,lts)utte
a<t
(1) = (t) + (1)(t) + n2 Σ Cs(t,
(1)) = (t) + (t) (t) N s)
(1))) = (²(() + Σ ()) ~ (()xt()))().)))
+ = wt) nu (t) + n2² Σ Cglt, s)(ss)
(t) (t) + Σ Rt(t,s)ff s) ux(() ~ N
lt, 8) Σ duf(t) Ra(t,s) = (t)
Ouf(s)
(s)
Colt, = Σ Ae (v)(t))((s)) Ca(t,s) Σ (1)(
(26a)
Closing the DMFT Correlation and Response as Time Time Matrices We can try using these equations to write a closed form expression for et(t) which determines the generalization
error. First, we start by solving the equations for o Vee ux (t) We introduce a step
function matrix which is just lower triangular matrix [ 1/0(1 ())
ok [ 0C Oug H of
H = [I - 111-m00
(27)
Now, combining this with the equation for of = Vec(uo(t)) we find
h + m Cw) (ut + Rz42)]
H = | d H C.)
Rs -1
(28)
20

---

Published as a conference paper at ICLR 2025
The key response function is R3 with entries [Rali,s Rs(t, s) satisfies the following equation
M
R3
I+
Rs.
R
H H + Cm)
N
M
R3
I
X H HHE + mC) R3
N
k=1
M
1
Ao As (H Cw) Ra] (H) C Rs
(29)
Lastly, we can compute the correlation matrix C from the covariance C33
C = 33] o
[ 77 -1T
(30)
The remaining correlation functions are defined as
Co = Σ
A H
(w T + (H + C,)
1 NNO
C3 +
d B
Radiag (C)) (HE mw) [H2]
h
C₂ =
A (I H Cs) diag(Co) (I H + CC)w
B k
Σ H
11 (1t)) + 1 (HE +mC + + N
H
h C3
R3C2Rg
(31)
D OFFLINE TRAINING: TRAIN AND TEST LOSS UNDER SAMPLE REUSE
Our theory can also handle the case where samples are reused in a finite dataset T E PxM, To simplify this setting we focus on the gradient flow limit (this will preserve all of the interesting finite-P effects while simplifying the expressions)
d
1 P
dt (1) A(1)
T
(1)
d
(T. (TV) vv) (H)) A(o)
[
di A(t) (1)
(32)
We introduce the fields
1 T (1)
v (1) = Vt)(t), (1) = p
(33)
(1) = A(0) (1), v (1) = N A()) A(0)TT²00
(34)
so that the dynamics can be expressed as
dt w(t) = (t))
di A(t) (t)
(35)
As before we also introduce the following field which shows up in the (t) dynamics
v (t)(t) N A() w(t)
wt))
(36)
Data Average The average over the frozen data matrix E PxM
(t) = (t) + A / (t, (s) =i((t) ~ N(0 P-
(37)
('(t) = (t) + P l'' dt (s) n²(() (()
(38)
21

---

Published as a conference paper at ICLR 2025
Feature Projection Average (t) (t(() + !. ds ((t
Next average over A(O) RNNM with N/M which yields
(39)
(t) n (t) N S. ds
((t,,x()
(40)
We can now simply plug these equations into the dynamics of un(t), nd (t), w0 (t) to obtain the final DMFT equations.
Final DMFT Equations for Data Reuse Setting The complete governing equations for the test loss evolution after averaging over the random matrices (v, A} can be obtained from the following stochastic processes which are driven by Gaussian noise sources {!)( (1), ut()). Letting (.) represent averages over these sources of noise, the equations close
21 = w - (() S (t,
0 (1) = v1) +7 I (sCa(l,8)uk1
(t) = (t) + P I ds ds s)u' (((t))((,CC((.t.))
(1) (1 + A /
do
~10 ~ (o. (o (c(((,)))
(t) = u (t) +
Σ
(s) (t) ~ No. s)) C((tttt)
N
s<t
w(t+ 1) = w(t) + tu (I) + 1/ Σ s3(l))
(1 = u/(()= + Σ
Ra(t, 3) (s) ~
a<t
Oug())) Rt(t,s) = (t)
(v1(()
Ro,((, s) = Σ A
(s)
Rt,a(t, Σ (
dug(()
Ro(t,s) (t)
Out(s)
(u)(s)
Col1, 8) Σ da (vR())(((s) Ci(1s)) = (u' (D))
K C2(t,)) Σ (vR()))(()) Ca(t,s) = (t)
(41a)
The 7 limit of these equations recovers the DMFT equations from Bordelon et al. (002a))
which analyzed the random feature (static A) case.
E BOTTLENECK SCALINGS FOR POWER LAW FEATURES
In this setting, investigate the scaling behavior of the model under the source and capacity ditions described in the main text:
A ~ k- (ww))² AB ~ k -88--
(42)
E. TIME BOTTLENECK
In this section we compute the loss dynamics in the limit of N, - co. We start with perturbative argument that predicts scaling law of the form L(I) (-2(2-9) for 1. We then this
22

---

Published as a conference paper at ICLR 2025
approximation to bootstrap more and more precise estimates of the exponent, The final prediction is 2a
the limit of infinitely many approximation steps which recovers (tt) ~ self-consistency derivation of this exponent verify that is the stable fixed point of the exponent.
We then provide a
E.1.1 WARMUP: PERTURBATION EXPANSION OF THE DMFT ODER PARAMETERS
First, in this section, we investigate the N, B - x limit of the DMFT equations and study what happens for small finite 7, This perturbative approximation will lead to an approximation L ~ t-912-0), In later sections, we will show how to refine this approximation to arrive at our self- consistently computed exponent 1+0 In the N, B oo limit, the DMFT equations simplify to
R3 + I, u 0, u 0, ot + Au C3 + C2.
(43)
The dynamics in this limit have the form
(t) w (t) T ((((s))
sct
"/ (4 1) - (1) (1) + (C((ss)0(()
SCt
w(t + - w = (t) + m Σ s2(t,))u(s)
S<1
C2(1,)) = Σ X (v2(0)2())
Ct,(t = (wi())((s))
(44)
These exact dynamics can be simulated of convergence by studying the following Markovian continuous time approximation of the above dynamics where neglect the extra O(y) term in the of (t) dynamics
do in Figure However, we obtain the correct rate
d (t(t) = Act((t) - Ct
(t) (t,
dt
O,C, (f) A C2(() + +())) C2(() = (t,, The solution for the error along the k-th eigenfunction will take the form
(45)
-Xxst - 7>x >>> f dsCCu(s) w
(1) x exp
(46)
We next solve for the dynamics of C2(t) and (t. in the leading order y - 0 limit (under the linear dynamies)
C2(() ~ Σ e ~ |kk dkk--- Ba-l - ~ 0-
(t) 15-88
g<1 1
(47)
(c0)
3>1
where (0o) is limiting finite value of C.. (t).
Texp (-Aul
B 8<1 1
(48)
exp [-xx[1+nC,n (co))t)
B>1 1
For B < 1, the feature learning term will eventually dominate, learned at time / satisfies
The mode k.(() which is being
(t) ~ (2-/8)/o
(49)
which implies that the
L Σ (wt)² ~ -
(50)
However, this solution actually over-estimates the exponent. To derive a better approximation the exponent, turn Markovian perspective the dynamics which holds as N, B co.
23

---

Published as a conference paper at ICLR 2025
E.1.2 MATRIX PERTURBATION PERSPECTIVE ON THE TIME BOTTLENECK SCALING WITH MARKOVIAN DYNAMICS
The limiting dynamics in the N, B + limit can also be expressed Markovian system in terms of the vector o (t) and matrix (t) which preconditions the gradient flow dynamics
dt v'()) (t))Ao M(t) [N A(t) A(t) + N 1w(t))
dt (t) = = (w. (t)) (t) + 7v (w. - (t)) _ + + - (t) (t)
(51)
We can rewrite this system in terms of the function ( = (1) with = 1 and the Hermitian kernel matrix K = A (1) then
d = -())=)K((A(()
di
d
dt K(t)= = > - A (1) +744 - +2v(y A A A
(52)
The test loss can be expressed as L(t) A (t)
Loss Dynamics Dominated by the Last Term in Kernel Dynamics We note that the loss dy- namics satisfy the following dynamics at large time
dt ((t) = A (t) (t))
((t)A((t) 27 ( ds((y A(s) ((t)]A(t) (A(s)
/ ds(y (s)) A(s)
(1) T (A)
A A4)) 271y (1)) J d= 27((() (A L ds y (s))
Lazy Limit
Subleading
Dominant
(53)
I dsy (1))
(1) AA)(()())
1 27
(54)
One can straightforwardly verify that the middle term is subleading compared to the final term is that under the ansatz that (+( ~ exp (-Agt2-x) where X S 1 for 1. We can therefore focus on the last term when deriving corrections to the scaling law.
Intuition Pump: Perturbative Level Approximation In the lazy limit 7 0, K(I) A for all However, for > this effective kernel matrix ((t) evolves in a task-dependent manner. To compute K will approximate
(t) with its leading order dynamics in y, which obtained by evaluating the (t) dynamics with the lazy learning ? - 0 solution. We can thus approximate the kernel matrix (tt dynamics
K(t) A + YVY (I (-p((At)) + exp(- yy 27 y (I-e---A(AA y) A
(55)
From this perspective we see that the kernel has two dynamical components. First, a low rank spike grows in the kernel, eventually converging the rank one matrix yy T. In addition, there is scale growth of the existing eigenvalues due to the last term yy' A (I exp( At)) y) A, which will approach the value the RKHS norm of the target function as t - oc. The eigenvalues (t) of the kernel K (1) evolve leading order the diagonal entries. Assuming that 3 these terms
24

---

Published as a conference paper at ICLR 2025
increase with t
(1) ~ Au + 2 (1 -e--"" Abt) + 27/x Σ y1
(1 e Act)
Xe
A 2 (1-63 e1-e-Art) + 271xt!
(56)
The dynamics for the errors can be approximated as
0t (x( - Kc (1)(((((())
= (x(1) exp (-f," ds ((dck(lo)) Ka((s)
v Vxaui
exp (-Xat 27)a(w)) 2x) Vauu
(57)
For sufficiently large t, the final term dominates and the mode k (t) which is being learned time / is
(1) 8
(58)
The test loss is simply the variance in the unlearned modes
L(t) (w ~tt 882--9))
(59)
Bootstrapping a More Accurate Exponent From the previous argument, we started with the lazy learning limiting dynamics for ((((() ~ e->ut= and used these dynamics to estimate the rate
which (() (or equivalently k()) changes. This lead to an improved rate of convergence for
the mode errors (t), which under this next order approximation decay as e((() e A wi. Supposing that the errors decay at this rate, we can estimate the dynamics of M (t)
d (1) d Σ (wt)² Aee - - Re Aaat 012-9(2-9)
(60)
dt
(t) ~ exp (-N6t2-0(2-33) w (Level 2 Approximation)
(61)
We can imagine continuing this approximation scheme to higher and higher levels which will yield series of better approximations the power law
Level 0 Approximation
(2(-9)
Level Approximation
C(I)
t -122-9(2-8)
Level Approximation
(62)
--10-8122-02-000
Level Approximation
We plot the first few of these in Figure showing that they approach a limit the number of levels diverges. As n - oc, this geometric series will eventually converge to t
28
E.2 SELF-CONSISTENT DERIVATION OF THE INFINITE LEVEL SCALING LAW EXPONENT
From the above argument, it makes to wonder whether or not there exists fixed point this series of approximations that will actually yield the correct exponent in the limit of infinitely many steps. Indeed in this section, we find that this limit can be computed self-consistently
d M(()) 7dx Σ oi (e))((())
(63)
d1
= M(( = 11 1" do Ewidente
(64)
ux(t) exp (-Me (' at' (e wt
(65)
25

---

Published as a conference paper at ICLR 2025
1.2
1.0
0.8
Level 0
Xn
Level 1
0.6
Level 2
Level 3
0.4
Level 4
Level 5
0.2
Level 6
Level
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Figure 11: Predictions for the loss scaling exponent (B) at varying levels of the approximation
scheme. Our final prediction is the infinite level limit which gives I43- consistency argument.
2 This agrees with a self-
We can define the intermediate variable
B(I) = Σ w1))v((1)
(66)
which has self-consistent equation
~>k L di' 1 1 S' dsB(s))
B(I) >x((w)) exp
(67)
We seek a solution of the form B(I) x_x, This yields
2
x = t- m0.00.0022200002
x B max
1, 8
(68)
Using the solution for (t), We note that this argument also leads to an approximate doubling of the exponent compared to the lazy for < 1, however this is slightly disagreement with the perturbative approach which yields ( B) for $ < 1.
also derive the scaling for the loss which is identical C(t) t
This argument is where developed the general expression that determines X which was provided in the main text
Σ (wt) exp (->x Au [t 7/2 xx)
X = lim too0 In
(69)
E.2.1
EFFECT OF y ON THE SCALING LAW
Using the results of the previous sections, we see that the loss curve transitions from the lazy scaling at time furnsition which satisfies
furansiioon y
fransition R3 ?
(70)
After this time, the loss will be dominated by the contributions from the feature learning term (which involves 7). At time / the mode which is being learned is k: 1//0 2=x The loss will scale
L =z
(w ~ B122 x)
(71)
k>*k
26

---

Published as a conference paper at ICLR 2025
This indicates that modifies the prefactor but will not change the asymptotic exponent provided that it is nonzero. We summarize these results as
1. For times < T-x(o] the dynamics closely track the lazy learning curve.
2. At timescale t R3 -xp the power law transitions from the lazy learning curve to the power law.
3. For all t > y1-x(0), the loss looks like L = -t-
Thus our learning curve in the N, B
limit has the following form.
1-8
1 <<7
C(t)
(72)
x t 1>7
E.3 FINITE MODEL SIZE BOTTLENECK
In the limit of t + oo, the dynamics for (t) will converge to a fixed point that depends on To ascertain the value of this fixed point, first must compute the asymptotics. First, note that correlation and response functions reach the following fixed point behaviors
lim (( dt' Relss(t's)) r3ō(t 8)
6,5
lim R2, = =12,4 Ott -8) 8)
lim Cw(t,s) Cw
7,00000
[' dt'Cstt',,
lim
(73)
8,80000
which gives the following long time behavior
(t) ~ w S( dt' uu (() - >u S dt' L' ds)
(74)
wh S'd dt' delut(()) Axr3el(1)
(75)
Av
72,4 I+ A ART'
(76)
Using the asymptotic relationship between 13/24 arrive at the following self-consistent equation for r's
ApTS
(77)
N A4
For power law features this gives
rra N dk k-ar +1
[rs]]]]]]
1/a
r3 ~3~~~ No
(78)
which recovers the correct scaling law with model size N.
Σ (2
lim C)(1.)) 1,8+000
Naaz
2 N 20 dk k 1+20 + Io
000
dkk Ba- ~ N min (2
(79a)
27

---

Published as a conference paper at ICLR 2025
E.4 FINITE DATA BOTTLENECK (DATA REUSE SETTING)
The data bottleneck when training on repeated P training examples is very similar. In this case, the relevant response functions track are R1 s) and (t s) which have the following large time properties as f, + 00
L dt' R1 riô(t s)
2,, 8) ro2 O s)
Under this ansatz, we find the following expression for ri as +
1 Σ
ARTI
(80)
P 1+ Apri k
Following an identical argument above we find that pa, resulting in the following asymptotic test loss
d (wg)²
lim Co(l, 8) = E 4,8-00
A Par2
~ p-2a 2a '"' dkik dhik-a-- - + f dkk dkk Ba-1 ~ p-o (2,02
(81a)
We that in the offline case, this scaling law in dataset size matches the dataset scaling in the lazy regime. We specifically recover the same exponents as the 7 + 0 limit which was computed in prior works Bordelon et al. (2024)): Paquette al. (2024).
F TRANSIENT DYNAMICS
To compute the transient 1/N and 1/B B effects, it suffices to compute the scaling of a response function Volterra kernel at leading order.
F.1 LEADING BIAS CORRECTION FINITE N
At leading order in 1/N the bias corrections from finite model size can be obtained from the follow- ing leading order approximation of the response function Ra(1, s)
sa(t,s) ~ 5(t ()
Σ
ARE Autto/a (x/1) + O(N-2)
N
5(1 3) (1x/8 8x/8)
1
(82)
1+ Following Paquette et al. 2024), we note that the scaling of Rs s) 5(t 8) determines the scaling of the finite width transient. Thus the finite width effects can be approximated
where x = 3 max (1, 2
L(1, N) - B max/11 1 N min (2,8)
(1-1//a) max(1, riz
(83)
Limiting Dynamics
Model
Finite Model Transient
In the case where this agrees with the transient derived in Paguette et al.) (2024). However for 1, the feature learning dynamics accelerate the decay rate of this term.
28

---

Published as a conference paper at ICLR 2025
F.2 LEADING SGD CORRECTION FINITE B
We start by computing the N + oo limit of the SGD dynamics can be approximated as
Σ ( ) Ax exp (-2Aa + (1+ x(2x)) ( + B ! ds (1,))
Co(l)=
(84)
where X = max 1, I+Ø and (t, s) the Volterra-kenn for SGD Paquette (201
2024
which in our case takes the form
k(t,s) = Σ d exp (-2 [[t + 7x2-x) - xs²-x)))
max 9-xx
(1
((--1)
(85)
Since the transient dynamics are again generated by the long time behavior of (), we can approximate the SGD dynamics as
C(t, B)
max 17 (1- max (f
(86)
Gradient Flow
SGD Noise
As before, the B >> 1 case is consistent with the estimate for the Volterra kernel scaling in Paquette al. 2024).
G LINEAR NETWORK DYNAMICS UNDER SOURCE AND CAPACITY
In this section, show how in simple linear network, the advantage in the scaling properties of the loss due to larger y is evident. Here, we consider a simple model of a two-layer linear neural network trained with vanilla SGD online. We explicitly add feature learning parameter 7, We study when this linear network can outperform the rate of -ー3 given by linear regression directly from input space. Despite its linearity, this setting is already rich enough capture many of the power laws behaviors observed in realistic models.
G.1 MODEL. DEFINITION
Following Chizat al. 2019), the network function parameterized as:
f(x;t) = (f(=:t) f(;; 0)), f(x; = w Ax.
(87)
We let RR and take hidden layer width N so that A E N E RV in the main text. We train the network on power law data of the following form (0, A), y -==".a
(88)
We impose the usual source and capacity conditions on A and w² as in equation 6
G.2 IMPROVED SCALINGS ONLY BELOW 8 <
We empirically find that when 9 > 1, large y networks do not achieve better scaling than small ? ones. By contrast, when 3 < we see improvement to the loss scaling, We illustrate both of these behaviors in Figure 12 Empirically, we observe rate of (-29/(1+8) for these linear networks. This is the same improvement derived for the projected gradient descent model studied in the main text.
G.3 TRACKING THE RANK ONE SPIKE
In the linear network setting, one can show that this improved scaling is due to the continued the growth of rank spike in the first layer weights A of the linear network. By balancing, as in Du et al. (2018), this is matched by the growth 2 This which will continue to grow extensively in time only when 8 < 1. We illustrate these cases in Figure 13
29

---

Published as a conference paper at ICLR 2025
Source Capacity Linear Networs, 2.220008-1.22
Source Capacity Linear Network, a 2,00, pm000
100,
10- 1c-00
10
ys
1o
10
LE55
10
10-3
101
101
10 Steps
10
10*
109 10
10a
10a
10*
Steps
(h)
Figure 12: Linear Networks a) > 1, where across values of y, observe the same asymptotic scaling going as 1-8 as predicted by kernel theory. b) < 1, where feature learning linear networks achieve improved scaling as predicted by our theory.
Linear Network, == 2.00, 8= 1.20
Linear Network, == 2.00, 8= 0 M 100000, N 100, B 128
10000,00,0 N 100, 8 128
Ye 13:03
10a
Ve = 03
10a
y 1e-02
V = 1e-02
=
Vo
V- 1c+00
V. 1c+00
10a
(1-0
27-200 -
10a
10a
10a 10a 109
10a 10
10a
10a
101
10a
10a
10
t
(b)
We study the growth of the spike as measured by /w/2. (a) For easy tasks, where w" is finite, the spike grows finite size, and then plateaus. This leads multiplicative improvement in the loss, but does not change the scaling exponent. (b) When the task is hard, wo continues to grow without bound. Both the perturbative scaling of 21-9 and the scaling 21220(1+3) obtained from the self-consistent equation 10 are plotted. We excellent agreement with the latter scaling.
Figure 13:
H ON THE DEFINITION OF FEATURE LEARNING
Prior works of the lazy/rich dichotomy of training neural networks define two regimes of deep network training (Chizat et al. 2019 Yang et al. 2022 Bordelon et al. 2023). The lazy regime is where the finite with empirical neural tangent kernel (eNTK) does not change (or changes negligibly) over the course of training. The rich or feature learning regime is what results when the network is trained beyond the lazy learning limit.
Definition A A feature learning network is one that is not in the lazy regime. That is, its eNTK changes noticeably over the course of training.
One might want to give a more stringent definition, as the above definition does not answer whether the network has learned "useful features". In general, characterizing and comparing the learned features of a deep network wide open research problem (Kornblith et al, 2019). However, several works (Baratin al. 2021 Fort et al. 2020 Atanasov al. 2022) have consistently observed
30

---

Published as a conference paper at ICLR 2025
that the kernel aligns itself to relevant task directions. This is generally measured by the kernel alignment, given by a cosine-similarity of the task vector y in the eigenbasis of the evolving kernel (see Figure 8 b) (Canatar & Pehlevan, 2022). This motivates more stingent definition of feature learning which reflects fask-relevant adaptation of the kernel.
yy or alternaivelyy in terms of the decomposition |K|ly//
Definition B network is said to learn useful features if the kernel-task alignment improves over its initial value the start of training.
In our model, the networks at y > 0 satisfy both Definition and Definition B. The networks at 0 satisfy neither. We note that neither A or B are sufficient to see improved scaling law, and that one also requires the task to be "hard" in the linear network setting.
31


Reviews:
[{"review_id": "1M92Uf6xsM", "reviewer": "Reviewer_6VnB", "summary": "The paper does a mean-field analysis of feature learning in a model similar to a shallow linear network. Scaling laws are obtained, where depending on the source exponent, feature learning can lead to a improvement in the exponent.", "questions": "The effect observed in this paper differs significantly from other feature learning paper in that it does not assume any form of sparse structure (actually feature learning appears to help when the signal is closest to being of the same intensity along all directions), i.e. low-index models for shallow networks, or low-rank matrices for linear networks. This suggest that the type of feature learning observed here is quite different from these other line of results. My intuition from your paper, is that feature learning may in some sense improve the conditioning of the Hessian/NTK, allowing faster convergence, whereas in the presence of sparsity feature learning seems to rather make the NTK low-rank and more ill-conditioned. I was therefore wondering whether the effects observed here might disappear when switching from online learning to GD on a fixed dataset, because it would untangle the training speed from the number of datapoints observed, and maybe we could see there that feature learning makes convergence faster but maybe one still converges to approximately the same solution (and thus same test error).\n\nTypos:\n- on page 5: The sentence \"The first terms represent bottleneck/resolution-limited scalings in the sense that taking all other quantities to infinity and studying scaling with the last ...\" does not finish.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper uses advanced theoretical tools to tackle the important question of the effect of feature learning. A complete phase diagram is obtained, showing under which condition feature learning leads to a significant advantage. The theoretical scaling laws are checked empirically, and seem to even extend to some degree to deep nonlinear networks (which are not covered by the theory).", "weaknesses": "The paper does not give much intuition for why this improvement of feature learning is observed, instead the authors simply mention the existence of a self-consistent dynamic derived in the appendix.\n\nI also feel that the authors do not relate much to the already existing litterature on the dynamics of linear networks (e.g. https://arxiv.org/abs/2211.16980), where feature learning and self-consistent dynamics have been described in https://arxiv.org/abs/2406.06158 or https://arxiv.org/abs/2405.17580 . It would be interesting to know whether the dynamics observed in this paper is similar to the ones observed in these papers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3HUxWz1aUn", "reviewer": "Reviewer_tuhi", "summary": "The paper introduces a solvable two-layer linear model trained using a variant of stochastic gradient descent (SGD) to predict the scaling laws beyond the kernel (lazy training) regime. The authors identify three scaling regimes—hard, easy, and super easy tasks—under which feature learning can improve the scaling of the loss with respect to training time and compute. The paper also provides experimental validation of these theoretical predictions by training deep, nonlinear models to demonstrate the applicability of the scaling laws.", "questions": "See weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. Tractable Theoretical Setting: The paper presents a solvable theoretical model, which allows for potentially clearer and rigorous exploration of neural scaling laws beyond the kernel regime. This tractable setting potentially makes it easier to understand and predict the dynamics of feature learning in neural networks.\n\n2. Extension of Prior Work: The authors improve upon simpler one-layer linear models from prior work by introducing a more sophisticated two-layer model. This extension potentially can provides deeper insights into the scaling behavior of neural networks.", "weaknesses": "1. Complexity and Assumptions: The paper is difficult to follow due to the heavy reliance on prior work. This reliance can make it challenging for readers who are not deeply familiar with the specific foundational literature to fully grasp the contributions.\n\n2. Unclear Definition of Feature Learning: Throughout the paper, the concept of \"feature learning\" is not clearly defined. It remains unclear how the authors measure feature learning or provide a tangible understanding of it. Although allowing \nA(t) to evolve leads to better performance, this change doesn’t adequately explain how features are being learned or evolved.\n\n3. Limited Model Scope: The model seems quite limited in its design. Allowing \nA(t) to change is reasonable, especially since the target function is 𝑊∗𝜓_{inf}(x). However, it is unclear why this is believed to represent or mimic feature learning in practical, real-world scenarios. The connection to real feature learning remains vague. This is also related to 2nd point, that I do not understand how authors define feature learning.\n\n4. Transferability to Deep Nonlinear Models: It is not clear why the results derived from the proposed linear model are expected to transfer well to deep, nonlinear models. \n\noverall I feel like the paper is scattered and I can not grasp the main point authors are trying to make about feature learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QbAUB77FMR", "reviewer": "Reviewer_q6mk", "summary": "This paper studies the scaling laws of neural networks in two regimes: lazy learning and feature learning. Eq (8) and (9) describe the power-law variation of the loss with training time \\( t \\), number of parameters \\( N \\), and batch size \\( B \\) in these two regimes which show that feature learning accelerates the convergence of hard tasks.", "questions": "From the experimental results in Figures 9 and 12, we can see that the scaling law does not change suddenly at $\\gamma = 0$, but changes gradually at a very small value of gamma. Can this paper's analysis cover this aspect?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper constructs a comprehensive scaling law framework, reveals the power law relationship between loss and the number of parameters, training time, and batch size, and provides a theoretical basis for understanding the performance of neural networks.\n\n2. This paper conducts a series of experiments to verify the theoretical results, making the conclusions of this paper more convincing.", "weaknesses": "1. This paper ignores the impact of gamma on the feature learning scaling law.\n\n2. The paper lacks an analysis of the critical state between the two regimes.\n\n3. (Minor) The colors of lines in the figures are not differentiated enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3oJFwEa4BG", "reviewer": "Reviewer_QCyn", "summary": "The paper studies scaling laws of the training loss with respect to relevant parameters (training time, model size and batch size), with a focus on online training, where dataset size is assumed to be unlimited.\nThey study training dynamics through the lens of the eigenspace decomposition of the NTK. Such eigenspace is used both to (1) formally define capacity and source of model and task and (2) to express the loss behaviour explicitly, both in the lazy learning regime ($\\gamma=0$) and in the feature learning regime ($\\gamma>0$), the former can be solved explicitly while the second is approximated with DMFT. Such analysis reveals two very different regimes depending on the value of the source, specifically in the hard regime (source<1) they prove better scaling laws than state of the art. Such scaling laws are studied and discussed in each component. Empirically are shown to be tight on a variety of task, including synthetic and MNIST vision, but they are shown to no be tight in the CIFAR case.", "questions": "In line 160 the \"allow\" pops a bit out of nowhere and it hard to grasp. Following with Lines 175-176, my understanding is that you choose an arbitrary kind of feature learning dynamics, with the implicit assumption that SDG is sufficient (but not necessary) to obtain such dynamics. Being sufficient means that the scaling law you derive is at least an upper bound on the loss. While not being necessary means that a different choice of modelling dynamics for $A(t)$ may lead to tighter scaling low (which may for example explain the CIFAR experiment). Is this understanding correct? Is such implicit assumption true? Can you please argument a bit more over this choice of feature learning dynamics?\n\nThe reference \"Bordelon et al. (2024a)\" appears very often in the paper. It would be nice to have a short paragraph (maybe in the appendix) to clarify the common point with such paper and the novel contribution of this one over that one. Can you reply with a draft of such paragraph?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is very well written and easy to follow, overall very pleasant to read.\nSettings and limitations are well clarified: the authors are very open about the limits of their theory and the non-success of the CIFAR experiment. This behaviour is super valuable and very helpful for future works and possible improvements.\nDerivations are explained both intuitively and formally: key concepts are present in the main paper with a good trade-off between formalism and explanation; technical details not directly useful to understand the concepts are properly deferred to the appendix.\nResults are clear and well discussed: for example the scaling laws are divided in several components, each of them is further discussed and empirically studied with ad-hoc experiments.", "weaknesses": "The paper is overall solid and I don't see major weakness.\n\nA big assumption made is the online setting, and the extension to finite dataset setting in Appendix D is somehow limited. However such kind of limitation has to be expected from a theoretical paper. Of course a general case analysis would be preferable, but that would very ambitious and I don't think this assumption can be considered a major weakness.\n\nThe dynamics mean field theory is not at all explained in the main paper, understandably. An informal explanation attempt is done in Line 202-211. While I consider this attempt valuable, the Equation (in line 206) is too much out of context, the notation is not defined and it ends up being practically useless for the reader. I recommend either removing (and using that space to expand the intuition informal explanation) or extending it, explaining the terms involved.\n\nMinor detail. In Line 133 \"we then diagonalize...\" can be misinterpreted as \"we discard the non-diagonal elements\". I think overall the message is clear because right after (in Line 134) you refer to eigenfunctions, but I'd recommend trying to reformulate this sentence to avoid ambiguities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PsBael0OCr", "reviewer": "Reviewer_YcZy", "summary": "This paper studies a minimal model for scaling laws in generalization error under feature learning non-linear neural networks. This model induces improved scaling exponents compared to kernel regression through feature learning (only) for target functions outside of the initial NTK’s RKHS. Consequently, different scaling rules of parameters and training time are asymptotically compute-optimal. These scaling rules are empirically verified in toy settings.", "questions": "Partially critical questions ordered from most to least important:\n\n1. **Large feature learning regime.** Which $\\gamma_0$ is optimal and is this another hyperparameter that has to be tuned, or are there useful heuristics for choosing it? In your experiments it looks like the larger the better, but the $\\gamma_0$ you consider are still pretty small. I am guessing in Figure 6 (b), larger values of $\\gamma_0$ would result in a better than predicted scaling exponent even earlier in training? If so, this would be an acceptable but important limitation of the studied model, which should be acknowledged. I suggest running the experiment for larger $\\gamma_0$ and discussing this limitation more deeply.\n2. **Importance of constants.** While reporting the pure scalings is nice for readability, in practice, how important/huge are the omitted constants? This could arbitrarily change the global optimum of the test error given finite resources. Towards answering this question of how practically useful the results are, I suggest an experiment that compares the predicted compute-optimal scaling to empirical loss measurements like in Figure 4 but for the more practical datasets in Figure 6.\n3. **Error of the approximation.** How good is your approximation in eq. (9)? In which cases does it fail?\n4. **Extension to label noise.** How would label noise affect the results?\n5. **Long training regime.** What is happening under long training in Figure 6b? Do you have an informed guess? Are structures in the images learned that are not covered by your simple distributional assumption?\n6. In eq. (6), you state $\\lambda_k\\sim k^{-\\alpha}$ and $\\lambda_k^{-\\beta}\\sim k^{-\\alpha \\beta}$. Is this a typo?\n7. In l. 192, ‘the components of $\\omega^*, v^0(t)$ in the k-th eigenspace of $\\Lambda$’, but $\\Lambda$ was defined as a diagonal matrix in eq. (2). I guess here the k-th eigenspace of $K_\\infty$ is meant?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is very well written and easy to read. To the best of my knowledge, this is the first paper that theoretically explains compute-optimal scaling laws in non-linear feature learning networks beyond the lazy regime. Separating the four bottleneck mechanisms in eq. (9) and achieving improved scaling laws through feature learning are particularly interesting accomplishments.\n\nOverall, although this paper only studies a toy learning algorithm, I believe it makes a valuable contribution towards a theory of scaling laws for finite, feature learning neural networks, which is why I recommend acceptance.", "weaknesses": "Besides partially critical questions below, I consider the following points to be the main weaknesses of this paper.\n\n- **Artificial learning setup.** The projected gradient descent with frozen first-layer weights is a constructed training procedure. This may make the analysis easier, but it remains unclear how well the results are transferable to more practical optimization procedures like SGD or Adam on **all** weights. Do you think your results can be generalized to realistic training? Do you train all weights in Section 5? This is not explicitly stated.\n- **Oversimplifications.** The simplifications that make the paper very easy to read sometimes lead to oversimplifications and imprecision. For example, in l. 191, what is $N,B\\gg 1$ formally supposed to mean? Overall, the approach taken in the paper seems very heuristic. For example, starting with finite $M$ and taking the limit $M\\to\\infty$ omits many mathematical technicalities, why the claimed limiting properties and statements are well-defined and hold.\n- A small limitation is that long training times are not well covered (Figure 6b). But this is acceptable as the long training regime is, in general, still very open.\n\n**Typos in lines:** 128, 201, 202, 476.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)