Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks
Authors: Francesco D'Amico
Dario Bocchi
Matteo Negri
Venue: ICLR 2026 Poster (2026)
Keywords: Neural scaling laws
Implicit bias
Learning curves
Spectral complexity norm
Perceptron theory
Primary area: optimization
TL;DR: We connect neural scaling laws in deep networks with the implicit bias induced by logistic losses through a surprisingly simple perceptron theory.

Abstract: Scaling laws in deep learning -- empirical power-law relationships linking model performance to resource growth -- have emerged as simple yet striking regularities across architectures, datasets, and tasks. These laws are particularly impactful in guiding the design of state-of-the-art models, since they quantify the benefits of increasing data or model size, and hint at the foundations of interpretability in machine learning. However, most studies focus on asymptotic behavior at the end of training. In this work, we describe a richer picture by analyzing the entire training dynamics:  we identify two novel \textit{dynamical} scaling laws that govern how performance evolves as function of different norm-based complexity measures. Combined, our new laws recover the well-known scaling for test error at convergence. Our findings are consistent across CNNs, ResNets, and Vision Transformers trained on MNIST, CIFAR-10 and CIFAR-100. Furthermore, we provide analytical support using a single-layer perceptron trained with logistic loss, where we derive the new dynamical scaling laws, and we explain them through the implicit bias induced by gradient-based training.

Published as a conference paper at ICLR 2026
IMPLICIT BIAS PRODUCES NEURAL SCALING LAWS IN LEARNING CURVES, FROM PERCEPTRONS TO DEEP NETWORKS
Francesco D'Amico*22 Dario Bocchi Matteo Negri1.2f IPhysics Department, University of Rome Sapienza, Piazzale Aldo Moro 5, Rome 00185 2CNR Nanotec, Rome Unit, P.le Aldo Moro 5, 00185 Rome, Italy {francesco damico, dario. bocchi, mwttao. mat.ae......./eammnteeeee.. (oggrr))@niromm..1.
ABSTRACT
Scaling laws in deep learning - empirical power-law relationships linking model performance to resource growth have emerged as simple yet strik- ing regularities across architectures, datasets, and tasks. These laws are particularly impactful guiding the design of state-of-the-art models, since they quantify the benefits of increasing data or model size, and hint at the foundations of interpretability studies focus on asymptotic behavior at the end of training. In this work, we describe a richer picture by analyzing the entire training dynamics: we identify two novel dynamical scaling laws that govern how performance evolves as function of different norm-basen complexity measures. Combined, new laws recover the well-known scaling for test error at convergence. Our findings are consistent across CNNs, ResNets, and Vision Transformers trained on MNIST, CIFAR-10 and CIFAR-100. Furthermore, analytical support using a single-layer percoptron trained with logistic loss, where we derive the new dynamical scaling laws, and we explain them through the implicit bias induced by gradient-based training.
machine learning. However, most
provide
INTRODUCTION
Neural scaling laws have emerged as powerful empirical description of how model per- formance improves data and model size grow. The first kind of scaling laws that were identified show that test error (or loss) often follows predictable power-law declines when plotted against. increasing training data model parameters. For example, deep networks exhibit approximately power-law scaling of crror with dataset size and network width or depth, phenomenon observed across vision and language tasks (Hestness et al.) 2017 Sun et al. 2017 Rosenfeld et al.] 2019). Such results highlight the macroscopic regularities of neural network training, yel. they largely summarize only the eud-of-truininn behavior.
Since the advent of large language models, neural scaling laws started to include the training time, especially in the form of computational budget spent train a given model. A seminal work (Kaplan et al.) 2020) demonstrated that cross-entropy loss scales as a power law in model size, data size, and compute budget, up an irreducible error floor. These empirical neural scaling laws, including those for generative modcling beyond language (Henighan et al. 2020), indicate remarkably smooth improvement of generalization performance as resources increase. The main interest of this research line is, given fixed compute budget, to find optimal way to allocate between model size and training data such that final performance is maximized (Hoffmann et al. 2022 Even though empirical results show clean scaling laws spanning for many decades, in particular for language models, there are cases where there different regimes with different exponents (Caballero et al. 2023). A review that compares various recent methodologies for mcasuring neural scaling laws can be
"These authors contributed equally to this work. Current address: I.PTM, CY Cergy Paris Université, 2 avenue A. Chanvin, Pontoise 95302

---

Published as a conference paper at ICLR 2026
found in Li et al. 2025 Notably, in contradiction to scaling laws, which are scale-free, some capabilities of large language models emerge at a certain scale (Wei et al.) 2022]. However, it is debated if such phenomena are intrinsic properties of scale or rather of the metrics used Schaeffer et al. 2023).
A complementary line of research studied the so-called implicit bias of gradient-based learning dynamics. Implicit bias refers to the inherent tendencies of optimization algorithms to favor certain types of solutions, even without explicit regularization or constraints. For instance, gradient descent. often finds solutions that generalize well in overparameterized models [Neyshabur al.) 2014 Zhang et al. 2017 Arnaboldi et al. 2024). Theoretical results have shown that for linearly separable classificaiio tasks, gradient descent on exponential or logistic losses converges in direction to the marimum-maryin classifier (Soudry et al., 2018), and analogous bias toward maximizing margins has been proven for deep homogeneous networks such as fully-connected ReLU networks (Lyu and Li 2020 as well as certain wide two-layer networks (Chizat and Bach 2020).
In this work we join these perspectives together by asking whether the implicit bias of gradient descent might itself induce prediciable scaling behavior throughout the training process, in models trained with logistic loss. The results are organized as follows. Section 2 focuses on perceptrons. We first observe a surprisingly good agreement between dynamical learning curves and analytical predictions from the static models with norms fixed at values corresponding to each training stage. We interpret this agreement as training-time implicit bias. Then we use the analytical predictions to highlight new dynamical scaling laws, by plotting learning curves as a function of the model's increasing norm. Finally, we show how the new scaling laws can be used to derive established neural end-of- training scaling laws. Section B focuses on deep architectures. By using generalized notion of norm, we reveal that the same set of scaling laws is present in deep networks, consistently across architectures and datasets, robust against. alternative choices of norm, training algorithms and regularization (the exponents do depend on those details). In section 4 we discuss the limits and potential consequences of these results.
Related works. The perceptron has long been a canonical model in the statistical me- chanics of learning. Early work established its storage capacity using replica methods, identifying the critical nattern-t-ddimenionn ratio beyond which classification fails (Gardner)
1987 Gardner and Derrida 1988). J. Later studies analyzed learning dynamics, including exact
convergence times (Opper 1988), the superior generalization of maximum-margin solutions
1990 and Bayes-optimal learning curves as performance benchmarks (Opper (Opper et al.
and Haussler 1991.. ). Online learning was also investigated, with analyses of sequential up- dates [Behl and Riegler 1994), exact teacher student dynamics in multilayer and committee machines (Saad and Solla 1995a bj, and Bayesian online approaches (Solla and Winther 1998).
Our main focus is to highlight the role of the norm growth to describe the learning dynamics, which is a perspective that is absent in the classic works. To do that, we use the solution of logistic regression with fixed norm that was studied in Aubin et al. (2020). In our work we present an equivalent calculation that reveals the implicit bias at training time and, as a consequence, the new scaling laws. The idea that implicit bias can extend to the whole learning trajectory can also be found in Wu al. (2025) restricted to the overparametrized regime.
Few studies on scaling laws include training time independently of the computational cost. Simple models in controlled settings exhibit a power law in the number of training steps (Velikanov and Yarotsky 2021 Bordelon et al. 2024), favoring the discussion on the trade-off between model scale and training time (Boopathy and Ficto 2024) that is central to the compute-optimal scalings. Particularly relevant is Montanari and Urbani (2025), where the authors connect the different dynamical regimes of a committee machine to its norm, suggesting that the same ideas that we present in our work can apply even outside the setting of logistic loss. In fact, in the case of regression with square loss, gradient descent is biased toward minimal no-m solutions when there are many interpolating solutions (Gunasekar
et. al. 2017
2

---

Published as a conference paper at ICLR 2026
2 10-1.
Generalization error Egen
Hebb
Generalization error Egen
10-1
Solution
Training free norm perceptron
fixed norm perceptron
10-1.
Max Stability
Bayes
10-2
d 6.25
d- 50.00
12.50
100.00
10
10
100
10
10 10-1
10a
10'
102
Norm
Norm
Figure 1: The learning curve of a perceptron with free norm resembles that of fixed-norm problems, which interpolate between known learning rules. Left panel: We plot the generalization error of the minimizers of the cross-enlropy loss in a stadher-student setup a fixed ratio a = 5 of number of data over size of the system. The blue curve represents the analytical result obtained under fixed-norm constraint (with À as the hyperparameter of the loss), while the multicolored curve where color varies with training
the result of numerical training in the free-norm case, where À corresponds to the norm of the weights; the model is trained with 10 steps of gradient descent. The horizontal lines indicate the generalization error of classical learning rules. Right panel: Same analysis for different values of a; solid curves are analytical solutions at fixed norm, dashed curves are trajectories with free norm.
2 SCALING LAWS IN LEARNING CURVES OF PERCEPTRONS
This section introduces the core intuitions that we will use for deep architectures plotting learning curves as function of the model's norm in a setting where we have analytical control of the optimization process.
In the case of perceptron trained on lincarly separable data, it is known that the implicit bias of gradient descent drives the weights toward the maximum-stability solution (the direction that maximizes the classification margin) while the norm grows over time (Soudry et al.] 2018). In this section, we ask if the implicit bias has a role intermediate slages of Training. Using the well-established (eacher student framework (Gardner and Derridal 1988), we show that the model's behavior throughout training is qualitatively captured by the solution to the problem in which the norm is held fixed (Aubin et al. 2020). This correspondence allows us to relate the evolution of the perceptron's 's norm during training to classical perceptron learning rules, offering picture on how the implicit bias influences learning dynamics.
Model definition in Teacher-Student scenario. To have an analytical prediction of the generalization error, consider framework where student perceptron w E RN attempts to learn an unknown teacher perceptron w" e R.W from P = aN labeled examples. Each example x N is random vector with i.i.d. components =f sampled from Rademacher distribution P(x!") 2 _0(2," 1) 00x + 1). The corresponding labels are generated by the teacher as y = sign(x" - w²). We assume both w" and w to lic on the N-sphere, i.c., | 1| |||||2 N. In this setting, the generalization error (or test error), defined the expected fraction of misclassified examples on new data, can be writlen as e = aarcoos( (R), where R = (w w²) w²)/N is the normalized overlap between student and teacher. The student minimizes a loss function L(w). We study the logistic loss, which reads:
L\(w)= -- (A4" log cosh (AA4)) Ev
(1)
3

---

Published as a conference paper at ICLR 2026
Relative generalization error Egan
10a
(Appt, E
10"
10
Aupt
10
30'
Eont
3 10
4 10
112
2 10
2x10
25.0000
50.00
0-12.50
a-100.00
10"
10 1
10
10
109
10
10
10
10
10
Norm A
AfAopt
Figure 2: Fixed-norm perceptrons exhibit scaling laws in the curves of relative generalization error vs norm. Left panel: we plot the generalization error of the minimizers of the cross-entropy loss in the lixed-norm teacher student setup of the perceptron, rescaled by the error of the Hebb rule eq, as a function of the hyperparameter for different values of a. The stars correspond to the optimal points (Nope, fopt); i.e., the minima of the generalization error for each curve. Right panel: we show the same curves after rescaling each one by its corresponding optimal point. The insets display the power-law dependencies of dopt and Copt as functions of a.
where we defined the margin of the j-th example as A yd (w=x"
vN and hyperpa- rameter controlling the sharpness of the logistic loss. Note that for the logistic cost Vi(A) chose the expression in Eq. (I) instead of the more common (but equivalent) In e eA because the former is more convenient to discuss the limits in X. For large N, the properties of the minimizers of Eq. (1) can be analyzed via the semi-rigorous replica method from the statistica mechanics of disordered systems, which out outputs the average value R from the solutions that minimize Lx- In Appendix A we present derivation alternative to that in Aubin et al. (2020). where we focus the analysis on the role of the growing norm, which allows notice the implicit bias training time. This observation led notice that we can use the norm A(t) as a measure of training status at time t, which is one of the key contributions of our work.
A- Regimes the Logistic Loss. In Figure 1 we show the analytical generalization error as function of X, revealing three regimes:
1. Small À regime (A 0): The second term of Eq. vanishes O(A), yielding VA) = -A, A, which corresponds (see Engel and Van den Brocck (2001) to the Hebbian learning, and defines baseline generalization error eg.
2. Intermediate regime and optimal At finite value Aupt the generalization error is minimum. We find that this optimal copt matches the generalization error achieved by the Bayes-optimal predictor (Opper and Haussler 1991), suggesting that the logistic loss rule can achieve Bayes-optimality when À is properly tuned. The dependence of )apt(a) on a is shown in the top insct of the right panel of Figure 2
3. Large À regime (A+ o0): The loss becomes: (A) --20)))))(((A))) -2)0(-A), where we defined the step function 0(x) has degenerate set of minima in A for A A20 0. In contrast, for any finite value X, the minimizer of VA(A) is unique. For this reason, we cannot apply our method directly to this potential. To recover the generalization error 6o in the limit À oo, one must first solve for finite À and then take the limit - oc. We find that this limiting behavior corresponds to the generalization error of the maximally stable perceptron WmaxStable = argmax [min, A (w)) (Gardner 1987 Opper et al. 1990).
x and 0(x) = elsewhere. This loss
w
In Figure I we presented curves for a > 1 because the scaling laws appear more clearly, but the same regimes are present also when a (see Fig in Appendix B
4

---

Published as a conference paper at ICLR 2026
Norm scaling and interpretation. An important observation is that the logistic loss defined in Eq. (1) depends only on the product YA (up to an overall multiplicative factor of À that does not affect the location of the minimizers), where A is linear in the norm of the perceptron weights ||w|| Rescaling the weight norm is thus equivalent to adjusting X, meaning that analyzing a fixed-norm perceptron with varying À is equivalent to studying the minimizers of the loss at fixed À and varying norm. This insight also helps explain the behavior of coo: it is known (Soudry et al., 2018 Montanari al., 2024) that in the infinite-norm limit, the perceptron converges to the maximally stable solution during training (implicit bias). Building on this observation, we compare two scenarios: the fixed-norm case, where the norm |//l²²= = N is fixed and À is treated as lunable hyperparameter of the loss (the results in this setting are obtained with the replica method); and the free-norm case, where the parameter in the loss is fixed to 1 (i.e., we use the classical logistic loss), and the norm [[w(t) X(t) is left free to evolve during training (here the perceptron is trained using standard gradient descent optimization techniques, and the results are obtained from mumerical simulations).
In Figure 1 we compare the generalization curves under these two scenarios. We remark that in the fixed-norm case, each point on the curve corresponds to the endpoint of training for a different rerceptron (at given A), while in the free-norm case, the curve represents the trajectory of a single perceptron during training, with each point corresponding to a different time step as the norm evolves. We see that the free-norm trajectory is qualitatively well described by the set of fixed-norm optimal solutions, indicating that the fixed-norm static analysis captures the essential features of the learning dynamics.
Scaling laws in learning curves at training time. From the left panel of Fig. I we observe that for sufficiently large a the curves share the same slope but differ in their starting point - that is the generalization error en of Hebbian learning (for large a, 60 ~ a -1/2). To highlight the power law scaling in X, in the left. panel of Fig. P we plot relative error Egen = Egen/Eo /eo as a function of x. We observe that for sufficiently large values of a, the learning curves of the relative error split into two distinct regimes, which behave differently as we vary a.
1. An early power-law regime, independent of a. The initial part of each learning curves follows the same shape for any a, up to a value delbow where it saturates. The curves collapse for X < Xelbow (a) on the power law
&gen k2 + Q1-
Here we introduce the term @i to be general, but in the perceptron we have qı 0. Keeping Q will be useful in the next scction deep networks, where it we will connect to the èrreducible error floor of rcalistic settings.
2. A late regime, which depends a. After (a)) the learning curves deviate from the power law and saturate or overfit following a curve whose hoight depends on a.
It is possible to find proper scalings that collapse also the late-phase curves (actually, the whole training curves will collapse). First, we need to discuss the scaling law for the point of minimum test crror (a). In the inset of the right panel of Fig. 2 we observe that the curves follow the power law
Aop== = 22a + Q2
(3)
Like Qı, the term d2 not needed in the fixed-norm perceptron, but we introduce it to obtain more general law applicable to deep networks. Now we can compute 6apt = ? &(Aape) and rescale the learning curves of the left panel horizontally by (apr(a) and vertically by copt (see Fig, 2 right panel). For large values of a, the curves collapse onto single master curve, ie. égen/Eopt = /gen/topt (>//Aopt)
(4)
for some universal function @. Note that it is a common practice when studying neural scaling laws to drop models trained with too-small datasets (see Li et al. (2025)) ) and the fact that our scaling laws appear only for large a provides a natural justification for this practice. We also stress that these scaling laws are not a general phenomenon with any choice of loss function: in Appendix [ as a counterexample, we plot learning curves for Mean Square Error (MSF), which do not show scaling laws.
5

---

Published as a conference paper at ICLR 2026
CNN
ResNet
VIT
MNIST
16
15-1
10
40m0
100
10
Generalization error CIFAR1O
3x10
1014
100
200
40
300
3010
CIFAR100
3K30
10
6x10
1430
1
30
10
10M
1000
101
400
Norm A(t)
Figure 3: Early-training learning curves collapse into power law when plotted as a function of the spectral complexity norm. We plot the generalization error as function of the norm X()) for different datasets and model architectures. Different colors in the same panel refer to training curves with increasing values of the dataset size P, ranging from small (blue tones) large (orange tones). The specific values of P used for each dataset-model combination are listed in Appendix
Connection to end-of-training scaling law. It is tempting to combine the two scaling laws in Eq. 2 and 3 to recover the well know scaling law (g) a² at the end of training (Hestness et al. 2017). However, Eq. is valid only for À < Aelbow(()) while Aapt(a) (a) > Aathow (a). Theretore, substituting Eq. into Eq. 2 scems
invalid step. Still,
in the limit of large a, Eq. g implies that the whole learning curve has the same power-law scaling with a, and therefore
Eq. 3 for any X. Plugging Eq. 3 in Eq. 2
we obtain
(g) = k (k 42)
92)=~ 71 + 41
(5)
For the perceptron Eq. simplifies to egen (a)
7172 and
recover 7 72 For
the fixed-norm perceptron we obtain 71 = - 1/2 (Fig. 2 left panel) and 72 1 (Fig. 2
right panel, upper inset), which recovers 71 7 1 (Fig. 2 right panel, lower inset..
Exponents computed for free-norm perceptron are we are unable estimate y in the free-norm case because training number of gradient deseent steps that is exponential in À (Soudry et al. 2018). In Appendix Dwe provide analtical arguments to obtain the exponent in the fixed norm case and describe the numerical methods that we used to compute exponents in both cases.
0.4901 + 0.0005 and 72 = 0.96 + 0.25;
large and À requires
3 SCALING LAWS IN LEARNING CURVES OF DEEP ARCHITECTURES
Methods. Motivated by results on perceptrons, we repeat for deep architectures the analysis of the test. error versus increasing during training X(t). We test simple

---

Published as a conference paper at ICLR 2026
CNN
ResNet
VIT
MNIST
10
10.
10
1.5%% Upp 1p0
100
L.
100
E/E CIFARIO
1.8% 100
300
100
11%
10
2010
10
200
10"10
10
100
100
300
LAx tp9
CIFARLOO
Lib 100
2D
300
10
10"
10-
10
30
10t
A(t)/Aopt
The whole learning curves collapse at large with the proper scalings. We plot the generalization error function of the norm X(t) for different datasets and model archilectures, rescaling each curve by its optimal point (Aope enpt (a:).. Different colors in the same panel refer to training curves with increasing values of the dataset size P, ranging from small (blue tones) to large (orange tones). The values of P used for each dataset-mode combination are listed in Appendix G
Figure 4:
CNN model (that in the following we will simply call "CNN") [LeCun al., 1998a), ResNet He et al. 2016) and Vision Transformer (Dosovilskly et al. 2021 architect ures for image classificatin over MNIST (LeCun al. 1998b). CIFAR10 and CIFAR100 (Krizhevsky and Hinton 2009) datasets. For each dataset and architecture we make a standard choice of lyperparameters (see Appendix G), without using a weight decay. Results with moderate select a random subset and we train for fixed number of epochs, large enough to see the test error overfit or saturate. We do this procedure for all values of P selected and then we repeat the training a number times varying the random subset and the initial condition of the training. See Appendix G for more details.
weight-decay are reported in Appendix H For each experiment, of elements from training
For the norm definition in the case of deep networks, in the main analysis we opt for the spectral complexity defined in Bartlett et al. (211)), In that work, the authors show that this quantity has desirable propertics for a norm, such as yielding a converging margin distributions that reflect the complexity of the datasct. In Sec. g Given the set A of weight matrices A;, the spectral complexity norm RA of the models reads
i p: lll.
Σ
L
JA
3/2
RA =
|| 2/3
where L is the total number of layers in the network, P; is the Lipschitz constant the activation function (e.g. for ReLU: A 1), A, the linear operator layer for dense

---

Published as a conference paper at ICLR 2026
Model
Dataset
Ppred
Ymmas
Goncralization ciror c(P)
P.
CNN
MNIST
0.60
0.55
0.09
CNN
CIFAR10
0.28
0.25
0.07
CNN
CIFAR100
0.16
0.16
0.03
ResNet
MNIST
0.57
0.69
0.08
ResNet
CIFAR10
0.54
0.56
0.04
ResNet
CIFAR100
0.31
0.37
0.03
10
ViT
MNIST
0.47
0.54
0.03
VIT
CIFARIO
0.23
0.21
0.03
10'
10a
10a
10a
1010
Number of data
ViT
CIFAR100
0.14
0.12
0.04
Figure 5: The combination of two power laws reproduces known scal- ings. We plot the combined power-law scaling of the generalization error as a function of the number of data (Equation (). The parameters of the power-law are chosen of the same order of magnitude as typical results obtained for deep networks.
Table Predicted vs. measured (P) ex- ponents across datasets and architectures. The exponent "pred is computed by indepen- dently fitting Yi and 72, and combining them as Ypred = 7/12-- The exponent "meas is obtained by fitting the c(P) curves directly. The value of a represents an estimate of the variability of the overall process (see Appendix F for details).
layers and it is an appropriate matrix for convolutional layers (see Bartlett et al. (0017) for complete explanation). The so-called reference matrix M; is chosen 0 for linear or convolutional layers and as the identity for residual layers. Then, ||A:llo is defined as the largest singular value of A, and ||Al|,,1 1 is defined as the average of the f--norms of the column vectors.
Throughout rest of the paper, when we write X(t) for deep architectures we mean the spectral complexity norm RA(t). measured after t training epochs. We can give an intuition on Eq. by analyzing the contribution of the two terms. Given a layer i, first term is the maximum amount. that an input vector can be expanded in the output space, and second term is correction that estimates the effective rank of the outputs of the layer, that is the number of columns that have weights substantially different from zero. In Appendix D (Fig, 10 we show that the relation between and is non trivial, and that simply plotting e(t) does not reveal the same scalings that plotting c(A(t) does. We always observe the monotonicity of X(t) if a weight-decay is not present.
Main result Dynamical scaling laws. In this section we consider three architectures without changing their sizes, so P and are interchangeable. In Fig. 3 we report the learning curves mediated over different runs, for many values (the values change for each datasets and are reported in Appendix G. together with the other details of the training process). Notably, we find the same dynamical scaling laws that we observed for perceptrons: the learning curves have an early training regime independent on P and late training regime which depends on P (compare Fig. 3 to the left panel of Fig. 2). In Fig. we rescale the learning curves in the same way did for perceptrons (dividing the axes by the optimal norm and optimal error) fiding that they collapse for large P (compare Fig. 4 to the right panel of Fig. 2.. Note that, at variance with perceptrons, it is sulficient to plot the generalization error to reveal the scaling laws (and not the relative error). We stress that the collapse of the learning curves is surprising because we are far from the regime where P is effectively infinite (since increasing P still decreases the generalization error of models): we have a different loss landscape for each value of P, and the carly stage curves at large P includes the carly stage curve of all loss landscapes at lower P.
Recovery of scaling laws at convergence. A natural question is wether we can use the measured values of y and 72 to recover the end-of-trainngg scaling law in Eq. 5 also for deep models. In this case, since q2 # 0 in general, we need to isolate the proper power law regime. As we show in the sketch in Fig. 5 it is possible to identify
Main result 2:
8

---

Published as a conference paper at ICLR 2026
two thresholds P- (22/k2) and P+ (K1kg //1) 1/(mn)), which distinguish between three regimes: 1) P < P-, where <(P) k192 + q1- In this regime, expect <(P) to be close to random guessing, which for classification is k192" + Q1 - (n 1)/n, with n the number of classes. 2) P- P < P+, where <(P) ~ ki p- The exponent y = corresponds to the neural scaling law observed in Hestness et al. (2017). 3) P > P+, where c(P) + 91 Here we approach the lowest possible error of the dataset and the performance saturates.
For each archilecture and dataset we consider, we measure "), 72 with a procedure described in Appendix F (see the results in Tab. 2 Appendix F). In analogy with perceptrons, can recover the exponent of end of training scaling law as "pred = 7122, and compare it to the value "mmeas that fit directly from the minima of the learning curves at different values of P. We observe from Tab. that in all cases the two values are compatible within the accuracy permitted by the fitting procedure. See Fig. 12 in Appendix F for more detailed comparison.
Effect of regularizations, alternative optimizers and different norms. Appendix H we show that the qualitative picture of scaling laws in learning curves holds also in the presence of a moderate weight decay. Exponents 7/1 and 72 change depending on the amount of weight decay, but the values of Tpree remain compatible within errors with the case without weight decay. In Appendix | we show that using SGD optimizer instead Adamn in CNN architecture changes the dynamical lcarning curves, and consequently obtain different values " and 7/2. However, they produce the same tit-of--rrinigg exponents Ypred = 7/1 as in the main analysis by using Adam. In short, we reproduce the scaling law from Hestneess et al. (2017) even when we employ weight, decay and an alternative optimizer. In Appendix J we show that also four other notions of norm reproduce the qualitative picture of the two scaling laws, but they all find incompatible values of "pred and Tmeas, suggesting that only the spectral complexity norm properly captures the scaling behavior.
DISCUSSION
Summary of results. Inspired by the implicit bias in perceptrons trained with logistic loss, our study uncovers new neural scaling laws in deep architectures that govern how test error evolves throughout training, not just convergence.
In perceptrons, observe that the whole learning curve is biased towards specific solutions. Early in the training the percept.ron implements Hebbian learn- ing, then it reaches a Bayes-optimal solution and finally it overfits by approaching max-stability rule.
The key point that we learn from perceptrons is to plot the learning curves as function of the increasing norm (we the spectral-complexity norm for deep architectures). The resulting learning curves show two distinct regimes: an early-training regimes that follows a power law that is independent of the size of the training set, and a late-training regime that depends on the size of the training set.
In deep networks, when the whole curves are rescaled by the optimal model norm and the corresponding minimum test error, learning trajectories from different large-dataset regimes collapse onto a single curve.
Together, these scaling laws recover the classic end-of-training scaling of test error with data.
The analogies between the scaling laws of perceptrons and deep architectures suggests an implicit bias throughout the whole learning procedure also for deep architectures. Overfitting can be seen as follows: although the asymptotic solution maximizes classification margins, the learning trajectory may pass near solutions with fixed spectral complexity and better generalization (cf. perceptrons, Fig. [.. An interesting future research line could be to train a deep architecture while constraining its spectral complexity to follow predetermined trend over time X(1) and study if such training procedure would
Possible implications.
9

---

Published as a conference paper at ICLR 2026
produce the same learning curves <(A) A second view comes from the self-similarity of early learning: the process first finds a simple solution (low complexity), then gradually increases the norm until reaching the maximum allowed by the dataset size. This provides a pictorial explanation of implicit bias: trajectories with larger datasets shadow those of smaller ones, until late training where overfitting may arise. A third, practical implication comes from the collapse of learning curves over an asymptotic master curve (Fig. 4 measure the shape of the generalization error curve on small dataset and predict the same shape for larger datascts, which can be of great practical convenience. However, this method requires a validation with an extensive analysis of robustness across models and datasets, as done for instance in Rosenfeld el. al. (0119)
it is possible to
Limitations of the comparison between perceptrons and deep architectures The idea of a training-timi time bias for perceptrons is fascinating, but in this work it remains mainly qualitative. To obtain quantitative guarantees, one would need an approach similar to Wu et al. (2025). or alternatively a full solution of the training dynamics using dynamical mean-field theory (see for example Montanari and Urbani (2025))- Extending these ideas to deep architectures is compelling, but while in perceptrons we can access analytically solutions fixed norm, there is no obvious analogous picture for deep architectures. The spectral complexity norm seems a good candidate, but the extent to which this property can be made quantitative is unknown. Here we provide a possible intuition for the success of the comparison between perceptrons and deep models: the classification margin A enters the cross-entropy loss in both in perceptrons (where it is normalized by the L.2 norm of weights) and in deep networks. In Bartlett et al. (2017), it was shown that spectral complexity norm reproduces the "correct" normalization of margins in deep architectures. This may be the reason why the spectral complexity norm reveals in deep architectures the same scaling laws of perceptrons (sce Bartlett et al. (2017) for a more detailed definition of "correct").
Limitations and possible extensions of our numerical analysis. The main short- coming of our analysis is that experiments were limited to image classification. We made this choice because we wanted to form a clean conceptual picture before addressing other domains, such as language models, that require larger-scale experiments. For similar rcasons we did not vary the number of parameters for each archilecture, limiting experiments to few standard architect ures. Moreover, our new scaling laws are motivated by the comparison with simple and fully understood model, and we lacked a similarly well-undesstood model for multi-layer perceptrons (in perceptrons we cannot increase arbitrarily the number of parameters because everything depends on the ratio P/N and there is no hidden layer). Recently, some promising works Monntnaar and Urbani (2025): Barbier et al. (2025), and we are optimistic that our analysis can be extended in the near future. Extending our analysis to the joint scaling with width and depth will be essential to understand how our result may
impact compute-optinal predictions (Kaplan al., 2020 Henighan al. 2020 Hofimann et al.) 2022) (especially in larger models, where these predictions are vital). We expect this direction be particularly promising, since the spectral complexity norm scales properly with the width and depth of architectures. Moreover, varying the number of parameters will clarify the role of overparametrization in escaping early-training plateaus, as suggested in Arnaboldi et al. 2024.
Final remarks. In this work consolidate the evidence of dynamical scaling laws consis- tently across dataset and architectures. At the same time, by linking implicit optimization bias with empirical scaling laws, we propose a picture in which norm growth is the variable that controls neural scaling laws during training. Our findings suggest that the same implicit bias that drives gradient descent toward solutions with maximum margins may also shape the learning trajectory throughout the entire training process, potentially providing a new theoretical framework to understand the emergence neural scaling laws, and possibly con- necting with dynamical scaling laws oblained with other methods (Velikanov and Yarotsky 2021 Bordelon el. al. 2024 Arnaboldi et al. 2024 Montanari and Urbani 2025).
We thank Brandon Livio Amnesi and Chiara Cammarota for in- sightful discussions. FD thanks Universidad Complutense de Madrid for its hospitality
Acknowledgements
10

---

Published as a conference paper at ICLR 2026
during his stay, during which part of this work conducted. This research has been supported by FIS (Italian Science Fund) 2021 funding scheme (FIS783 SMaC Statistical Mechanics and Complexity) from MUR, Italian Ministry of University and Research. FD acknowledges funding from the Bando Ricerca Scientifica. 2025 Avvio alla Ricerca, (D. R. 2155/22022 of Sapienza Università di Roma, project B83C25004300005 VESTA. MN aknowledges the support of the PNRR project PE0000013-PAIR, funded by the European Union NextGennrationEU. This study was conducted using the DARIAH HPC-AI cluster at CNR-NANOTEC in Lecce, funded by the "MUR PON Ricerca Innovazione 2014-2020" project, code PIR01 00022 and H2IOSC Project Humanities and cultural Heritage Italian Open Science Cloud funded by the European Union NexGGenera NRRP M4C2- Project code IR0000029.
REFERENCES
Arnaboldi, L., Krzakala, F., Loureiro, B., and Stephan, L. (2024). Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD. arXiv:2305.1850 18502 [stat].
Aubin, B., Krzakala, F., Lu, Y., and Zdeborová, L. (2020). Generalization error in high- dimensional perceptrons: Approaching Bayes error with convex optimization. In Advances in Neural Information Processing Systems, volume 33, pages 12199 12210. Curran Asso- ciates, Inc.
Barbier, J., Camilli, F., Nguyen, M.-T., Pastore, M., and Skork, R. (2025). Statistical physics of deep learning: Optimal learning of a multi-layer perceptron near interpolation.
Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. (2017). Spectrally-normaaized margin bounds for neural networks. In Advances in Neural Information Processing Systems, volume 30, pages 6240 6249.
Biehl, M. and Riegler, P. (1994). On-line learning with perceptron. Europhysics Letters, 2877):525-530.
Boopathy, A. and Ficte, I. (2024). Unified neural network scaling laws and scale-time equivalence. arXiv preprint arXiv:2409.0572
Bordelon, B., Atanasov, A., and Pehlevan, C. (2024). A dynamical model of neural scaling laws. arXiv preprint Xiv: 2402:2402.00092.
Caballero, E., Gupta, K., Rish, I., and Krueger, D. (2023). Broken neural scaling laws.
Chizat, L. and Bach, F. (2020). Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory (COLT).
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkorcit, J., and Houlsby, N. (2021). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR).
Engel, A. and Van den Broeck, C. (2001). Statislical Mechanics of Learning. Cambridge University Press.
Gardner, E. (1987). Maximum storage capacity in neural networks. Europhysics Lellers, 4(4):481-48) 485.
Gardner, E. and Derrida, B. (1988). Optimal storage properties of neural network models. Journal of Physics A: Mathematical and general, 21(1):271.
Gunasckar, S., Woodworth, B., Bhojanapalli, S., Neyshabur, B., and Srebro, N. (2017). Im- plicit regularization in matrix factorization. In Advances in Newrol Information Processing Systems, volume 30, pages 6151 -1156.
11

---

Published as a conference paper at ICLR 2026
Hc, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, IEEE.
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020). Scaling laws for autoregressive generative modeling. Xiv preprind arXiv:2010.14700
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. (2017). Deep learning scaling is predictable, empirically. Xiv preprint 1rxiv:1112.000
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. Xiv preprint 2rxi::203. 15556.
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., et al. (2020).
Scaling laws for neural language models. arXiv preprint or Xiv:20001 083611.
Krishevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images (cifar-10 dataset). Technical Report Technical Report 0, University of Toronto.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998a). Gradient- Sradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324.
LeCun, Y., Cortes, C., and Burges, C. J. C. (1998b). MNIST handwritten digit database. Atttp://yami lecun. com/ wxdee hord//mnis///// Accessed: 2025-05-14
Li, M., Kudugunta, S., and Zettlemoyer, L. (2025). (mis)fitting: A survey of scaling laws.
Lyu, K. and Li, J. (2020). Gradient descent maximizes the margin homogeneous neural networks. In International Conference Learning Representations (ICLR).
Mézard, M., Parisi, G., and Virasoro, M. A. (1987). Spin glass theory and beyond: An Introduction to the Replico Method and Its Applications, volume 9. World Scientific Publishing Company.
Montanari, A. and Urbani, P. (2025). Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks. arXiv:2502.213 21229 [stat].
Montanari, A., Zhong, Y., and Zhou, K. (2024). Tractability from overparametriiaatoon: The example of the negative perceptron. Probability Theory and Related Fields, 188(3-4):800) 910. arXiv:2110.1588 158224
Neyshabur, B., Tomioka, R., and Srebro, N. (2014). In scarch of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv:1412. 6614.
Opper, M. (1988). Learning times of neural networks: Exact solution for a perceptron algorithm. Physical Review A, (8(8):3824-382)
Opper, M. and Haussler, D. (1991). Generalization performance of the optimal bayes algorithm for learning perceptron. Physical Review Letfers, 66(21):2677-2)) 2680.
Opper, M., Kinzel, W., Kleinz, J., and Nehl, R. (1990). On the ability of the optimal perceptron to generalize. Journal of Physics A: Mathemaatical and General, 23(11): L586.
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. (2019). A constructive prediction of the generalization error across scales. arXiv preprint ar arXi:: 12673.
Saad, D. and Solla, S. A. (1995a). Exact solution for on-line learning in multilayer neural networks. Physical Review Letters, 74(21):4337-43340
Saad, D. and Solla, S. A. (1995b). On-line learning in soft committee machines. Physical Review F, 52(4):4225 4224-33334.
12

---

Published as a conference paper at ICLR 2026
Schaeffer, R., Miranda, B., and Koyejo, S. (2023). Are emergent abilitics of large language models a mirage?
Solla, S. A. and Winther, O. (1998), Optimal perceptron learning: an on-line bayesian approach. In Saad, D., editor, On-line Learning in Neural Networks, pages 157 178. Cambridge University Press.
Soudry, D., Hoffer, E.,, Nacson, M. S., Gunasckar, S., and Srebro, N. (2018). The implicit bias of gradient descent on separable data. Journal of Machine Learning Research, 19(70):1-57.
Sun, C., Shrivastava, A., Singh, S., and Gupta, A. (2017). Revisiting unreasonable effective- ness of data in deep learning In Proceedings of the IEEE international conference computer vision, pages 843-852.
Velikanov, M. and Yarotsky, D. (2021). Explicit loss asymptotics in the gradient descent training of neural networks. Advances Neural Information Processing Systems, 34:2570- 2582.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogalama, D., Bosma, M., Zhou, D., Metzler, D., Chi, H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. (2022). Emergent abilities of large language models.
Wu, J., Bartlett, P., Telgarsky, M., and Yu, B. (2025). Benefits of Early Stopping in Gradient Descent for Overparameterrieed Logistic Regression. Aiv::2200 13283 [cs].
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations (ICLR).
13

---

Published as a conference paper at ICLR 2026
APPENDIX
Acknowledgment of LLMs usage. The authors acknowledge the usage of LLMs for pol- ishing the text and to produce standard functions in the code for deep networks experiments. All texts and codes produced by LLMs have been carefully analyzed and validated by the authors.
Code to reproduce the results of the paper All the codes, data, hyperparameters and results on deep architectures can be found in the GitHub repository https://gitie com/Franci1199/ /eepd git
A REPLICA ANALYSIS
In this section, provide sketch of the necessary computations to obtain the analytical curve for the fixed-norm perceptron. We are interested in computing the generalization error, defined the expected fraction of misclassified examples new data. In the teacher-student setup for the perceptron presented in the main text, this is given by e arrcoos R.), where R (w- w") //N is the normalized overlap between the student and the teacher.
Given a loss function the form
PEON L(w) Σ V(A"),
where A =" y" w-z"" is the margin of the p-th example, we therefore need to compute the vN
typical overlap R between minimizer of Equation 7 and the teacher. To do this, can study the averaged free energy, defined as
((3) = lim (-EN 1 N+000
((In 2))
BN
where 3 is the inverse temperature, (),,,, w denotes the average over the distribution of the data points {x²} and the teacher vector w" Z is the partition function defined, as
Z(w) = f /dp(w) e e-AL(w)
(9)
where n(w) is the probability distribution of the student vectors, assumed to be uniform on the N-sphere. In the thermodynamic limit N + co, only subset of students, characterized by an overlap with the teacher R(3), contributes to f(5). By taking the limit / + xc, one can obtain the typical overlap considering only the minimizers of the loss.
To compute the average of In Z in Equation g we apply the replica method (Mézard et al.
1987,, which involves rewriting the logarithmic average
(Onn 2))
lim n000
7
where z is the replicated partition function defined by
z(n)_ = ((" w))) w-
II a=1
II exp (-3L(w0))
(10)
One can introduce new variables R = (w" W/N and qab = /NN,N, which represent the normalized overlap of student with the teacher, and the overlap between student vectors and 6, respectively. The free energy function can then be rewritten in terins of these new variables. Under the replica symmetric ansatz, i.e., choosing solutions of the form
R R Va E [1,n), Yab dab q(1 Eab) Va, b E [1,n].
(11)
14

---

Published as a conference paper at ICLR 2026
one obtains
1
R
f(3) extr
In(1 9) +
9.R
2
25(1 -9)
1
exp (-SVIA) (A (A-vTe)) vTt}
2(1 9) )
In
dA
(12)
v 2n 1 9)
where (x)) erfc
erf
v2
If the potential ((A) has unique minimum, one can evaluate the zero-temperature limit of Equation yielding
1
R
dt
-2/2
Rt
f(T 0)
extr 2,R
2a
H (Au(t.z) t)²
2x
2x
v1 R²
v(Do(t,z))
e(xx R),
(13)
2x.
where x = = - and Ao(t, x) argmins (V(A) + (A-1)
By solving the saddle-poing point
2x
equations
de
de
0,
= 0,
Ōx === R=R
aR
R=R
one can finally recover the value R and, consequently, the generalization error.
B PERCEPTRON IN THE OVER PARAMETRIZED REGIME
In this section we show that the analysis of the different regimes in X, shown in the main text for > 1, is qualitatively equivalent in the regime a 1. In Figure [ we plot the generalization error as function of the parameter X for = 0.5.
0.5
3.4 x 10
63.375 10 11
Hebb
Generalization error
3.35 x 1
3.325 x 10
10-1
Stability
x
3.25 x 10
Bayes
3.225 x 10
3.2 x 10
10-2
10
10"
Norm A
Figure 6: The fixed-norm problem is qualitatively the same in the over- parametrized regime. We show the generalization error of the minimizers of the cross- entropy loss in the leacher-student setup for = 0.5,
C MSE LOSS IN PERCEPTRON
In Fig. we show numerical results obtained with the same perceptron settings in the main analysis with the only difference that loss is chosen as MSE. The qualitative picture is fundamentally different and do not observe the same phenomenology.
15

---

Published as a conference paper at ICLR 2026
2x
Generalization erro
10
0=3.1
x
a=15
0 25.0
10:
20t
Norm A(t)
Figure 7: MSE loss does not produce the scaling laws in learning curves as Cross-Entropy. Norm X(t) increases during training epochs up to certain value, where training stops. We do not observe a scaling law in carly training of the form c d and (a) is constant, not following sccond scaling-law.
D ANALYSIS OF THE SCALING LAWS IN PERCEPTRON
Fixed-norus analytical perceptron We provide an analytical argument to obtain the first scaling law of the perceptron. We begin from the zero-temperature free energy per neuron:
- 122
Cmin (a) (a)== =-
extr
2a
DI H (-R////111 R²) min (Vx()) + (A (A-1)²
-1
2x
where DI de 1112
H(u) = = Dt.
/2T"
Reusing the definition
(t,(t, arg min (1(A)) + (A-1)
stationarily w.r.t. and R yields the coupled equalions:
1- R 2a Dt (Ao.1 t)2 ((- H(-Rt/VII- R22)
(14a)
R
2a
Dt Do, exp
R²r2 2(1 R
(14b)
22T (1 (1-
We focus on the regime a + 0C, where R 1- < x < 1, and the generalization error E = 1/T arccos It) se /28/m n. For x < 1, we can solve the equation for Ao.
v'(A)) Ao -t
(15)
order by order. By assuming that the derivative of the potential negligible with respect to 1/x, al first order A, = since & 1. We then assume As ~ l+ ea. The minimizing equation leads to V'(t e r) e V'(t) + c xt) + - = -v-V'(t), where we have implicitly assumed that (t( is negligible respect to 1/z. At the end we have
A ~t V((t) r
(16)
As R 1
H(-Rt/VI R2
Ott),
exp
21
31-R" RAj
R²
exp
1 76
By plugging these two expressions into
14a), we get
1 - R² x 26,
20
Dt (Ao. 1)2 20 ( (A - -t)2) 1200
110
16

---

C Published as a conference paper at ICLR 2026
Since (Au,) t)2 ~ V we have
8~a a~ax² IA())
where we have introduced
Eo(A) = f DI
(An,)(1,t) -
/ DI [
x²2
120
Similarly, from
14b) with the combined Gaussian:
2x
/
Die (((48) (t,
f
V2Tō [Veno =-- (e-12/2 (o,((,,))
dt
V
This integral exhibits delta sequence structure in the 8 + 0 limit, since the prefactor
V2Tō
acts approximation to the Dirac della funccion 0p()). Therefore, the integral localizes around / = 0, and we obtain:
R a. (0,(0,2)) a0)
Since R ~ 1:
(17)
(V(
Substituting x into ā ~ x²2 EA()):
~~a a~aa (V30) Eo(A) = En(A) aV((
(18)
For (A) A
In(2cosh(AA)) compute
(i(A)= tanh(AA), V((0) 1.
We now turn to
(A = / DI DI(V(()) = // Dt - tanh(a]))
For large \ we have;
tanh(Xt) ~ 1 -22 2e--2A +....
Then, at leading order:
Eo(A)
Dt
1ht
=4
dt
12
t' J/z0
4
dt'
-1 ((2x)) -4' ut
A
1>0
120
2
22T
4
dt'
1
(19)
2T
X
Putting this back into 8, one finally gets for the log cosh potential
~ ad e~ 11
(20)
In order to understand the regime of validity of this scaling law, we analyze the second derivative of the potential
Vi(A): = - 2
so in particular,
Vi( -A.
17

---

Published as a conference paper at ICLR 2026
This means that the hypothesis (*(t)æ A/a < 1 is no longer valid when À a, implying that the regime of validity of this power law is
< A
(21)
We now provide numerical evidence of the convergence -1/2 exponent in the e(X) curve for the perceptron by analyzing the theoretical curves. In Fig. we plot dlogg for different values of a, showing that as a increases there appears a broader region of À where the effective exponent approaches 1
alog
Derivative dloge/dloga.
0.0
-0.1
Phog5
abgA
1=1000 000
0.3
e-10
0.4
Q+3
a-0.1
-0.
10
10P
10
10a
Figure 8: Convergence of the perceptron learning exponent. We plot dlog for a different values of a. As a increases, an extended region of À develops where the effective exponent approaches 11/2, which corresponds to the asymptotic behavior e 1-1/2 The dashed red line marks the reference slope -1/2 while the black dotted line marks the zcro derivative point.
Unbounded numerical perceptron We compute the two exponents of the unbounded perceptron. For consistency, have chosen to follow the same procedure that resulted to be the best for deep networks experiments, reported in Appendix F Fig. we report the fitting plot for 71 and "2 exponents. The two exponents result not compatible considering errors with the analytical result for fixed-norm perceptrons 711 = 0.5, 72 = 1.0, but the differences only of the order of 5%. So not only the fixed-norm analytical case predict qualitatively the dynamical behavior of the unbounded perceptron, it also approximates quantitatively the values of the dynamical exponents.
V1 = 0. 0.49 + 0.0005
10
100
10
3 100
eltap
102
x 100
10
data
4x 109
fit: 02-0.957+0.225
10a
10-2
10
10P
3 104 60
10:
x
Figure Dynamical exponents of unbounded Perceptron are close to the fixed- norm analytical prediction. (left) Curves collapsed by rescaling axes for the minima, using values of a > 25. (right) Fit of the scaling minima of curves, Aape(a), using only curves for which the minimum have been reached during numerical simulation.
18

---

Published as a conference paper at ICLR 2026
E TRAINING CURVES IN FUNCTION OF TIME (NUMBER OF EPOCHS)
We show in fig. 10 that plotting e versus time instead of do not make the curves collapse. In particular X(t) nonlinear, meaning that the two plots c(t) and c()) are qualitatively different.
NNIST
P-89
30
R-188
109
F-750-
F-575
P-0
F-1500
1505
Test Error
P-0000
1012
5-0000
12000
105
Pw2403C
4-40000
40000
104
10-2
102
107
107
10'
Figure 10: The function x()) is highly non-trivial. The left panel shows the general- ization error of a CNN trained on MNIST as a function of the number of epochs for different dataset sizes P. The right panel shows the behavior of the spectral complexity as function of the number of epochs.
F RESULTS OF E(P) POWER LAW EXPONENT COEFFICIENTS AND COMPUTATION OF ERRORS
The aim this section is to explain the procedure used to compute the exponents 71, 7/ of the power laws
E c K1y + q1,
Aopt = k2p +
It is possible to combine the two power laws only in the regime of P large enough such that
A
Copt
Aopt
with a master curve function & that does not depend on P.
The first passage is to decide the minimum P to consider for the procedure. We observed that a value of P slightly bigger or smaller than the chosen one did not change substantially the estimate of 71. In almost all cases we used P 26000 the minimum value.
Then, in the collapsed graph in Fig. 11 least-squares fil. is performed over the pure power-law region to obtain a prediction of ?1 for each value of P. The final 7 value is the mean, and the associated error is the error of the mean.
To obtain 72 the minimum of the curves X is plotted versus P Fig. and from the fit 72 is obtained with the associated error.
Then Ypred 7/ and the error is
02
T Ored Ypred
71
72
The exponent to compare with is 7meas- For each value of P, we considered the minimum of the curve during training, obtaining the empirical curve of c(P). Then a power-law is performed over that curve, obtaining Timas and the dmeas of the fit. Numerical comparisons are reported in Tab. and the empirical and predicted power-laws are compared visually in Fig. 12
19

---

Published as a conference paper at ICLR 2026
Table 2: Results of the fit for the exponents 71 and 72. We report the numerical
values of the power-law exponents and 72, along with their respective uncertaintics, across different datascts and model architectures.
Model
Dataset
71
01
72
02
CNN
MNIST
0.59
0.06
1.01
0.11
CNN
CIFARIO
0.21
0.01
1.32
0.32
CNN
CIFAR100
0.112
0.003
1.44
0.22
ResNet
MNIST
1.15
0.14
0.50
0.02
ResNet
CIFAR10
0.53
0.03
1.01
0.04
ResNet
CIFARI00
0.31
0.01
1.03
0.07
VIT
MNIST
0.139
0.005
3.41
0.11
VIT
CIFAR1O
0.0124
0.0002
18.4
2.1
VIT
CIFARI00
0.0068
0.0004
21
The assigned to the comparison of exponents is computed We observe that the magnitude of is similar across experiments, while exponents change from the maximum of Tpred = 0.60 for CNN MNIST to the minimum 7pred = 0.14 of ViT CIFAR100. For this reason the relative error is higher the lower is the exponent. Being in possess of more computational power it would be possible to mitigate this effect producing more statistics for models and datasets with lower exponents.
/o a2 meas
CIFAR10, gamma 1 0.0124 +- 0.0002
2.4 10a
P-26000
1024
data
22
fit: y: 18.392+22.069
2x
P=41.000
1054
166
Agpt
10%
12x10
10-33 20-21
10-24
A(t)/Agpt
10-
13
10"3
1014
12
0':10"
2x 5x 4*200
Figure 11: The curve collapse helps predict the numerical exponents. (left) Rescaled generalization error curves used to obtain 71 from the fit. The fitted power laws shown as dashed lines. (right) The numerical fit used to ostimate "/2-
20

---

Published as a conference paper at ICLR 2026
begirical Pnd riteed
tropikal Prodeted filed
brpeical Reeittee rüse
MNIST
10
10
10a
10
30
Generalization error
Yond 0.25 Veaw - 0.25
Yime - 0.54, Yrum -0.66
Yome 0.2.. Yeerr 0.23
107
CIHAR1O
10
Empiriaal Pitedd
Driphical heekited
Irpetical Prekceed
10
Preas
Yeup
30a
CIFARLOO
10
Enga
trepk Kol
trepiical Prekite
20
Mile
10f
10" ResNet
10
CNN
VI
Number of data P
Figure 12: The predicted power laws closely match the empirical ones. We graphically present the numerical results from Table The power laws fitted the data are compared with the predicted ones. For the predicted power laws, only the exponent is known; the coefficient is chosen to enable visual comparison.
All intermediate plots as Fig. compulations and choices of value of P and X to compute power-laws reported in the paper are reported the supplementary material, as notebooks in the repository of codes with plots and data. We did not report in the paper all details because would have been necessary to show O(100) plots to evaluate all cases.
G ARCHITECTURES, DATASETS, TRAINING AND RESOURCES IN DETAILS
Architectures and hyperparameters We used PyTorch Adam optimizer for CNNs and ResNets and AdamW for ViT, in all cases with lerning rate 0.001. We used the standard and most simple possible definitions of the architectures, taken from the original papers. Please refer to the code in the supplementary the precise definition of each block and width and number of layers.
Trainings and values of P We trained for 500 epochs CNNs and for 1000 epochs ResNets and ViTs. Values of are
For MNIST in all cases 89, 188, 375, 750, 1500, 3000, 6000, 1 12000, 24000, 30000, 36000, 42000, 48000.
For CIFAR10 and CIFAR100 with ResNet and ViT in all cases: from 6000 to 46000 every 5000.
For CNN in CIFAR10/100 in the main analysis from 4000 to 48000 every 4000, and in computation of norms and the effect of weight decay from 6000 to 46000 every 5000.
Resources to replicate the study For perceptron curves the necessary resources are irrelevant. All deep network trainings have been carried on 18 V100 GPUs using 4 CPUs for each, for period of two months. We set maximum number of 30 repetitions for each training to get a statistic of learning curves and a month of computation. For smaller models finished all 30 repetitions while for the slowest one obtain a total of 5 repetitions.
21

---

Published as a conference paper at ICLR 2026
H EFFECT OF WEIGHT DECAY
We reapeted the experiment with the 3 deep architectures analyzed in the paper CIFAR10 datasct, but with an increasing level of weight decay (WD). In Fig. 13 and 14 we that in all cases the qualitative picture remain the same, even if the norm of the models doesn't I increase monotonically as the case without weight-decay. In Tab. 3 we observe that the values of 7i and "/2 exponents change depending the amount of weight decay, but their product "pred remains compatible with "meas within the accuracy permitted by the fitting procedure. For ResNet architecture, with a fixed computing budget we found difficult to find the right hyperparameters to obtain overfitting or to saturate the generalization error with the weight decays used for other two architectures, so we reported the result for smaller weight-decays. Duc to the increase in training time and corresponding decrease in statistics, the exponents fitted and predicted are affected by a larger error than in other two cases.
Model
WD
"preed
Tmmus
σ
Model
WD
71
01
722
CNN
1e-3
0.163
0.212
0.033
CNN
1c-3
0.2773
0.0184
0.5883
0.0943
CNN
le-4
0.136
0.193
0.050
CNN
1e-4
0.1880
0.0133
0.7257
0.2551
CNN
lc-5
0.133
0.184
0.024
CNN
0.1343
0.0177
0.9906
0.0815
ResNet
1e-6
0.269
0.525
0.090
ResNet
1e-6
0.6487
0.0247
0.4149
0.1342
ResNet
1c-7
0.611
0.550
0.079
ResNet
1c-7
0.6572
0.0298
0.9298
0.1101
ResNet
1e-8
0.450
0.567
0.075
ResNet
1e-8
0.6641
0.0272
0.6780
0.1047
VIT
1c-3
0.205
0.176
0.014
VIT
1c-3
0.0132
0.0003
15.5590
0.7670
VIT
1e-4
0.198
0.174
0.023
VIT
1c-4
0.0121
0.0003
16.3150
1.8161
VIT
1c-5
0.193
0.173
0.016
VIT
1e-5
0.0124
0.0003
15.5182
1.0233
Table 3: Results CIFAR10 dataset and increasing levels of weight decay. (left)
Predicted and measured exponents. (right) "n and 72 exponents computed by fitting the
data.
ResNiet
(1c-6)
33300
Generalization error E 1o4 (1c-7)
(1c-8)
Norm A(t)
Figure 13: Curves from experiments with weight decay on CIFAR10 dataset. Values of weight decay in parentheses refer to ResNet.
22

---

Published as a conference paper at ICLR 2026
16-3 (166)
elElAmp))
1o-4
10-5 (1e-8)
ta La
3
A(tilAapt
Figure 14: Curves after rescaling collapse onto a master curve also in the presence of moderate weight-decay. Values of weight decay in parentheses refer to ResNet.
216
0.2
(1e6)
1e3
Pot
Ceneralization errrr E
Orppica tine
1e-d (Te-71
22"
Yee
3e-5 (1e40
P2nd
flxd
Reshiet
Number of data
Figure 15: Comparison between predicted scaling laws by combining 7 and t2 and the empirical one measured at end-of-training. Values of weight decay in parentheses refer to ResNet.
I USING SGD OPTIMIZER INSTEAD OF ADAM ON CNNS
We reapeted the experiment using SGD optimizer instead of Adam, with only CNN archi- tecture over CIFARIO and CIFAR100 datascts. We did not repeat the experiment over the other two more complex architecture (ResNet, ViT) because Adam and Adamn (respectively used for ResNets and ViTss) are fundamental to make these architectures work appropriately. In Fig. 16 and 17 we see that all cases the qualitative picture remain the same as the main analysis also for these other norm definitions. At same time exponents predicted are compatible with the ones measured, and compatible as well with the exponents measured in the main analysis using Adam optimizer. This experimental result. suggests that the
23

---

Published as a conference paper at ICLR 2026
CIFAR10
CIFAR100
2AK UP
14 300
E/E(popp)
1.5 100
1.6
12
1Ax32
12x 105
224400
A(t)/Aopt
Figure 17: Curves after rescaling collapse onto master curve also in the of SGD optimizer instead of Adam.
optimizer is not relevant for the end-of-training scaling law exponent 7, in eT, This instead not true for the dynamics to reach the optimal value weights: for example the mumber of epochs increases dramatically using SGD instead of Adam. This difference in the dynamics captured from the dynamical exponent. Even though "pred 712 is equal with Adam and SGD, we observe that y > >dddmm while y < 22
Adam
Model
Norm
7pred
Timeas
Model
Norm
7/1
7/2
02
CNN
CIFAR10
0.202
0.225
0.047
CNN
CIFAR10
0.4735
0.0268
0.4276
0.0962
CNN
CIFAR100
0.150
0.122
0.013
CNN
CIFARI00
0.1469
0.0098
1.0233
0.0170
Table 4: Results on CIFAR10/100 datasets with CNN using SGD optimizer.
(left) Predicted and measured exponents. (right) ?1 and exponents computed by fitting the data.
CIFAR10
CIFAR100
100
Generalization error &
0x10-11
101
Norm A(t)
Figure 16: Curves from experiments using SGD optimizer instead of Adam with CNN architecture.
24

---

4. (Spectral) norm product: I Aillo, where |illo is the largest singular value of Published as a conference paper at ICLR 2026
Generalization error e
Pive Freess
Your 0.15, Frows
10
CIFARIO
CIFARIOD
Empinca
Empincel
Precicted
Predicted
10
Rdted
Rited
101
10 3
10*
10"
2x
3x 104 10*
Number of data
Comparison between predicted scaling laws by combining 71 and 72 and the empirical one measured at end-of-training.
Figure 18:
USING OTHER DEFINITIONS OF NORM X
We reapeted the experiment with the deep architectures analyzed in the paper over CIFAR10 dataset, but measuring other norms instead of the spectral complexity. The norms are:
1. (I.1) Entry-wise & norm: |||| = Eie Eji (AAi)iik
2. (I2) Frobenius (entry-wisc E2) norm: ||A|| = - Egn 1/2
3. (G21) Group (2 norm ||A|22,1 Eter Eg (Ee(A))s)) i.c. the sum over columns of their le norms.
Ag-
In Fig. 19 and 20 we see that in all cases the qualitative picture remain the same as in the main analysis also for these other norm definitions: plotting learning curves against every tested definition of norm produces the two scaling laws with exponents Y and 72, and rescaling by minima make the curves collapse over a master curve. However, the exponents predicted are not compatible with the ones measured. This result suggest that the correct quantity that generalizes in deep networks the role of 1.2 norm in the Perceptron analysis.
Spectral Complexity norm Eq.
Even if 7meas F Tpred = 71 we observe a compensation mechanism wetween 71 and 72 exponents: bigger Yi implies in almost all cases smaller 72 with respect to other norms for the same model.
Model
Norm
Tpred
Tieaa
Model
Norm
71
01
7/2
472
CNN
L11
0.083
(.1881
0.023
CNN
L1
0.5687
0.0300
0.1458
0.1358
CNN
L2
0.118
0.181
0.028
CNN
L22
0.5894
0.0157
0.2000
0.0426
CNN
G21
0.107
0.181
0.026
CNN
G21
0.5482
0.0187
0.1958
0.0428
CNN
Spectral
0.081
0.181
0.042
CNN
Spectral
0.1861
0.0041
0.4339
0.2161
ResNet
L1
0.634
0.500
0.013
ResNet
1.1634
0.0107
0.5447
0.0087
ResNet.
L2
0.750
0.500
0.013
ResNet
L2
L.4406
0.0130
0.5208
0.0063
RosNet
G21
0.680
0.500
0.018
RosNet
G21
1.1997
0.0157
0.5669
0.0121
ResNet. Spectral
0.641
(.500
0.011
ResNet
Spectral
0.5699
0.0058
1.1239
0.0119
VIT
L1
0.252
0.175
0.027
ViT
L1
0.4089
0.0171
0.6170
0.0556
VIT
12
0.323
0.175
0.040
VIT
L2
0.5491
0.0392
0.5881
0.0571
VIT
G21
0.262
0.175
0.028
ViT
G21
0.4313
0.0173
0.6075
0.0567
VIT
Spectral
0.193
0.175
0.018
VIT
Spectral
0.0133
0.0001
14.4951
1.1217
Table Results on CIFAR10 dataset and different. norm definitions. (left)
Predicted and measured exponents are not compatible using these norm definitions instead of Spectral Complexity norm. (right) 71 and 72 exponents computed by fitting the data.
25

---

Published as a conference paper at ICLR 2026
CNN
ResNet
9
Generalization errOr
c21
Speectrl
4
Norm A(t)
Figure Curves from experiments with different norm definitions on CIFAR10 dataset.
CNN
ResNet
VI
Mh
La
Nav
CNN
RezNet
VIT
sa Tago sasvel sarsa
Navea
tiE
CNN
Lavie
CNN
ResNett
Seve
VIT
Speccra
Mtt
Figure 20: Curves after rescaling collapse onto a master curve also for the other norm considered.
26

---

Published as a conference paper at ICLR 2026
Ner 203 Pre +010
Nar 0.2 Pow +0.0
Pop 222, Yrea +0.0
10
10'
12"
10'
10"
Generr Generallzation zattonn enror E
Eog Ret
Red Rood
Rood
Reod
10
107
107
12
107
Sent 200 Yoww 0.10
Key 261
0.59
Yost 314
0.19
167
22"
107
32a
10
10"
Number of data
Figure 21: Comparison between predicted scaling laws by combining Yi and Y2 and the empirical one measured al end-of-training. Other norms considered predict exponents at end-of-training not always compatible with the empirical ones, even if we can consider them as an approximation of the correct exponent that can be computed using spectral complexity as the norm A.
27


Reviews:
[{"review_id": "1y3Xchgyqs", "reviewer": "Reviewer_kC4U", "summary": "The authors find interesting connections between scaling laws and implicit bias in SGD to come up with new scaling laws for models with logistic loss, where learning curves as a function of the model’s increasing norm. They additionally show that these findings generalize across architectures and datasets in computer vision.", "questions": "- How you think these findings would translate to LLMs? Do you think the norm justification will still hold, or you will need to change the form? In practice, early training points/low FLOP datapoints of LLMs are dropped while fitting scaling laws (https://arxiv.org/abs/2502.18969). \n- What determines where the elbow is? data quality/scale/model capacity etc?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- the connection is interesting, elegant and theoretically well motivated\n- the intuition builiding using the perceptron model is helpful\n- theory is empirically confirmed across architectures and datasets\n- I think this work has connections to several empirical observations in LLM scaling law fitting which are interesting, even though the authors stick to vision datasets.\n- work is well presented", "weaknesses": "- The piece-wise scaling law proposed is not new and has been empirically demonstrated before: https://arxiv.org/abs/2210.14891 \n- The paper would benefit from a literature review of related scaling law work. Eg.\n  - https://arxiv.org/abs/2210.14891 \n  - https://arxiv.org/abs/2304.15004\n- Experiments are on a single model scale and it is not clear if it holds for LLMs and generative models where scaling law research is more useful\n- From related work and observed practice (https://arxiv.org/abs/2502.18969), outliers early in training are dropped. Connecting this to the proposed form would greatly improve the draft and I'd consider raising my score if this is done.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nMNwmAtpwn", "reviewer": "Reviewer_yVN5", "summary": "Neural scaling laws predict test error as a function of training properties (architecture, data, compute, ...). However, they typically predict the test error at the end of training, not during training itself. This article provides scaling laws for neural networks (CNNs + ViTs trained on MNIST / CIFAR for image classification) that relate test error to the model's norm. The results show highly systematic and predictable behavior. The paper's specific insights from these experiments are neatly summarized at the beginning of Section 4.\n\nReviewing caveat: while I'm somewhat familiar with the area overall (neural scaling laws), I'm not familiar with the theory in this area and therefore can't provide an informed assessment of the math.", "questions": "Beyond the questions mentioned above:\n1. Is the relationship between norm and generalization error of a correlational or causal nature? If causal, in which direction?\n2. Related to question #1: what happens if the norm is regularized during training, does this increase / decrease generalization error as predicted by the scaling law or does the relationship break if one intervenes on the norm?\n\n\nMISC:\n- the authors describe testing \"CNN, ResNet and ViT architectures\" - this sounds a bit odd since ResNets are CNNs.\n- caption formatting for Table 1 as centered is an unusual choice\n- line 58: \"at the corresponding fixed norm\" - this sentence was a bit unclear to me, perhaps there's a way to rephrase.\n- citations within a sentence should be \"as shown by Authors (YYYY)\" not \"as shown by (Authors, YYYY)\" if they're part of the sentence", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- outstanding figure design: very clear and compelling figures\n- very well written article, a pleasure to read\n- neat combination of theory and experiments across different model families\n- deriving scaling laws that predict test error during training, instead of just at the end of it, could potentially be a useful contribution (though I have some questions below). Either way, the observation that generalization error is highly related to model norm is interesting.", "weaknesses": "1. **No ImageNet scale results**: since not all results from small and toy-scale datasets like MNIST and CIFAR generalize to larger settings, an easy way to improve the reader's trust in the presented results would be to include at least an ImageNet-scale training setting. With libraries like FFCV (https://github.com/libffcv/ffcv-imagenet), ImageNet training can be done within less than an hour.\n\n2. Please correct me if my understanding is wrong, but **does relating the model's norm to test error means that we essentially can't do a prediction of test error ahead of time?** In classic scaling laws, a big advantage is that we can predict the (a priori unknown) test error as a function of settings like dataset size / compute that we can determine ahead of time. In the case of the model norm as investigated in this article, we might need to train the network to determine its norm and predict generalization error?\n\n3. Related to #2, it remains a bit unclear (to me) why the findings matter, beyond a general scientific contribution that improves our understanding of how different properties relate to each other. To be clear if it's \"just\" a scientific contribution that's perfectly valid too, but the paper's impact could potentially be strengthened by clearly explaining - or better yet, demonstrating - **why the findings matter in practice** (for people looking to train their networks), if at all. For example, the article mentions that we can \"use the norm as a measure of training time\", but provocatively speaking, couldn't we also just use a wallclock for that purpose?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GB7SsZQxPJ", "reviewer": "Reviewer_M5WM", "summary": "This paper attempts to connect the learning dynamics of neural networks to the evolution of weight norms. Inspired by a simple perceptron model, where the norm directly relates to the scaling of the test loss, the authors introduce a spectral complexity measure for the weights of deep networks. They show that the loss curves of equally sized models across different dataset sizes $P$ collapse when plotted against this spectral complexity measure.", "questions": "1. Is the collapse of the risk curves at large $P$ surprising? Shouldn’t all curves collapse to identical dynamics as you approach gradient descent on the population risk? \n2. Have the authors thought about how to characterize models of different sizes? Many scaling law results consider jointly increasing parameters and data (steps). Are the complexity measures well defined when parameters diverge? My suspicion is that the loss should still be reasonable as you approach a mean-field infinite width limit but the complexity measures may diverge. \n3. Is there any distinction between the behavior of networks trained in the online regime where data points are not repeated (where I expect losses to be monotone with steps) and training in this multiple-pass setting? How many epochs are required to see these overfitting effects?\n4. Do normalization layers (which don't encourage scale growth of weights) change the picture provided here?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes an interesting hypothesis: that some complexity norm of a neural network should be directly related to performance and summarize scaling law behavior. This hypothesis is motivated by the perceptron model where they provide substantial evidence of this effect. They also examine a variety of norms for deeper networks in the Appendix to find which of them is most promising in summarizing scaling behavior.", "weaknesses": "While this paper studies a very interesting hypothesis, there are a few concerns. \n\n**Connection between Norms and Performance in Deep Models** It is not apriori obvious how the spectral complexity measure connects to performance. In the perceptron model, it is clear that only the weight norm and the weight overlap with the target direction fully capture the generalization performance, but this is not known for deeper models. \n\n**Online vs Offline Training** Is the relationship between norms and loss primarily driven by offline training? \n\n**Large Parameter Limits** It is unclear how the spectral complexity measures depend on total parameter count which is also often scaled jointly with training time (total processed tokens = batch size * steps ).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fB4IEIEvp0", "reviewer": "Reviewer_c6fD", "summary": "The paper first studies dynamical scaling laws in perceptrons (single neurons) under a teacher-student setting. The authors find that perceptreons undergo a particular pattern of generalization error vs weight norm (or loss smoothness, which is in a sense dual to the weight norm). In particular, the authors find power-law scaling curves during different training regimes. This is shown both theoretically and validated empirically. Inspired by this, the authors then empirically evaluate generalization error vs weight norm in deep networks. The authors empirically make two observations: 1) for multiple dataset sizes, learning curves are initially similar but then diverge later on depending on dataset size, 2) the scaling law exponent at convergence can be predicted from other statistics of the learning curve, a finding found in the perceptron as well.", "questions": "- Stronger mechanistic connections between perceptron and deep network\n- Stronger empirical predictions for deep networks\n- See minor comments above", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "On the positive side, the paper shows that looking at generalization error vs weight norm curves reveals interesting and consistent patterns across datasets, architectures and other training hyperparameters. The perceptron theory also appears sound and links to empirical observations on deep networks, which is notable. Empirically, the authors consider 3 architectures and 3 datasets which gives confidence in the robustness of their results.", "weaknesses": "The paper suffers from a few key weaknesses in my view. First, the perceptron theory introduced in this paper is not particularly novel or surprising: plenty of prior works have investigated dynamical scaling laws for linear networks in a student-teacher setting. Of course, the particular setting considered by the authors is different from prior work (for instance, the choice of a logistic loss with variable sharpness), but it's unclear what value this setting offers relative to prior work.\n\nSecond, (as the authors acknowledge in Section 4) the connection between the perceptron and deep networks is weak: the only real similarity in my mind is that they are both models with some notion of norm and are trained with gradient-based methods. This concern could be remedied if the authors could somehow show that deep networks somehow behaved like perceptrons more mechanistically (beyond just looking at their generalization behavior).\n\nNow, these first two weaknesses could be put aside if the perceptron model provided strong empirical predictioons about scaling behavior in deep networks. Unfortunately, the results are lacking here as well: the paper's main result 1 is purely qualitative. There could plausibly be many other models of learning in a deep network that obey main result 1. Main result 2 is stronger, but again the results are not completely convincing: looking at table 1, for certain results, the predicted and actual $\\gamma$ look outside the range of the variability $\\sigma$. Moreover, main result 2 doesn't make prediction about the scaling laws of the networks from scratch; instead, it really just evaluates whether the scaling law obeys a certain property ($\\gamma=\\gamma_1 \\gamma_2$) predicted by the perceptron model. Again, there could be other models of deep network learning that also obey this property.\n\nIn summary, at the current stage, the paper is hampered by its weak connection between the theory and experiments. I recommend making stronger mechanistic connections between the perceptron and deep network as well as stronger empirical predictions for the deep networks.\n\nMinor comments:\n- \"perceprons\" on lines 56-57\n- Figures 3 and 4 are too small", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)