Implicit bias produces neural scaling laws in learning curves, from perceptrons to deep networks

## Abstract

Abstract Scaling laws in deep learning – empirical power-law relationships linking model performance to resource growth – have emerged as simple yet striking regularities across architectures, datasets, and tasks. These laws are particularly impactful in guiding the design of state-of-the-art models, since they quantify the benefits of increasing data or model size, and hint at the foundations of interpretability in machine learning. However, most studies focus on asymptotic behavior at the end of training. In this work, we describe a richer picture by analyzing the entire training dynamics: we identify two novel dynamical scaling laws that govern how performance evolves as function of different norm-based complexity measures. Combined, our new laws recover the well-known scaling for test error at convergence. Our findings are consistent across CNNs, ResNets, and Vision Transformers trained on MNIST, CIFAR-10 and CIFAR-100. Furthermore, we provide analytical support using a single-layer perceptron trained with logistic loss, where we derive the new dynamical scaling laws, and we explain them through the implicit bias induced by gradient-based training.

## 1 Introduction

Neural scaling laws have emerged as a powerful empirical description of how model performance improves as data and model size grow. The first kind of scaling laws that were identified show that test error (or loss) often follows predictable power-law declines when plotted against increasing training data or model parameters.
For example, deep networks exhibit approximately power-law scaling of error with dataset size and network width or depth, a phenomenon observed across vision and language tasks (Hestness et al.,, 2017; Sun et al.,, 2017; Rosenfeld et al.,, 2019).
Such results highlight the macroscopic regularities of neural network training, yet they largely summarize only the end-of-training behavior.

Since the advent of large language models, neural scaling laws started to include the training time, especially in the form of computational budget spent to train a given model.
A seminal work (Kaplan et al.,, 2020) demonstrated that cross-entropy loss scales as a power law in model size, data size, and compute budget, up to an irreducible error floor. These empirical neural scaling laws, including those for generative modeling beyond language (Henighan et al.,, 2020), indicate a remarkably smooth improvement of generalization performance as resources increase.
The main interest of this research line is, given a fixed compute budget, to find optimal way to allocate it between model size and training data such that final performance is maximized (Hoffmann et al.,, 2022).

A complementary line of research studied the so-called implicit bias of gradient-based learning dynamics. Implicit bias refers to the inherent tendencies of optimization algorithms to favor certain types of solutions, even without explicit regularization or constraints. For instance, gradient descent often finds solutions that generalize well in overparameterized models (Neyshabur et al.,, 2014; Zhang et al.,, 2017; Arnaboldi et al.,, 2024).
Theoretical results have shown that for linearly separable classification tasks, gradient descent on exponential or logistic losses converges in direction to the maximum-margin classifier
(Soudry et al.,, 2018), and analogous bias toward maximizing margins has been proven for deep homogeneous networks such as fully-connected ReLU networks (Lyu and Li,, 2020) as well as certain wide two-layer networks (Chizat and Bach,, 2020).

In this work we join these perspectives together by asking whether the implicit bias of gradient descent might itself induce predictable scaling behavior throughout the training process, in models trained with logistic loss.
The results are organized as follows.
Section [2](https://arxiv.org/html/2505.13230v2#S2) focuses on perceprons.
We first observe a surprisingly good agreement between numerical learning curves and analytical predictions from models at the corresponding fixed norm. We interpret this agreement as a training-time implicit bias.
Then we use the analytical predictions to highlight new dynamical scaling laws, by plotting learning curves as a function of the model’s increasing norm.
Finally, we show how the new scaling laws can be used to derive established neural end-of-training scaling laws.
Section [3](https://arxiv.org/html/2505.13230v2#S3) focuses on deep architectures. By using a generalized notion of norm, we reveal that the same set of scaling laws is present in deep networks, consistently across architectures and datasets, robust against alternative choices of norm, training algorithms and regularization (the exponents do depend on those details).
In section [4](https://arxiv.org/html/2505.13230v2#S4) we discuss the limits and potential consequences of these results.

### Related works.

The perceptron has long been a canonical model in the statistical mechanics of learning. Early work established its storage capacity using replica methods, identifying the critical pattern-to-dimension ratio beyond which classification fails (Gardner,, 1987; Gardner and Derrida,, 1988). Later studies analyzed learning dynamics, including exact convergence times (Opper,, 1988), the superior generalization of maximum-margin solutions (Opper et al.,, 1990), and Bayes-optimal learning curves as performance benchmarks (Opper and Haussler,, 1991).
Online learning was also investigated, with analyses of sequential updates (Biehl and Riegler,, 1994), exact teacher–student dynamics in multilayer and committee machines (Saad and Solla, 1995a,; Saad and Solla, 1995b,), and Bayesian online approaches (Solla and Winther,, 1998).

Our main focus is to highlight the role of the norm growth to describe the learning dynamics, which is a perspective that is absent in the classic works. To do that, we use the solution of logistic regression with fixed norm that was studied in Aubin et al., (2020). In our work we present an equivalent calculation that reveals the implicit bias at training time and, as a consequence, the new scaling laws.
The idea that implicit bias can extend to the whole learning trajectory can also be found in Wu et al., (2025), restricted to the overparametrized regime.

Few studies on scaling laws include training time independently of the computational cost. Simple models in controlled settings exhibit a power law in the number of training steps (Velikanov and Yarotsky,, 2021; Bordelon et al.,, 2024), favoring the discussion on the trade-off between model scale and training time (Boopathy and Fiete,, 2024) that is central to the compute-optimal scalings.
Particularly relevant is Montanari and Urbani, (2025), where the authors connect the different dynamical regimes of a committee machine to its norm, suggesting that the same ideas that we present in our work can apply even outside the setting of logistic loss.
In fact, in the case of regression with square loss, gradient descent is biased toward minimal $\ell_{2}$-norm solutions when there are many interpolating solutions (Gunasekar et al.,, 2017).

Figure: Figure 1: The learning curve of a perceptron with free norm resembles that of fixed-norm problems, which interpolate between known learning rules. Left panel: We plot the generalization error of the minimizers of the cross-entropy loss in a teacher–student setup at a fixed ratio $\alpha=5$ of number of data over size of the system. The blue curve represents the analytical result obtained under a fixed-norm constraint (with $\lambda$ as the hyperparameter of the loss), while the multicolored curve—where color varies with training time—represents the result of numerical training in the free-norm case, where $\lambda$ corresponds to the norm of the weights; the model is trained with $10^{6}$ steps of gradient descent. The horizontal lines indicate the generalization error of classical learning rules. Right panel: Same analysis for different values of $\alpha$; solid curves are analytical solutions at fixed norm, dashed curves are trajectories with free norm.
Refer to caption: PL_unbounded_vs_bounded_alpha5.0_arrow.png

## 2 Scaling laws in learning curves of perceptrons

This section introduces the core intuitions that we will use for deep architectures – plotting learning curves as function of the model’s norm – in a setting where we have analytical control of the optimization process.

In the case of a perceptron trained on linearly separable data, it is known that the implicit bias of gradient descent drives the weights toward the maximum-stability solution (the direction that maximizes the classification margin) while the norm grows over time (Soudry et al.,, 2018). In this section, we ask if the implicit bias has a role at intermediate stages of training. Using the well-established teacher–student framework Gardner and Derrida, (1988), we show that the model’s behavior throughout training is qualitatively captured by the solution to the problem in which the norm is held fixed (Aubin et al.,, 2020).
This correspondence allows us to relate the evolution of the perceptron’s norm during training to classical perceptron learning rules, offering a picture on how the implicit bias influences learning dynamics.

### Model definition in Teacher-Student scenario.

To have an analytical prediction of the generalization error, we consider a framework where a student perceptron $\boldsymbol{w}\in\mathbb{R}^{N}$ attempts to learn an unknown teacher perceptron $\boldsymbol{w}^{*}\in\mathbb{R}^{N}$ from $P=\alpha N$ labeled examples. Each example $\boldsymbol{x}^{\mu}\in\mathbb{R}^{N}$ is a random vector with i.i.d. components $x^{\mu}_{i}$ sampled from a Rademacher distribution
$P(x^{\mu}_{i})=\frac{1}{2}\delta(x^{\mu}_{i}-1)+\frac{1}{2}\delta(x^{\mu}_{i}+1).$
The corresponding labels are generated by the teacher as $y^{\mu}=\text{sign}(\boldsymbol{x}^{\mu}\cdot\boldsymbol{w}^{*})$. We assume both $\boldsymbol{w}^{*}$ and $\boldsymbol{w}$ to lie on the $N$-sphere, i.e., $\|\boldsymbol{w}^{*}\|^{2}=\|\boldsymbol{w}\|^{2}=N$. In this setting, the generalization error (or test error), defined as the expected fraction of misclassified examples on new data, can be written as $\epsilon=\frac{1}{\pi}\text{arccos}(R)$, where $R\equiv(\boldsymbol{w}\cdot\boldsymbol{w}^{*})/N$ is the normalized overlap between student and teacher.
The student minimizes a loss function $L(\boldsymbol{w})$. We study the logistic loss, which reads:

$$ $L_{\lambda}(\boldsymbol{w})=-\sum_{\mu=1}^{P}\frac{1}{\lambda}\left(\lambda\Delta^{\mu}-\log 2\cosh\left(\lambda\Delta^{\mu}\right)\right)=\sum_{\mu=1}^{P}V_{\lambda}(\Delta^{\mu}),$ (1) $$

where we defined the *margin* of the $\mu$-th example as
$\Delta^{\mu}\equiv y^{\mu}\left(\frac{\boldsymbol{w}\cdot\boldsymbol{x}^{\mu}}{\sqrt{N}}\right),$
and $\lambda$ is a hyperparameter controlling the sharpness of the logistic loss. Note that for the logistic cost $V_{\lambda}(\Delta)$ we chose the expression in Eq. ([1](https://arxiv.org/html/2505.13230v2#S2.E1)) instead of the more common (but equivalent) $\ln(1+e^{-\lambda\Delta})$ because the former is more convenient to discuss the limits in $\lambda$.
For large $N$, the properties of the minimizers of Eq. ([1](https://arxiv.org/html/2505.13230v2#S2.E1)) can be analyzed via the semi-rigorous *replica method* from the statistical mechanics of disordered systems, which outputs the average value of $R$ from the solutions $\boldsymbol{w}$ that minimize $L_{\lambda}$. In Appendix [A](https://arxiv.org/html/2505.13230v2#A1) we present a derivation alternative to that in Aubin et al., (2020), where we focus the analysis on the role of the growing norm, which allows us to notice the implicit bias at training time. This observation led us to notice that we can use the norm as a measure of training time, which is one of the key contributions of our work.

Figure: Figure 2: Fixed-norm perceptrons exhibit scaling laws in the curves of relative generalization error vs norm. Left panel: we plot the generalization error of the minimizers of the cross-entropy loss in the fixed-norm teacher–student setup of the perceptron, rescaled by the error of the Hebb rule $\epsilon_{0}$, as a function of the hyperparameter $\lambda$ for different values of $\alpha$. The stars correspond to the optimal points $(\lambda_{\text{opt}},\hat{\epsilon}_{\text{opt}})$, i.e., the minima of the generalization error for each curve. Right panel: we show the same curves after rescaling each one by its corresponding optimal point. The insets display the power-law dependencies of $\lambda_{\text{opt}}$ and $\hat{\epsilon}_{\text{opt}}$ as functions of $\alpha$.
Refer to caption: x2.png

### λ \lambda -Regimes of the Logistic Loss.

In Figure [1](https://arxiv.org/html/2505.13230v2#S1.F1), we show the analytical generalization error as a function of $\lambda$, revealing three regimes:

- 1.
Small $\lambda$ regime ($\lambda\to 0$): The second term of Eq. ([1](https://arxiv.org/html/2505.13230v2#S2.E1)) vanishes as $\mathcal{O}(\lambda)$, yielding
$V_{\lambda\to 0}(\Delta)=-\Delta,$
which corresponds (see Engel and Van den Broeck, (2001)) to the Hebbian learning, and defines a baseline generalization error $\epsilon_{0}$.
- 2.
Intermediate regime and optimal $\lambda$: At a finite value $\lambda_{\mathrm{opt}}(\alpha)$, the generalization error is minimum. We find that this optimal $\epsilon_{\mathrm{opt}}$ matches the generalization error achieved by the Bayes-optimal predictor (Opper and Haussler,, 1991), suggesting that the logistic loss rule can achieve Bayes-optimality when $\lambda$ is properly tuned. The dependence of $\lambda_{\mathrm{opt}}(\alpha)$ on $\alpha$ is shown in the top inset of the right panel of Figure [2](https://arxiv.org/html/2505.13230v2#S2.F2).
- 3.
Large $\lambda$ regime ($\lambda\to\infty$): The loss becomes:
$V_{\lambda\to\infty}(\Delta)=-2\Delta\theta(-\Delta),$
where we defined the step function $\theta(x)=1$ if $x>0$ and $\theta(x)=0$ elsewhere.
This loss has a degenerate set of minima in $\Delta$ for $\Delta\geq 0$. In contrast, for any finite value $\lambda$, the minimizer of $V_{\lambda}(\Delta)$ is unique. For this reason, we cannot apply our method directly to this potential. To recover the generalization error $\epsilon_{\infty}$ in the limit $\lambda\to\infty$, one must first solve for finite $\lambda$ and then take the limit $\lambda\to\infty$. We find that this limiting behavior corresponds to the generalization error of the maximally stable perceptron
$\boldsymbol{w}_{\text{maxStable}}=\underset{\boldsymbol{w}}{\text{argmax}}\left[\min_{\mu}\Delta^{\mu}(\boldsymbol{w})\right]$ (Gardner,, 1987; Opper et al.,, 1990).

In Figure [1](https://arxiv.org/html/2505.13230v2#S1.F1) we presented curves for $\alpha>1$ because the scaling laws appear more clearly, but the same regimes are present also when $\alpha<1$ (see Fig.[6](https://arxiv.org/html/2505.13230v2#A2.F6) in Appendix [B](https://arxiv.org/html/2505.13230v2#A2)).

### Norm scaling and interpretation.

An important observation is that the logistic loss defined in Eq. ([1](https://arxiv.org/html/2505.13230v2#S2.E1)) depends only on the product $\lambda\Delta$ (up to an overall multiplicative factor of $\lambda$ that does not affect the location of the minimizers), where $\Delta$ is linear in the norm of the perceptron weights $\|\boldsymbol{w}\|$. Rescaling the weight norm is thus equivalent to adjusting $\lambda$, meaning that analyzing a fixed-norm perceptron with varying $\lambda$ is equivalent to studying the minimizers of the loss at fixed $\lambda$ and varying norm. This insight also helps explain the behavior of $\epsilon_{\infty}$: it is known (Soudry et al.,, 2018; Montanari et al.,, 2024) that in the infinite-norm limit, the perceptron converges to the maximally stable solution during training (implicit bias).
Building on this observation, we compare two scenarios: the fixed-norm case, where the norm $\|\boldsymbol{w}\|^{2}=N$ is fixed and $\lambda$ is treated as a tunable hyperparameter of the loss (the results in this setting are obtained with the replica method); and the free-norm case, where the parameter in the loss is fixed to 1 (i.e., we use the classical logistic loss), and the norm $\|\boldsymbol{w}(t)\|\equiv\lambda(t)$ is left free to evolve during training (here the perceptron is trained using standard gradient descent optimization techniques, and the results are obtained from numerical simulations).

In Figure [1](https://arxiv.org/html/2505.13230v2#S1.F1), we compare the generalization curves under these two scenarios.
We remark that in the fixed-norm case, each point on the curve corresponds to the endpoint of training for a different perceptron (at given $\lambda$), while in the free-norm case, the curve represents the trajectory of a single perceptron during training, with each point corresponding to a different time step as the norm evolves.
We see that the free-norm trajectory is qualitatively well described by the set of fixed-norm optimal solutions, indicating that the fixed-norm static analysis captures the essential features of the learning dynamics.

### Scaling laws in learning curves at training time.

From the left panel of Fig. [1](https://arxiv.org/html/2505.13230v2#S1.F1), we observe that for sufficiently large $\alpha$ the curves share the same slope but differ in their starting point – that is the generalization error $\epsilon_{0}$ of Hebbian learning (for large $\alpha$, $\epsilon_{0}\sim\alpha^{-1/2}$). To highlight the power law scaling in $\lambda$, in the left panel of Fig. [2](https://arxiv.org/html/2505.13230v2#S2.F2) we plot relative error $\hat{\epsilon}_{\mathrm{gen}}\equiv\epsilon_{\mathrm{gen}}/\epsilon_{0}$ as a function of $\lambda$. We observe that for sufficiently large values of $\alpha$, the learning curves of the relative error split into two distinct regimes, which behave differently as we vary $\alpha$.

- 1.
An early power-law regime, independent of $\alpha$. The initial part of each learning curves follows the same shape for any $\alpha$, up to a value $\lambda_{\mathrm{elbow}}(\alpha)$ where it saturates. The curves collapse for $\lambda<\lambda_{\text{elbow}}(\alpha)$ on the power law
$\hat{\epsilon}_{\mathrm{gen}}=k_{1}\lambda^{-\gamma_{1}}+q_{1}.$
(2)
Here we introduce the term $q_{1}$ to be general, but in the perceptron we have $q_{1}=0$. Keeping $q_{1}$ will be useful in the next section on deep networks, where it we will connect to the *irreducible error floor* of realistic settings.
- 2.
A late regime, which depends on $\alpha$. After $\lambda_{\mathrm{elobw}}(\alpha)$, the learning curves deviate from the power law and saturate or overfit following a curve whose height depends on $\alpha$.

It is possible to find proper scalings that collapse also the late-phase curves (actually, the whole training curves will collapse).
First, we need to discuss the scaling law for the point of minimum test error $\lambda_{\mathrm{opt}}(\alpha)$. In the inset of the right panel of Fig. [2](https://arxiv.org/html/2505.13230v2#S2.F2), we observe that the curves follow the power law

$$ $\lambda_{\mathrm{opt}}=k_{2}\alpha^{\gamma_{2}}+q_{2},$ (3) $$

Like $q_{1}$, the term $q_{2}$ is not needed in the fixed-norm perceptron, but we introduce it to obtain a more general law applicable to deep networks.
Now we can compute $\hat{\epsilon}_{\mathrm{opt}}=\hat{\epsilon}(\lambda_{\mathrm{opt}})$ and rescale the learning curves of the left panel horizontally by $\lambda_{\mathrm{opt}}(\alpha)$ and vertically by $\hat{\epsilon}_{\mathrm{opt}}$ (see Fig. [2](https://arxiv.org/html/2505.13230v2#S2.F2), right panel). For large values of $\alpha$, the curves collapse onto a single master curve, i.e.

$$ $\hat{\epsilon}_{\mathrm{gen}}/\hat{\epsilon}_{\mathrm{opt}}=\epsilon_{\mathrm{gen}}/\epsilon_{\mathrm{opt}}=\Phi(\lambda/\lambda_{\mathrm{opt}}),$ (4) $$

for some universal function $\Phi$.

### Connection to end-of-training scaling law.

It is tempting to combine the two scaling laws in Eq. [2](https://arxiv.org/html/2505.13230v2#S2.E2) and [3](https://arxiv.org/html/2505.13230v2#S2.E3) to recover the well know scaling law $\hat{\epsilon}_{\mathrm{gen}}(\alpha)\sim\alpha^{-\gamma}$ at the end of training (Hestness et al.,, 2017).
However, Eq. [2](https://arxiv.org/html/2505.13230v2#S2.E2) is valid only for $\lambda<\lambda_{\mathrm{elbow}}(\alpha)$, while $\lambda_{\mathrm{opt}}(\alpha)>\lambda_{\mathrm{elbow}}(\alpha)$. Therefore, substituting Eq. [3](https://arxiv.org/html/2505.13230v2#S2.E3) into Eq. [2](https://arxiv.org/html/2505.13230v2#S2.E2) seems an invalid step.
Still, in the limit of large $\alpha$, Eq. [4](https://arxiv.org/html/2505.13230v2#S2.E4) implies that the whole learning curve has the same power-law scaling with $\alpha$, and therefore we can use Eq. [3](https://arxiv.org/html/2505.13230v2#S2.E3) for any $\lambda$.
Plugging Eq. [3](https://arxiv.org/html/2505.13230v2#S2.E3) in Eq. [2](https://arxiv.org/html/2505.13230v2#S2.E2) we obtain

$$ $\hat{\epsilon}_{\mathrm{gen}}(\alpha)=k_{1}\bigl(k_{2}\alpha^{\gamma_{2}}+q_{2})^{-\gamma_{1}}+q_{1}.$ (5) $$

For the perceptron Eq. [5](https://arxiv.org/html/2505.13230v2#S2.E5) simplifies to $\hat{\epsilon}_{\mathrm{gen}}(\alpha)\sim\alpha^{-\gamma_{1}\gamma_{2}}$, and we can recover $\gamma$ as $\gamma_{1}\gamma_{2}$. For the fixed-norm perceptron we obtain $\gamma_{1}=-1/2$ (Fig. [2](https://arxiv.org/html/2505.13230v2#S2.F2), left panel) and $\gamma_{2}=1$ (Fig. [2](https://arxiv.org/html/2505.13230v2#S2.F2), right panel, upper inset), which recovers $\gamma_{1}\gamma_{2}=\gamma=-1/2$ (Fig. [2](https://arxiv.org/html/2505.13230v2#S2.F2), right panel, lower inset). Exponents computed for free-norm perceptron are $\gamma_{1}=0.4901\pm 0.0005$ and $\gamma_{2}=0.96\pm 0.25$; we are unable to estimate $\gamma$ in the free-norm case because training at large $\alpha$ and $\lambda$ requires a number of gradient descent steps that is exponential in $\lambda$ (Soudry et al.,, 2018). In Appendix [C](https://arxiv.org/html/2505.13230v2#A3) we describe the numerical methods that we used to compute exponents in both cases.

## 3 Scaling laws in learning curves of deep architectures

### Methods.

Motivated by results on perceptrons, we repeat for deep architectures the analysis of the test error $\epsilon$ versus increasing norm during training $\lambda(t)$. We test CNN (LeCun et al., 1998a,), ResNet (He et al.,, 2016) and Vision Transformer (Dosovitskiy et al.,, 2021) architectures for image classification over MNIST (LeCun et al., 1998b,), CIFAR10 and CIFAR100 (Krizhevsky and Hinton,, 2009) datasets.
For each dataset and architecture we make a standard choice of hyperparameters (see Appendix [F](https://arxiv.org/html/2505.13230v2#A6)), without using a weight decay. Results with moderate weight-decay are reported in Appendix [3](https://arxiv.org/html/2505.13230v2#A7.T3). For each experiment, we select a random subset of $P$ elements from training set and we train for a fixed number of epochs, large enough to see the test error overfit or saturate. We do this procedure for all values of $P$ selected and then we repeat the training a number of times varying the random subset and of the initial condition of the training. See Appendix [F](https://arxiv.org/html/2505.13230v2#A6) for more details.

For the norm definition in the case of deep networks, in the main analysis we opt for the spectral complexity defined in Bartlett et al., (2017),
In that work, the authors show that this quantity has desirable properties for a norm, such as yielding a converging margin distributions that reflect the complexity of the dataset. In Sec. [3](https://arxiv.org/html/2505.13230v2#S3.SS0.SSS0.Px3).
Given the set $A$ of weight matrices $A_{i}$, the spectral complexity norm $R_{A}$ of the models reads

$$ $R_{A}=\Biggl(\prod_{i=1}^{L}\rho_{i}\,\|A_{i}\|_{\sigma}\Biggr)\Biggl(\sum_{i=1}^{L}\frac{\|A_{i}^{\top}-M_{i}^{\top}\|_{2,1}^{\,2/3}}{\|A_{i}\|_{\sigma}^{\,2/3}}\Biggr)^{3/2},$ (6) $$

where L is the total number of layers in the network, $\rho_{i}$ is the Lipschitz constant of the activation function (e.g. for ReLU: $\rho_{i}=1$), $A_{i}$ is the linear operator at layer $i$ for dense layers and it is an appropriate matrix for convolutional layers (see Bartlett et al., (2017) for a complete explanation). The so-called reference matrix $M_{i}$ is chosen as 0 for linear or convolutional layers and as the identity for residual layers.
Then, $\|A_{i}\|_{\sigma}$ is defined as the largest singular value of $A_{i}$ and $\|A\|_{2,1}$ is defined as the average of the $\ell_{2}$-norms of the column vectors.

Throughout rest of the paper, when we write $\lambda(t)$ for deep architectures we mean the spectral complexity norm $R_{A(t)}$, measured after $t$ training epochs.
We can give an intuition on Eq. [6](https://arxiv.org/html/2505.13230v2#S3.E6) by analyzing the contribution of the two terms. Given a layer $i$, first term is the maximum amount that an input vector can be expanded in the output space, and second term is a correction that estimates the effective rank of the outputs of the layer, that is the number of columns that have weights substantially different from zero.
In Appendix [D](https://arxiv.org/html/2505.13230v2#A4) (Fig. [9](https://arxiv.org/html/2505.13230v2#A4.F9)) we show that the relation between $\lambda$ and $t$ is non trivial, and that simply plotting $\epsilon(t)$ does not reveal the same scalings that plotting $\epsilon(\lambda(t))$ does. We always observe the monotonicity of $\lambda(t)$ if a weight-decay is not present.

### Main result 1: Dynamical scaling laws.

In this section we consider three architectures without changing their sizes, so $P$ and $\alpha$ are interchangeable.
In Fig. [3](https://arxiv.org/html/2505.13230v2#S3.F3) we report the learning curves mediated over different runs, for many values $P$ (the values change for each datasets and are reported in Appendix [F](https://arxiv.org/html/2505.13230v2#A6), together with the other details of the training process).
Notably, we find the same dynamical scaling laws that we observed for perceptrons: the learning curves have an early training regime independent on $P$ and a late training regime which depends on $P$ (compare Fig. [3](https://arxiv.org/html/2505.13230v2#S3.F3) to the left panel of Fig. [2](https://arxiv.org/html/2505.13230v2#S2.F2)).
In Fig. [4](https://arxiv.org/html/2505.13230v2#S3.F4) we rescale the learning curves in the same way we did for perceptrons (dividing the axes by the optimal norm and optimal error) fiding that they collapse for large $P$ (compare Fig. [4](https://arxiv.org/html/2505.13230v2#S3.F4) to the right panel of Fig. [2](https://arxiv.org/html/2505.13230v2#S2.F2)).
Note that, at variance with perceptrons, it is sufficient to plot the generalization error to reveal the scaling laws (and not the relative error).

Figure: Figure 3: Early-training learning curves collapse into a power law when plotted as a function of the spectral complexity norm. We plot the generalization error $\epsilon$ as a function of the norm $\lambda(t)$ for different datasets and model architectures. Different colors in the same panel refer to training curves with increasing values of the dataset size $P$, ranging from small (blue tones) to large (orange tones). The specific values of $P$ used for each dataset-model combination are listed in Appendix [F](https://arxiv.org/html/2505.13230v2#A6).
Refer to caption: images_rebuttal/eps_vs_lambda.png

Figure: Figure 4: The whole learning curves collapse at large $P$ with the proper scalings. We plot the generalization error $\epsilon$ as a function of the norm $\lambda(t)$ for different datasets and model architectures, rescaling each curve by its optimal point $(\lambda_{\text{opt}}(\alpha),\epsilon_{\text{opt}}(\alpha))$. Different colors in the same panel refer to training curves with increasing values of the dataset size $P$, ranging from small (blue tones) to large (orange tones). The values of $P$ used for each dataset-model combination are listed in Appendix [F](https://arxiv.org/html/2505.13230v2#A6).
Refer to caption: images_rebuttal/phi_collapse.png

### Main result 2: Recovery of scaling laws at convergence.

A natural question is wether we can use the measured values of $\gamma_{1}$ and $\gamma_{2}$ to recover the end-of-training scaling law in Eq. [5](https://arxiv.org/html/2505.13230v2#S2.E5) also for deep models. In this case, since $q_{2}\neq 0$ in general, we need to isolate the proper power law regime. As we show in the sketch in Fig. [5](https://arxiv.org/html/2505.13230v2#S3.F5), it is possible to identify two thresholds
$P_{-}\sim({q_{2}}/{k_{2}})^{1/\gamma_{2}}$ and
$P_{+}\sim({k_{1}k_{2}^{-\gamma_{1}}}/{q_{1}})^{1/(\gamma_{1}\gamma_{2})}$,
which distinguish between three regimes:
1) $P\ll P_{-}$, where $\epsilon(P)\,\simeq\,k_{1}q_{2}^{-\gamma}+q_{1}$. In this regime, we expect $\epsilon(P)$ to be close to random guessing, which for classification is $k_{1}q_{2}^{-\gamma}+q_{1}=(n-1)/n$, with $n$ the number of classes.
2) $P_{-}\ll P\ll P_{+}$, where $\epsilon(P)\,\simeq\,k_{1}k_{2}^{-\gamma_{1}}\,P^{-\gamma_{1}\gamma_{2}}$. The exponent $\gamma=\gamma_{1}\gamma_{2}$ corresponds to the neural scaling law observed in Hestness et al., (2017).
3) $P\gg P_{+}$, where $\epsilon(P)\,\to\,q_{1}.$ Here we approach the lowest possible error of the dataset and the performance saturates.

For each architecture and dataset we consider, we measure $\gamma_{1},\gamma_{2}$ with a procedure described in Appendix [E](https://arxiv.org/html/2505.13230v2#A5) (see the results in Tab. [2](https://arxiv.org/html/2505.13230v2#A5.T2), Appendix [E](https://arxiv.org/html/2505.13230v2#A5)). In analogy with perceptrons, we can recover the exponent of end of training scaling law as $\gamma_{\mathrm{pred}}=\gamma_{1}\gamma_{2}$, and compare it to the value $\gamma_{\mathrm{meas}}$ that we fit directly from the minima of the learning curves at different values of $P$. We observe from Tab. [1](https://arxiv.org/html/2505.13230v2#S3.T1) that in all cases the two values are compatible within the accuracy permitted by the fitting procedure. See Fig. [11](https://arxiv.org/html/2505.13230v2#A5.F11) in Appendix [E](https://arxiv.org/html/2505.13230v2#A5) for a more detailed comparison.

### Effect of regularizations, alternative optimizers and different norms.

In Appendix [3](https://arxiv.org/html/2505.13230v2#A7.T3) we show that the qualitative picture of scaling laws in learning curves holds also in the presence of a moderate weight decay. Exponents $\gamma_{1}$ and $\gamma_{2}$ change depending on the amount of weight decay, but the values of $\gamma_{\mathrm{pred}}$ remain compatible within errors with the case without weight decay. In Appendix [H](https://arxiv.org/html/2505.13230v2#A8) we show that using SGD optimizer instead of Adam in CNN architecture changes the dynamical learning curves, and consequently we obtain different values $\gamma_{1}$ and $\gamma_{2}$. However, they produce the same end-of-training exponents $\gamma_{\text{pred}}=\gamma_{1}\gamma_{2}$ as in the main analysis by using Adam. In short, we reproduce the scaling law from Hestness et al., (2017) even when we employ weight decay and an alternative optimizer. In Appendix [5](https://arxiv.org/html/2505.13230v2#A9.T5) we show that also four other notions of norm reproduce the qualitative picture of the two scaling laws, but they all find incompatible values of $\gamma_{\text{pred}}$ and $\gamma_{\text{meas}}$, suggesting that only the spectral complexity norm properly captures the scaling behavior.

Figure: Figure 5: The combination of two power laws reproduces known scalings. We plot the combined power-law scaling of the generalization error as a function of the number of data (Equation ([5](https://arxiv.org/html/2505.13230v2#S2.E5))). The parameters of the power-law are chosen of the same order of magnitude as typical results obtained for deep networks.
Refer to caption: combination.png

## 4 Discussion

### Summary of results.

Inspired by the implicit bias in perceptrons trained with logistic loss,
our study uncovers new neural scaling laws in deep architectures that govern how test error evolves throughout training, not just at convergence.

- •
In perceptrons, we observe that the whole learning curve is biased towards specific solutions.
Early in the training the perceptron implements Hebbian learning, then it reaches a Bayes-optimal solution and finally it overfits by approaching max-stability rule.
- •
The key point that we learn from perceptrons is to plot the learning curves as function of the increasing norm (we use the spectral-complexity norm for deep architectures).
The resulting learning curves show two distinct regimes: an early-training regimes that follows a power law that is independent of the size of the training set, and a late-training regime that depends on the size of the training set.
- •
In deep networks, when the whole curves are rescaled by the optimal model norm
and the corresponding minimum test error,
learning trajectories from different large-dataset regimes collapse onto a single curve.
- •
Together, these scaling laws recover the classic end-of-training scaling of test error with data.

### Possible implications.

The analogies between the scaling laws of perceptrons and deep architectures suggests an implicit bias throughout the whole learning procedure also for deep architectures.
Overfitting can be seen as follows: although the asymptotic solution maximizes classification margins, the learning trajectory may pass near solutions with fixed spectral complexity and better generalization (cf. perceptrons, Fig. [1](https://arxiv.org/html/2505.13230v2#S1.F1)). An interesting future development may be to train a deep architecture while constraining its spectral complexity.
A second view comes from the self-similarity of early learning: the process first finds a simple solution (low complexity), then gradually increases the norm until reaching the maximum allowed by the dataset size. This provides a pictorial explanation of implicit bias: trajectories with larger datasets shadow those of smaller ones, until late training where overfitting may arise.

### Limitations of the comparison between perceptrons and deep architectures

The idea of a training-time bias for perceptrons is fascinating, but in this work it remains mainly qualitative. To obtain quantitative guarantees, one would need an approach similar to Wu et al., (2025), or alternatively a full solution of the training dynamics using dynamical mean-field theory (see for example Montanari and Urbani, (2025)). Extending these ideas to deep architectures is compelling, but while in perceptrons we can access analytically solutions at fixed norm, there is no obvious analogous picture for deep architectures. The spectral complexity norm seems a good candidate, but the extent to which this property can be made quantitative is unknown.

### Limitations and possible extensions of our numerical analysis.

The main shortcoming of our analysis is that experiments were limited to image classification. We made this choice because we wanted to form a clean conceptual picture before addressing other domains, such as language models, that require larger-scale experiments.
For similar reasons we did not vary the number of parameters for each architecture, limiting our experiments to few standard architectures.
Extending our analysis to the joint scaling with width and depth will be essential to understand how our result may impact compute-optimal predictions (Kaplan et al.,, 2020; Henighan et al.,, 2020; Hoffmann et al.,, 2022) (especially in larger models, where these predictions are vital). We expect this direction to be particularly promising, since the spectral complexity norm scales properly with the width and depth of architectures.
Moreover, varying the number of parameters will clarify the role of overparametrization in escaping early-training plateaus, as suggested in Arnaboldi et al., (2024).

### Final remarks.

In this work we consolidate the evidence of dynamical scaling laws consistently across dataset and architectures. At the same time, by linking implicit optimization bias with empirical scaling laws, we propose a picture in which norm growth is the variable that controls neural scaling laws during training. Our findings suggest that the same implicit bias that drives gradient descent toward solutions with maximum margins may also shape the learning trajectory throughout the entire training process, potentially providing a new theoretical framework to understand the emergence of neural scaling laws, and possibly connecting with dynamical scaling laws obtained with other methods (Velikanov and Yarotsky,, 2021; Bordelon et al.,, 2024; Arnaboldi et al.,, 2024; Montanari and Urbani,, 2025).

### Code availability

All the code and the data that we used in this work are available at [https://github.com/Francill99/deep_norm.git](https://github.com/Francill99/deep_norm.git).

## Acknowledgements

We thank Chiara Cammarota and Brandon Livio Annesi for important discussions. MN acknowledges the support of PNRR MUR project PE0000013-FAIR. We
acknowledge support from the computational infrastructure DARIAH.IT, PON Project code PIR01-00022, National
Research Council of Italy.

## References

- Arnaboldi et al., (2024)
Arnaboldi, L., Krzakala, F., Loureiro, B., and Stephan, L. (2024).
Escaping mediocrity: how two-layer networks learn hard generalized linear models with SGD.
arXiv:2305.18502 [stat].
- Aubin et al., (2020)
Aubin, B., Krzakala, F., Lu, Y., and Zdeborová, L. (2020).
Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization.
In Advances in Neural Information Processing Systems, volume 33, pages 12199–12210. Curran Associates, Inc.
- Bartlett et al., (2017)
Bartlett, P. L., Foster, D. J., and Telgarsky, M. J. (2017).
Spectrally-normalized margin bounds for neural networks.
In Advances in Neural Information Processing Systems, volume 30, pages 6240–6249.
- Biehl and Riegler, (1994)
Biehl, M. and Riegler, P. (1994).
On-line learning with a perceptron.
Europhysics Letters, 28(7):525–530.
- Boopathy and Fiete, (2024)
Boopathy, A. and Fiete, I. (2024).
Unified neural network scaling laws and scale-time equivalence.
arXiv preprint arXiv:2409.05782.
- Bordelon et al., (2024)
Bordelon, B., Atanasov, A., and Pehlevan, C. (2024).
A dynamical model of neural scaling laws.
arXiv preprint arXiv:2402.01092.
- Chizat and Bach, (2020)
Chizat, L. and Bach, F. (2020).
Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss.
In Conference on Learning Theory (COLT).
- Dosovitskiy et al., (2021)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. (2021).
An image is worth 16x16 words: Transformers for image recognition at scale.
In International Conference on Learning Representations (ICLR).
- Engel and Van den Broeck, (2001)
Engel, A. and Van den Broeck, C. (2001).
Statistical Mechanics of Learning.
Cambridge University Press.
- Gardner, (1987)
Gardner, E. (1987).
Maximum storage capacity in neural networks.
Europhysics Letters, 4(4):481–485.
- Gardner and Derrida, (1988)
Gardner, E. and Derrida, B. (1988).
Optimal storage properties of neural network models.
Journal of Physics A: Mathematical and general, 21(1):271.
- Gunasekar et al., (2017)
Gunasekar, S., Woodworth, B., Bhojanapalli, S., Neyshabur, B., and Srebro, N. (2017).
Implicit regularization in matrix factorization.
In Advances in Neural Information Processing Systems, volume 30, pages 6151–6159.
- He et al., (2016)
He, K., Zhang, X., Ren, S., and Sun, J. (2016).
Deep residual learning for image recognition.
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778. IEEE.
- Henighan et al., (2020)
Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. (2020).
Scaling laws for autoregressive generative modeling.
arXiv preprint arXiv:2010.14701.
- Hestness et al., (2017)
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. (2017).
Deep learning scaling is predictable, empirically.
arXiv preprint arXiv:1712.00409.
- Hoffmann et al., (2022)
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022).
Training compute-optimal large language models.
arXiv preprint arXiv:2203.15556.
- Kaplan et al., (2020)
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., et al. (2020).
Scaling laws for neural language models.
arXiv preprint arXiv:2001.08361.
- Krizhevsky and Hinton, (2009)
Krizhevsky, A. and Hinton, G. (2009).
Learning multiple layers of features from tiny images (cifar-10 dataset).
Technical Report Technical Report 0, University of Toronto.
- (19)
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998a).
Gradient-based learning applied to document recognition.
Proceedings of the IEEE, 86(11):2278–2324.
- (20)
LeCun, Y., Cortes, C., and Burges, C. J. C. (1998b).
MNIST handwritten digit database.
[http://yann.lecun.com/exdb/mnist/](http://yann.lecun.com/exdb/mnist/).
Accessed: 2025-05-14.
- Lyu and Li, (2020)
Lyu, K. and Li, J. (2020).
Gradient descent maximizes the margin of homogeneous neural networks.
In International Conference on Learning Representations (ICLR).
- Mézard et al., (1987)
Mézard, M., Parisi, G., and Virasoro, M. A. (1987).
Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications, volume 9.
World Scientific Publishing Company.
- Montanari and Urbani, (2025)
Montanari, A. and Urbani, P. (2025).
Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks.
arXiv:2502.21269 [stat].
- Montanari et al., (2024)
Montanari, A., Zhong, Y., and Zhou, K. (2024).
Tractability from overparametrization: The example of the negative perceptron.
Probability Theory and Related Fields, 188(3–4):805–910.
arXiv:2110.15824.
- Neyshabur et al., (2014)
Neyshabur, B., Tomioka, R., and Srebro, N. (2014).
In search of the real inductive bias: On the role of implicit regularization in deep learning.
arXiv preprint arXiv:1412.6614.
- Opper, (1988)
Opper, M. (1988).
Learning times of neural networks: Exact solution for a perceptron algorithm.
Physical Review A, 38(8):3824–3826.
- Opper and Haussler, (1991)
Opper, M. and Haussler, D. (1991).
Generalization performance of the optimal bayes algorithm for learning a perceptron.
Physical Review Letters, 66(21):2677–2680.
- Opper et al., (1990)
Opper, M., Kinzel, W., Kleinz, J., and Nehl, R. (1990).
On the ability of the optimal perceptron to generalize.
Journal of Physics A: Mathematical and General, 23(11):L581–L586.
- Rosenfeld et al., (2019)
Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. (2019).
A constructive prediction of the generalization error across scales.
arXiv preprint arXiv:1909.12673.
- (30)
Saad, D. and Solla, S. A. (1995a).
Exact solution for on-line learning in multilayer neural networks.
Physical Review Letters, 74(21):4337–4340.
- (31)
Saad, D. and Solla, S. A. (1995b).
On-line learning in soft committee machines.
Physical Review E, 52(4):4225–4243.
- Solla and Winther, (1998)
Solla, S. A. and Winther, O. (1998).
Optimal perceptron learning: an on-line bayesian approach.
In Saad, D., editor, On-line Learning in Neural Networks, pages 157–178. Cambridge University Press.
- Soudry et al., (2018)
Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. (2018).
The implicit bias of gradient descent on separable data.
Journal of Machine Learning Research, 19(70):1–57.
- Sun et al., (2017)
Sun, C., Shrivastava, A., Singh, S., and Gupta, A. (2017).
Revisiting unreasonable effectiveness of data in deep learning era.
In Proceedings of the IEEE international conference on computer vision, pages 843–852.
- Velikanov and Yarotsky, (2021)
Velikanov, M. and Yarotsky, D. (2021).
Explicit loss asymptotics in the gradient descent training of neural networks.
Advances in Neural Information Processing Systems, 34:2570–2582.
- Wu et al., (2025)
Wu, J., Bartlett, P., Telgarsky, M., and Yu, B. (2025).
Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression.
arXiv:2502.13283 [cs].
- Zhang et al., (2017)
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017).
Understanding deep learning requires rethinking generalization.
In International Conference on Learning Representations (ICLR).

## Appendix

### Acknowledgment of LLMs usage.

The authors acknowledge the usage of LLMs for polishing the text and to produce standard functions in the code for deep networks experiments. All texts and codes produced by LLMs have been carefully analyzed and validated by the authors.

## Appendix A Replica Analysis

In this section, we provide a sketch of the necessary computations to obtain the analytical curve for the fixed-norm perceptron. We are interested in computing the generalization error, defined as the expected fraction of misclassified examples on new data. In the teacher-student setup for the perceptron presented in the main text, this is given by $\epsilon=\frac{1}{\pi}\text{arccos}(R)$, where $R\equiv(\boldsymbol{w}\cdot\boldsymbol{w}^{*})/N$ is the normalized overlap between the student and the teacher.

Given a loss function of the form

$$ $L(\boldsymbol{w})=\sum_{\mu=1}^{P\equiv\alpha N}V(\Delta^{\mu}),$ (7) $$

where
$\Delta^{\mu}\equiv y^{\mu}\left(\frac{\boldsymbol{w}\cdot\boldsymbol{x}^{\mu}}{\sqrt{N}}\right)$ is the *margin* of the $\mu$-th example, we therefore need to compute the typical overlap $\bar{R}$ between a minimizer of Equation ([7](https://arxiv.org/html/2505.13230v2#A1.E7)) and the teacher. To do this, one can study the averaged free energy, defined as

$$ $f(\beta)=\lim_{N\to\infty}\left(-\frac{1}{\beta N}\left\langle\langle\ln Z\right\rangle\rangle_{\boldsymbol{x}^{\mu},\,\boldsymbol{w}^{*}}\right),$ (8) $$

where $\beta$ is the inverse temperature, $\left\langle\cdot\right\rangle_{\boldsymbol{x}^{\mu},\,\boldsymbol{w}^{*}}$ denotes the average over the distribution of the data points $\{\boldsymbol{x}^{\mu}\}$ and the teacher vector $\boldsymbol{w}^{*}$. $Z$ is the partition function defined, as

$$ $Z(\boldsymbol{w})\equiv\int d\mu(\boldsymbol{w})\;e^{-\beta L(\boldsymbol{w})},$ (9) $$

where $\mu(\boldsymbol{w})$ is the probability distribution of the student vectors, assumed to be uniform on the $N$-sphere. In the thermodynamic limit $N\to\infty$, only a subset of students, characterized by an overlap with the teacher $\bar{R}(\beta)$, contributes to $f(\beta)$. By taking the limit $\beta\to\infty$, one can obtain the typical overlap considering only the minimizers of the loss.

To compute the average of $\ln Z$ in Equation ([8](https://arxiv.org/html/2505.13230v2#A1.E8)), we apply the replica method (Mézard et al.,, 1987), which involves rewriting the logarithmic average as

$$ $\langle\langle\ln Z\rangle\rangle=\lim_{n\to 0}\frac{\langle\langle Z^{n}\rangle\rangle-1}{n},$ $$

where $Z^{n}$ is the replicated partition function defined by

$$ $Z^{(n)}\equiv\left\langle\!\left\langle Z^{n}(\boldsymbol{x}^{\mu},\boldsymbol{w}^{*})\right\rangle\!\right\rangle_{\boldsymbol{x}^{\mu},\,\boldsymbol{w}^{*}}=\left\langle\!\left\langle\int\prod_{a=1}^{n}d\mu(\boldsymbol{w}^{a})\prod_{a=1}^{n}\exp\left(-\beta L(\boldsymbol{w}^{a})\right)\right\rangle\!\right\rangle_{\boldsymbol{x}^{\mu},\,\boldsymbol{w}^{*}}.$ (10) $$

One can introduce new variables
$R^{a}=(\boldsymbol{w}^{*}\cdot\boldsymbol{w}^{a})/N$ and $q_{ab}=(\boldsymbol{w}^{a}\cdot\boldsymbol{w}^{b})/N$, which represent the normalized overlap of student $a$ with the teacher, and the overlap between student vectors $a$ and $b$, respectively. The free energy function can then be rewritten in terms of these new variables. Under the replica symmetric ansatz, i.e., choosing solutions of the form

$$ $R^{a}=R\quad\forall a\in[1,n],\quad q_{ab}=\delta_{ab}+q(1-\delta_{ab})\quad\forall a,b\in[1,n].$ (11) $$

one obtains

$$ $\displaystyle f(\beta)=-\underset{q,R}{\text{extr}}$ $\displaystyle\left[\frac{1}{2\beta}\ln(1-q)+\frac{q-R^{2}}{2\beta(1-q)}\right.$ $\displaystyle\times\left.\ln\int d\Delta\frac{1}{\sqrt{2\pi(1-q)}}\exp\left(-\beta V(\Delta)-\frac{(\Delta-\sqrt{q}t)^{2}}{2(1-q)}\right)\right],$ (12) $$

where $H(x)=\frac{1}{2}\,\text{erfc}\left(\frac{x}{\sqrt{2}}\right)=\frac{1}{2}\left(1-\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$.

If the potential $V(\Delta)$ has a unique minimum, one can evaluate the zero-temperature limit of Equation ([12](https://arxiv.org/html/2505.13230v2#A1.E12)), yielding

$$ $\displaystyle f(T=0)=-$ $\displaystyle\underset{x,R}{\text{extr}}\left[\frac{1-R^{2}}{2x}-2\alpha\int\frac{dt}{\sqrt{2\pi}}e^{-t^{2}/2}\,H\left(-\frac{Rt}{\sqrt{1-R^{2}}}\right)\right.$ $\displaystyle\left.\times\left(V(\Delta_{0}(t,x))+\frac{(\Delta_{0}(t,x)-t)^{2}}{2x}\right)\right]\equiv e(x,R),$ (13) $$

where $x\equiv\beta(1-q)$ and
$\Delta_{0}(t,x)\equiv\text{argmin}_{\Delta}\left(V(\Delta)+\frac{(\Delta-t)^{2}}{2x}\right)$. By solving the saddle-point equations

$$ $\left.\frac{\partial e}{\partial x}\right|_{x=\bar{x},\,R=\bar{R}}=0,\quad\left.\frac{\partial e}{\partial R}\right|_{x=\bar{x},\,R=\bar{R}}=0,$ $$

one can finally recover the value $\bar{R}$ and, consequently, the generalization error.

## Appendix B Perceptron in the over parametrized regime

In this section we show that the analysis of the different regimes in $\lambda$, shown in the main text for $\alpha>1$, is qualitatively equivalent in the regime $\alpha<1$. In Figure [6](https://arxiv.org/html/2505.13230v2#A2.F6), we plot the generalization error as a function of the parameter $\lambda$ for $\alpha=0.5$.

Figure: Figure 6: The fixed-norm problem is qualitatively the same in the overparametrized regime. We show the generalization error of the minimizers of the cross-entropy loss in the teacher-student setup for $\alpha=0.5$,
Refer to caption: PL_fixed_only_with_baselines_alpha0.5.png

## Appendix C Analysis of the scaling laws in Perceptron

### Fixed-norm analytical perceptron

We provide numerical evidence of the convergence to a $-1/2$ exponent in the $\epsilon(\lambda)$ curve for the perceptron by analyzing the theoretical curves. In Fig. [7](https://arxiv.org/html/2505.13230v2#A3.F7) we plot $\tfrac{d\log\epsilon}{d\log\lambda}$ for different values of $\alpha$, showing that as $\alpha$ increases there appears a broader region of $\lambda$ where the effective exponent approaches $-1/2$.

Figure: Figure 7: Convergence of the perceptron learning exponent. We plot $\tfrac{d\log\epsilon}{d\log\lambda}$ for different values of $\alpha$. As $\alpha$ increases, an extended region of $\lambda$ develops where the effective exponent approaches $-1/2$, which corresponds to the asymptotic behavior $\epsilon\sim\lambda^{-1/2}$. The dashed red line marks the reference slope $-1/2$ while the black dotted line marks the zero derivative point.
Refer to caption: x4.png

### Unbounded numerical perceptron

We compute the two exponents of the unbounded perceptron. For consistency, we have chosen to follow the same procedure that resulted to be the best for deep networks experiments, reported in Appendix [E](https://arxiv.org/html/2505.13230v2#A5). In Fig. [8](https://arxiv.org/html/2505.13230v2#A3.F8) we report the fitting plot for $\gamma_{1}$ and $\gamma_{2}$ exponents. The two exponents result not compatible considering errors with the analytical result for fixed-norm perceptrons $\gamma_{1}=0.5,\gamma_{2}=1.0$, but the differences are only of the order of 5%. So not only the fixed-norm analytical case predict qualitatively the dynamical behavior of the unbounded perceptron, it also approximates quantitatively the values of the dynamical exponents.

Figure: Figure 8: Dynamical exponents of unbounded Perceptron are close to the fixed-norm analytical prediction. (left) Curves collapsed by rescaling axes for the minima, using values of $\alpha>25$. (right) Fit of the scaling of minima of curves, $\lambda_{opt}(\alpha)$, using only curves for which the minimum have been reached during numerical simulation.
Refer to caption: images_rebuttal/phi_perc.png

## Appendix D Training curves in function of time (number of epochs)

We show in fig. [9](https://arxiv.org/html/2505.13230v2#A4.F9) that plotting $\epsilon$ versus time instead of $\lambda$ do not make the curves collapse. In particular $\lambda(t)$ is nonlinear, meaning that the two plots $\epsilon(t)$ and $\epsilon(\lambda)$ are qualitatively different.

Figure: Figure 9: The function $\lambda(t)$ is highly non-trivial. The left panel shows the generalization error of a CNN trained on MNIST as a function of the number of epochs for different dataset sizes $P$. The right panel shows the behavior of the spectral complexity as a function of the number of epochs.
Refer to caption: err_vs_t.png

## Appendix E Results of ϵ ​ ( P ) \epsilon(P) power law exponent coefficients and computation of errors

The aim of this section is to explain the procedure used to compute the exponents $\gamma_{1},\gamma_{2}$ of the power laws

$$ $\epsilon=k_{1}\lambda^{-\gamma_{1}}+q_{1},$ $$

$$ $\lambda_{\mathrm{opt}}=k_{2}P^{\gamma_{2}}+q_{2}.$ $$

It is possible to combine the two power laws only in the regime of $P$ large enough such that

$$ $\frac{\epsilon}{\epsilon_{\mathrm{opt}}}=\Phi\left(\frac{\lambda}{\lambda_{\mathrm{opt}}}\right),$ $$

with a master curve function $\Phi$ that does not depend on $P$.

The first passage is to decide the minimum $P$ to consider for the procedure. We observed that a value of $P$ slightly bigger or smaller than the chosen one did not change substantially the estimate of $\gamma_{1}$. In almost all cases we used $P\sim 26000$ as the minimum value.

Then, in the collapsed graph in Fig. [10](https://arxiv.org/html/2505.13230v2#A5.F10) a least-squares fit is performed over the pure power-law region to obtain a prediction of $\gamma_{1}$ for each value of $P$. The final $\gamma_{1}$ value is the mean, and the associated error is the error of the mean.

To obtain $\gamma_{2}$ the minimum of the curves $\lambda^{*}$ is plotted versus $P$ in Fig. [10](https://arxiv.org/html/2505.13230v2#A5.F10), and from the fit $\gamma_{2}$ is obtained with the associated error.

Then $\gamma_{\mathrm{pred}}=\gamma_{1}\gamma_{2}$ and the error is

$$ $\sigma_{\mathrm{pred}}=\gamma_{\mathrm{pred}}\sqrt{\left(\frac{\sigma_{1}}{\gamma_{1}}\right)^{2}+\left(\frac{\sigma_{2}}{\gamma_{2}}\right)^{2}}.$ $$

The exponent to compare with is $\gamma_{\mathrm{meas}}$. For each value of $P$, we considered the minimum of the curve during training, obtaining the empirical curve of $\epsilon(P)$. Then a power-law fit is performed over that curve, obtaining $\gamma_{\textrm{meas}}$ and the $\sigma_{\text{meas}}$ of the fit. Numerical comparisons are reported in Tab. [1](https://arxiv.org/html/2505.13230v2#S3.T1) and the empirical and predicted power-laws are compared visually in Fig. [11](https://arxiv.org/html/2505.13230v2#A5.F11).
The error assigned to the comparison of exponents is computed as $\sigma=\sqrt{\sigma_{\mathrm{pred}}^{2}+\sigma_{\mathrm{meas}}^{2}}$. We observe that the magnitude of $\sigma$ is similar across experiments, while exponents change from the maximum of $\gamma_{\text{pred}}=0.60$ for CNN MNIST to the minimum $\gamma_{\text{pred}}=0.14$ of ViT CIFAR100. For this reason the relative error is higher the lower is the exponent. Being in possess of more computational power it would be possible to mitigate this effect producing more statistics for models and datasets with lower exponents.

Figure: Figure 10: The curve collapse helps predict the numerical exponents. (left) Rescaled generalization error curves used to obtain $\gamma_{1}$ from the fit. The fitted power laws are shown as dashed lines. (right) The numerical fit used to estimate $\gamma_{2}$.
Refer to caption: A_D_a.png

**Table 2: Results of the fit for the exponents $\gamma_{1}$ and $\gamma_{2}$. We report the numerical values of the power-law exponents $\gamma_{1}$ and $\gamma_{2}$, along with their respective uncertainties, across different datasets and model architectures.**
| Model | Dataset | $\gamma_{1}$ | $\sigma_{1}$ | $\gamma_{2}$ | $\sigma_{2}$ |
| --- | --- | --- | --- | --- | --- |
| CNN | MNIST | 0.59 | 0.06 | 1.01 | 0.11 |
| CNN | CIFAR10 | 0.21 | 0.01 | 1.32 | 0.32 |
| CNN | CIFAR100 | 0.112 | 0.003 | 1.44 | 0.22 |
| ResNet | MNIST | 1.15 | 0.14 | 0.50 | 0.02 |
| ResNet | CIFAR10 | 0.53 | 0.03 | 1.01 | 0.04 |
| ResNet | CIFAR100 | 0.31 | 0.01 | 1.03 | 0.07 |
| ViT | MNIST | 0.139 | 0.005 | 3.41 | 0.11 |
| ViT | CIFAR10 | 0.0124 | 0.0002 | 18.4 | 2.1 |
| ViT | CIFAR100 | 0.0068 | 0.0004 | 21 | 6 |

Figure: Figure 11: The predicted power laws closely match the empirical ones. We graphically present the numerical results from Table [1](https://arxiv.org/html/2505.13230v2#S3.T1). The power laws fitted on the data are compared with the predicted ones. For the predicted power laws, only the exponent is known; the coefficient is chosen to enable visual comparison.
Refer to caption: images_rebuttal/fit_results.png

All intermediate plots as Fig. [10](https://arxiv.org/html/2505.13230v2#A5.F10), computations and choices of value of $P$ and $\lambda$ to compute power-laws reported in the paper are reported in the supplementary material, as notebooks in the repository of codes with plots and data. We did not report in the paper all details because it would have been necessary to show $\mathcal{O}(100)$ plots to evaluate all cases.

## Appendix F Architectures, datasets, training and resources in details

### Architectures and hyperparameters

We used PyTorch Adam optimizer for CNNs and ResNets and AdamW for ViT, in all cases with lerning rate 0.001. We used the standard and most simple possible definitions of the architectures, taken from the original papers. Please refer to the code in the supplementary to the precise definition of each block and width and number of layers.

### Trainings and values of P

We trained for 500 epochs CNNs and for 1000 epochs ResNets and ViTs. Values of P are

- •
For MNIST in all cases 89, 188, 375, 750, 1500, 3000, 6000, 12000, 24000, 30000, 36000, 42000, 48000.
- •
For CIFAR10 and CIFAR100 with ResNet and ViT in all cases: from 6000 to 46000 every 5000.
- •
For CNN in CIFAR10/100, in the main analysis from 4000 to 48000 every 4000, and in computation of norms and the effect of weight decay from 6000 to 46000 every 5000.

### Resources to replicate the study

For perceptron curves the necessary resources are irrelevant. All deep network trainings have been carried on 18 V100 GPUs using 4 CPUs for each, for a period of two months. We set a maximum number of 30 repetitions for each training to get a statistic of learning curves and a month of computation. For smaller models we finished all 30 repetitions while for the slowest one we obtain a total of 5 repetitions.

## Appendix G Effect of weight decay

We reapeted the experiment with the 3 deep architectures analyzed in the paper over CIFAR10 dataset, but with an increasing level of weight decay (WD). In Fig. [12](https://arxiv.org/html/2505.13230v2#A7.F12) and [13](https://arxiv.org/html/2505.13230v2#A7.F13) we see that in all cases the qualitative picture remain the same, even if the norm of the models doesn’t increase monotonically as the case without a weight-decay.
In Tab. [3](https://arxiv.org/html/2505.13230v2#A7.T3) we observe that the values of $\gamma_{1}$ and $\gamma_{2}$ exponents change depending on the amount of weight decay, but their product $\gamma_{pred}$ remains compatible with $\gamma_{meas}$ within the accuracy permitted by the fitting procedure. For ResNet architecture, with a fixed computing budget we found difficult to find the right hyperparameters to obtain overfitting or to saturate the generalization error with the weight decays used for other two architectures, so we reported the result for smaller weight-decays. Due to the increase in training time and corresponding decrease in statistics, the exponents fitted and predicted are affected by a larger error than in other two cases.

**Table 3: Results on CIFAR10 dataset and increasing levels of weight decay. (left) Predicted and measured exponents. (right) $\gamma_{1}$ and $\gamma_{2}$ exponents computed by fitting the data.**
| Model | WD | $\gamma_{pred}$ | $\gamma_{meas}$ | $\sigma$ |
| --- | --- | --- | --- | --- |
| CNN | 1e-3 | 0.163 | 0.212 | 0.033 |
| CNN | 1e-4 | 0.136 | 0.193 | 0.050 |
| CNN | 1e-5 | 0.133 | 0.184 | 0.024 |
| ResNet | 1e-6 | 0.269 | 0.525 | 0.090 |
| ResNet | 1e-7 | 0.611 | 0.550 | 0.079 |
| ResNet | 1e-8 | 0.450 | 0.567 | 0.075 |
| ViT | 1e-3 | 0.205 | 0.176 | 0.014 |
| ViT | 1e-4 | 0.198 | 0.174 | 0.023 |
| ViT | 1e-5 | 0.193 | 0.173 | 0.016 |

Figure: Figure 12: Curves from experiments with weight decay on CIFAR10 dataset. Values of weight decay in parentheses refer to ResNet.
Refer to caption: images_rebuttal/eps_vs_lambda_wd.png

Figure: Figure 13: Curves after rescaling collapse onto a master curve also in the presence of a moderate weight-decay. Values of weight decay in parentheses refer to ResNet.
Refer to caption: images_rebuttal/phi_wd.png

Figure: Figure 14: Comparison between predicted scaling laws by combining $\gamma_{1}$ and $\gamma_{2}$ and the empirical one measured at end-of-training. Values of weight decay in parentheses refer to ResNet.
Refer to caption: images_rebuttal/phi_collapsed_wd.png

## Appendix H Using SGD optimizer instead of Adam on CNNs

We reapeted the experiment using SGD optimizer instead of Adam, with only CNN architecture over CIFAR10 and CIFAR100 datasets. We did not repeat the experiment over the other two more complex architecture (ResNet, ViT) because Adam and AdamW (respectively used for ResNets and ViTs) are fundamental to make these architectures work appropriately. In Fig. [15](https://arxiv.org/html/2505.13230v2#A8.F15) and [16](https://arxiv.org/html/2505.13230v2#A8.F16) we see that in all cases the qualitative picture remain the same as in the main analysis also for these other norm definitions. At same time exponents predicted are compatible with the ones measured, and compatible as well with the exponents measured in the main analysis using Adam optimizer. This experimental result suggests that the optimizer is not relevant for the end-of-training scaling law exponent $\gamma$, in $\epsilon\sim\epsilon^{\gamma}$. This instead is not true for the dynamics to reach the optimal value of weights: for example the number of epochs increases dramatically using SGD instead of Adam. This difference in the dynamics is captured from the dynamical exponent. Even though $\gamma_{pred}=\gamma_{1}\gamma_{2}$ is equal with Adam and SGD, we observe that $\gamma_{1}^{\textrm{SGD}}>\gamma_{1}^{\textrm{Adam}}$, while $\gamma_{2}^{\textrm{SGD}}<\gamma_{2}^{\textrm{Adam}}$.

**Table 4: Results on CIFAR10/100 datasets with CNN using SGD optimizer. (left) Predicted and measured exponents. (right) $\gamma_{1}$ and $\gamma_{2}$ exponents computed by fitting the data.**
| Model | Norm | $\gamma_{pred}$ | $\gamma_{meas}$ | $\sigma$ |
| --- | --- | --- | --- | --- |
| CNN | CIFAR10 | 0.202 | 0.225 | 0.047 |
| CNN | CIFAR100 | 0.150 | 0.122 | 0.013 |

Figure: Figure 15: Curves from experiments using SGD optimizer instead of Adam with CNN architecture.
Refer to caption: images_rebuttal/eps_vs_lambda_SGD.png

Figure: Figure 16: Curves after rescaling collapse onto a master curve also in the case of SGD optimizer instead of Adam.
Refer to caption: images_rebuttal/phi_SGD.png

Figure: Figure 17: Comparison between predicted scaling laws by combining $\gamma_{1}$ and $\gamma_{2}$ and the empirical one measured at end-of-training.
Refer to caption: images_rebuttal/fit_final_SGD.png

## Appendix I Using other definitions of norm λ \lambda

We reapeted the experiment with the 3 deep architectures analyzed in the paper over CIFAR10 dataset, but measuring other norms instead of the spectral complexity. The norms are:

- 1.
(L1) Entry-wise $\ell_{1}$ norm: $\|A\|_{1}=\sum_{i=1}^{L}\sum_{j,k}|(A_{i})_{j,k}|$
- 2.
(L2) Frobenius (entry-wise $\ell_{2}$) norm: $\|A\|_{F}=\left(\sum_{i=1}^{L}\sum_{j,k}(A_{i})_{j,k}^{2}\right)^{1/2}$
- 3.
(G21) Group $(2,1)$ norm $\|A\|_{2,1}=\sum_{i=1}^{L}\sum_{j}\left(\sum_{k}(A_{i})_{k,j}^{2}\right)^{1/2}$
i.e. the sum over columns of their $\ell_{2}$ norms.
- 4.
(Spectral) norm product: $\prod_{i=1}^{L}\|A_{i}\|_{\sigma}$, where $\|A_{i}\|_{\sigma}$ is the largest singular value of $A_{i}$.

In Fig. [18](https://arxiv.org/html/2505.13230v2#A9.F18) and [19](https://arxiv.org/html/2505.13230v2#A9.F19) we see that in all cases the qualitative picture remain the same as in the main analysis also for these other norm definitions: plotting learning curves against every tested definition of norm produces the two scaling laws with exponents $\gamma_{1}$ and $\gamma_{2}$, and rescaling by minima make the curves to collapse over a master curve. However, the exponents predicted are not compatible with the ones measured. This result suggest that Spectral Complexity norm of Eq. [6](https://arxiv.org/html/2505.13230v2#S3.E6) is the correct quantity that generalizes in deep networks the role of L2 norm in the Perceptron analysis.
Even if $\gamma_{meas}\neq\gamma_{pred}=\gamma_{1}\gamma_{2}$, we observe a compensation mechanism between $\gamma_{1}$ and $\gamma_{2}$ exponents: a bigger $\gamma_{1}$ implies in almost all cases a smaller $\gamma_{2}$ with respect to other norms for the same model.

**Table 5: Results on CIFAR10 dataset and different norm definitions. (left) Predicted and measured exponents are not compatible using these norm definitions instead of Spectral Complexity norm. (right) $\gamma_{1}$ and $\gamma_{2}$ exponents computed by fitting the data.**
| Model | Norm | $\gamma_{pred}$ | $\gamma_{meas}$ | $\sigma$ |
| --- | --- | --- | --- | --- |
| CNN | L1 | 0.083 | 0.181 | 0.023 |
| CNN | L2 | 0.118 | 0.181 | 0.028 |
| CNN | G21 | 0.107 | 0.181 | 0.026 |
| CNN | Spectral | 0.081 | 0.181 | 0.042 |
| ResNet | L1 | 0.634 | 0.500 | 0.013 |
| ResNet | L2 | 0.750 | 0.500 | 0.013 |
| ResNet | G21 | 0.680 | 0.500 | 0.018 |
| ResNet | Spectral | 0.641 | 0.500 | 0.011 |
| ViT | L1 | 0.252 | 0.175 | 0.027 |
| ViT | L2 | 0.323 | 0.175 | 0.040 |
| ViT | G21 | 0.262 | 0.175 | 0.028 |
| ViT | Spectral | 0.193 | 0.175 | 0.018 |

Figure: Figure 18: Curves from experiments with different norm definitions on CIFAR10 dataset.
Refer to caption: images_rebuttal/eps_vs_lambda_norms.png

Figure: Figure 19: Curves after rescaling collapse onto a master curve also for the other norm considered.
Refer to caption: images_rebuttal/phi_norms.png

Figure: Figure 20: Comparison between predicted scaling laws by combining $\gamma_{1}$ and $\gamma_{2}$ and the empirical one measured at end-of-training. Other norms considered predict exponents at end-of-training not always compatible with the empirical ones, even if we can consider them as an approximation of the correct exponent that can be computed using spectral complexity as the norm $\lambda$.
Refer to caption: images_rebuttal/fit_final_norms.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: - How you think these findings would translate to LLMs? Do you think the norm justification will still hold, or you will need to change the form? In practice, early training points/low FLOP datapoints of LLMs are dropped while fitting scaling laws (https://arxiv.org/abs/2502.18969). 
- What determines where the elbow is? data quality/scale/model capacity etc?
soundness: 2
strengths: - the connection is interesting, elegant and theoretically well motivated
- the intuition builiding using the perceptron model is helpful
- theory is empirically confirmed across architectures and datasets
- I think this work has connections to several empirical observations in LLM scaling law fitting which are interesting, even though the authors stick to vision datasets.
- work is well presented
summary: The authors find interesting connections between scaling laws and implicit bias in SGD to come up with new scaling laws for models with logistic loss, where learning curves as a function of the model’s increasing norm. They additionally show that these findings generalize across architectures and datasets in computer vision.
weaknesses: - The piece-wise scaling law proposed is not new and has been empirically demonstrated before: https://arxiv.org/abs/2210.14891 
- The paper would benefit from a literature review of related scaling law work. Eg.
  - https://arxiv.org/abs/2210.14891 
  - https://arxiv.org/abs/2304.15004
- Experiments are on a single model scale and it is not clear if it holds for LLMs and generative models where scaling law research is more useful
- From related work and observed practice (https://arxiv.org/abs/2502.18969), outliers early in training are dropped. Connecting this to the proposed form would greatly improve the draft and I'd consider raising my score if this is done.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: Beyond the questions mentioned above:
1. Is the relationship between norm and generalization error of a correlational or causal nature? If causal, in which direction?
2. Related to question #1: what happens if the norm is regularized during training, does this increase / decrease generalization error as predicted by the scaling law or does the relationship break if one intervenes on the norm?


MISC:
- the authors describe testing "CNN, ResNet and ViT architectures" - this sounds a bit odd since ResNets are CNNs.
- caption formatting for Table 1 as centered is an unusual choice
- line 58: "at the corresponding fixed norm" - this sentence was a bit unclear to me, perhaps there's a way to rephrase.
- citations within a sentence should be "as shown by Authors (YYYY)" not "as shown by (Authors, YYYY)" if they're part of the sentence
soundness: 3
strengths: - outstanding figure design: very clear and compelling figures
- very well written article, a pleasure to read
- neat combination of theory and experiments across different model families
- deriving scaling laws that predict test error during training, instead of just at the end of it, could potentially be a useful contribution (though I have some questions below). Either way, the observation that generalization error is highly related to model norm is interesting.
summary: Neural scaling laws predict test error as a function of training properties (architecture, data, compute, ...). However, they typically predict the test error at the end of training, not during training itself. This article provides scaling laws for neural networks (CNNs + ViTs trained on MNIST / CIFAR for image classification) that relate test error to the model's norm. The results show highly systematic and predictable behavior. The paper's specific insights from these experiments are neatly summarized at the beginning of Section 4.

Reviewing caveat: while I'm somewhat familiar with the area overall (neural scaling laws), I'm not familiar with the theory in this area and therefore can't provide an informed assessment of the math.
weaknesses: 1. **No ImageNet scale results**: since not all results from small and toy-scale datasets like MNIST and CIFAR generalize to larger settings, an easy way to improve the reader's trust in the presented results would be to include at least an ImageNet-scale training setting. With libraries like FFCV (https://github.com/libffcv/ffcv-imagenet), ImageNet training can be done within less than an hour.

2. Please correct me if my understanding is wrong, but **does relating the model's norm to test error means that we essentially can't do a prediction of test error ahead of time?** In classic scaling laws, a big advantage is that we can predict the (a priori unknown) test error as a function of settings like dataset size / compute that we can determine ahead of time. In the case of the model norm as investigated in this article, we might need to train the network to determine its norm and predict generalization error?

3. Related to #2, it remains a bit unclear (to me) why the findings matter, beyond a general scientific contribution that improves our understanding of how different properties relate to each other. To be clear if it's "just" a scientific contribution that's perfectly valid too, but the paper's impact could potentially be strengthened by clearly explaining - or better yet, demonstrating - **why the findings matter in practice** (for people looking to train their networks), if at all. For example, the article mentions that we can "use the norm as a measure of training time", but provocatively speaking, couldn't we also just use a wallclock for that purpose?

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Is the collapse of the risk curves at large $P$ surprising? Shouldn’t all curves collapse to identical dynamics as you approach gradient descent on the population risk? 
2. Have the authors thought about how to characterize models of different sizes? Many scaling law results consider jointly increasing parameters and data (steps). Are the complexity measures well defined when parameters diverge? My suspicion is that the loss should still be reasonable as you approach a mean-field infinite width limit but the complexity measures may diverge. 
3. Is there any distinction between the behavior of networks trained in the online regime where data points are not repeated (where I expect losses to be monotone with steps) and training in this multiple-pass setting? How many epochs are required to see these overfitting effects?
4. Do normalization layers (which don't encourage scale growth of weights) change the picture provided here?
soundness: 3
strengths: This paper proposes an interesting hypothesis: that some complexity norm of a neural network should be directly related to performance and summarize scaling law behavior. This hypothesis is motivated by the perceptron model where they provide substantial evidence of this effect. They also examine a variety of norms for deeper networks in the Appendix to find which of them is most promising in summarizing scaling behavior.
summary: This paper attempts to connect the learning dynamics of neural networks to the evolution of weight norms. Inspired by a simple perceptron model, where the norm directly relates to the scaling of the test loss, the authors introduce a spectral complexity measure for the weights of deep networks. They show that the loss curves of equally sized models across different dataset sizes $P$ collapse when plotted against this spectral complexity measure.
weaknesses: While this paper studies a very interesting hypothesis, there are a few concerns. 

**Connection between Norms and Performance in Deep Models** It is not apriori obvious how the spectral complexity measure connects to performance. In the perceptron model, it is clear that only the weight norm and the weight overlap with the target direction fully capture the generalization performance, but this is not known for deeper models. 

**Online vs Offline Training** Is the relationship between norms and loss primarily driven by offline training? 

**Large Parameter Limits** It is unclear how the spectral complexity measures depend on total parameter count which is also often scaled jointly with training time (total processed tokens = batch size * steps ).

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: - Stronger mechanistic connections between perceptron and deep network
- Stronger empirical predictions for deep networks
- See minor comments above
soundness: 3
strengths: On the positive side, the paper shows that looking at generalization error vs weight norm curves reveals interesting and consistent patterns across datasets, architectures and other training hyperparameters. The perceptron theory also appears sound and links to empirical observations on deep networks, which is notable. Empirically, the authors consider 3 architectures and 3 datasets which gives confidence in the robustness of their results.
summary: The paper first studies dynamical scaling laws in perceptrons (single neurons) under a teacher-student setting. The authors find that perceptreons undergo a particular pattern of generalization error vs weight norm (or loss smoothness, which is in a sense dual to the weight norm). In particular, the authors find power-law scaling curves during different training regimes. This is shown both theoretically and validated empirically. Inspired by this, the authors then empirically evaluate generalization error vs weight norm in deep networks. The authors empirically make two observations: 1) for multiple dataset sizes, learning curves are initially similar but then diverge later on depending on dataset size, 2) the scaling law exponent at convergence can be predicted from other statistics of the learning curve, a finding found in the perceptron as well.
weaknesses: The paper suffers from a few key weaknesses in my view. First, the perceptron theory introduced in this paper is not particularly novel or surprising: plenty of prior works have investigated dynamical scaling laws for linear networks in a student-teacher setting. Of course, the particular setting considered by the authors is different from prior work (for instance, the choice of a logistic loss with variable sharpness), but it's unclear what value this setting offers relative to prior work.

Second, (as the authors acknowledge in Section 4) the connection between the perceptron and deep networks is weak: the only real similarity in my mind is that they are both models with some notion of norm and are trained with gradient-based methods. This concern could be remedied if the authors could somehow show that deep networks somehow behaved like perceptrons more mechanistically (beyond just looking at their generalization behavior).

Now, these first two weaknesses could be put aside if the perceptron model provided strong empirical predictioons about scaling behavior in deep networks. Unfortunately, the results are lacking here as well: the paper's main result 1 is purely qualitative. There could plausibly be many other models of learning in a deep network that obey main result 1. Main result 2 is stronger, but again the results are not completely convincing: looking at table 1, for certain results, the predicted and actual $\gamma$ look outside the range of the variability $\sigma$. Moreover, main result 2 doesn't make prediction about the scaling laws of the networks from scratch; instead, it really just evaluates whether the scaling law obeys a certain property ($\gamma=\gamma_1 \gamma_2$) predicted by the perceptron model. Again, there could be other models of deep network learning that also obey this property.

In summary, at the current stage, the paper is hampered by its weak connection between the theory and experiments. I recommend making stronger mechanistic connections between the perceptron and deep network as well as stronger empirical predictions for the deep networks.

Minor comments:
- "perceprons" on lines 56-57
- Figures 3 and 4 are too small


Decision: Accept (Poster)