Multi-scale Feature Learning Dynamics: Insights for Double Descent
Authors: Mohammad Pezeshki
Amartya Mitra
Yoshua Bengio
Guillaume Lajoie
Venue: ICLR 2022 Submitted (2022)
Keywords: generalization
neural networks
dynamics
double descent

Abstract: A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions between the large number of network parameters. Such non-trivial interactions lead to intriguing model behaviors such as the phenomenon  of "double descent" of the generalization error. The more commonly studied aspect of this phenomenon corresponds to model-wise double descent where the test error exhibits a second descent with increasing model complexity, beyond the classical U-shaped error curve. In this work, we investigate the origins of the less studied epoch-wise double descent in which the test error undergoes two non-monotonous transitions, or descents as the training time increases. We study a linear teacher-student setup exhibiting epoch-wise double descent similar to that in deep neural networks. In this setting, we derive closed-form analytical expressions for the evolution of generalization error over training. We find that double descent can be attributed to distinct features being learned at different scales: as fast-learning features overfit, slower-learning features start to fit, resulting in a second descent in test error. We validate our findings through numerical experiments where our theory accurately predicts empirical findings and remains consistent with observations in deep neural networks.

Under review as a conference paper at ICLR 2022
MULTI-SCALE FEATURE LEARNING DYNAMICS: INSIGHTS FOR DOUBLE DESCENT
Anonymous authors Paper under double-blind review
ABSTRACT
A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high- dimensional interactions between the large number of network parameters. Such non-trivial dynamics lead to intriguing model behaviors such as the phenomenon of "double descent" of the generalization error. The more commonly studied as- pect of this phenomenon corresponds to model-wise double descent where the test error exhibits second descent with increasing model complexity, beyond the classical U-shaped error curve. In this work, we investigate the origins of the less studied epoch-wise double descent in which the test error undergoes two non- monotonous transitions, or descents as the training time increases. By leveraging tools from statistical physics, we study a linear teacher-student setup exhibiting epoch-wise double descent similar to that in deep neural networks. In this setting, we derive closed-form analytical expressions for the evolution of generalization error over training. We find that double descent can be attributed to distinct fea- tures being learned at different scales: as fast-learning features overfit, slower- learning features start to fit, resulting in a second descent in test error. We validate our findings through numerical experiments where our theory accurately predicts empirical findings and remains consistent with observations in deep neural net- works.
INTRODUCTION
Classical wisdom in statistical learning theory predicts trade-off between the generalization ability of a machine learning model and its complexity, with highly complex models less likely to gener- alize well (Friedman et al., 2001). If the number of parameters measures complexity, deep learning models sometimes go against this prediction (Zhang et al., 2016): deep neural networks trained by stochastic gradient descent exhibit a so-called double descent behavior (Belkin al., 2019b) with increasing model parameters. Specifically, with increasing complexity, the generalization error first obeys the classical U-shaped curve consistent with statistical learning theory. However, second regime emerges as the number of parameters is further increased past a transition threshold where generalization error drops again, hence the "double descent" more accurately model-wise double descent (Nakkiran et al., 2019).
Nakkiran et al. (2019) showed that the phenomenon of double descent is not limited varying model size but is also observed as a function of training time or epochs. In this case as well, the so-called epoch-wise double descent is in apparent contradiction with the classical understanding of over- over-fitting (Vapnik, 1998), where one expects that longer training of a sufficiently large model beyond certain threshold should result in over-fitting This has important implications for prac- titioners and raises questions about one of the most widely used regularization method in deep learning (Goodfellow al., 2016): early stopping. Indeed, while one might expect early stopping to prevent over-fitting. it might in fact prevent models from being trained at their fullest potential.
Since the 1990s, there has been much interest in understanding the origins of non-trivial general- ization behaviors of neural networks (Opper, 1995; Opper & Kinzel, 1996). The authors of Krogh & Hertz (1992b) were among the first to provide theoretical explanations for (model-wise) double descent in linear models. Summarily, at intermediate levels of complexity, where the model size is equal to the number of training examples, the model is very sensitive to noise in training data and hence, generalizes poorly. This sensitivity to noise reduces if the model complexity is either de-

---

Under review as a conference paper at ICLR 2022
Isotropic latent features
Anisctropic Input features
MSE Generriizttinn Error
Fast fealure
0.4
Slow feature
0.3
0.2
0.1
0.0
10a
100
0.5
R
20 F
E R
NSE Genera Ization Ereer
0.4
Both
W WER"
W E R
0.2
==
9 x
0.1
Noisy teacher
Student
10a
10-
10a
10a
10
30a
10
Training time
Figure I: Left: The teacher the data generating process that operates on isotropic Gaussian inputs z. The student is trained on dataset generated by the teacher, D = {z,, Vi |.= where := FTs follow an anisotropic Gaussian distribution such that the directions with larger/smaller variance are learned faster/slower. The condition number of F determines how much faster some features are learned than the others. One can think of z as the latent factors variation on which the teacher operates, while can be thought as the pixels that the student learns from. Right: The generalization error as the training time proceeds. (top): The case where only the fast-learning feature or slow- learning feature are trained. (bottom): The case where both features. Features that are learned on a faster time-scale are responsible for the classical U-shaped generalization curve, while the second descent can be attributed to the features that are learned at slower time-scale.
creased or increased. More recently, the double descent phenomena has been also studied for more complex models such as two-layer neural networks and random feature models (Ba et al., 2019; Mei & Montanari, 2019; D' Ascoli al., 2020; Gerace et al., 2020).
The majority of previous work in this direction focuses on understanding the asymptotic behavior of model performance, i.c., where training time - oc. In recent years, there has been an interest in studying the non-asymptotic (finite training time) performance, suggesting that several intriguing properties of neural networks can be attributed to different features being learned at different scales. Among the limited work studying the particular epoch-wiss double descent, Nakkiran et al. (2019) introduces the notion of effective model complexity and hypothesizes that it increases with training time and hence unifies both model-wise and epoch-wise double descent. Through combination of theory and empirical results, Heckel & Yilmaz (2020) find that the dynamics of evolution of single and two layer networks under gradient descent, can be perceived to be the superposition of two bias/variance curves with different minima times, thus leading to non-monotonic test error curves.
In this work, we build Bös al. (1993); Bös (1998); Advani & Saxe (2017); Mei & Montanari (2019) which analyze model-wise double descent through the lens of linear models, to probe the origins of epoch-wise double descent. Particularly,
We introduce linear teacher-student model which, despite its simplicity, exhibits some of intriguing properties of generalization dynamics in deep neural networks. (Section 2.1)
In the limit of high dimensions, we leverage the replica method developed in statistical physics to derive closed-form expressions for the generalization dynamics of our teacher- student setup, as function of training time and regularization strength. (Section 2.2)
Consistent with recent findings, provide explanation for the existence of epoch-wise double descent through the lens of multi-scale feature learning. (Figure 1)
We perform simulation experiments to validate our analytical predictions. We also conduct experiments with deep networks, showing that our teacher-student setup exhibits general- ization behavior which is qualitatively similar to that of deep networks. (Figure
2

---

Under review as a conference paper at ICLR 2022
2 ANALYTICAL RESULTS
Stochastic Gradient Descent (SGD) - the de facto optimization algorithm for neural networks exhibits complex dynamics arising from large number of parameters (Kunin et al., 2020). How- ever, it is possible to describe some aspects of the high-dimensional microscopic dynamics of neural networks in terms of now-dimensional understandable macroscopic entities. In series of seminal papers by Gardner (Gardner, 1988; Gardner & Derrida, 1988; 1989), the replica method of statisti- cal physies was adopted derive expressions describing the generalization behavior of large linear models trained using SGD. In this paper, we employ Gardner's 's analysis to build upon an established line of work studying linear and generalized linear models (Seung et al., 1992; Kabashima et al., 2009; Krzakala et al., 2012). While most of previous work study the asymptotic (t + oo) gener- alization behavior, we adapt these methods to study transient learning dynamics of generalization for finite training time. In the following, we first introduce a teacher-student model that exhibits interesting characteristics of modern neural networks. We then adapt the replica method to study the generalization performance as function of training time and amount of regularization.
2.1 A TEACHER-STUDENT SETUP
Teacher: We study a supervised linear regression problem in which the training labels y. are generated by noisy linear model (Figure 1),
= y== + y" = W.W. 24 ~~~N
Va
(1)
where: z I2 is the teacher's input and yy E are the teacher's noiseless and noisy outputs, respectively. W E Rd represents the (fixed) weights of the teacher and e E R is the noise. Both W and e are drawn i.i.d. from Gaussian distributions with zero means and variances of and o? respectively.
Student: A student model is correspondingly chosen to be a similar shallow network with train- able weights W E Idd The student model is trained on n training pairs {(x", "y") with the labels y being generated by the above teacher network, as,
ÿ = x W.
s.t.
x x:= 2,
(2)
where the matrix F E Rdxd predefined and fixed modulation matrix regulating the student's access to the true input z. One can think of z as the latent factors of variation on which the teacher operates, while æ can be thought as the pixels that the student learns from.
Learning paradigm: To train our student network,
stochastic gradient descent (SGD) on the regularized mean squared loss, evaluated on the n training examples as,
LT
Σ (y" #=1
gn)² |
2n
where À E [0, (SGD) yields the typical update rule,
oo) is the regularization coefficient. Optimizing Eq. 3 with stochastic gradient descent
W. Wi-i CT +
in which t denotes the training step and 77 is the learning rate. Additionally, 5 ~ NN of) models the stochasticity noise of the optimization algorithm (Bottou et al., 1991).
The quantity of interest in this work, is the expected generalization error of the student, determined by averaging the student's 's error over all possible input-target pairs and noise realizations, as,
Macroscopic variables:
Lg = E. ((0 3]
(5)
2
As shown in Bös et al. (1993), if d, + do with a constant ratio z < oc, Eq. 5 can be written as function of two macroscopic scalar variables R, Q E R.
1 Lg 1 (1+Q-22 222),
(6)
3

---

Under review as a conference paper at ICLR 2022
where of is the variance of the teacher's output noise and,
WWFW,
1 = d WT FT FW,
See App. B.1 for the proof.
Both R and Q have clear interpretations; is the dot-product between the teacher's weights W and the student's modulated weights FW, hence can be interpreted as the alignment between the teacher and the student. Similarly, Q can be interpreted as the student's modulated norm. The negative sign of R. in Eq. 6 suggests that the larger R is, the smaller the generalization eπror gets. At the same time, Q appears with a positive sign suggesting the students with smaller (modulated) norm generalize better.
As remark, note that both R and Q are functions of W which itself is a function of training iteration t and the regularization strength Therefore, hereafter, we denote the above quantities as Lg (1, A), R(t, A), and (tt, A).
2.2 MAIN RESULTS
In this Section, we present our main analytical results, with Section 2.3 containing a sketch of our derivations. For brevity of the results, here, only present the results for of X = 0. See App. B for the general case and the detailed proofs.
General matrix F. Let Z = |== Rnxd and X
Ji=1 E Raxd denote the input
matrices for the teacher and student such that X := ZF. For a general modulation matrix F, the input covariance matrix has the following singular value decomposition (SVD).
X = zT VAVV
in which the diagonal matrix A contains the eigenvalues of the student's input covariance matrix. Solving the dynamics of gradient descent as in Eq. 4, we arrive the following exact analytical expressions for R(t) and Q(!),
1 R(t)=
- TT(D) d
where, D: D1-(1-M--nAl)
1 Q(t) ā (A where, A FVDVV
(10)
in which Tr(.) is the trace operator. See App. B.2 the proof.
Remark: The solution in Eqs. and 10 are exact, however, they require the empirical computation of the eigenvalues A. Below, we treat special case of the dynamics that allow to derive approximate solutions that do not explicitly depend on A.
Fast and slow features. We now study a where the modulation matrix F has a specific structure described in Assumption
Special case:
The modulation matrix, F, under a SVD, F == UEVT has two sets of singular values such that the first singular values are equal to and the remaining d - p singular values are equal to 02 We let the condition number of to be denoted by K == a >
Assumption 1.
By employing the replica method of statistical physics (Gardner, 1988; Gardner & Derrida, 1988), now derive approximate expressions for R(t) and Qtt). To begin with, we first define the fol- lowing auxiliary variables.
n a1
Â d
d 5e d p noo!
P
02
d-p P
nnoli
(11)
and also let,
0, 1+
for (1,2)
(12)
(1-a; a: 1-aa-Ai 2)++ + V(1 a4 - -x,3² + 4i,
4

---

Under review as a conference paper at ICLR 2022
The closed-froe from scalar expression for R(t) is then given by, R(t) = R1 + R2, where,
n
R2 == n
R1 = 01 d'
and,
(13)
a2d
For Q(t), we accordingly define two more auxiliary variables,
2R, n n
for with which the closed-from scalar expression for Q(t) reads,
b;
G = --
i i (1,2),
(14)
0 ai
b16202 + b1c1
and, Q2 := bib2c1 + b202
(t) = Q1 + Q2. where,
Q1 1 b1
(15)
1 b1b2
By plugging Eqs. 13 and 15 into Eq. 6, one obtains closed- closed-form expression for Lg (t) function of the training time. See App. B.3 for the proof.
Remark: Eq. 11 indicates that the singular values of F, are directly multiplied by t. That implies that the learn- ing speed of each feature is scaled by the magnitude of its corresponding singular value. As an illustration, the figure on the right shows the evolution of R2, Rg, and R R1 + R2 for a case where d/2, σ1 1, and 01 0.01 implying a condition number of 100.
= Ri Rg
1
2.3 SKETCH OF DERIVATIONS
In this Section, we sketch the key steps in the derivation of our main results. For the sake of sim- plicity, here we only treat the case where a. = = The general case with detailed proofs are presented in App B.
Exact dynamics of SGD, Recall the gradient descent update rule in Eq. 4. For the linear model defined in Eqs. 1-2, learning is governed by the following discrete-time dynamics,
W. Wi-1 17 W._ LT,
(16)
We-1 - n| X XW.
(17)
With the assumption that Wi- the dynamics admit the following exact closed-form solution,
(1 (==1--""x))xx>>>-->>>>> (().
(18)
With SVD on x X. Eqs. 9-10 can then be obtained by substituting W,, in Eqs. 7. As remark, note that one can recover the results of Advani & Saxe (2017) by setting F 1. In that case, the cigenvalues of x follow Marchenko-Paae Pastur distribution (Marchenko & Pastur, 1967).
Induced probability density of SGD, It is well-known (Kuhn & Bos, 1993; Solla, 1995) that probability distribution weight configurations for network weights W trained via SGD on a loss L(W), tend to the Gibbs distribution such that,
P(W) -CC(W)
(19)
ZB"
in which Z3 is the partition function (ʃ (saw exp(-3c(11))) and , is called the inverse temper- ature and is inversely proportional the stochastic noise of SGD, &. defined in Eq. 4. Intuitively, for small B, the distribution of P(W) is almost uniform, while as B + oc, P(W) becomes more concentrated around the minimum of the training loss.
is important to highlight that Eq. 19 describes the equilibrium distribution of the student network's weights, i.e., at the end of training (t. oo). However, interested in studying the trajectory

---

Under review as a conference paper at ICLR 2022
of student's weights during the course of training, i.e., for finite t. To that end, we derive the time- dependent probability density over W,
P(W, t)
1 Zz,t
#E(W.))
where,
(20)
2n Σ 10' (²(1)) + A +2 211i1 1
1
(T ;t==
(21)
(0" x² W(t))² + 2 1
(W(t) defined in Eq. 18)
(22)
2n
CT(W) + 2 (^+ 7 1
(23)
Remark: Cr(W 8) is a modified loss such that its min- imum (equilibrium distribution) is achieved at the 1hh it- erate of gradient descent L(W). The schematic dia- gram on right illustrates this equivalence, such that, arg ming (r(W)t) = We, where We is the defined in Eq. 4.
E(W.o
SGD steps
t step on C(W)
argmin L(W, .))
I
2(W)
W
The typical generalization error. To determine the typical generalization performance at time one proceeds by first computing the free-energy of the system as,
Ew In Zz,a] = 3d
(24)
Free-energy is a self-averaging property where its typical/mosi probable value coincides with its av- erage over proper probability distributions Engel & Van den Broeck (2001). Therefore, determine the typical values of R and Q, we extremize the free-energy w.r.t. those variables.
Due to the logarithm inside the expectation, analytical computation of Eq. 24 is intractable. How- ever, the replica method (Mézard et al., 1987) allows us to tackle this through the following identity,
Ew 12
Ew., [Iu Z,,,]
lim r00
(25)
Computation of the free-energy via replica method and its subsequent extremization w.r.t R and Q, we arrive at Eqs. 13 and 15. See App. B.3 for more details.
To summarize, using the replica method, we are able to cast the high-dimensional dynamics of SGD into simple scalar equations governing I and Q and, consequently, the generalization error La- While analysis is limited to the specific teacher and student setup. this simple model already exhibits dynamics qualitatively similar to those observed in more complex networks, as we now illustrate.
EXPERIMENTAL RESULTS
In this Section, we conduct numerical simulations to validate our analytical results and provide clear insights on the macroscopic dynamics of generalization. We also conduct experiments on real- world neural networks showing a close qualitative match between the generalization behavior of neural networks and our teacher-student setup.
For real-world experiments, we train ResNet18 (He et al., 2016) with large layer widths [64, 2 x 64, 4 x 64, x 64). We follow the training setup of Nakkiran et al. (2019); Label noise with probability 0.15 randomly assign incorrect label training examples. Noise is sampled only once before the training starts. We train using Adam (Kingma & Ba, 2014) with learning rate of le - 4 for IK epochs. Real-world experiments averaged over 50 random seeds. To ensure reproducibility, we include the complete source code Gi Hit repository as well as an anonymous labl notebook.

---

Under review as a conference paper at ICLR 2022
(a) ResNet18 Cifar
(b)
10
0.0
0.50
0.45
MSE genoralization loss
0:99-
Training time
0.4g
0:40
0
0:35
10
0.30
0:30
109
10
Regulatization strength
Training time /
(c) Analytical resalts
(d)
+..00
MSE generalization loss
0.15
Small
10
0.85
0:35
Training
30
0.30
10
0:25
122
10
10 10
10-7
0.00
0.15
11/
122
107
10
Regularization strength
Training time f
Figure 2: qualitative comparison between a ResNet-18 and our analy tical results. (a): Heat- map of empirical generalization error (0-1 classification error) for the ResNet-18 trained on CIFAR- 10 with 15% label noise. X-axis denotes the inverse of weight-decar regularization strength and Y- axis represents the training time. (c): Heat-map the analytical generalization error (mean squared error) for the linear teacher-student setup with к 100, the condition number of the modulation matrix. (b, d): Three slices the heat-maps for large, intermediate, and small amounts of regu- larization. Analysis: As predicted by Eqs. 13 and 15, are learned 100 times faster that the rest. Intuitively, large amounts of regularization allow for the fast-learning features to be learned not to overfit. Intermediate levels of regularization result in classical U-shaped generalization curve but prevent slow features from learning. Small amounts of regularization allow for both fast and slow features to be learned, leading double descent.
100 implies that subset of features
3.1 MATCH BETWEEN THEORY AND
EXPERIMENTS
We conduct an experiment on the classification task of CIFAR-10 (Krizhevsky et al., 2009) with varying amount of weight decay regularization strength A. We monitor the generalization error (0- test error) during the course of training and visualize heat-map of the generalization error for different A's in Figure (a).
We also conduct similar experiment with the teacher-student setup presented in Section 2.1. We visualize heat-map of the generalization error which is the mean squared error (MSE) over test distribution in Figure 2 (b). Particularly, we plot Eqs. 13 and 15 with constant = 100. As remark, we note that a & = 100 implies that a subset of features are learned 100 times faster than other features.
observed that in both experiments, a model with intermediate levels of regularization displays typical overfitting behavior where the generalization error decreases first and then overfits. This is consistent with Eq. 87 which indicates larger amounts of regularization prevent slow feature from being learned as À and the inverse of are summed. In other words, learning of slow features requires large weights, something that is penalized by the weight-decay. On the other hand, a model with smaller amount of regularization exhibits the double descent generalization curve.
We also validate our derived analytical expressions by running numerical simulations which are presented in Figure 4.

---

Under review as a conference paper at ICLR 2022
3.2 THE PHASE DIAGRAM
To further investigate the transition between the two phases of classical single descent and double descent, explore the phase diagram. Recall that with Eq. 6, one fully characterize the lution of the generalization dynamics in terms of two scalar variables instead of the d-dimensional parameter space. R and presented Eq. 7 macroscopic variables where R represents the alignment between the teacher and the student and Q is the student's (modulated) norm. Hence, a better generalization performance is achieved with larger R and smaller Q
R and Q are not free parameters and both depend on the training dynamics through Eqs. 13 and 15. Nevertheless,
instructive to visualize the generalization error for all pairs of RR Q). In Figure 3, we visualize the RQ-plane for (R, Q) E [0.0, 0.8] x Q E [0.0, 1.6]. At the time of initialization, (R, Q) = (0, 0) as the models are initialized at the origin. As training time proceeds, values of R and Q follow the depicted trajectories. In Figure 3, different trajectories correspond to different values of k, the condition number the modulation matrix F in Eq. 2. important to note that the closer trajectory is to the lower-right, the better the generalization error gets.
The yellow curve which corresponds to the case with large K 1e5 meaning that a subset of features extremely slower than the others that practically do not get learned. In that case, generalization error exhibits traditional over-fitting due to over-training. On the phase diagram, the yellow trajec- tory starts at (0, 0) and moves towards Point A which has the lowest generalization error of this curve. Then as the training continues, Q increases and as / + oo the trajectory lands at Point B which has the worse generalization error. The curves in orange, green and blue correspond tra- jectories with k 1e3, = 1e2, k lel, respectively. They follow the case of к 1e5 up to the vicinity of Point B, but then the trajectories slowly incline towards another fixed point, Point C signalling a second descent in the generalization error.
The phase diagram along with the corresponding generalization curves Figure 2 illustrate that features that are learned on a faster time-scale are responsible for the initial conventional U-shaped generalization curve, while the second descent can be attributed to the features that are learned at slower time-scale.
0.5
12
student' moddlltedd borm
MSB genealization
-15
0:3-
Q
1.0
0.2
0.5
0.3
06
1.0
R: teucher ttuccher-student alignment
Training time
Figure 3: Left: Phase diagram of the generalization error as a function of R(t)) and Q(t) (Eqs. 13
and 15). The generalization error for all pairs of (R, Q) [0.0, 0.8] x [0.0, 1.6] is contour- plotted in the background in shades of beige, with the best generalization performance being attained on the lower right part the plot. The trajectories describe the evolution of R(t) and Q(t) as training proceeds. Each trajectory correspond to different k,, the condition number of the modulation matrix F in Eq. 2. x describes the ratio of the rates at which two sets of features are learned. Right: The corresponding generalization curves for different plotted over the training time axis. Analysis: The trajectory with k 1e5 (bright yellow) starts at the origin and advances towards point A (a descent generalization error). Then by over-training, it converges to point B (an ascent in generalization error). For the other trajectories with smaller the point A, then an ascent happens, but they no longer converge to point B. Instead, by further training, these trajectories converge to point C' implying second descent.
first descent in generalization error occurs up to
8

---

Under review as a conference paper at ICLR 2022
4 RELATED WORK AND DISCUSSION
Although the term double descent has been introduced rather recently (Belkin et al., 2019a), similar behaviors had already been observed and studied in several decades-old works form statistical physics perspective (Krogh & Hertz, 1992a; Opper, 1995; Opper & Kinzel, 1996; Bös, 1998; Engel & Van den Broeck, 2001). More recently, these behaviors have been investigated in the context of modern machine learning, both from an empirical (Nakkiran et al., 2019; Amari et al., 2020; Yang al., 2020) and theoretical (Belkin et al., 2019a; Geiger al., 2019; Advani & Saxe, 2017; Mei & Montanari, 2019; Gerace et al., 2020; d' Ascoll et al., 2020; Ba et al., 2019; d' d'Ascoli et al., 2021) perspectives.
Hastie et al. (2019); Advani et al. (2020); Belkin et al. (2020) use random matrix theory (RMT) tools characterize the asymptotic generalization behavior of over-parameterized linear and random feature models. In an influential work, Mei & Montanari (2019) extend the same analysis to random feature model and theoretically derive the model-wise double descent curve for model with Tikhonov regularization. Jacot et al. (2020) also study double descent in ridge estimators and show an equivalence to kernel ridge regression. Pennington Worah (2019) used RMT to study the curvature of hingle-hidden-layer neural network in an attempt understand the efficacy of first-order optimization methods in training DNNs. In addition, Liang& & Rakhlin (2020) take a similar approach to investigate implicit regularization in high dimensional ridgeless regression with nonlinear kernels.
While most of the related work study the non-monon monotonicity of the generalization error as a function of the model size or sample size, Nakkiran et al. (2019) introduced the spoch-wise double descent. Epoch-wise double descent refers to the phenomenon where the generalization error undergoes two descents as the training time increases. There has been limited work on studying of epoch-wise double descent. Very recently, Heckel & Yilmaz (2020) and Stephenson & Lee (2021) have focused on finding the roots of this phenomenon.
Heckel & Yilmaz (2020) provides upper bounds on the risk of single and two layer models in a regression setting where the input data has distinct feature variances. Heckel & Yilmaz (2020) demonstrate that superposition of two more bias-variance tradeoff curves leads to epoch-wise double descent. The authors also show that different layers of the network are learned at different epochs. For that reason, epoch- wichewise&dee double descent can be eliminated by appropriate selection of learning rates for individual network weights. Consistent with these findings, our work formalizes this phenomenon in terms of feature learning scales and provides closed-form predictions.
Stephenson & Lee (2021) arrives at similar conclusions. Authors in Stephenson & Lee (2021) take data model that exhibits epoch-wise double descent. The
random matrix theory approach data model is constructed so that the noise is explicitly added only to the fast-lcarning features while slow-learning features remain noise-free. Consequently, the fast-learning features noisy and hence show a U-shaped generalization curve while slow-learning features are noiseless.
Our findings and those of Heckel & Yilmaz (2020) and Stephenson & Lee (2021) reinforce one another with a common central finding that the epoch-wise double descent results from different features/layers being learned at different time-scale. However, we also highlight that both Heckel & Yilmaz (2020) and Stephenson & Lee (2021) built upon tool from random matrix theory and study distinct data models from our teacher-student setup. We study the same phenomenon from different perspective. By leveraging the replica method from statistical physics, we characterized the generalization behavior using a set of informative macroscopic parameters. While supporting the notion that the interaction of different feature learning speeds causes epoch-wise double descent, our work provides formal predictions of the dynamics that unfold during training.
We believe our theoretical framework sets the stage for further understanding of generalization dy- namics neural networks beyond the double descent. A future direction to study is case which the first descent is strong enough to bring down the training loss to very small values to the point that learning slower features is practically impossible or happens after a very large number of epochs. Power et al. (2021) reports instance of such behavior called Grokking where the model abruptly learss to perfectly generalize but long after the training loss has reached very small values.
Limitations. It should be noted that studying finer details of the dynamics would require a more precise model of the neural networks. Clearly, our proposed model is not a universal and unique way to model the dynamics of the complex, over-parameterized deep neural networks.

---

Under review as a conference paper at ICLR 2022
Social Impact. The authors do not foresee negative social impact specifically arising from this rather theoretical work.
REFERENCES
Madhu S Advani and Andrew M Saxe. High-dimensional dynamics of generalization error neural networks. arXiv preprint arXiv:1110.03667, 2017.
Madhu S Advani, Andrew M Saxe, and Haim Sompolinsky. High-dimensional dynamics of gener- alization error in neural networks. Neural Nenworks, 132:428 446, 2020.
Alnur Ali, J Zico Kolter, and Ryan Tibshirani. continuous-tima view of carly stopping for least squares regression. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1370-1378. PMLR, 2019.
Alnur Ali, Edgar Dobriban, and Ryan Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In International Conference Machine Learning, pp. 233 244. PMLR, 2020.
Shun-ichi Amari, Jimmy Ba, Roger Grosse, Xuechen Li, Atsushi Nitanda, Taiji Suzuki, Denny Wu, and Ji Xu. When does preconditioning help hurt generalization? arXiv preprint arXiv:2006. 10732, 2020.
Jimmy Ba, Murat Erdogdu, Taiji Suzuki, Denny Wu, and Tianzong Zhang. Generalization of two- layer neural networks: An asymptotic viewpoint. In International conference on learning repre- sentations, 2019.
Yu Bai and Jason D. Lee. Beyond linearization: On quadratic and higher-order approximation of wide neural networks, 2020.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical birs-variance trade-off. Proceedings of the National Academy of Sciences, 116(32):15849 15844915544 2019a.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical varinvariance trade-off. Proceedings of the National Academy of Sciences, 156339):1549--155
2019b.
Mikhail Belkin, Daniel Hsu, and Ji Xu. Two models of double descent for weak features. SIAM Journal on Mathematics of Data Science, 2(4):1167-1180, 2020.
Carl M Bender and Steven A Orszag. Advanced mathematical methods for scientists and engineers I: Asymptotic methods and perturbation theory. Springer Science & Business Media, 2013.
S Bös, W Kinzel, and Opper. Generalization ability of perceptrons with continuous outputs. Physical Review F, 47(2): 1993.
Siegfried Bös. Statistical mechanics approach to early stopping and weight decay. Physical Review E, (55(188(33 1998.
Léon Bottou et al. Stochastic gradient learning in neural networks. Proceedings of Neuro-Nimes, (8):12, 1991.
Lin Chen, Yifei Min, Mikhail Belkin, and Amin Karbasi. Multiple descent: Design your own generalization curve. arXiv preprint (xXi:20008.01336 2020.
Lenaic Chizat and Francis Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference Learning Theory, pp. 1305-1338. PMLR, 2020.
Stéphane D' A'sooi,,, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in dou- ble descent: Bias and variance(s) in the lazy regime. In Hal Daumé III and Aarti Singh Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 2280-2290. PMLR, 13-18 Jul 2020. URL https: htaps://preee http:///ooooddeecc ngs ml pre.sppeesseee.........r...ee / / 9 d-ascol 120a html.
(eds.). Proceedings of the 37th International Conference
10

---

Under review as a conference paper at ICLR 2022
Stéphane d' Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfitting: Where & why do they appear? arXiv preprint (xXi::2006003509, 2020.
Stéphane Ascoli, Marylou Gabrié, Levent Sagun, and Giulio Biroli. On the interplay between data structure and loss function in classification problems. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
Stéphane Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in dou- ble descent: Bias and variance (s) in the lazy regime. In International Conference Machine Learning, pp. 2280-22900 PMLR, 2020.
Andreas Engel and Christian Van den Broeck. Statistical mechanics of learning. Cambridge Uni- versity Press, 2001.
Jerome Friedman, Trevor Hastie, Robert Tibshirani, al. The elements of statistical learning, volume 1. Springer series in statistics New York, 2001.
Elizabeth Gardner. The space interactions neural network models. Journal of physics A: Mathematical and general, 21(1):257, 1988.
Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models.
Journal of Physics A: Mathematical and general, 21(1):271, 1988.
Elizabeth Gardner and Bernard Derrida. Three unfinished works on the optimal storage capacity of networks. Journal Physics A: Mathematical and General, 22(12): 1989.
Mario Geiger, Stefano Spigler, Stephane d' Ascoli, Levent Sagun, Marco Baity-Jesi, Giulio Biroli, and Matthieu Wyart. Jamming transition as paradigm to understand the loss landscape of deep neural networks. Physical Review E, 100(1):01211,, 2019.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
Federica Gerace, Bruno Loureiro, Florent Krzakala, Marc Mézard, and Lenka Zdeborová. Gener- alisation error in learning with random features and the hidden manifold model. In International Conference on Machine Learning, pp. 3452-3462. PMLR, 2020.
Sebastian Goldt, Galen Reeves, Marc Mézard, Florent Krzakala, and Lenka Zdeborová. The gaus- sian equivalence of generative models for learning with two-layer neural networks. arXiv e-prints, pp. 2006, 2020.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning. MIT press Cambridge, 2016.
Trevor Hastic, Andrea Montanari, Saharon Rosset, and Ryan J Tibshirani. Surprises in high- dimensional ridgeless least squares interpolation. arXiv preprint arXiv:1903.08560, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Reinhard Heckel and Fatih Furkan Yilmaz. Early stopping in deep networks: Double descent and how to eliminate it. arXiv preprint ar axXhv:2007.10099, 2020.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and gen- cralization in neural networks. Advances in neural information processing systems, pp. 8571- 8580, 2018.
Arthur Jacot, Berfin Simsek, Francesco Spadaro, Clément Hongler, and Franck Gabriel. Implicit regularization of random feature models. In International Conference on Machine Learning, pp- 4631 4641- PMLR, 2020.
Yoshiyuki Kabashima, Tadashi Wadayama, and Toshiyuki Tanaka. A typical reconstruction limit for compressed sensing based on Ip-norm minimization. Journal of Statistical Mechanics: Theory and Experiment, L000039),L000003, 2009.

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Anders Krogh and John A Hertz. Generalization in linear perceptron in the presence of noise. Journal of Physics A: Mathematical and General, 25(5):1135, 1992a.
Anders Krogh and John A Hertz. A simple weight decay can improve generalization. In Advances in neural information processing systems, pp. 950-957, 1992b.
Florent Krzakala, Mare Mézard, François Sausset, YF Sun, and Lenka Zdeborová. Statistical- physics-based reconstruction in compressed sensing, Physical Review X, 2(2):021005. 2012.
R Kuhn and S Bos. Statistical mechanics for neural networks with continuous-time dynamics.
Journal of Physics A: Mathematical and General, 26(4):831, 1993.
Daniel Kumin, Javier Sagastuy-Bren Surya Ganguli, Daniel LK Yamins, and Hidenori Tanaka. Neural mechanics: Symmetry and broken conservation laws in deep learning dynamics. arXiv preprint arXiv:2012.04722, 2020.
Yann Le Cun, Ido Kanter, and Sara Solla. Eigenvalues of covariance matrices: Application to neural-network learning. Physical Review Letters, 66(1 (1 (6)12)),39,, 1991.
Tengyuan Liang and Alexander Rakhlin. Just interpolate: Kernel "ridgeless" regression can gen- eralize. The Annals of Statistics, 48(3), Jun 2020. ISSN 0090-5364. doi: 10.1214/19-aos 849. URL http://///// / doi. oro.ollgg//.oooooooo 1ttp//1...11/1191
Vladimir Alexandrovich Marchenko and Leonid Andreevich Pastur. Distribution of eigenvalues for some sets of random matrices. Matematicheski Sbornik, 114(4):507-536, 1967.
Song Mei and Andrea Montanari. The generalization error of random features regression: precise asymptotics and double descent curve. arXiv preprint 19X8 2019.
Mare Mézard, Giorgio Parisi, and Miguel Virasoro. Spin glass theory and beyond: an introduction to the Replica Method and its applications, volume 9. World Scientific Publishing Company, 1987.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Ilya Sutskever. Deep double descent: where bigger models and more data hurt. ICLR 2020, arXiv preprint arXiv:1912.02292. 2019.
Brady Neal, Sarthak Mittal, Aristide Baratin, Vinayak Tantia, Matthew Scicluna, Simon Lacoste- Julien, and Ioannis Mitliagkas. A modern take on the bias-variance tradeoff in neural networks. arXiv preprint arXiv: 2018
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. search of the real inductive bias: On the role of implicit regularization in deep learning. arXiv preprint arXiv: 1411vv14/2.66/4 2014.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. In Advances Neural Information Processing Systems, pp. 5947-5956,
2017.
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. Towards understanding the role of over werrparameetrzzaion in generalization of neural networks, 2018
Manfred Opper. Statistical mechanics learning: Generalization. The handbook of brain theory and neural networks, pp. 922-929, 1995.
Manfred Opper and Wolfgang Kinzel. Statistical mechanics of generalization. In Models of neural networks III, pp. 151-209. Springer, 1996.
Jeffrey Pennington and Pratik Worah. Nonlinear random matrix theory for deep learning. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 2019.
12

---

Under review as a conference paper at ICLR 2022
Mohammad Pezeshki, Sékou- Kaba, Yoshua Bengio, Aaron C. Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: A learning proclivity in neural networks. CoRR, abs/2011.09468 2020. URL
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gener- alization beyond overfitting small algorithmic datasets. In ICLR MATH-AI Workshop, 2021.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Confer- ence on Machine Learning. pp. 5301-5310. PMLR, 2019.
Frederick Reif. Fundamentals of statistical and thermal physics. Waveland Press, 2009.
Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby, Statistical mechanics of learn- ing from examples. Physical review A, 45(8):6056, 1992.
Sara A Solla. A bayesian approach to learning in neural networks. International Journal of Neural Systems, 6:161-170, 1995.
Cory Stephenson and Tyler Lee. When and how epochwise double descent happens. arXiv preprint arXiv:2108. /2006, 2021.
Vladimir N. Vapnik. The nature of statistical learning theory. Wiley, New York, 1st edition, Septem- ber 1998. ISBN 978-0-471-03003--.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking variinne trade- off for generalization of neural networks. In International Conference Machine Learning, pp- 10767-10777. PMLR, 2020.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530 2016.
Xiao Zhang and Dongrui Wu. Rethink the connections among generalization, memorization and the spectral bias of dnns. CoRR, abs/2004. 13954, 2020. URL 2004. 13954.
A FURTHER RELATED WORK AND DISCUSSION
If we consider plots where the generalization error on the y-axis is plotted against other quantities on the r-axis, we find earlier works that have identified double descent behavior for quantities such as the number of parameters, the dimensionality of the data, the number of training samples, or the training time on the z-axis. In this paper, we studied epoch-wist wiiee double descent, i.e. plot the training time I, or the number of training epochs, on the z-axis. Literature displaying double descent phenomena in generalization behavior w.r.t. other quantities do so in the limit of / xc.
From a random matrix theory perspective, Le Cun et al. (1991); Hastie et al. (2019); Advani et al. (2020), and Belkin et al. (2020) are among works which have analytically studied the spectral density of the Hessian matrix. According to their analyses, at intermediate levels of complexity, the presence of small but non-zero eigenvalues in the Hessian matrix results in high generalization error as the inverse of the Hessian is calculated for the pseudo-invers inverse solution.
Neyshabur et al. (2014) demonstrated that over over- peemperraeteri networks does not necessarily overfit thus suggesting the need of a new form of measure of model complexity other than network size. Subsequently, Neyshabur et al. (2018) suggest novel complexity measure based on unit-wise ca- pacities which correlates better with the behavior of test error with increasing network size. Chizat & Bach (2020) study the global convergence and superior generalization behavior of infinitely wide two-layer neural networks with logistic loss. Goldt et al. (2020) make use of the Gaussian Equiv- alence Theorem to study the generalization performance of two-layer neural networks and kernel models trained data drawn from pre- trained generative models. Bai & Lee (2020) investigated the gap between the empirical performance of over- par-perameterized networks and their NTK counter- parts, first proposed by Jacot al. (2018).
13

---

Under review as a conference paper at ICLR 2022
From the perspective of bias/variance trade-off, Geman et al. (1992), and more recently, Neal et al. (2018) empirically observe that while bias is monotonically decreasing, variance could be decreasing too or unimodal as the number of parameters increases, thus manifesting a double descent general- ization curve. Hastie et al. (2019) analytically study the variance. More recently, Yang et al. (2020) provides a new bias/variance decomposition of bias exhibiting double desc-nt in which the variance follows bell-shaped curve. However, the decrease in variance as the model size increases remains unexplained. For high dimensional regression with random features, d' d'Ascoli et al. (2020) provides an asymptotic expression for the bias/variance decomposition and identifies three sources of vari- ance with un-moonttonuss behavior as the model size or dataset size varies. d' Ascoli et al. (2020) also employs the analysis of random feature models and identifies two forms of overfitting which leads to the so-called sample-wise triple descent. More recently, Chen et al. (2020) show that as a result of the interaction between the data and the model, one may design generalization curves with multiple descents.
From statistical physics perspective, Opper (1995); Bös et al. (1993); Bös (1998); Opper & Kinzel (1996) are among the first studies which theoretically observe sample-wise double-descent in a ridge regression setup where the solution is obtained by the pseudo-inverse method. Most of these studies employ the "Gardner analysis" (Gardner, 1988; Gardner & Derrida, 1988; 1989) for models where the number of parameters and the dimensionality of data are coupled and hence the observed form of double descent is different from that observed in deep neural networks. A beautiful extended review of this line of work is provided in Engel & Van den Broeck (2001). Among recent works. Gerace et al. (2020) also apply the Gardner analysis but to novel generalized data generating process called the hidden manifold model and derive the model-wise double-descent equations analytically.
Finally, recall that towards providing an explanation for the epoch-wise double descent, we argue that the epoch-wise double descent can be attributed to different features being learned different time-scales, resulting in a non- monotonous generalization curve. In relation to the aspect of different feature learning scales, Rahaman et al. (2019) had observed that DNNs have a tendency towards learning simple target. functions first that can allow for good generalization behavior of various data samples. Pezeshki et al. (2020) also identify and provide cxplanation for a feature learning imbalance exhibited by wver-parameterized networks trained via gradient descent cross-entropy loss, with the networks learning only a subset of the full feature spectrum over training. More recently though, Zhang & Wu (2020), show that certain DNNs models prioritize learning high- frequency components first followed by the learning of slow but informative features, leading to the second descent of the test error observed in epoch-wise double descent.
On the difference between model-wise and epoch-wise double descent curves. In accordance with its name, model-wise double descent (in the test error) occurs due to an increase in model- size (number of its parameters), i.e., as the model transitions from an under-parameterized to an over-parameterized regime. A variety of works have tried to understand this phenomenon from the lens of implicit regularization (Neyshabur et al., 2014) or defining novel complexity measures (Neyshabur et al., 2017). On the other hand, epoch-wis double descent (in the test error) treated in our work, is observed to occur for both over-parameterized (Nakkiran et al., 2019) and under- parameterized (Heckel & Yilmaz, 2020) setups. As found in our work along with the latter reference, this phenomenon seems to be a result of different feature learning speeds rather than the extent of model parameterization. The overlap of the test-error contributions from the different weights with varying scales of learning henceforth leads to a non-monotonous evolution of the model test error as exemplified by epoch-wise double descent.
We also note that the peak in model-wise double descent is associated with the model's capacity to perfectly interpolate the data, we do not think an analogous notion exists for the case of epoch-wise double descent. Our understanding of the peak in the latter is that it corresponds to a training time configuration whereby a subclass of features are already learnt (due a larger associated signal-to- notse-ratio) and are being overfitted upon to fit the target. As training proceeds further, the remaining set of features are eventually learnt thus allowing for a lowering of the test error.
On the implicit regularization of SGD and rigge-regularized loss. The results presented in Eqs. 20-23 have core dependence the findings of Ali et al. (2019; 2020). These works first formalize the connection between (continuous-timg time) GD or SGD-based training of an ordinary least squares (OLS) setup and that of ridge regression, providing bounds the test error under these algorithms
14

---

Under review as a conference paper at ICLR 2022
over training time t, in terms of ridge setup with ridge parameter 1/t. We utilize these results in the sense that by evaluating the generalization error LG of our student-teacher setup with explicit ridge regularization, we invoke the connection between the ridge coefficient À and training time as described in these works, to obtain the behavior of (ridgeless) LG over training. This determination of expression of (t(t) is what allows study the epoch-wise DI phenomenon.
B TECHNICAL PROOFS
B.1 THE GENERALIZATION ERROR A FUNCTION OF R AND Q (Eq. 6)
Recall that the teacher is the data generator and is defined as,
V
TWW
24
(O0
Va
(26)
where Idd is the teacher's 's input and vy,y R are the teacher's noiseless and noisy outputs, respectively. W E Rd represents the (fixed) weights of the teacher and c E is the noise.
And student is defined as,
= x W,
s.f.
2,
(27)
where the matrix F E Raxd is predefined and fixed modulation matrix regulating the student's access to the true input z.
The average generalization error of the student, determined by averaging the student's error over all possible input-target pairs and noise realizations given by,
Lg = 2 E... [Vy" ))
(28)
in which the variables (y", form bi-variate Gaussian distribution with zero mean and a covari- ance of,
< y" y" >= <
1 R R
E
(29)
< y.. 10'00> <ú.0>
Q
in which,
R E. TT 1 Eg . T FW
..
FWV
and,
(30)
= E. f E. W F 22 T FW]
2 // F FW
(31)
Eq. 29 implies a correlation between y² and obstructing the calculation of the average in Eq. 28. Following (Bös, 1998; Krogh & Hertz, 1992a), we define decoupled variables ÿ and as follows,
y 0" and ÿ Rij Q R
(32)
The variables T' and ý are independent Gaussian variables such that < σ. >,= 0. Therefore, two expectations can be applied independently,
Cg
2 Es.w w [(y> 9)
(33)
2 gg "(0"" - -(Ri" + QQQ -00-RP RI -
(34)
(1 + - -2R).
(35)
2
Finally, we note that expectation w.r.t. a Gaussian variable x is defined as,
dr x² exp ( f((x).
Ex[f(x)) ==
(36)
22m
2
15

---

Under review as a conference paper at ICLR 2022
B.2 THE GENERAL CASE EXACT DYNAMICS (Eqs. 9-10)
Recall that to train our student network, we use stochastic gradient descent (SGD) on the regularized mean squared loss, evaluated on the training examples as,
1 2n
CT
18"
8²1)
+
W 2
(37)
2
==1
where À E [0, oo) is the regularization coefficient.
The minimum of the loss function, denoted by W, is achieved at,
V Ww = 0 w - x - [1 = 0
(38)
= x- - VW =0
(39)
W = X))
(40)
An exact gradient descent has the following dynamics,
We = W-1 - V W.- fT:
(41)
We-1 - 1/| - x - - + AW
(42)
(1 n -- 7x XW, 1-1 7Xxx y.
(43)
[(1 TAAT - '7X'' XJW, 1 778 y,
(44)
[[1 MA)I - 7,'' x + 7xx X +
(45)
[(1 TA) '/X XX][x 7XT + +,W,
(46)
= [(1 MAA; - 7x'"" X (1X*x + +MdDW,
(47)
[(1 mA)L 7x XJW - (nXTx- X + (n) - 1)II + W.
(48)
which leads to,
X We - W = [11 -|1-1AA-XX''X(000 W),
(49)
[[[- - " x] W).
(50)
Assuming Wo = 0, we arrive at the following closed-form equation,
We = [ - (1 - m))r -x-x''x' W
(51)
where W is defined in Eq 40.
Now back to definition of R in Eq. 84 and by substitution of Eq. 51, we have,
R(t) WT FW,
(52)
WT F (1 -1) -x-x W.
(53)
d WWTF F ( [ [[1- -
- x (X
(54)
WT FV (1 (I- 7A)I -
x
(55)
2 WWT FV (1- (1- - 7A)1 -A + Akte)
(56)
WT FV (1- [[1 - 7A)z >A]" (A++)) + Ate),
(57)
T (1 [11 / - -))1%) X7
(58)
16

---

Under review as a conference paper at ICLR 2022
Similarly for Q, Det D: = (1 [[1 n/)l - n.l"). then we have,
Q(t) d
WT FT FW,
(59)
W" [(1 -1'' -1
(60)
1 W VVDTF"
(61)
d
1 W" VDF
;= FV,= = UA ===
U
(62)
A
DF A (VTF +A
à (WTF F-F V +
(63)
A+ A
A+
(WTF-
F-17 + +1-1/22)
DITDD
(- + + 1122))
(64)
A+
A+
1 WWF-11
W
A F-1"
DF
FD
A F-I,,
(65)
d
A+ A
A+NI
1/2g 1-12
1
A
DE
FD
FD
A 1/22
(66)
d
A+
A+
+ 2
Tr
AT
A
Tr B
(67)
d
d
where,
A: F (1 [1- [1- nAAL - =na]")
A
F-1
and,
(68)
XI
A
B: F (T [11 -
A A-2_
(69)
A A AI
For simplicity and brevity of the results, in the main text, we only present the results where of = 0 leads to the following expressions,
and X = 0. Substittinng of
R(t) Tr (I-I- I-- -aA)]- ---)))
(70)
Q(t) = / Tr AAA
where, A:= 1--1-nAl')VTP
(71)
and that concludes the proof.
B.3 THE SPECIAL CASE APPROXIMATE DYNAMICS (Eqs. 13 AND 15)
Recall that the teacher and student are defined as,
y = y e, y == 2 W,
f==*" W,
===
(72)
where e ~ (0 ) is the label noise, F is the modulation matrix, and The training and generalization losses are defined as,
||>| ||
1 LT 2n
La La B(O
(73)
2
According to Eq. 6, the generalization loss can be written in terms of two scalar variables R and Q,
La = 7
where,
(74)
R T -. 22 FW]- 2 WT -WFFW, FW, and,
(75)
Q : = B. E p" = FW] 2 rF
(76)
17

---

Under review as a conference paper at ICLR 2022
Now, applying t steps of SGD LT results in the following distribution for the student's weights,
1 e-BLr(W.1),
P(W.t) t)
(77)
ZØ.1
in which (T(,, (() is modified loss where its equilibrium coincides with the 1th iterate of SGD on the original loss Cr (W).
In Eq. 77 the scalar variable 3 depends on the noise of SGD and Z3, is the partition function which is defined as,
/WW 2o) !(W..!
Z3
(78)
- Ih d(W.) (WW FW, Q0))
in which, Qo can be perceived to be target norm the student weights W are being constrained to and d is the dimensionality of the data. can be interpreted that the partition function Zg, counts the students.
We are now interested in finding R and Q of the typical (most probable) students. Therefore, suffices to find the students that dominate the partition function (or more precisely the free-energy). The free-energy is defined as,
3d Ew. In Z
(79)
where W and are the teacher's weight and input, respectively.
Due to the logarithm inside the expectation, analytical computation of Eq. 79 is intractable. How- ever, the replica method (Mézard et al., 1987) allows us tackle this through the following identity,
lim F.
[Z,,]
IEw, [In 2
(80)
The case where I. As first step, we first study a case where F I. In that case, as derived in Bös (1998), Eq. 79 can be simplified to,
1 1Q R²
n3 G - 2HR + Q
-8f
2 Q - Q In(Qo Q) 2d
In[1+ Q)]
(81)
2d 1+ B(Qo Q)
in which the scalar variables G and H are defined as,
H = Ey [y* *T y] = Ey Jy (v" +c]] =
(82)
G: Ey [yT v) [(y + "v + +c)] 1 o².
(83)
At this point, in order to find the most probable students,
extremize the free-energy
fRR, Q, Qo) in Eq. 81. The solution to this extermination is derived in Bös et al. (1993) and reads,
R n
Rzf= =0
(84)
d
Vqf= =0
Q d a²
n
1
G
n 2 a d
(85)
12/d
21
vo.f=0 =0
a 1+
(86)
n/d
i v
n/d X)2 + 47
in which,
a= 1+
and,
A= x 9
(87)
B(Qo Q)
18

---

Under review as a conference paper at ICLR 2022
The case where F follows Assumption 1.
Assumption. The modulation matrix, F, under a SVD, F := UEVT has two sets of singular values such that the first p singular values are equal to on and the remaining d d singular values are equal to 02- We let the condition number of to be denoted by n = mn
Without loss of generality, assume that U = V = I. Consequently, the (noiseless) teacher and the student can be written as the composition of two sub-models as following,
W2. 0 51 + 82 क W1 + 02 W2.
y vī + v2 = = =f
(teacher decomposition)
(88)
(student decomposition)
(89)
in which 21 IR and 22 Rd
Let ds denote the output of the h component of the student. Also let yf and denote the noiseless and noisy targets, respectively. Therefore, for the student components E 1, 2, we have,
 W1,
D2 0222 W2
vi 2TW1.
y/ 2/ W2.
y1 = vi + 23 W2 - 022 W/ te, 1 =1))=)
y22 = vi + o Wi - -1 +c,
vi -i=e1=e
in which e is the explicit noise, added to the teacher's output while (t) is an implicit variable noise which decreases the component j A i learns to match D, and yj-
Accordingly, the variables H; and G; for each component are re-defined as.
H1 = Elvi yıl = Evi 1v1 T vi'l d'
H₂ = Bly²" y2| = ys Ivā T Všl = d-p p
d
G Elyl vi),
G2 Bly² y2l.
Evi + 32 02) "yri + 1/2 12)1 + of.
B[(2 + vi 3" (23 + - -1)1 + o?
= -Elyi" F vil + Ely² T vτl + Eli 0zl.
= =Ely²" vτ1 Elyi vil + E.
22E(v² [2] +
- -2Elyi TT
+ 51
- d P + Q2- 2R + 0Z:
d d-P p
+ Q1 2R1 + 0Z
d d
dd =d + d
= =1+ - 2R +
= =1+ - 2R +0.
in which R, and Qi are defined as,
1 W W. and, Q: 1:: 01 WT o
:= B. 0z] d
where og denotes the singular values of the matrix F defined in Assumption 1. Rewriting Eqs. 84, 85, and 86 for each the student's components, we arrive at,
1
1
Ri d aa
R₂ d 02
Q pad
n
1 Q2 2R2 + 0² n 01
Q2 (d p)at
n
1 Q1 -2R + +
n
22 - 02
d 01
d 02
0j
a2 1 +
1- n R1 + V(1 (1 -x1)² 4
1
n
i / (1 - J-
2m - - 4
, : d 1
1
(}+
A d p
+
fod
1pt
19

---

Under review as a conference paper at ICLR 2022
where Q1 depends Q2 and vice versa. However, with simple calculations, arrive the following standalone equation. Let.
01
02
(90)
P
p
and also let,
a4
C
2R,
12 ai
for {1,2),
(91)
aj
ai
with which the closed-from scalar expression for Q(I, A) reads,
where, Q1 bib202 + b1c
Q2 = bj bibri1 + b202
Q A) = Q1 Q2,
and,
(92)
bibb
bib2
B.4 REPLICA TRICK
In the following, we detail the mathematical arguments leading the replica trick expression. For some r - O, we can write for any scalar x:
x' exp(r In 2:
lim + m
lim In lim r
xx -1
x² - lim In 2
(93)
E[x""
B In 2
lim
E averaging
B.5 COMPUTATION OF THE FREE-ENERGY
The self-averaged free energy (per unit weight) of our student network, is given by (Engel & Van den Broeck, 2001),
1 -Bf = ((nn ,,)
(94)
Here, 3 1/T is the inverse temperature parameter corresponding to our statistical ensemble, d the (teacher) student network width, and the partition function of the system defined as (n: number of training examples).
As Gaussian variables (with n, + oo), the partition function, obtain,
IIII| / du (Wa) dydd -NNET().."")
(1")),,
))
y
Vd WT x
8
W x4
Vd
2.W
(95)
dy dDt dy
du (Wa)
BNNN (Vo. "
2T
2x
1=1
i "
exp (-
WT 2014
a.W where in the last line above, have expressed the inserted functions using their integral repre- sentations. To make further progress, we introduce the auxiliary variables,
W W =dR..
(96)
3ja
Σ
WiT W3 dQab
(97)
(),
20

---

Under review as a conference paper at ICLR 2022
via the respective ò functions, to arrive at,
/ du (W.) dy dy""
(( = II
e (N. "00"" +iy 92
2m
2T
1.a.b
WW' -RRa 8 (
x
/ pdqub
PdR
Σ
WTT,, w?? PQub
Jja
(j)(a,)
(( exp
(-2
Q0 2
(ot)
Σ
8e 16 Qab
Σ
IgnOt Ru
Σ (1)2)2
Ria
(.(a,))
W
(98)
Repeating the procedure of expressing the above ā functions using their integral representations, we then get (a = n/d),
dQu
dQoa
dQu
iP
Σ Qo + +iP)
Σ
QaaQqn
x, w, w
exp
2T
4m
2
2T
2x/d
acb
dW
+
RR R.
exp
(-2
Σ
Qua Wot
2T
2
i.a
Σ
Que W.T W
Rs Ai W
khachh
dyddi dy
(NE exp
(-1
(-2
1 Σ
(1) (102 +1 Σ Sa
2TT
22T
P.R
1
1ia
1
Σ
(1
R (122)
Σ
(Qab
R2 n2))
- i Σ 1/ " Re
2
0
(a(a.))
nia
(99)
If we now, perform singular value decomposition of the covariance matrix r as, r = SU
= VTV, where S: matrix of singular values of T, and we have expressed, V = S//2U, then one can proceed to write,
1
/
dQu dQaa
dQu dR. Rs exp
i.P EQoeas
(1Z")),
II a.b
det IV
V 2TT
4T
2
2T/d
/
dwe
+ iP
Qay Qab +iP Σ
R Rr
cxp
Σ Qua (wa)
V
2T
Qas Wi
Σ
R WW)
/
II
dy
dy
-BNET
2T
v 2t
i.j.a
n.a
exp
Σ
( +i Σ
Σ (1 R2)
(Na)
Σ
y
" R.
2
2
a.
1
Σ
Na (Qna
R R*RR))
2
(.(a,))
(100)
having expressed, W. VW.. and identifying A u from our definitions. Now, since in the above, the W integrals factorize in i, and similarly the yh 0h and dy factorize in μ. one can proceed to write:
dQudQua oa dQ dR. R. exp (P[2 Σ QoQue 2r/d
((2")),w = del [V]
II a.b
2m
2 2x/d
2 a
(101)
Σ Qub + Σ RD Rm Gs(Qua Gub, )ub, + aG +GE(( (Q))), R²
1
b<b
21

---

Under review as a conference paper at ICLR 2022
where,
dW
Gs(Qua- QT0 ,A*)
In
exp
Qualiigy
Qao Wi E Wo)
2
ach
dy dûa
dy"
-NNE exp (-pp((1))) 1-(0))²+ Σ VaDa
1
Gg(Quh, ,R)) In
2π
2t
R2y )
Σ {0,b)
Jado (Qat RR RRRR)
iy
Σ Un Ftt
2
(102)
Now, in the limit d + oo, Eq. 101 can be approximated using the saddle- saddle-point approach (Bender & Orszag, 2013),
(("")),w x extr Quhh Sah nt Aor exp (P) Eo000s 2000s +
1 2
acb
(103)
Σ ne ft + Gs(Qoa Qay, feo) + aGr(Qat, n²)))
where, extr corresponds extremization of ((2m)),,w w over the respective order parameters. Per- forming this extremization over Qoa. Qab and R", then generates an expression of the form,
((Z))),, = extr Qo...
nNV
2 QQ Q + In( - Q) In [1 + 3(Q - Q))
1Q R²
a
Q) }
a3 1 2R
2 + 3(Q 1
(104)
where we have invoked replica symmetry in the form, Qab = Q and R = R. and that Ey (y* u /)2/2. Plugging this back into Eq. ?? then finally yields,
1Q R² 2Qu - Q
In(Qu Q) a In - 1 - Q)]
Bf extro.. Qn
(105)
1-2R Q a.
2 1+ 3(0 Q)
The remaining pair of order parameters generate the following set of transcendental equations on extremization (Bös, 1998):
R
α
a 1
Q a
1
(106)
Qu Q +
(aa
where, a = max[1, at] for -
Now, the above determined values of R, Q and Qa can be perceived as the maximally likely values of R, Q and Qo of our teacher-student setup, for an inverse temperature 3 parameterizing the system.
C EXTENDED EXPERIMENTS
Figure presents the analytical generalization dynamics for two values of x: and provides compar- ison between the theory and simulation results of the same model. We observe that the theory and
22

---

Under review as a conference paper at ICLR 2022
1.6
MSE ceneralization error
Theory Experiment
1.4
1.0
0.8
0.1
10 20
10'
10
Training time
Figure 4: The teacher-student set-up in Sec. equation 2.1. We compare the analytical solutions to simulations performed on our teacher-student setup with d 100, p 50, 150 and we plot the error bars over 100 random seeds. The solutions and the simulations match closely and we observe double descent over the generalization error.
simulations accurately match. Further experiments are provided in the following anonymous Colab notebook.
Before diving into the theory, we invite the reader to recall a simple equation from thermodynam- ics. Consider ideal gas in a container with its large number of molecules moving around, col- liding with each other, all while obeying Newton's 's laws. While the exact dynamics of each of such molecules is intractable, the system's macroscopic behavior can be characterized in terms of handful of scalar quantities, namely, the pressure P, the volume V, and the temperature T. By av- eraging over suitable probability measures and applying the principle of free-energy minimization, one arrives at remarkably simple relationship between these three macroscopic variables, i.e., the well-known PV = nRT (n: number moles of gas, R: gas constant) (Reif, 2009).
23


Reviews:
[{"review_id": "zThpMQ1w39", "reviewer": "Reviewer_tRFb", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n--The analysis of a simple model that exhibits epoch-wise double descent is illuminating and worthwhile, so the topic is of general interest.\n\n--The concrete findings from the paper (exhibited in the numerics of Fig 2, Fig 3) help build intuition for the phenomenon; the agreement between finite-size experiments and analytic theory is good.\n\n\n\n\nWeaknesses:\n\n--Numerous typos in the main text and appendix that need to be addressed & divert the flow of the calculations for the reader. See more on this below. This also made it a bit challenging to check the calculations in full.\n\n--I am not certain as to whether there are notable technical advancements in the paper. Relatedly, I am not positive if the intuition involved this paper (e.g. introducing multiple scales into the problem) has appeared before; for instance, can there be more discussion of how this paper relates to Heckel & Yilmiz (2020), as well as Stephenson & Lee (2021)?\n\n--I think this paper could improve a bit on how the student-teacher model could be connected to the realistic setting (deep neural network with label noise on training data). Although I appreciate full analysis of a simple setting, I wonder if there is more that can be done (numerically or otherwise) to make a connection. Does the label noise effectively create two scales for the problem?\n\n--I would like to better understand some of the prior results this work relies on for the analytic calculation; a self-contained discussion of the exact results drawn from older literature in the appendix would be helpful. For instance, regarding Eq. (17) and the probability distribution induced by SGD. Is this for a fixed choice of initialization, and considering the stochasticity / distributional nature arises from the Gaussian noise in SGD (Eq. 4)? In that case, how robust is this to other choices of noise? What justifies then the time-dependent distribution in Eq. 18, since there isn't a notion of equilibrium in the finite time case? Overall, I also think this paper could be clearer about how it treats the various sources of randomness (e.g. why it chooses to do averaging over SGD noise separately from the average over x and W).\n\n--(Not a major source of weakness, but a suggestion.) Some of the writing and terminology seem too imprecise to be useful. For instance, I would suggest removing the first paragraph on Sec. 2.1 since it is somewhat tangential (discussion of microscopic & macroscopic quantities in the following paragraph are sufficient). The authors also write about the \"interaction\" of different feature learning speeds, which makes me think of a precise notion of interaction in the sense of physics, although I believe the authors simply mean the \"presence\" of different scales.\n\n--The abstract mentions usage of tools from random matrix theory -- where does this appear?\n\n\nTypos in manuscript & other comments:\n\n--Eq.(5): Since there is an expectation over the teacher noise \\epsilon, why isn't the loss in terms of (y-\\hat{y}) instead of (y* - \\hat{y})?\n\n--It is common to average over the draw of finite-size training dataset. I am not sure how this appears in Eq. 5; could the authors comment on this? I understand E_{x} to be over the population distribution on x, and E_{W, \\epsilon} average over the choice of teacher model. \n\n--Above Eq. 8, it is written X = F^T Z, but X = ZF is used in Eq. 8.\n\n--Eq. (20) there is a reference to W* but I do not find it introduced earlier. (It seems these are the teacher weights, W -> W*?)\n\n--Figure 2: a/b/c/d figures are mislabeled relative to the captions. In caption (b), k -> \\kappa.\n\n--Eq. (23): from the first to second line, how did z -> x without an appearance of F?\n\n--In Eq. (24)/(25), the average over x gives rise to Kronecker delta orthogonality; since z ~ Normal, how does this hold for generic F? Is there an assumption of orthogonality on F?\n\n--Eq. (28): Z^n should be labeled Z^r.\n\n--Eq. (29): what is x^*? As with earlier equations, I assume this equation refers to the teacher network but don't see input z referenced.\n\n--Eq. (30): capital N is introduced. \\mu index also previously referenced training sample index (up to n).\n\n--Eq. (31): usage of capital P, is this same as lowercase p?\n\n--Eq. (35): two lines are copied identically.\n\n\n", "summary_of_the_paper": "This paper theoretically analyzes a student-teacher setting for which epoch-wise double descent occurs. (Epoch-wise double descent, previously discussed in the literature, refers to the non-monotonicity of the population error as a function of training time.) More specifically, the setup is: (a) a linear teacher with Gaussian noise; (b) a linear student whose inputs are a linear transformation F (\"modulation matrix\") applied to the teacher's inputs. The inputs are d-dimensional and there are n training samples. Exact analytic expressions for the test error can be derived by utilizing the replica method in the limit of n, d -> \\infty. The case analyzed in most depth is a setting in which F has two scales for its singular values (with high degeneracy) -- c.f. Assumption 1. Both the theory and finite-size simulations indeed exhibit epoch-wise double descent depending on the choice of setting parameters (e.g. strength of regularization and condition number of F). The intuition behind this is that when the scales are well-separated and learnable (e.g. regularization not too strong), overfitting of the faster features occurs (leading to a rise in the test error) before learning of the slower features (which subsequently decreases the test error).", "summary_of_the_review": "My slightly lower score is based on the following factors:\n--Since the focus of this paper is not empirical (e.g. the observation in realistic networks has appeared before), the contributions come primarily from the theoretical side. In this respect, I am not sure if similar insights in linear models & control of multiple scales for epoch-wise double descent have appeared in earlier works and would appreciate if the authors could comment on this in detail. \n--The typos / errors made the calculations more ambiguous and harder to assess the correctness of the final result.\n\nI would be happy to consider adjusting my score based on the author's response.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I939fR65Pdt", "reviewer": "Reviewer_ZC59", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My major concerns include:\n1. The implicit regularization effects of SGD/GD on least square linear regression may perform as a Ridge regularization. (Ali and Ray Tibshrani's work at ICML 2020), where the number of learning epochs/steps are connected to the inverse of lambda (strength of Ridge effects). It is not superise that when you tune the lambda of Ridge, certain double descent would appear in testing accuracy.  Shall authors discuss the connections between this work and Rayn Tibshirani's works on ICML 2020 and AISTATS 2019 on Ridge-style implicit regularization of GD and SGD for OLS?\n\n2. whether authors tried to connec their work to the linear regression problem under HDLSS settings (d>>p). In such setting, the analytical results of SGD convergence might be different (inclusion of pseudo inverse). In that sense, the result would lead to Ridgeless regression. There are quite a lot of works studying the double descent of Ridgeless. Shall authors discuss the connections between this work and those works?", "summary_of_the_paper": "This work studied epoch-wise double descent using linear model as a proxy. Basically, authors proposed a linear teacher-student models to established analysis and used random matrix theory (rmt) to interpret the learning dynamics. Some simulation on the well-posed (n>p) linear regression problems backups the theory. Some real-world experiments based on ResNet-18 and noisy labels also demonstrate the relevance of proposed theorem.", "summary_of_the_review": "It is a solid work with theoretical analysis and empirical evaluation. Though it may connect to many pioneering works in the field, authors should discuss these connections.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1FND8aDfwA7", "reviewer": "Reviewer_Tczc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This insightful work demonstrates that another apparently exotic behavior of deep neural networks, non-monotonic generalization error over the course of training, is already present in simple, analytically tractable linear models. The paper is clearly written, the theory is well-motivated, and the results are neatly presented. I only wish that the authors dove more deeply into interpreting the behavior of their analytical theory: (1) what happens when features of more than two scales are present? is there triple descent in the generalization error? (2) how does epoch-wise double descent differ from model-size double descent? one difference seems to me that linear teacher-student models trained on isotropic data exhibit model-size double descent but apparently not epoch-wise double descent, which requires anisotropic data. What explains the difference? Moreover, the peak in the model-size double descent curve has a nice interpretation as the model size necessary to perfectly interpolate the training data. Is there a similar interpretation for the peak of the epoch-wise descent curve? More broadly, epoch-wise double descent is sometimes explained by training time controlling model complexity in analogy to model-size double descent - but this work appears to suggest a different mechanism. It would be interesting to discuss this further. (3) How do the results change as a function of the the overlap between the teacher and the anisotropy in the data? \n\nNote: I believe the color code in Fig. 2 is incorrect, and is misleading. I think the colors of the large and intermediate regularization strength curves should be flipped.", "summary_of_the_paper": "An influential line of work has revealed that deep neural networks can exhibit non-monotonic behavior in their generalization error (double descent) as a function of model size, dataset size, and training time. Recent (and old) theoretical work has demonstrated that even simple models like linear regression exhibit the same non-monotonic behavior as a function of model and dataset size. The authors of this work build on this literature by demonstrating that linear teacher-student models trained with gradient descent can exhibit double descent as  a function of training time. Using replica theory, they derive a closed-form expression for the generalization error of this model as a function of training time. They show that double descent can arise when the student is trained on anisotropic data which includes a set of high SNR features and a set of low SNR features. Finally, the authors demonstrate a qualitative match between the generalization error behavior of linear teacher-student regression and a ResNet18 trained on CIFAR10 as a function of training time and regularization strength.", "summary_of_the_review": "I think this work is a valuable contribution which captures an interesting feature of the training dynamics of generalization error in deep neural networks in a simple, analytically tractable model. The close qualitative match between the behavior of the simple model and a ResNet on CIFAR10 suggests that the mechanisms identified here may be general, and the theory derived by the authors lays the groundwork for deeper investigations into the training dynamics of generalization error in neural networks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper examines the time-dependent generalization behavior of high-dimensional student-teacher linear regression models. It introduces a simple two-scale covariance model and examines the exact solutions for the dynamics, finding a tradeoff between the fast- and slow-learning features, leading to epoch-wise double descent. Qualitative comparisons are made with the SGD dynamics of ResNe18 on CIFAR-10.

The reviewers offer split opinions on this work, with most reviewers finding strength in exhibiting the complex behavior of epoch-wise double descent in a simple and analytically-tractable setting. Weaknesses highlighted in the discussion include clarity, discussion of prior work, and rigor of the analyses.

I believe a clear demonstration and analytical explanation for epoch-wise double-descent would certainly be of interest to the ICLR community, and I concur with the reviewers who emphasize these strengths of the paper. However, as one reviewer mentioned, this paper is primarily a theoretical work, and as such, the main theoretical advancements over prior work should be clear, and the novel results should be sufficiently rigorous. In this regard, the paper is lacking, as detailed below.

First of all, the discussion of SGD is imprecise, with no explicit definition of the optimization method that is actually being performed. What is the batch size? How is the sampling performed? What is the learning rate/schedule? The formulas in Secs. 2.1-2.2 suggest that full-batch gradient descent is being performed. In Sec. 2.3, stochasticity from SGD is induced via a Gibbs distribution. However, contrary to the discussion, I don't think that this is a "well-known" **result** (though of course it is a well-known **model**), and in high-dimensions I am not sure it is even correct (see e.g. [1]).

Second of all, even assuming the Gibbs distribution, the substitution on line (23) is only justified in words, whereas the cited results from Ali et al., 2020, only provide a bound. What is meant by "$\approx$"? Some discussion is given about this step of the derivation, but more precise statements would really help make the argument convincing.

Finally, the derivations seem to rely on the replica method from statistical physics, which is not rigorous. While I am generally supportive of such methods for technically challenging problems that do not readily admit alternative analyses, given the simplicity of the linear model setup here, I believe a more rigorous approach would not be prohibitively difficult. At the very least, some acknowledgement should be given about the lack of rigor in the derivation.

Overall, this paper presents a simple and analytically tractable model that sheds light on the importance phenomenon of epoch-wise double descent. Unfortunately, the presentation is not sufficiently clear and the derivations not sufficiently rigorous to merit publication at this time.

[1] Paquette, Courtney et al. “SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality.” COLT (2021).
Decision: Reject