Year: 2021

Abstract:
A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the highdimensional interactions between the large number of network parameters. Such non-trivial dynamics lead to intriguing model behaviors such as the phenomenon of “double descent” of the generalization error. The more commonly studied aspect of this phenomenon corresponds to model-wise double descent where the test error exhibits a second descent with increasing model complexity, beyond the classical U-shaped error curve. In this work, we investigate the origins of the less studied epoch-wise double descent in which the test error undergoes two nonmonotonous transitions, or descents as the training time increases. By leveraging tools from statistical physics, we study a linear teacher-student setup exhibiting epoch-wise double descent similar to that in deep neural networks. In this setting, we derive closed-form analytical expressions for the evolution of generalization error over training. We find that double descent can be attributed to distinct features being learned at different scales: as fast-learning features overfit, slowerlearning features start to fit, resulting in a second descent in test error. We validate our findings through numerical experiments where our theory accurately predicts empirical findings and remains consistent with observations in deep neural networks.

heading: 1 INTRODUCTION
text: Classical wisdom in statistical learning theory predicts a trade-off between the generalization ability of a machine learning model and its complexity, with highly complex models less likely to generalize well (Friedman et al., 2001). If the number of parameters measures complexity, deep learning models sometimes go against this prediction (Zhang et al., 2016): deep neural networks trained by stochastic gradient descent exhibit a so-called double descent behavior (Belkin et al., 2019b) with increasing model parameters. Specifically, with increasing complexity, the generalization error first obeys the classical U-shaped curve consistent with statistical learning theory. However, a second regime emerges as the number of parameters is further increased past a transition threshold where generalization error drops again, hence the “double descent” or more accurately model-wise double descent (Nakkiran et al., 2019).
Nakkiran et al. (2019) showed that the phenomenon of double descent is not limited to varying model size but is also observed as a function of training time or epochs. In this case as well, the so-called epoch-wise double descent is in apparent contradiction with the classical understanding of over-fitting (Vapnik, 1998), where one expects that longer training of a sufficiently large model beyond a certain threshold should result in over-fitting. This has important implications for practitioners and raises questions about one of the most widely used regularization method in deep learning (Goodfellow et al., 2016): early stopping. Indeed, while one might expect early stopping to prevent over-fitting, it might in fact prevent models from being trained at their fullest potential.
Since the 1990s, there has been much interest in understanding the origins of non-trivial generalization behaviors of neural networks (Opper, 1995; Opper & Kinzel, 1996). The authors of Krogh & Hertz (1992b) were among the first to provide theoretical explanations for (model-wise) double descent in linear models. Summarily, at intermediate levels of complexity, where the model size is equal to the number of training examples, the model is very sensitive to noise in training data and hence, generalizes poorly. This sensitivity to noise reduces if the model complexity is either de-
creased or increased. More recently, the double descent phenomena has been also studied for more complex models such as two-layer neural networks and random feature models (Ba et al., 2019; Mei & Montanari, 2019; D’Ascoli et al., 2020; Gerace et al., 2020).
The majority of previous work in this direction focuses on understanding the asymptotic behavior of model performance, i.e., where training time t → ∞. In recent years, there has been an interest in studying the non-asymptotic (finite training time) performance, suggesting that several intriguing properties of neural networks can be attributed to different features being learned at different scales. Among the limited work studying the particular epoch-wise double descent, Nakkiran et al. (2019) introduces the notion of effective model complexity and hypothesizes that it increases with training time and hence unifies both model-wise and epoch-wise double descent. Through a combination of theory and empirical results, Heckel & Yilmaz (2020) find that the dynamics of evolution of single and two layer networks under gradient descent, can be perceived to be the superposition of two bias/variance curves with different minima times, thus leading to non-monotonic test error curves.
In this work, we build on Bös et al. (1993); Bös (1998); Advani & Saxe (2017); Mei & Montanari (2019) which analyze model-wise double descent through the lens of linear models, to probe the origins of epoch-wise double descent. Particularly,
• We introduce a linear teacher-student model which, despite its simplicity, exhibits some of intriguing properties of generalization dynamics in deep neural networks. (Section 2.1)
• In the limit of high dimensions, we leverage the replica method developed in statistical physics to derive closed-form expressions for the generalization dynamics of our teacherstudent setup, as a function of training time and regularization strength. (Section 2.2)
• Consistent with recent findings, we provide an explanation for the existence of epoch-wise double descent through the lens of multi-scale feature learning. (Figure 1)
• We perform simulation experiments to validate our analytical predictions. We also conduct experiments with deep networks, showing that our teacher-student setup exhibits generalization behavior which is qualitatively similar to that of deep networks. (Figure 2)

heading: 2 ANALYTICAL RESULTS
text: Stochastic Gradient Descent (SGD) — the de facto optimization algorithm for neural networks — exhibits complex dynamics arising from a large number of parameters (Kunin et al., 2020). However, it is possible to describe some aspects of the high-dimensional microscopic dynamics of neural networks in terms of low-dimensional understandable macroscopic entities. In a series of seminal papers by Gardner (Gardner, 1988; Gardner & Derrida, 1988; 1989), the replica method of statistical physics was adopted to derive expressions describing the generalization behavior of large linear models trained using SGD. In this paper, we employ Gardner’s analysis to build upon an established line of work studying linear and generalized linear models (Seung et al., 1992; Kabashima et al., 2009; Krzakala et al., 2012). While most of previous work study the asymptotic (t → ∞) generalization behavior, we adapt these methods to study transient learning dynamics of generalization for finite training time. In the following, we first introduce a teacher-student model that exhibits interesting characteristics of modern neural networks. We then adapt the replica method to study the generalization performance as a function of training time and amount of regularization.

heading: 2.1 A TEACHER-STUDENT SETUP
text: Teacher: We study a supervised linear regression problem in which the training labels y, are generated by a noisy linear model (Figure 1),
y := y∗ + , y∗ := zTW, zi ∼ N (0, 1√ d ), (1)
where z ∈ Rd is the teacher’s input and y∗, y ∈ R are the teacher’s noiseless and noisy outputs, respectively. W ∈ Rd represents the (fixed) weights of the teacher and ∈ R is the noise. Both W and are drawn i.i.d. from Gaussian distributions with zero means and variances of 1 and σ2 , respectively. Student: A student model is correspondingly chosen to be a similar shallow network with trainable weights Ŵ ∈ Rd. The student model is trained on n training pairs {(xµ, yµ)}nµ=1, with the labels yµ being generated by the above teacher network, as,
ŷ := xT Ŵ , s.t. x := FTz, (2)
where the matrix F ∈ Rd×d is a predefined and fixed modulation matrix regulating the student’s access to the true input z. One can think of z as the latent factors of variation on which the teacher operates, while x can be thought as the pixels that the student learns from. Learning paradigm: To train our student network, we use stochastic gradient descent (SGD) on the regularized mean squared loss, evaluated on the n training examples as,
LT := 1
2n n∑ µ=1 (yµ − ŷµ)2 + λ 2 ||Ŵ ||22 (3)
where λ ∈ [0,∞) is the regularization coefficient. Optimizing Eq. 3 with stochastic gradient descent (SGD) yields the typical update rule,
Ŵt ← Ŵt−1 − η∇ŴLT + ξ, (4) in which t denotes the training step and η is the learning rate. Additionally, ξ ∼ N (0, σ2ξ ) models the stochasticity noise of the optimization algorithm (Bottou et al., 1991).
Macroscopic variables: The quantity of interest in this work, is the expected generalization error of the student, determined by averaging the student’s error over all possible input-target pairs and noise realizations, as,
LG := 1 2 Ez [ (y∗ − ŷ)2 ] . (5)
As shown in Bös et al. (1993), if n, d→∞ with a constant ratio nd <∞, Eq. 5 can be written as a function of two macroscopic scalar variables R,Q ∈ R,
LG = 1
2 (1 +Q− 2R), (6)
where σ2 is the variance of the teacher’s output noise and,
R := 1
d WTFŴ , Q :=
1 d ŴTFTFŴ , (7)
See App. B.1 for the proof.
Both R and Q have clear interpretations; R is the dot-product between the teacher’s weights W and the student’s modulated weights FŴ , hence can be interpreted as the alignment between the teacher and the student. Similarly, Q can be interpreted as the student’s modulated norm. The negative sign of R in Eq. 6 suggests that the larger R is, the smaller the generalization error gets. At the same time, Q appears with a positive sign suggesting the students with smaller (modulated) norm generalize better.
As a remark, note that bothR andQ are functions of Ŵ which itself is a function of training iteration t and the regularization strength λ. Therefore, hereafter, we denote the above quantities as LG(t, λ), R(t, λ), and Q(t, λ).

heading: 2.2 MAIN RESULTS
text: In this Section, we present our main analytical results, with Section 2.3 containing a sketch of our derivations. For brevity of the results, here, we only present the results for σ2 = λ = 0. See App. B for the general case and the detailed proofs.
General matrix F . Let Z := [zµ]nµ=1 ∈ Rn×d and X := [xµ]nµ=1 ∈ Rn×d denote the input matrices for the teacher and student such that X := ZF . For a general modulation matrix F , the input covariance matrix has the following singular value decomposition (SVD),
XTX = FTZTZF = V ΛV T , (8)
in which the diagonal matrix Λ contains the eigenvalues of the student’s input covariance matrix. Solving the dynamics of gradient descent as in Eq. 4, we arrive at the following exact analytical expressions for R(t) and Q(t),
R(t) = 1
d Tr(D) where, D :=
( I − [I − ηΛ]t ) , (9)
Q(t) = 1 d Tr ( ATA ) where, A := FV DV TF−1, (10)
in which Tr(.) is the trace operator. See App. B.2 the proof.
Remark: The solution in Eqs. 9 and 10 are exact, however, they require the empirical computation of the eigenvalues Λ. Below, we treat a special case of the dynamics that allow us to derive approximate solutions that do not explicitly depend on Λ.
Special case: Fast and slow features. We now study a case where the modulation matrix F has a specific structure described in Assumption 1. Assumption 1. The modulation matrix, F , under a SVD, F := UΣV T has two sets of singular values such that the first p singular values are equal to σ1 and the remaining d− p singular values are equal to σ2. We let the condition number of F to be denoted by κ := σ1σ2 > 1.
By employing the replica method of statistical physics (Gardner, 1988; Gardner & Derrida, 1988), we now derive approximate expressions for R(t) and Q(t). To begin with, we first define the following auxiliary variables,
α1 := n
p , α2 :=
n d− p , λ̃1 := d p 1 ησ21t , λ̃2 := d d− p 1 ησ22t , (11)
and also let,
ai = 1 + 2λ̃i (1− αi − λ̃i) + √ (1− αi − λ̃i)2 + 4λ̃i , for i ∈ {1, 2}. (12)
The closed-from scalar expression for R(t) is then given by,
R(t) = R1 +R2, where, R1 := n
a1d , and, R2 :=
n
a2d (13)
For Q(t), we accordingly define two more auxiliary variables,
bi = αi
a2i − αi , ci = 1− 2Ri −
n
d 2− ai ai
for i ∈ {1, 2}, (14)
with which the closed-from scalar expression for Q(t) reads,
Q(t) = Q1 +Q2, where, Q1 := b1b2c2 + b1c1
1− b1b2 , and, Q2 := b1b2c1 + b2c2 1− b1b2 . (15)
By plugging Eqs. 13 and 15 into Eq. 6, one obtains a closed-form expression for LG(t) as a function of the training time. See App. B.3 for the proof.
R = R1 + R2 R1 R2
t
SGD steps tth step on argmin L(Ŵ ) <latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
L(Ŵ ) <latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
SGD steps tth step on argmin L(Ŵ ) <latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
L(Ŵ ) <latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
Remark: Eq. 11 indicates that the singular values of F , are directly multiplied by t. That implies that the learning speed of each feature is scaled by the magnitude of its corresponding singular value. As an illustration, the figure on the right shows the evolution of R1, R2, and R = R1 + R2 for a case where p = d/2, σ1 = 1, and σ1 = 0.01 implying a condition number of κ = 100.

heading: 2.3 SKETCH OF DERIVATIONS
text: In this Section, we sketch the key steps in the derivation of our main results. For the sake of simplicity, here we only treat the case where σ = λ = 0. The general case with detailed proofs are presented in App B.
Exact dynamics of SGD. Recall the gradient descent update rule in Eq. 4. For the linear model defined in Eqs. 1-2, learning is gove ned by the following discrete-time dynamics,
Ŵt = Ŵt−1 − η∇Ŵt−1LT , (16) = Ŵt−1 − η [ −XT (y −XŴt−1) ] . (17)
With the assumption that Ŵt=0 = 0, the dynamics admit the following exact closed-form solution, Ŵt = ( I − [ I − ηXTX ]t) (XTX)−1XT y := W̃ (t). (18)
With a SVD on XTX , Eqs. 9-10 can then be obtained by substituting Ŵt in Eqs. 7. As a remark, note that one can recover the results of Advani & Saxe (2017) by setting F = I . In that case, the eigenvalues of XTX follow a Marchenko–Pastur distribution (Marchenko & Pastur, 1967).
Induced probability density of SGD. It is well-known (Kuhn & Bos, 1993; Solla, 1995) that probability distribution of weight configurations for network weights Ŵ trained via SGD on a loss L(Ŵ ), tend to the Gibbs distribution such that,
P (Ŵ ) = 1
Zβ e−βL(Ŵ ), (19)
in which Zβ is the partition function (∫ dŴ exp(−βL(Ŵ )) )
and β is called the inverse temperature and is inversely proportional the stochastic noise of SGD, ξ, defined in Eq. 4. Intuitively, for small β, the distribution of P (Ŵ ) is almost uniform, while as β → ∞, P (Ŵ ) becomes more concentrated around the minimum of the training loss.
It is important to highlight that Eq. 19 describes the equilibrium distribution of the student network’s weights, i.e., at the end of training (t → ∞). However, we are interested in studying the trajectory
of student’s weights during the course of training, i.e., for finite t. To that end, we derive the timedependent probability density over Ŵ ,
P (Ŵ , t) = 1
Zβ , t e−βL̃(Ŵ ,t), where, (20)
L̃T (Ŵ , t) : = 1
2n
∑ (ŷµ − ỹµ(t))2 + λ
2 ||Ŵ ||22, (21)
= 1
2n
∑ (ŷµ − xµT W̃ (t))2 + λ
2 ||Ŵ ||22, (W̃ (t) defined in Eq. 18) (22)
≈ LT (Ŵ ) + 1
2
( λ+ 1
ηt
) ||Ŵ ||22. (23)
R = R1 + R2 R1 R2
t
SGD steps tth step on argmin L(Ŵ ) <latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
L(Ŵ ) <latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
SGD steps tth step on argmin L(Ŵ ) <latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit>
L(Ŵ ) <latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit><latexit sha1_base64="KrE7GKXcbjd4i3q6mbD83colcc4=">AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg==</latexit>
L̃(Ŵ , t) <latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit><latexit sha1_base64="32x2N2VoNbVPu4FGsSV5PhoASno=">AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo=</latexit> Remark: L̃T (Ŵ , t) is a modified loss such that its minimum (equilibrium distribution) is achieved at the tth iterate of gradient descent on L(Ŵ ). The schematic diagram on the right illustrates this equivalence, such that, arg minŴ L̃T (Ŵ , t) = Ŵt, where Ŵt is the defined in Eq. 4.
The typical generalization error. To determine the typical generalization performance at time t, one proceeds by first computing the free-energy of the system as,
f := − 1 βd
EW,z [ lnZβ,t ] . (24)
Free-energy is a self-averaging property where its typical/most probable value coincides with its average over proper probability distributions Engel & Van den Broeck (2001). Therefore, to determine the typical values of R and Q, we extremize the free-energy w.r.t. those variables.
Due to the logarithm inside the expectation, analytical computation of Eq. 24 is intractable. However, the replica method (Mézard et al., 1987) allows us to tackle this through the following identity,
EW,z[lnZβ,t] = lim r→0 EW,z[Zrβ,t]− 1 r . (25)
Computation of the free-energy via replica method and its subsequent extremization w.r.t R and Q, we arrive at Eqs. 13 and 15. See App. B.3 for more details.
To summarize, using the replica method, we are able to cast the high-dimensional dynamics of SGD into simple scalar equations governing R and Q and, consequently, the generalization error LG. While our analysis is limited to the specific teacher and student setup, this simple model already exhibits dynamics qualitatively similar to those observed in more complex networks, as we now illustrate.

heading: 3 EXPERIMENTAL RESULTS
text: In this Section, we conduct numerical simulations to validate our analytical results and provide clear insights on the macroscopic dynamics of generalization. We also conduct experiments on realworld neural networks showing a close qualitative match between the generalization behavior of neural networks and our teacher-student setup.
For real-world experiments, we train a ResNet18 (He et al., 2016) with large layer widths [64, 2× 64, 4 × 64, 8 × 64]. We follow the training setup of Nakkiran et al. (2019); Label noise with a probability 0.15 randomly assign an incorrect label to training examples. Noise is sampled only once before the training starts. We train using Adam (Kingma & Ba, 2014) with learning rate of 1e − 4 for 1K epochs. Real-world experiments are averaged over 50 random seeds. To ensure reproducibility, we include the complete source code in a GitHub repository as well as an anonymous Collab notebook.

heading: 3.1 MATCH BETWEEN THEORY AND REAL-WORLD EXPERIMENTS
text: We conduct an experiment on the classification task of CIFAR-10 (Krizhevsky et al., 2009) with varying amount of weight decay regularization strength λ. We monitor the generalization error (0- 1 test error) during the course of training and visualize a heat-map of the generalization error for different λ’s in Figure 2 (a).
We also conduct a similar experiment with the teacher-student setup presented in Section 2.1. We visualize a heat-map of the generalization error which is the mean squared error (MSE) over test distribution in Figure 2 (b). Particularly, we plot Eqs. 13 and 15 with a constant κ = 100. As a remark, we note that a κ = 100 implies that a subset of features are learned 100 times faster than other features.
It is observed that in both experiments, a model with intermediate levels of regularization displays a typical overfitting behavior where the generalization error decreases first and then overfits. This is consistent with Eq. 87 which indicates larger amounts of regularization prevent slow feature from being learned as λ and the inverse of t are summed. In other words, learning of slow features requires large weights, something that is penalized by the weight-decay. On the other hand, a model with smaller amount of regularization exhibits the double descent generalization curve.
We also validate our derived analytical expressions by running numerical simulations which are presented in Figure 4.

heading: 3.2 THE PHASE DIAGRAM
text: To further investigate the transition between the two phases of classical single descent and double descent, we explore the phase diagram. Recall that with Eq. 6, one can fully characterize the evolution of the generalization dynamics in terms of two scalar variables instead of the d-dimensional parameter space. R and Q presented in Eq. 7 are macroscopic variables where R represents the alignment between the teacher and the student andQ is the student’s (modulated) norm. Hence, a better generalization performance is achieved with larger R and smaller Q.
R and Q are not free parameters and both depend on the training dynamics through Eqs. 13 and 15. Nevertheless, it is instructive to visualize the generalization error for all pairs of (R,Q). In Figure 3, we visualize the RQ-plane for (R,Q) ∈ [0.0, 0.8]×Q ∈ [0.0, 1.6]. At the time of initialization, (R,Q) = (0, 0) as the models are initialized at the origin. As training time proceeds, values of R and Q follow the depicted trajectories. In Figure 3, different trajectories correspond to different values of κ, the condition number of the modulation matrix F in Eq. 2. It is important to note that the closer a trajectory is to the lower-right, the better the generalization error gets.
The yellow curve which corresponds to the case with large κ = 1e5 meaning that a subset of features are extremely slower than the others that practically do not get learned. In that case, generalization error exhibits traditional over-fitting due to over-training. On the phase diagram, the yellow trajectory starts at (0, 0) and moves towards Point A which has the lowest generalization error of this curve. Then as the training continues, Q increases and as t → ∞ the trajectory lands at Point B which has the worse generalization error. The curves in orange, green and blue correspond to trajectories with κ = 1e3, κ = 1e2, κ = 1e1, respectively. They follow the case of κ = 1e5 up to the vicinity of Point B, but then the trajectories slowly incline towards another fixed point, Point C signalling a second descent in the generalization error.
The phase diagram along with the corresponding generalization curves in Figure 2 illustrate that features that are learned on a faster time-scale are responsible for the initial conventional U-shaped generalization curve, while the second descent can be attributed to the features that are learned at a slower time-scale.

heading: 4 RELATED WORK AND DISCUSSION
text: Although the term double descent has been introduced rather recently (Belkin et al., 2019a), similar behaviors had already been observed and studied in several decades-old works form a statistical physics perspective (Krogh & Hertz, 1992a; Opper, 1995; Opper & Kinzel, 1996; Bös, 1998; Engel & Van den Broeck, 2001). More recently, these behaviors have been investigated in the context of modern machine learning, both from an empirical (Nakkiran et al., 2019; Amari et al., 2020; Yang et al., 2020) and theoretical (Belkin et al., 2019a; Geiger et al., 2019; Advani & Saxe, 2017; Mei & Montanari, 2019; Gerace et al., 2020; d’Ascoli et al., 2020; Ba et al., 2019; d’Ascoli et al., 2021) perspectives.
Hastie et al. (2019); Advani et al. (2020); Belkin et al. (2020) use random matrix theory (RMT) tools to characterize the asymptotic generalization behavior of over-parameterized linear and random feature models. In an influential work, Mei & Montanari (2019) extend the same analysis to a random feature model and theoretically derive the model-wise double descent curve for a model with Tikhonov regularization. Jacot et al. (2020) also study double descent in ridge estimators and show an equivalence to kernel ridge regression. Pennington & Worah (2019) used RMT to study the curvature of single-hidden-layer neural network in an attempt to understand the efficacy of first-order optimization methods in training DNNs. In addition, Liang & Rakhlin (2020) take a similar approach to investigate implicit regularization in high dimensional ridgeless regression with nonlinear kernels.
While most of the related work study the non-monotonicity of the generalization error as a function of the model size or sample size, Nakkiran et al. (2019) introduced the epoch-wise double descent. Epoch-wise double descent refers to the phenomenon where the generalization error undergoes two descents as the training time increases. There has been limited work on studying of epoch-wise double descent. Very recently, Heckel & Yilmaz (2020) and Stephenson & Lee (2021) have focused on finding the roots of this phenomenon.
Heckel & Yilmaz (2020) provides upper bounds on the risk of single and two layer models in a regression setting where the input data has distinct feature variances. Heckel & Yilmaz (2020) demonstrate that a superposition of two or more bias-variance tradeoff curves leads to epoch-wise double descent. The authors also show that different layers of the network are learned at different epochs. For that reason, epoch-wise double descent can be eliminated by appropriate selection of learning rates for individual network weights. Consistent with these findings, our work formalizes this phenomenon in terms of feature learning scales and provides closed-form predictions.
Stephenson & Lee (2021) arrives at similar conclusions. Authors in Stephenson & Lee (2021) take a random matrix theory approach on a data model that exhibits epoch-wise double descent. The data model is constructed so that the noise is explicitly added only to the fast-learning features while slow-learning features remain noise-free. Consequently, the fast-learning features are noisy and hence show a U-shaped generalization curve while slow-learning features are noiseless.
Our findings and those of Heckel & Yilmaz (2020) and Stephenson & Lee (2021) reinforce one another with a common central finding that the epoch-wise double descent results from different features/layers being learned at different time-scale. However, we also highlight that both Heckel & Yilmaz (2020) and Stephenson & Lee (2021) built upon tool from random matrix theory and study distinct data models from our teacher-student setup. We study the same phenomenon from a different perspective. By leveraging the replica method from statistical physics, we characterized the generalization behavior using a set of informative macroscopic parameters. While supporting the notion that the interaction of different feature learning speeds causes epoch-wise double descent, our work provides formal predictions of the dynamics that unfold during training.
We believe our theoretical framework sets the stage for further understanding of generalization dynamics in neural networks beyond the double descent. A future direction to study is a case in which the first descent is strong enough to bring down the training loss to very small values to the point that learning slower features is practically impossible or happens after a very large number of epochs. Power et al. (2021) reports an instance of such behavior called Grokking where the model abruptly learns to perfectly generalize but long after the training loss has reached very small values.
Limitations. It should be noted that studying finer details of the dynamics would require a more precise model of the neural networks. Clearly, our proposed model is not a universal and unique way to model the dynamics of the complex, over-parameterized deep neural networks.
Social Impact. The authors do not foresee a negative social impact specifically arising from this rather theoretical work.

heading: A FURTHER RELATED WORK AND DISCUSSION
text: If we consider plots where the generalization error on the y-axis is plotted against other quantities on the x-axis, we find earlier works that have identified double descent behavior for quantities such as the number of parameters, the dimensionality of the data, the number of training samples, or the training time on the x-axis. In this paper, we studied epoch-wise double descent, i.e. we plot the training time t, or the number of training epochs, on the x-axis. Literature displaying double descent phenomena in generalization behavior w.r.t. other quantities do so in the limit of t→∞. From a random matrix theory perspective, Le Cun et al. (1991); Hastie et al. (2019); Advani et al. (2020), and Belkin et al. (2020) are among works which have analytically studied the spectral density of the Hessian matrix. According to their analyses, at intermediate levels of complexity, the presence of small but non-zero eigenvalues in the Hessian matrix results in high generalization error as the inverse of the Hessian is calculated for the pseudo-inverse solution.
Neyshabur et al. (2014) demonstrated that over-parameterized networks does not necessarily overfit thus suggesting the need of a new form of measure of model complexity other than network size. Subsequently, Neyshabur et al. (2018) suggest a novel complexity measure based on unit-wise capacities which correlates better with the behavior of test error with increasing network size. Chizat & Bach (2020) study the global convergence and superior generalization behavior of infinitely wide two-layer neural networks with logistic loss. Goldt et al. (2020) make use of the Gaussian Equivalence Theorem to study the generalization performance of two-layer neural networks and kernel models trained on data drawn from pre-trained generative models. Bai & Lee (2020) investigated the gap between the empirical performance of over-parameterized networks and their NTK counterparts, first proposed by Jacot et al. (2018).
From the perspective of bias/variance trade-off, Geman et al. (1992), and more recently, Neal et al. (2018) empirically observe that while bias is monotonically decreasing, variance could be decreasing too or unimodal as the number of parameters increases, thus manifesting a double descent generalization curve. Hastie et al. (2019) analytically study the variance. More recently, Yang et al. (2020) provides a new bias/variance decomposition of bias exhibiting double desc-nt in which the variance follows a bell-shaped curve. However, the decrease in variance as the model size increases remains unexplained. For high dimensional regression with random features, d’Ascoli et al. (2020) provides an asymptotic expression for the bias/variance decomposition and identifies three sources of variance with non-monotonous behavior as the model size or dataset size varies. d’Ascoli et al. (2020) also employs the analysis of random feature models and identifies two forms of overfitting which leads to the so-called sample-wise triple descent. More recently, Chen et al. (2020) show that as a result of the interaction between the data and the model, one may design generalization curves with multiple descents.
From a statistical physics perspective, Opper (1995); Bös et al. (1993); Bös (1998); Opper & Kinzel (1996) are among the first studies which theoretically observe sample-wise double-descent in a ridge regression setup where the solution is obtained by the pseudo-inverse method. Most of these studies employ the “Gardner analysis” (Gardner, 1988; Gardner & Derrida, 1988; 1989) for models where the number of parameters and the dimensionality of data are coupled and hence the observed form of double descent is different from that observed in deep neural networks. A beautiful extended review of this line of work is provided in Engel & Van den Broeck (2001). Among recent works, Gerace et al. (2020) also apply the Gardner analysis but to a novel generalized data generating process called the hidden manifold model and derive the model-wise double-descent equations analytically.
Finally, recall that towards providing an explanation for the epoch-wise double descent, we argue that the epoch-wise double descent can be attributed to different features being learned at different time-scales, resulting in a non-monotonous generalization curve. In relation to the aspect of different feature learning scales, Rahaman et al. (2019) had observed that DNNs have a tendency towards learning simple target functions first that can allow for good generalization behavior of various data samples. Pezeshki et al. (2020) also identify and provide explanation for a feature learning imbalance exhibited by over-parameterized networks trained via gradient descent on cross-entropy loss, with the networks learning only a subset of the full feature spectrum over training. More recently though, Zhang & Wu (2020), show that certain DNNs models prioritize learning highfrequency components first followed by the learning of slow but informative features, leading to the second descent of the test error as observed in epoch-wise double descent.
On the difference between model-wise and epoch-wise double descent curves. In accordance with its name, model-wise double descent (in the test error) occurs due to an increase in modelsize (number of its parameters), i.e., as the model transitions from an under-parameterized to an over-parameterized regime. A variety of works have tried to understand this phenomenon from the lens of implicit regularization (Neyshabur et al., 2014) or defining novel complexity measures (Neyshabur et al., 2017). On the other hand, epoch-wise double descent (in the test error) as treated in our work, is observed to occur for both over-parameterized (Nakkiran et al., 2019) and underparameterized (Heckel & Yilmaz, 2020) setups. As found in our work along with the latter reference, this phenomenon seems to be a result of different feature learning speeds rather than the extent of model parameterization. The overlap of the test-error contributions from the different weights with varying scales of learning henceforth leads to a non-monotonous evolution of the model test error as exemplified by epoch-wise double descent.
We also note that the peak in model-wise double descent is associated with the model’s capacity to perfectly interpolate the data, we do not think an analogous notion exists for the case of epoch-wise double descent. Our understanding of the peak in the latter is that it corresponds to a training time configuration whereby a subclass of features are already learnt (due to a larger associated signal-tonoise-ratio) and are being overfitted upon to fit the target. As training proceeds further, the remaining set of features are eventually learnt thus allowing for a lowering of the test error.
On the implicit regularization of SGD and ridge-regularized loss. The results presented in Eqs. 20-23 have a core dependence on the findings of Ali et al. (2019; 2020). These works first formalize the connection between (continuous-time) GD or SGD-based training of an ordinary least squares (OLS) setup and that of ridge regression, providing bounds on the test error under these algorithms
over training time t, in terms of a ridge setup with ridge parameter λ = 1/t. We utilize these results in the sense that by evaluating the generalization error LG of our student-teacher setup with explicit ridge regularization, we invoke the connection between the ridge coefficient λ and training time t as described in these works, to obtain the behavior of (ridgeless) LG over training. This determination of an expression of LG(t) is what allows us to study the epoch-wise DD phenomenon.

heading: B TECHNICAL PROOFS
text: B.1 THE GENERALIZATION ERROR AS A FUNCTION OF R AND Q (EQ. 6)
Recall that the teacher is the data generator and is defined as,
y := y∗ + , y∗ := zTW, zi ∼ N (0, 1√ d ), (26)
where z ∈ Rd is the teacher’s input and y∗, y ∈ R are the teacher’s noiseless and noisy outputs, respectively. W ∈ Rd represents the (fixed) weights of the teacher and ∈ R is the noise. And student is defined as,
ŷ := xT Ŵ , s.t. x := FTz, (27)
where the matrix F ∈ Rd×d is a predefined and fixed modulation matrix regulating the student’s access to the true input z.
The average generalization error of the student, determined by averaging the student’s error over all possible input-target pairs and noise realizations is given by,
LG := 1
2 Ex,W
[ (y∗ − ŷ)2 ] , (28)
in which the variables (y∗, ŷ) form a bi-variate Gaussian distribution with zero mean and a covariance of,
Σ =
[ < y∗, y∗ >z < y
∗, ŷ >z < y∗, ŷ >z < ŷ, ŷ >z
] = [ 1 R R Q ] , (29)
in which,
R : = Ez[y∗T ŷ] = Ez[WT zzTFŴ ] = 1
d WTFŴ , and, (30)
Q : = Ez[ŷT ŷ] = Ez[ŴTFT zzTFŴ ] = 1
d ŴTFTFŴ . (31)
Eq. 29 implies a correlation between y∗ and ŷ obstructing the calculation of the average in Eq. 28. Following (Bös, 1998; Krogh & Hertz, 1992a), we define decoupled variables ỹ∗ and ˜̂y as follows,
y∗ =: ỹ∗, and ŷ =: Rỹ∗ + √ Q−R2 ˜̂y. (32)
The variables ỹ∗ and ˜̂y are independent Gaussian variables such that < ỹ∗, ˜̂y >z= 0. Therefore, two expectations can be applied independently,
LG : = 1
2 Ex,W
[ (y∗ − ŷ)2 ] , (33)
= 1
2 Eỹ∗,˜̂y
[ (ỹ∗ − (Rỹ∗ + √ Q−R2 ˜̂y))2 ] , (34)
= 1
2 (1 +Q− 2R). (35)
Finally, we note that expectation w.r.t. a Gaussian variable x is defined as, Ex[f(x)] := ∫ +∞ −∞ dx√ 2π exp ( − x 2 2 ) f(x). (36)
B.2 THE GENERAL CASE EXACT DYNAMICS (EQS. 9-10)
Recall that to train our student network, we use stochastic gradient descent (SGD) on the regularized mean squared loss, evaluated on the n training examples as,
LT := 1
2n n∑ µ=1 (yµ − ŷµ)2 + λ 2 ||Ŵ ||22, (37)
where λ ∈ [0,∞) is the regularization coefficient. The minimum of the loss function, denoted by W , is achieved at,
∇ŴLT = 0⇒ ∇Ŵ [1 2 ||y −XŴ ||22 + λ 2 ||Ŵ ||22 ] = 0 (38)
⇒ −XT (y −XŴ ) + λŴ = 0 (39) ⇒W := (XTX + λI)−1XT y. (40)
An exact gradient descent has the following dynamics,
Ŵt = Ŵt−1 − η∇Ŵt−1LT , (41) = Ŵt−1 − η [ −XT (y −XŴt−1) + λŴt−1 ] (42)
= (1− ηλ)Ŵt−1 − ηXTXŴt−1 + ηXT y, (43) = [(1− ηλ)I − ηXTX]Ŵt−1 + ηXT y, (44) = [(1− ηλ)I − ηXTX]Ŵt−1 + η(XTX + λI)(XTX + λI)−1XT y, (45) = [(1− ηλ)I − ηXTX]Ŵt−1 + η(XTX + λI)W, (46) = [(1− ηλ)I − ηXTX]Ŵt−1 + (ηXTX + ηλI)W, (47) = [(1− ηλ)I − ηXTX]Ŵt−1 + (ηXTX + (ηλ− 1)I)W +W, (48)
which leads to,
Ŵt −W = [(1− ηλ)I − ηXTX](Ŵt−1 −W ), (49) = [(1− ηλ)I − ηXTX]t(Ŵ0 −W ). (50)
Assuming Ŵ0 = 0, we arrive at the following closed-form equation, Ŵt = ( I − [ (1− ηλ)I − ηXTX ]t) W, (51)
where W is defined in Eq 40.
Now back to definition of R in Eq. 84 and by substitution of Eq. 51, we have,
R(t) : = 1
d WTFŴt, (52)
= 1
d WTF
( I − [ (1− ηλ)I − ηXTX ]t) W, (53)
= 1
d WTF
( I − [ (1− ηλ)I − ηXTX ]t) (XTX + λI)−1XT y, (54)
= 1
d WTFV
( I − [ (1− ηλ)I − ηΛ ]t) (Λ + λI)−1V TXT y, (XTX = V ΛV T ) (55)
= 1
d WTFV
( I − [ (1− ηλ)I − ηΛ ]t) (Λ + λI)−1(ΛV TF−1W + Λ 1 2 ), (56)
= 1
d WTFV
( I − [ (1− ηλ)I − ηΛ ]t) (Λ + λI)−1(ΛV TF−1W + Λ 1 2 ), (57)
= 1 d Tr [( I − [(1− ηλ)I − ηΛ]t ) Λ Λ + λI ] . (58)
Similarly for Q, let D := ( I − [ (1− ηλ)I − ηΛ ]t) , then we have,
Q(t) : = 1
d ŴTFTFŴ , (59)
= 1
d W
T ( I − [ (1− ηλ)I − ηXTX ]t) FTF ( I − [ (1− ηλ)I − ηXTX ]t) W, (60)
= 1
d W
T V DV TFTFV DV TW, (61)
= 1
d W
T V DF̃T F̃DV TW, (F̃ := FV,X = UΛ1/2V T , ̃ := UT ) (62)
= 1
d (WTF−1
T V + Λ−1/2̃) Λ
Λ + λI DF̃T F̃D
Λ
Λ + λI (V TF−1W + Λ−1/2̃), (63)
= 1
d (WT F̃−1
T + Λ−1/2̃) Λ
Λ + λI DF̃T F̃D
Λ
Λ + λI (F̃−1W + Λ−1/2̃), (64)
= 1
d WT F̃−1
T Λ
Λ + λI DF̃T F̃D
Λ
Λ + λI F̃−1W, (65)
+ 1
d Λ−1/2̃
Λ
Λ + λI DF̃T F̃D
Λ
Λ + λI Λ−1/2̃, (66)
= 1 d Tr [ ATA ] + σ2 d Tr [ BTB ] (67)
where,
A : = F̃ ( I − [ (1− ηλ)I − ηΛ ]t) Λ Λ + λI F̃−1 and, (68)
B : = F̃ ( I − [ (1− ηλ)I − ηΛ ]t) Λ Λ + λI Λ− 1 2 . (69)
For simplicity and brevity of the results, in the main text, we only present the results where σ2 = 0 and λ = 0. Substituting σ2 = λ = 0 leads to the following expressions,
R(t) = 1 d Tr [( I − I − ηΛ]t )] . (70) Q(t) = 1 d Tr [ ATA ] where, A := FV ( I − [ I − ηΛ ]t) V TF−1, (71)
and that concludes the proof.
B.3 THE SPECIAL CASE APPROXIMATE DYNAMICS (EQS. 13 AND 15)
Recall that the teacher and student are defined as,
y := y∗ + , y∗ := zTW, ŷ := xT Ŵ , x := FT z, (72)
where ∼ N (0, σ2 ) is the label noise, F is the modulation matrix, and ||z||22 = ||W ||22 = 1. The training and generalization losses are defined as,
LT := 1
2n
∑ (ŷ − y)2 + λ
2 ||Ŵ ||22, LG :=
1 2 Ez[(ŷ − y∗)2]. (73)
According to Eq. 6, the generalization loss can be written in terms of two scalar variables R and Q,
LG = 1
2 (1 +Q− 2R), where, (74)
R : = Ez[y∗T ŷ] = Ez[WT zzTFŴ ] = 1
d WTFŴ , and, (75)
Q : = Ez[ŷT ŷ] = Ez[ŴTFT zzTFŴ ] = 1
d ŴTFTFŴ . (76)
Now, applying t steps of SGD on LT results in the following distribution for the student’s weights,
P (Ŵ , t) = 1
Zβ,t e−βL̃T (Ŵ ,t), (77)
in which L̃T (Ŵ , t) is a modified loss where its equilibrium coincides with the tth iterate of SGD on the original loss LT (Ŵ ). In Eq. 77 the scalar variable β depends on the noise of SGD and Zβ,t is the partition function which is defined as,
Zβ,t =
∫∞ −∞ ∏d i=1 d ( Ŵi ) δ ( 1 dŴ T i F TFŴi −Q0 ) P (Ŵi, t)∫∞
−∞ ∏d i=1 d ( Ŵi ) δ ( 1 dŴ T i F TFŴi −Q0 ) , (78)
in which, Q0 can be perceived to be a target norm the student weights Ŵ are being constrained to and d is the dimensionality of the data. It can be interpreted that the partition function Zβ,t counts the students.
We are now interested in finding R and Q of the typical (most probable) students. Therefore, it suffices to find the students that dominate the partition function (or more precisely the free-energy). The free-energy is defined as,
f := − 1 βd
EW,z [ lnZβ,t ] , (79)
where W and z are the teacher’s weight and input, respectively.
Due to the logarithm inside the expectation, analytical computation of Eq. 79 is intractable. However, the replica method (Mézard et al., 1987) allows us to tackle this through the following identity,
EW,z[lnZβ,t] = lim r→0 EW,z[Zrβ,t]− 1 r . (80)
The case where F = I . As a first step, we first study a case where F = I . In that case, as derived in Bös (1998), Eq. 79 can be simplified to,
−βf = 1 2 Q−R2 Q0 −Q + 1 2 ln(Q0 −Q)− n 2d ln[1 + β(Q0 −Q)]− nβ 2d G− 2HR+Q 1 + β(Q0 −Q) , (81)
in which the scalar variables G and H are defined as,
H : = Ey∗ [y∗T y] = Ey∗ [y∗T (y∗ + )] = 1, (82) G : = Ey∗ [yT y] = Ey∗ [(y∗ + )T (y∗ + )] = 1 + σ2 . (83)
At this point, in order to find the most probable students, one can extremize the free-energy f(R,Q,Q0) in Eq. 81. The solution to this extermination is derived in Bös et al. (1993) and reads,
∇Rf = 0 ⇒ R = n
d
1 a , (84)
∇Qf = 0 ⇒ Q = n
d
1
a2 − n/d
( G− n
d 2− a a
) , (85)
∇Q0f = 0 ⇒ a = 1 + 2λ̃ 1− n/d− λ̃+ √ (1− n/d− λ̃)2 + 4λ̃ , (86)
in which,
a := 1 + 1
β(Q0 −Q) , and, λ̃ := λ+
1 ηt . (87)
The case where F follows Assumption 1.
Assumption. The modulation matrix, F , under a SVD, F := UΣV T has two sets of singular values such that the first p singular values are equal to σ1 and the remaining d−p singular values are equal to σ2. We let the condition number of F to be denoted by κ := σ1σ2 > 1.
Without loss of generality, we assume that U = V = I . Consequently, the (noiseless) teacher and the student can be written as the composition of two sub-models as following,
y∗ = y∗1 + y ∗ 2 = z T 1 W1 + z T 2 W2, (teacher decomposition) (88)
ŷ = ŷ1 + ŷ2 = σ1z T 1 Ŵ1 + σ2z T 2 Ŵ2, (student decomposition) (89)
in which z1 ∈ Rp and z2 ∈ Rd−p. Let ŷi denote the output of the ith component of the student. Also let y∗i and yi denote the noiseless and noisy targets, respectively. Therefore, for the student components i ∈ 1, 2, we have,
ŷ1 = σ1z T 1 Ŵ1, y∗1 = z T 1 W1, y1 = y ∗ 1 + z
T 2 W2 − σ2zT2 Ŵ2︸ ︷︷ ︸ y∗2−ŷ2= 2(t) + ,
ŷ2 = σ2z T 2 Ŵ2, y∗2 = z T 2 W2, y2 = y ∗ 2 + z
T 1 W1 − σ1zT1 Ŵ1︸ ︷︷ ︸ y∗1−ŷ1= 1(t) + ,
in which is the explicit noise, added to the teacher’s output while j(t) is an implicit variable noise which decreases as the component j 6= i learns to match ŷj and yj . Accordingly, the variables Hi and Gi for each component i are re-defined as,
H1 = E[y∗1 T y1] = Ey∗1 [y ∗ 1 T y∗1 ] =
p d ,
G1 = E[yT1 y1], = E[(y∗1 + y∗2 − ŷ2)T (y∗1 + y∗2 − ŷ2)] + σ2 , = E[y∗1T y∗1 ] + E[y∗2T y∗2 ] + E[ŷT2 ŷ2], − 2E[y∗2T ŷ2] + σ2 ,
= p d + d− p d +Q2 − 2R2 + σ2 , = 1 +Q2 − 2R2 + σ2 ,
H2 = E[y∗2 T y2] = Ey∗2 [y ∗ 2 T y∗2 ] = d− p d , G2 = E[yT2 y2], = E[(y∗2 + y∗1 − ŷ1)T (y∗2 + y∗1 − ŷ1)] + σ2 , = E[y∗2T y∗2 ] + E[y∗1T y∗1 ] + E[ŷT1 ŷ1], − 2E[y∗1T ŷ1] + σ2 ,
= d− p d + p d +Q1 − 2R1 + σ2 , = 1 +Q1 − 2R1 + σ2 ,
in which Ri and Qi are defined as,
Ri := Ez[y∗Ti ŷi] = 1
d WTi σiŴi, and, Qi := Ez[ŷTi ŷi] =
1 d ŴTi σ 2 i Ŵi,
where σi denotes the singular values of the matrix F as defined in Assumption 1.
Rewriting Eqs. 84, 85, and 86 for each of the student’s components, we arrive at,
R1 = n
d
1
a1 ,
Q1 = n
pa21 − n
( 1 +Q2 − 2R2 + σ2 − n
d 2− a1 a1
) ,
a1 = 1 + 2λ̃1 1− np − λ̃1 + √ (1− np − λ̃1)2 + 4λ̃1 ,
λ̃1 : = d
p
1
σ21 (λ+
1 ηt ),
R2 = n
d
1
a2 ,
Q2 = n
(d− p)a21 − n
( 1 +Q1 − 2R1 + σ2 − n
d 2− a2 a2
) ,
a2 = 1 + 2λ̃ 1− nd−p − λ̃+ √ (1− nd−p − λ̃)2 + 4λ̃ ,
λ̃2 : = d d− p 1 σ22 (λ+ 1 ηt ),
where Q1 depends on Q2 and vice versa. However, with simple calculations, we can arrive at the following standalone equation. Let,
α1 = n
p , α2 =
n
d− p , (90)
and also let,
bi = αi
a2i − αi , ci = 1− 2Ri −
n
d 2− ai ai
for i ∈ {1, 2}, (91)
with which the closed-from scalar expression for Q(t, λ) reads,
Q(t, λ) = Q1 +Q2, where, Q1 := b1b2c2 + b1c1
1− b1b2 , and, Q2 := b1b2c1 + b2c2 1− b1b2 . (92)
B.4 REPLICA TRICK
In the following, we detail the mathematical arguments leading to the replica trick expression. For some r → 0, we can write for any scalar x:
xr = exp(r lnx) = lim r→0 1 + r lnx
⇒ lim r→0 r lnx = lim r→0 xr − 1
⇒ lnx = lim r→0 xr − 1 r
∴ E[lnx] = lim r→0 E[xr]− 1 r , E : averaging
(93)
B.5 COMPUTATION OF THE FREE-ENERGY
The self-averaged free energy (per unit weight) of our student network, is given by (Engel & Van den Broeck, 2001),
−βf = 1 d 〈〈lnZ〉〉z,W (94)
Here, β = 1/T is the inverse temperature parameter corresponding to our statistical ensemble, d the (teacher) student network width, and Z the partition function of the system defined as (n: number of training examples).
As Gaussian variables (with n, d→∞), in the partition function, to obtain,
〈〈Zr〉〉z,W = r∏ a=1 d∏ µ=1 ∫ dµ (W a) dyµad(y ∗)µe−βNET (ya,y ∗)
× 〈〈 δ ( y∗µ − 1√
d WTx∗µ
) δ ( yµa −
1√ d WTa x µ )〉〉 z,W
= r∏ a=1 d∏ µ=1 ∫ dµ (W a) dyµadŷ µ a 2π dy∗µdŷ∗µ 2π e−βNET (ya,y ∗)eiy ∗µŷ∗µ+iyµa ŷ µ a
× 〈〈 exp ( − i√
d ŷ∗µWTx∗µ − i√ d ŷµaW T a x µ )〉〉 z,W
(95)
where in the last line above, we have expressed the inserted δ functions using their integral representations. To make further progress, we introduce the auxiliary variables,∑
ija W ia∆ijW ∗j = dRa, (96)∑
ij〈a,b〉
W iaΓijW j b = dQab (97)
via the respective δ functions, to arrive at, 〈〈Zn〉〉z,W = ∏ µ,a,b ∫ dµ (Wa) dyµadŷ µ a 2π dy∗µdŷ∗µ 2π e−βNET (ya,y ∗)eiy ∗µŷ∗µ+iyµa ŷ µ a
× ∫ PdQab ∫ PdRa δ ∑ i,j,a W ia∆i,jW ∗j − PRa  δ  ∑ ij〈a,b〉 W iaΓijW j b − PQab  × 〈〈 exp ( − Q0
2 ∑ µ,a (ŷµa ) 2 − 1 2 ∑ µ,〈a,b〉 ŷµa ŷ µ bQab − ∑ µ,a ŷ∗µŷµaRa − 1 2 ∑ µ
(ŷ∗µ)2 )〉〉
W
(98)
Repeating the procedure of expressing the above δ functions using their integral representations, we then get (α = n/d),
〈〈Zn〉〉x,x∗,W = ∫ ∏
a,b
dQ0√ 2π dQ̂0a 4π dQabQ̂ab 2π/d dRaR̂a 2π/d
exp ( iP
2 ∑ a Q0Q̂0a + iP ∑ a<b QabQ̂ab
+ iP ∑ a RaR̂a )∫ ∏ i,a dW ai√ 2π exp ( − i 2 ∑ i,j,a Q̂0aW i aΓijW j a
− i ∑ i,j,a<b Q̂abW i aΓijW j b − i ∑ i,j,a R̂a∆ijW j a ) ×∫ ∏
µ,a
dyµadŷ µ a
2π dy∗µ√ 2π e−βNET (ya,y
∗) exp ( − 1
2 ∑ µ (y∗µ)2 + i ∑ µ,a ŷµa ŷ µ a
− 1 2 ∑ a,µ ( 1−R2a ) (ŷµa ) 2 − 1 2 ∑ µ,〈a,b〉 ŷµa ŷ µ b ( Qab −RaRb ) − i ∑ µ,a y∗µŷµaR a )
(99)
If we now, perform a singular value decomposition of the covariance matrix Γ as, Γ = UTSU = VTV, where S: matrix of singular values of Γ, and we have expressed, V = S1/2U, then one can proceed to write,
〈〈Zn〉〉x,W = 1
det |V | ∫ ∏ a,b dQ0√ 2π dQ̂0a 4π dQabQ̂ab 2π/d dRaR̂a 2π/d exp ( iP 2 ∑ a Q0Q̂0a
+ iP ∑ a<b QabQ̂ab + iP ∑ a RaR̂a )∫ ∏ i,a dW̃ ai√ 2π exp ( − i 2 ∑ i,a Q̂0a ( W̃ ia )2 − i
∑ i,a<b Q̂abW̃ i aW̃ i b − i ∑ i,j,a R̂aW̃ j a ) × ∫ ∏ µ,a dyµadŷ µ a 2π dy∗µ√ 2π e−βNET (ya,y ∗)
exp ( − 1
2 ∑ µ (y∗µ) 2 + i ∑ µ,a ŷµa ŷ µ a − 1 2 ∑ a,µ ( 1−R2a ) (ŷµa ) 2 − i ∑ µ,a y∗µŷµaR a
− 1 2 ∑ µ,〈a,b〉 ŷµa ŷ µ b ( Qab −RaRb ) ) (100)
having expressed, W̃a = VWa, and identifying ∆ = S1/2U from our definitions. Now, since in the above, the W ai integrals factorize in i, and similarly the y µ a , ŷ µ a and dy
∗µ factorize in µ, one can proceed to write:
〈〈Zn〉〉x,W = 1
det |V | ∫ ∏ a,b dQ0dQ̂0a√ 2π4π dQabQ̂ab 2π/d dRaR̂a 2π/d exp ( P [ i 2 ∑ a Q0Q̂0a
+ i ∑ a<b QabQ̂ab + i ∑ a RaR̂a +GS(Q̂0a, Q̂ ab, R̂a) + αGE(Q ab, Ra) ]) (101)
where,
GS(Q̂0a, Q̂ ab, R̂a) = ln ∫ ∏ a dW̃ a√ 2π exp ( − i 2 ∑ a Q̂0aW̃ i aW̃ i a − i ∑ a<b Q̂abW̃aW̃b − i ∑ a R̂aW̃a ) GE(Q ab, Ra) = ln
∫ ∏ a dyadŷa 2π dy∗√ 2π e−βNET (ya,y ∗) exp ( − 1 2 (y∗)2 + i ∑ a ŷaŷa
− 1 2 ∑ a ( 1−R2a ) (ŷa) 2 − 1 2 ∑ 〈a,b〉 ŷaŷb ( Qab −RaRb ) − iy∗µ ∑ a ŷaR a )
(102)
Now, in the limit d→∞, Eq. 101 can be approximated using the saddle-point approach (Bender & Orszag, 2013),
〈〈Zn〉〉x,W ≈ extrQ0,Q̂0a,Qab,Q̂ab,Ra,R̂a exp ( P [ i
2 ∑ a Q0Q̂0a + i ∑ a<b QabQ̂ab
+ i ∑ a RaR̂a +GS(Q̂0a, Q̂ ab, R̂a) + αGE(Q ab, Ra) ]) (103)
where, extr corresponds to extremization of 〈〈Zn〉〉x,W over the respective order parameters. Performing this extremization over Q̂0a, Q̂ab and R̂a, then generates an expression of the form,
〈〈Zn〉〉x,W = extrQ0,Q,R exp { nN ( 1
2 Q−R2 Q0 −Q + 1 2 ln(Q0 −Q)− α 2 ln [1 + β(Q0 −Q)]
− αβ 2 1− 2R+Q 1 + β(Q0 −Q) )} (104)
where we have invoked replica symmetry in the form, Qab = Q and Ra = R, and that ET = (y∗ − y)2/2. Plugging this back into Eq. ??, then finally yields,
βf = −extrQ0,Q,R { 1
2 Q−R2 Q0 −Q + 1 2 ln(Q0 −Q)− α 2 ln [1 + β(Q0 −Q)]
− αβ 2 1− 2R+Q 1 + β(Q0 −Q) } (105) The remaining pair of order parameters generate the following set of transcendental equations on extremization (Bös, 1998):
R = α
a
Q = α
a2 − α
( 1− 2− a
a α ) Q0 = Q+ 1
β (a− 1)
(106)
where, a = max[1, α] for T → 0. Now, the above determined values ofR,Q andQ0 can be perceived as the maximally likely values of R,Q and Q0 of our teacher-student setup, for an inverse temperature β parameterizing the system.

heading: C EXTENDED EXPERIMENTS
text: Figure 4 presents the analytical generalization dynamics for two values of κ and provides comparison between the theory and simulation results of the same model. We observe that the theory and
simulations accurately match. Further experiments are provided in the following anonymous Colab notebook.
Before diving into the theory, we invite the reader to recall a simple equation from thermodynamics. Consider an ideal gas in a container with its large number of molecules moving around, colliding with each other, all while obeying Newton’s laws. While the exact dynamics of each of such molecules is intractable, the system’s macroscopic behavior can be characterized in terms of a handful of scalar quantities, namely, the pressure P , the volume V , and the temperature T . By averaging over suitable probability measures and applying the principle of free-energy minimization, one arrives at a remarkably simple relationship between these three macroscopic variables, i.e., the well-known PV = nRT (n: number of moles of gas, R: gas constant) (Reif, 2009).

Review summaries:
An influential line of work has revealed that deep neural networks can exhibit non-monotonic behavior in their generalization error (double descent) as a function of model size, dataset size, and training time. Recent (and old) theoretical work has demonstrated that even simple models like linear regression exhibit the same non-monotonic behavior as a function of model and dataset size. The authors of this work build on this literature by demonstrating that linear teacher-student models trained with gradient descent can exhibit double descent as  a function of training time. Using replica theory, they derive a closed-form expression for the generalization error of this model as a function of training time. They show that double descent can arise when the student is trained on anisotropic data which includes a set of high SNR features and a set of low SNR features. Finally, the authors demonstrate a qualitative match between the generalization error behavior of linear teacher-student regression and a ResNet18 trained on CIFAR10 as a function of training time and regularization strength.

This work studied epoch-wise double descent using linear model as a proxy. Basically, authors proposed a linear teacher-student models to established analysis and used random matrix theory (rmt) to interpret the learning dynamics. Some simulation on the well-posed (n>p) linear regression problems backups the theory. Some real-world experiments based on ResNet-18 and noisy labels also demonstrate the relevance of proposed theorem.

This paper theoretically analyzes a student-teacher setting for which epoch-wise double descent occurs. (Epoch-wise double descent, previously discussed in the literature, refers to the non-monotonicity of the population error as a function of training time.) More specifically, the setup is: (a) a linear teacher with Gaussian noise; (b) a linear student whose inputs are a linear transformation F ("modulation matrix") applied to the teacher's inputs. The inputs are d-dimensional and there are n training samples. Exact analytic expressions for the test error can be derived by utilizing the replica method in the limit of n, d -> \infty. The case analyzed in most depth is a setting in which F has two scales for its singular values (with high degeneracy) -- c.f. Assumption 1. Both the theory and finite-size simulations indeed exhibit epoch-wise double descent depending on the choice of setting parameters (e.g. strength of regularization and condition number of F). The intuition behind this is that when the scales are well-separated and learnable (e.g. regularization not too strong), overfitting of the faster features occurs (leading to a rise in the test error) before learning of the slower features (which subsequently decreases the test error).