Multi-scale Feature Learning Dynamics: Insights for Double Descent

1 INTRODUCTION . Classical wisdom in statistical learning theory predicts a trade-off between the generalization ability of a machine learning model and its complexity , with highly complex models less likely to generalize well ( Friedman et al. , 2001 ) . If the number of parameters measures complexity , deep learning models sometimes go against this prediction ( Zhang et al. , 2016 ) : deep neural networks trained by stochastic gradient descent exhibit a so-called double descent behavior ( Belkin et al. , 2019b ) with increasing model parameters . Specifically , with increasing complexity , the generalization error first obeys the classical U-shaped curve consistent with statistical learning theory . However , a second regime emerges as the number of parameters is further increased past a transition threshold where generalization error drops again , hence the “ double descent ” or more accurately model-wise double descent ( Nakkiran et al. , 2019 ) . Nakkiran et al . ( 2019 ) showed that the phenomenon of double descent is not limited to varying model size but is also observed as a function of training time or epochs . In this case as well , the so-called epoch-wise double descent is in apparent contradiction with the classical understanding of over-fitting ( Vapnik , 1998 ) , where one expects that longer training of a sufficiently large model beyond a certain threshold should result in over-fitting . This has important implications for practitioners and raises questions about one of the most widely used regularization method in deep learning ( Goodfellow et al. , 2016 ) : early stopping . Indeed , while one might expect early stopping to prevent over-fitting , it might in fact prevent models from being trained at their fullest potential . Since the 1990s , there has been much interest in understanding the origins of non-trivial generalization behaviors of neural networks ( Opper , 1995 ; Opper & Kinzel , 1996 ) . The authors of Krogh & Hertz ( 1992b ) were among the first to provide theoretical explanations for ( model-wise ) double descent in linear models . Summarily , at intermediate levels of complexity , where the model size is equal to the number of training examples , the model is very sensitive to noise in training data and hence , generalizes poorly . This sensitivity to noise reduces if the model complexity is either de- creased or increased . More recently , the double descent phenomena has been also studied for more complex models such as two-layer neural networks and random feature models ( Ba et al. , 2019 ; Mei & Montanari , 2019 ; D ’ Ascoli et al. , 2020 ; Gerace et al. , 2020 ) . The majority of previous work in this direction focuses on understanding the asymptotic behavior of model performance , i.e. , where training time t → ∞ . In recent years , there has been an interest in studying the non-asymptotic ( finite training time ) performance , suggesting that several intriguing properties of neural networks can be attributed to different features being learned at different scales . Among the limited work studying the particular epoch-wise double descent , Nakkiran et al . ( 2019 ) introduces the notion of effective model complexity and hypothesizes that it increases with training time and hence unifies both model-wise and epoch-wise double descent . Through a combination of theory and empirical results , Heckel & Yilmaz ( 2020 ) find that the dynamics of evolution of single and two layer networks under gradient descent , can be perceived to be the superposition of two bias/variance curves with different minima times , thus leading to non-monotonic test error curves . In this work , we build on Bös et al . ( 1993 ) ; Bös ( 1998 ) ; Advani & Saxe ( 2017 ) ; Mei & Montanari ( 2019 ) which analyze model-wise double descent through the lens of linear models , to probe the origins of epoch-wise double descent . Particularly , • We introduce a linear teacher-student model which , despite its simplicity , exhibits some of intriguing properties of generalization dynamics in deep neural networks . ( Section 2.1 ) • In the limit of high dimensions , we leverage the replica method developed in statistical physics to derive closed-form expressions for the generalization dynamics of our teacherstudent setup , as a function of training time and regularization strength . ( Section 2.2 ) • Consistent with recent findings , we provide an explanation for the existence of epoch-wise double descent through the lens of multi-scale feature learning . ( Figure 1 ) • We perform simulation experiments to validate our analytical predictions . We also conduct experiments with deep networks , showing that our teacher-student setup exhibits generalization behavior which is qualitatively similar to that of deep networks . ( Figure 2 ) 2 ANALYTICAL RESULTS . Stochastic Gradient Descent ( SGD ) — the de facto optimization algorithm for neural networks — exhibits complex dynamics arising from a large number of parameters ( Kunin et al. , 2020 ) . However , it is possible to describe some aspects of the high-dimensional microscopic dynamics of neural networks in terms of low-dimensional understandable macroscopic entities . In a series of seminal papers by Gardner ( Gardner , 1988 ; Gardner & Derrida , 1988 ; 1989 ) , the replica method of statistical physics was adopted to derive expressions describing the generalization behavior of large linear models trained using SGD . In this paper , we employ Gardner ’ s analysis to build upon an established line of work studying linear and generalized linear models ( Seung et al. , 1992 ; Kabashima et al. , 2009 ; Krzakala et al. , 2012 ) . While most of previous work study the asymptotic ( t → ∞ ) generalization behavior , we adapt these methods to study transient learning dynamics of generalization for finite training time . In the following , we first introduce a teacher-student model that exhibits interesting characteristics of modern neural networks . We then adapt the replica method to study the generalization performance as a function of training time and amount of regularization . 2.1 A TEACHER-STUDENT SETUP . Teacher : We study a supervised linear regression problem in which the training labels y , are generated by a noisy linear model ( Figure 1 ) , y : = y∗ + , y∗ : = zTW , zi ∼ N ( 0 , 1√ d ) , ( 1 ) where z ∈ Rd is the teacher ’ s input and y∗ , y ∈ R are the teacher ’ s noiseless and noisy outputs , respectively . W ∈ Rd represents the ( fixed ) weights of the teacher and ∈ R is the noise . Both W and are drawn i.i.d . from Gaussian distributions with zero means and variances of 1 and σ2 , respectively . Student : A student model is correspondingly chosen to be a similar shallow network with trainable weights Ŵ ∈ Rd . The student model is trained on n training pairs { ( xµ , yµ ) } nµ=1 , with the labels yµ being generated by the above teacher network , as , ŷ : = xT Ŵ , s.t . x : = FTz , ( 2 ) where the matrix F ∈ Rd×d is a predefined and fixed modulation matrix regulating the student ’ s access to the true input z . One can think of z as the latent factors of variation on which the teacher operates , while x can be thought as the pixels that the student learns from . Learning paradigm : To train our student network , we use stochastic gradient descent ( SGD ) on the regularized mean squared loss , evaluated on the n training examples as , LT : = 1 2n n∑ µ=1 ( yµ − ŷµ ) 2 + λ 2 ||Ŵ ||22 ( 3 ) where λ ∈ [ 0 , ∞ ) is the regularization coefficient . Optimizing Eq . 3 with stochastic gradient descent ( SGD ) yields the typical update rule , Ŵt ← Ŵt−1 − η∇ŴLT + ξ , ( 4 ) in which t denotes the training step and η is the learning rate . Additionally , ξ ∼ N ( 0 , σ2ξ ) models the stochasticity noise of the optimization algorithm ( Bottou et al. , 1991 ) . Macroscopic variables : The quantity of interest in this work , is the expected generalization error of the student , determined by averaging the student ’ s error over all possible input-target pairs and noise realizations , as , LG : = 1 2 Ez [ ( y∗ − ŷ ) 2 ] . ( 5 ) As shown in Bös et al . ( 1993 ) , if n , d→∞ with a constant ratio nd < ∞ , Eq . 5 can be written as a function of two macroscopic scalar variables R , Q ∈ R , LG = 1 2 ( 1 +Q− 2R ) , ( 6 ) where σ2 is the variance of the teacher ’ s output noise and , R : = 1 d WTFŴ , Q : = 1 d ŴTFTFŴ , ( 7 ) See App . B.1 for the proof . Both R and Q have clear interpretations ; R is the dot-product between the teacher ’ s weights W and the student ’ s modulated weights FŴ , hence can be interpreted as the alignment between the teacher and the student . Similarly , Q can be interpreted as the student ’ s modulated norm . The negative sign of R in Eq . 6 suggests that the larger R is , the smaller the generalization error gets . At the same time , Q appears with a positive sign suggesting the students with smaller ( modulated ) norm generalize better . As a remark , note that bothR andQ are functions of Ŵ which itself is a function of training iteration t and the regularization strength λ . Therefore , hereafter , we denote the above quantities as LG ( t , λ ) , R ( t , λ ) , and Q ( t , λ ) . 2.2 MAIN RESULTS . In this Section , we present our main analytical results , with Section 2.3 containing a sketch of our derivations . For brevity of the results , here , we only present the results for σ2 = λ = 0 . See App . B for the general case and the detailed proofs . General matrix F . Let Z : = [ zµ ] nµ=1 ∈ Rn×d and X : = [ xµ ] nµ=1 ∈ Rn×d denote the input matrices for the teacher and student such that X : = ZF . For a general modulation matrix F , the input covariance matrix has the following singular value decomposition ( SVD ) , XTX = FTZTZF = V ΛV T , ( 8 ) in which the diagonal matrix Λ contains the eigenvalues of the student ’ s input covariance matrix . Solving the dynamics of gradient descent as in Eq . 4 , we arrive at the following exact analytical expressions for R ( t ) and Q ( t ) , R ( t ) = 1 d Tr ( D ) where , D : = ( I − [ I − ηΛ ] t ) , ( 9 ) Q ( t ) = 1 d Tr ( ATA ) where , A : = FV DV TF−1 , ( 10 ) in which Tr ( . ) is the trace operator . See App . B.2 the proof . Remark : The solution in Eqs . 9 and 10 are exact , however , they require the empirical computation of the eigenvalues Λ . Below , we treat a special case of the dynamics that allow us to derive approximate solutions that do not explicitly depend on Λ . Special case : Fast and slow features . We now study a case where the modulation matrix F has a specific structure described in Assumption 1 . Assumption 1 . The modulation matrix , F , under a SVD , F : = UΣV T has two sets of singular values such that the first p singular values are equal to σ1 and the remaining d− p singular values are equal to σ2 . We let the condition number of F to be denoted by κ : = σ1σ2 > 1 . By employing the replica method of statistical physics ( Gardner , 1988 ; Gardner & Derrida , 1988 ) , we now derive approximate expressions for R ( t ) and Q ( t ) . To begin with , we first define the following auxiliary variables , α1 : = n p , α2 : = n d− p , λ̃1 : = d p 1 ησ21t , λ̃2 : = d d− p 1 ησ22t , ( 11 ) and also let , ai = 1 + 2λ̃i ( 1− αi − λ̃i ) + √ ( 1− αi − λ̃i ) 2 + 4λ̃i , for i ∈ { 1 , 2 } . ( 12 ) The closed-from scalar expression for R ( t ) is then given by , R ( t ) = R1 +R2 , where , R1 : = n a1d , and , R2 : = n a2d ( 13 ) For Q ( t ) , we accordingly define two more auxiliary variables , bi = αi a2i − αi , ci = 1− 2Ri − n d 2− ai ai for i ∈ { 1 , 2 } , ( 14 ) with which the closed-from scalar expression for Q ( t ) reads , Q ( t ) = Q1 +Q2 , where , Q1 : = b1b2c2 + b1c1 1− b1b2 , and , Q2 : = b1b2c1 + b2c2 1− b1b2 . ( 15 ) By plugging Eqs . 13 and 15 into Eq . 6 , one obtains a closed-form expression for LG ( t ) as a function of the training time . See App . B.3 for the proof . R = R1 + R2 R1 R2 t SGD steps tth step on argmin L ( Ŵ ) < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > L̃ ( Ŵ , t ) < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > L̃ ( Ŵ , t ) < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > L̃ ( Ŵ , t ) < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > L ( Ŵ ) < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > L̃ ( Ŵ , t ) < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > SGD steps tth step on argmin L ( Ŵ ) < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > L̃ ( Ŵ , t ) < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > L̃ ( Ŵ , t ) < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > L̃ ( Ŵ , t ) < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > L ( Ŵ ) < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > < latexit sha1_base64= '' KrE7GKXcbjd4i3q6mbD83colcc4= '' > AAAB/XicbVDLSsNAFJ34rPUVHzs3g0Wom5KIoMuiGxcuKtgHNKHcTKft0MkkzEyEGoK/4saFIm79D3f+jZM2C209MHA4517umRPEnCntON/W0vLK6tp6aaO8ubW9s2vv7bdUlEhCmyTikewEoChngjY105x2YkkhDDhtB+Pr3G8/UKlYJO71JKZ+CEPBBoyANlLPPvRC0CMCPL3Nqt4IdNrOTnt2xak5U+BF4hakggo0evaX149IElKhCQeluq4Taz8FqRnhNCt7iaIxkDEMaddQASFVfjpNn+ETo/TxIJLmCY2n6u+NFEKlJmFgJvOsat7Lxf+8bqIHl37KRJxoKsjs0CDhWEc4rwL3maRE84khQCQzWTEZgQSiTWFlU4I7/+VF0jqruU7NvTuv1K+KOkroCB2jKnLRBaqjG9RATUTQI3pGr+jNerJerHfrYza6ZBU7B+gPrM8fXfGVJg== < /latexit > L̃ ( Ŵ , t ) < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > < latexit sha1_base64= '' 32x2N2VoNbVPu4FGsSV5PhoASno= '' > AAACCHicbVDLSsNAFJ3UV62vqEsXDhahgpREBF0W3bhwUcE+oAllMpm2QycPZm6EErJ046+4caGIWz/BnX/jpM1CWw9cOJxzL/fe48WCK7Csb6O0tLyyulZer2xsbm3vmLt7bRUlkrIWjUQkux5RTPCQtYCDYN1YMhJ4gnW88XXudx6YVDwK72ESMzcgw5APOCWgpb556AAXPkudgMCIEpHeZlnNGRFIO9kphpO+WbXq1hR4kdgFqaICzb755fgRTQIWAhVEqZ5txeCmRAKngmUVJ1EsJnRMhqynaUgCptx0+kiGj7Xi40EkdYWAp+rviZQESk0CT3fm96p5Lxf/83oJDC7dlIdxAiyks0WDRGCIcJ4K9rlkFMREE0Il17diOiKSUNDZVXQI9vzLi6R9Vretun13Xm1cFXGU0QE6QjVkowvUQDeoiVqIokf0jF7Rm/FkvBjvxsestWQUM/voD4zPH7G3mbo= < /latexit > Remark : Eq . 11 indicates that the singular values of F , are directly multiplied by t. That implies that the learning speed of each feature is scaled by the magnitude of its corresponding singular value . As an illustration , the figure on the right shows the evolution of R1 , R2 , and R = R1 + R2 for a case where p = d/2 , σ1 = 1 , and σ1 = 0.01 implying a condition number of κ = 100 .

Summary: An influential line of work has revealed that deep neural networks can exhibit non-monotonic behavior in their generalization error (double descent) as a function of model size, dataset size, and training time. Recent (and old) theoretical work has demonstrated that even simple models like linear regression exhibit the same non-monotonic behavior as a function of model and dataset size. The authors of this work build on this literature by demonstrating that linear teacher-student models trained with gradient descent can exhibit double descent as  a function of training time. Using replica theory, they derive a closed-form expression for the generalization error of this model as a function of training time. They show that double descent can arise when the student is trained on anisotropic data which includes a set of high SNR features and a set of low SNR features. Finally, the authors demonstrate a qualitative match between the generalization error behavior of linear teacher-student regression and a ResNet18 trained on CIFAR10 as a function of training time and regularization strength.