Principal Components Bias in Deep Neural Networks

Recent work suggests that convolutional neural networks of different architectures1 learn to classify images in the same order . To understand this phenomenon , we2 revisit the over-parametrized deep linear network model . Our asymptotic analysis,3 assuming that the hidden layers are wide enough , reveals that the convergence rate4 of this model ’ s parameters is exponentially faster along directions corresponding5 to the larger principal components of the data , at a rate governed by the singular6 values . We term this convergence pattern the Principal Components bias ( PC-bias ) .7 We show how the PC-bias streamlines the order of learning of both linear and non-8 linear networks , more prominently at earlier stages of learning . We then compare9 our results to the spectral bias , showing that both biases can be seen independently,10 and affect the order of learning in different ways . Finally , we discuss how the11 PC-bias may explain some benefits of early stopping and its connection to PCA,12 and why deep networks converge more slowly when given random labels.13 1 Introduction14 The dynamics of learning in deep neural networks is an intriguing subject , not yet sufficiently15 understood . Diverse empirical data seems to support the hypothesis that neural networks start by16 learning a simple model , which then gains complexity as learning proceeds ( Gunasekar et al. , 2018 ; 17 Soudry et al. , 2018 ; Hu et al. , 2020 ; Nakkiran et al. , 2019 ; Gissin et al. , 2019 ; Heckel & Soltanolkotabi,18 2019 ; Ulyanov et al. , 2018 ; Valle-Perez et al. , 2018 ) . This phenomenon is sometimes called simplicity19 bias ( Dingle et al. , 2018 ; Shah et al. , 2020 ) .20 Recent work additionally shows that neural networks learn the training examples of natural datasets21 in a consistent order , and further impose a consistent order on the test set ( Hacohen et al. , 2020 ; 22 Pliushch et al. , 2021 ) . Below we call this effect Learning Order Constancy ( LOC ) . Currently , the23 characteristics of visual data , which may explain this consistently imposed order , remain unclear.24 Surprisingly , this universal order persists despite the variability introduced into the training of different25 models and architectures.26 To understand this phenomenon , we start by analyzing the deep linear network model ( Saxe et al.,27 2013 , 2019 ) , defined by the concatenation of linear operators . While not a universal approximator , it28 is nevertheless trained by minimizing a non-convex objective function with a multitude of minima.29 The investigation of such networks is often employed to shed light on the learning dynamics when30 complex geometric landscapes are explored by GD ( Fukumizu , 1998 ; Arora et al. , 2018 ) .31 In Section 2 , we prove that the convergence of the weights of deep linear networks is governed32 by the eigendecomposition of the raw data in a phenomenon we term PC-bias . These asymptotic33 results , valid when the hidden layers are wide enough , can be seen as an extension of the known34 behavior of the single-layer convex linear model ( Le Cun et al. , 1991 ) . Our work is closely related to35 ( Saxe et al. , 2013 , 2019 ) , where the deep linear model ’ s dynamics is analyzed as a function of the36 input and input-output statistics . Importantly , the analysis in ( Saxe et al. , 2013 , 2019 ; Arora et al.,37 Submitted to 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Do not distribute . 2018 ) incorporates the simplifying assumption that the data ’ s singular values are identical ( whitened38 data ) , an assumption which unfortunately obscures the main result of our analysis – the direct39 dependence of convergence rate on the singular values of the data.40 In Section 3 , we empirically show that this pattern of convergence is indeed observed in deep linear41 networks , validating the plausibility of our assumptions . We continue by showing that the LOC-effect42 in deep linear network is determined solely by their PC-bias . We prove a similar ( weaker ) result for43 the non-linear two-layer ReLU model introduced by Allen-Zhu et al . ( 2018 ) , where this model is44 presented as a certain extension of NTK ( Jacot et al. , 2020 ) . In this framework , convergence is fastest45 along the largest kernel ’ s principal components , a result related to the Spectral bias discussed below.46 In Section 4 , we extend the study empirically to non-linear networks , and investigate the relation47 between the PC-bias and the LOC-effect in general deep networks . We first show that the order48 by which examples are learned by linear networks is highly correlated with the order induced by49 prevalent deep CNN models . We then show directly that the learning order of non-linear CNN models50 is affected by the principal decomposition of the data . Moreover , the LOC-effect diminishes when51 data is whitened , indicating a tight connection between the PC-bias and the LOC-effect.52 Our results are reminiscent of another phenomenon , termed Spectral bias ( Rahaman et al. , 2019 ; 53 Cao et al. , 2019 ) , which associates the learning dynamics of neural networks with the Fourier54 decomposition of functions in the hypothesis space . Rahaman et al . ( 2019 ) empirically demonstrated55 that the complexity of classifiers learned by ReLU networks increases with time . Basri et al . ( 2019,56 2020 ) showed theoretically , by way of analyzing elementary neural network models , that these models57 first fit the data with low-frequency functions , and gradually add higher frequencies to improve the fit.58 Nevertheless , the spectral bias and PC-bias are inherently different . Indeed , the eigendecomposition59 of raw images is closely related to the Fourier analysis of images as long as the statistical properties60 of images are ( approximately ) translation-invariant ( Simoncelli & Olshausen , 2001 ; Torralba & Oliva,61 2003 ) . Still , the PC-bias is guided by spectral properties of the raw data and is additionally blind to62 class labels . On the other hand , the spectral bias , as well as the related frequency bias that has been63 shown to characterize NTK models ( Basri et al. , 2020 ) , are all guided by spectral properties of the64 learned hypothesis , which strongly depends on label assignment.65 In Section 4.3 we investigate the relation between the PC-bias , spectral bias , and the LOC-effect.66 We find that the LOC-effect is very robust : ( i ) when we neutralize the spectral bias by using low67 complexity models such as deep linear networks , the effect is still observed ; ( ii ) when we neutralize68 the PC-bias by using whitened data , the LOC-effect persists . We hypothesize that at the beginning of69 learning , the learning dynamics of neural models is controlled by the eigendecomposition of the raw70 data . As learning proceeds , control of the dynamics slowly shifts to other factors.71 The PC-bias has implications beyond the LOC-effect , as expanded in Section 5 and Suppl . §A:72 1 . Early stopping . It is often observed that when training deep networks with real data , the highest73 generalization accuracy is obtained before convergence . Consequently , early stopping is often74 prescribed to improve generalization . Following the commonly used assumption that in natural75 images the lowest principal components correspond to noise ( Torralba & Oliva , 2003 ) , our results76 predict the benefits of early stopping , and relate it to PCA . In Section 5 we investigate the relevance77 of this conclusion to real non-linear networks ( see , e.g. , Basri et al . ( 2019 ) ; Li et al . ( 2020 ) for78 complementary accounts ) .79 2 . Slower convergence with random labels . Zhang et al . ( 2016 ) showed that neural networks80 can learn any label assignment . However , training with random label assignments is known to81 converge slower as compared to training with the original labels ( Krueger et al. , 2017 ) . We report a82 similar phenomenon when training deep linear networks . Our analysis shows that when the principal83 eigenvectors are correlated with class identity , as is often the case in natural images , the loss decreases84 faster when given true label assignments as against random label assignments . In Section 5 we85 investigate this hypothesis empirically in linear and non-linear networks.86 3 . Weight initialization . Different weight initialization schemes have been proposed to stabilize the87 learning and minimize the hazard of `` exploding gradients '' ( e.g. , Glorot & Bengio , 2010 ; He et al.,88 2015 ) . Our analysis ( see Suppl . §A ) identifies a related variant , which eliminates the hazard when89 all the hidden layers are roughly of equal width . In the deep linear model , it can be proven that the90 proposed normalization variant in a sense minimizes repeated gradient amplification.91 2 Theoretical analysis92 Notations . Let X = { ( xi , yi ) } ni=1 denote the training data , where x ∈ Rq denotes the i-th data93 point and y ∈ { 0 , 1 } K its corresponding label . Let 1nimi denote the centroid ( mean ) of class i with94 ni points , and M = [ m1 . . .mK ] > . Finally , let X and Y denote the matrices whose ith column95 is xi and yi respectively . ΣXX = XX > and ΣY X = Y X > denote the covariance matrix of X96 and cross-covariance of X and Y respectively . We note that ΣXX captures the structure of the data97 irrespective of class identity.98 Definition 1 ( Principal coordinate system ) . The coordinate system obtained by rotating the data in Rq99 by an orthonormal matrixU > , where SV D ( ΣXX ) =UDU > . Now ΣXX =D , a diagnoal matrix whose100 elements are the singular values of XX > , arranged in decreasing order d1 ≥ d2 ≥ . . . ≥ dq ≥ 0.101 Definition 2 ( Compact representation ) . Let f ( x ) denote a deep linear network . Then f ( x ) =102 ( ∏1 l=LWl ) x = Wx , where W ∈ RK×q is called the compact representation of the network.103 Definition 3 ( Error matrix ) . For a deep linear network whose compact representation is W , the104 error matrix is Er = WΣXX − ΣY X . In the principal coordinate system , Er = WD −M .105 Assumptions . Our analysis assumes that the learning rate µ is infinitesimal , and therefore terms106 of size O ( µ2 ) can be neglected . We further assume that the width of the hidden layers lies in107 [ m , m+Mb ] , wherem→∞ denotes a very large number and Mb is fixed . Thus terms of size O ( 1m ) 108 can also be neglected . In Fig . 1 we show the plausibility of these assumptions , where the predicted109 dynamics is seen throughout the training of deep linear networks , even for small values ofm.110 2.1 The dynamics of deep over-parametrized linear networks111 Consider a deep linear network with L layers , and let112 L ( X ) = 1 2 ‖WX − Y ‖2F W : = 1∏ l=L Wl , Wl ∈ Rml×ml−1 ( 1 ) Above ml denotes the number of neurons in layer l , where m0 = q and mL = K.113 Theorem 1 . In each time point s , the compact matrix representation W obeys the following dynamics,114 when using the notation Ers defined in Def . 3:115 W s+1 = W s − µ L∑ l=1 Asl · Ers ·Bsl +O ( µ2 ) ( 2 ) Above µ denotes the learning rate . Asl and B s l are called gradient scale matrices , and are defined as116 Asl : = ( l+1∏ j=L W sj ) ( l+1∏ j=L W sj ) > ∈ RK×K Bsl : = ( 1∏ j=l−1 W sj ) > ( 1∏ j=l−1 W sj ) ∈ Rq×q ( 3 ) The proof can be found in Suppl . §B.117 Gradient scale matrices . Some statistical properties of such matrices are established in Suppl . §A.118 Note that when the number of hidden layers is 0 ( L = 1 ) , both gradient scale matrices reduce to the119 identity matrix and the dynamics in ( 2 ) is reduced to the following known result ( e.g. , Le Cun et al.,120 1991 ) : W s+1 = W s−µErs . Recall , however , that the focus of this paper is the over-parameterized121 linear model with L > 1 , in which the loss is not convex . Since the difference between the convex122 linear model and the over-parametrized deep model boils down to these matrices , our convergence123 analysis henceforth focuses on the dynamics of the gradient scale matrices.124 In accordance , we analyze the evolution of the gradient scale matrices as learning proceeds . Let125 m = min ( m1 , ... , mL−1 ) denote the size of the smallest hidden layer . Initially for s = 0 , all weight126 matrices W 0l are assumed to be initialized by sampling from a distribution with mean 0 and variance127 σ2l = O ( 1 m ) . The specific normalization factor , alluded to in O ( 1 m ) , is a variant of the Glorot128 initialization . Details and justification can be found in Suppl . §A.1.129 At time s , letAsl ( m ) andB s l ( m ) denote a sequence of random gradient scale matrices , corresponding130 to networks whose smallest hidden layer hasm neurons . From Suppl . §A we deduce that:131 Theorem 2 . Using p−→ to denote convergence in probability asm→∞ , and ∀s , l:132 Bsl ( m ) p−→ I , var [ Bl ( m ) ] = O ( 1 m ) Asl ( m ) p−→ I , var [ Al ( m ) ] = O ( 1 m ) Proof . Proof by induction on s. Initially when s = 0 , the claim follows from Thm 4 and Corr 5.1.133 The induction step validity follows from Thm 6 and Thm 7 ( see Suppl . §A.2 ) .134 The detailed proof shows that the relevant constants are amplified with s. While they remain moderate135 andm is sufficiently large , Bsl ( m ) ≈ I and Asl ( m ) ≈ I ∀l . In this case , the dynamics of the over-136 parameterized model is identical to the dynamics of the convex linear model , W s+1 = W s − µErs.137 Convergence rate . In §A.2 we show that the convergence of Bsl ( m ) to I is governed to some extent138 by O ( K m ) , while the convergence of Asl ( m ) is governed by O ( q m ) . Recall that while m → ∞,139 q is the dimension of the data space which is fixed in advance and can be fairly large , while K is140 the number of classes which is fixed and quite small . Typically , K q . Thus we expect the right141 gradient scale matrices Bsl ( m ) to remain approximately I much longer than the left matrices A s l ( m ) .142 Empirical validation . Since the results above are asymptotic , and to envision the difference between143 convergence governed by O ( K m ) vs. O ( q m ) , we resort to simulations whose results are shown in144 Fig . 1 . These empirical results , recounting linear networks with 4 hidden layers of width 1024 , clearly145 show that during a significant part of the training both gradient scale matrices remain approximately146 I . The difference between the convergence rate of Bsl and A s l is seen later on , when ∆A s l starts to147 increase shortly before convergence , while ∆Bsl remains essentially 0 throughout.148 2.2 Weight evolution149 K q entails that Bsl ( m ) remains approximately equal to I much longer than Asl ( m ) . This is150 substantiated by the simulation results in Fig . 1 . Consequently , while earlier on it is safe to assume151 that both Asl ≈ I and Bsl ≈ I , as learning proceeds only Bsl ≈ I is safe to assume.152 With this in mind , we obtain expressions for the evolution of W s separately for earlier and later in153 learning . We first shift to the principal coordinate system defined in Def 1 . In this system we can154 analyze each column of W s separately , where wsj and mj denote the respective columns of W s and155 M . At the beginning of learning when both Asl ≈ I and Bsl ≈ I ( see §B.3 for a detailed derivation ) :156 ws+1j = ( λj ) sw0j + [ 1− ( λj ) s ] mj dj λj = 1− µdjL ( 4 ) 157 Eq . 4 is reminiscent of the well understood dynamics of training the convex one layer linear model . It158 is composed of two additive terms , revealing two parallel and independent processes:159 1 . The dependence on random initialization tends to 0 exponentially with decline rate λj .160 2 . The final value is the sum of a geometrical series with a common ratio λj .161 In either case , convergence is fastest for the largest singular eigenvalue , or the first column of W ,162 and slowest for the smallest singular value . This behavior is visualized in Fig . 2a . Importantly , the163 rate of convergence depends on the singular value dj , the number of layers L , and the learning rate µ.164 In later stages of learning , when we can only assume that Bsl ≈ I , the dynamic becomes:165 ws+1j = s∏ ν=1 ( I − µdjAν ) w0j + µ [ s∑ ν=1 s∏ ρ=ν+1 ( I − µdjAρ ) Aν ] mj ( 5 ) where As = ∑L l=1A s l . The proof is provided in §B.3 . Although the dynamics now depends on166 matrices As as well , it is still the case that the convergence of each column is governed by its singular167 value dj . This suggests that while the PC-bias is more pronounced in earlier stages of learning , its168 effect persists throughout.169 The analysis above is extended to a simple non-linear ReLU model ( cf . Arora et al. , 2019 ) as detailed170 in §B.2 , with qualitatively similar results ( albeit under unrealistic assumptions ) . Empirical results,171 shown in Fig . 2b , indicate that the results are indicative beyond the assumed circumstances.172 3 PC-bias : empirical study173 In this section , we first analyze deep linear networks , showing that the convergence rate is indeed174 governed by the principal singular values of the data , which demonstrates the plausibility of the175 assumptions made in Section 2 . We continue by extending the scope of the investigation to non-linear176 neural networks , finding there evidence for the PC-bias mostly in the earlier stages of learning.177 3.1 Methodology178 We say that a linear network is L-layered when it has L − 1 hidden fully connected ( FC ) layers179 ( without convolutional layers ) . In our empirical study we relaxed some assumptions of the theoretical180 study , in order to increase the resemblance of the trained networks to networks in common use.181 Specifically , we changed the initialization to the commonly used Glorot initialization , replaced the182 L2 loss with the cross-entropy loss , and employed SGD instead of the deterministic GD . Notably,183 the original assumptions yielded similar results . The results presented summarize experiments with184 networks of equal width across all hidden layers , specifically the moderate value of m = 1024,185 keeping in mind that we test the relevance of asymptotic results form→∞ . Using a different width186 for each layer yielded similar qualitative results . Details regarding the hyper-parameters , architectures,187 and datasets can be found in §D.1 , §D.3 and §D.4 respectively.188 3.2 PC-bias in deep linear networks189 In this section , we train L-layered linear networks , then compute their compact representations190 W rotated to align with the canonical coordinate system ( Def . 1 ) . Note that each row wr in W191 essentially defines the one-vs-all separating hyper-plane corresponding to class r.192 To examine both the variability between models and their convergence rate , we inspect wr at different193 time points during learning . The rate of convergence can be measured directly , by observing the194 changes in the weights of each element in wr . These weight values1 should be compared with195 the optimal values in each row wr of Wopt = Y XT ( XXT ) . The variability between models is196 measured by calculating the standard deviation ( std ) of each wr across N models.197 We begin with linear networks . We trained 10 5-layered FC linear networks , and 10 linear st-VGG198 convolutional networks . When analyzing the compact representation of such networks we observe199 similar behavior – weights corresponding to larger principal components converge faster to the200 optimal value , and their variability across models converges faster to 0 ( Figs . 3a,3b ) . Thus , while the201 theoretical results are asymptotic , PC-bias is empirically seen throughout the entire learning process202 of deep linear networks.203 Whitened data . The PC-bias is neutralized when the data is whitened , at which point ΣXX is the204 scaled identity matrix . In Fig . 3c , we plot the results of the same experimental protocol while using a205 ZCA-whitened dataset . As predicted , the networks no longer show any bias towards any principal206 direction . Weights in all directions are scaled similarly , and the std over all models is the same in207 each epoch , irrespective of the principal direction . ( Additional experiments show that this is not an208 artifact of the lack of uniqueness when deriving the principal components of a white signal ) .209 1We note that the weights tend to start larger for smaller principal components , as can be seen in Fig . 3a left . 3.3 PC-bias in general CNNs210 In this section , we investigate the manifestation of the PC-bias in non-linear deep convolutional211 networks . As we can not directly track the learning dynamics separately in each principal direction of212 non-linear networks , we adopt two different evaluation mechanisms:213 Linear approximation . We considered several linear approximations , but since all of them showed214 the same qualitative behavior , we report results with the simplest one . Specifically , to obtain a linear215 approximation of a non-linear network , without max-pooling or batch-normalization layers , we216 follow the definition of the compact representation from Section 2 while ignoring any non-linear217 activation . We then align this matrix with the canonical coordinate system ( Def . 1 ) , and observe the218 evolution of the weights and their std across models along the principal directions during learning.219 Note that now the networks do not converge to the same compact representation , which is not unique.220 Nevertheless , we see that the PC-bias governs the weight dynamics to a noticeable extent.221 More specifically , in these networks a large fraction of the lowest principal components hardly changes222 during learning , as good as being ignored . Nevertheless , the PC-bias affects the higher principal223 components , most notably at the beginning of training ( see Fig . 3d ) . Thus weights corresponding to224 higher principal components converge faster , and the std across models of such weights decreases225 faster for higher principal components.226 Projection to higher PC ’ s . We created a modified test-set , by project-227 ing each test example on the span of the first P principal components.228 This is equivalent to reducing the dimensionality of the test set to P us-229 ing PCA . We trained an ensemble of N=100 st-VGG networks on the230 original small mammals training set , then evaluated these networks dur-231 ing training on 4 versions of the test-set , reduced to P=1,10,100,1000232 dimensions respectively . Mean accuracy is plotted in Fig . 4 . Similar233 results are obtained when training VGG-19 networks on CIFAR-10,234 see §C.3.235 Taking a closer look at Fig . 4 , we see that when evaluated on lower236 dimensionality test-data ( P=1,10 ) , the networks ’ accuracy peaks after237 a few epochs , at which point performance starts to decrease . This result suggests that the networks238 rely more heavily on these dimensions in the earlier phases of learning , and then continue to learn239 other things . In contrast , when evaluated on higher dimensionality test-data ( P=100,1000 ) , accuracy240 continues to rise , longer so for larger P . This suggests that significant learning of the additional241 dimensions continues in later stages of the learning.242 4 PC-bias : Learning Order Constancy243 In this section , we show that the PC-bias is significantly correlated with the learning order of deep244 neural networks , and can therefore partially account for the LOC-effect described in Section 1.245 Following Hacohen et al . ( 2020 ) , we measure the `` speed of learning '' of each example by computing246 its accessibility score . This score is given per example , and characterizes how fast an ensemble of247 N networks learns it . Formally , accessibility ( x ) = E [ 1 ( fei ( x ) = y ( x ) ) ] , where fei ( x ) denotes248 the outcome of the i-th network trained over e epochs , and the mean is taken over networks and249 epochs . For the set of datapoints { ( xj , yj ) } nj=1 , Learning Order Constancy is manifested by the high250 correlation between 2 instances of accessibility ( x ) , each computed from a different ensemble.251 PC-bias is shown to pertain to LOC in two ways : First , in Section 4.1 we show high correlation252 between the learning order in deep linear and non-linear networks . Since the PC-bias fully accounts253 for LOC in deep linear networks , this suggests it also accounts ( at least partially ) for the observed254 LOC in non-linear networks . Comparison with the critical principal component verifies this assertion.255 Second , we show in Section 4.2 that when the PC-bias is neutralized , LOC diminishes as well . In256 Section 4.3 we discuss the relationship between the spectral bias , PC-bias and the LOC-effect.257 4.1 PC-Bias is correlated with LOC258 We first compare the order of learning of non-linear models and deep linear networks by computing259 the correlation between the accessibility scores of both models . This comparison reveals high260 correlation ( r = 0.85 , p < 10−45 ) , as seen in Fig . 5a . To investigate directly the connection between261 the PC-bias and LOC , we define the critical principal component of an example to be the first262 principal component P , such that a linear classifier trained on the original data can classify the263 example correctly when projected to P principal components . We trained N=100 st-VGG networks264 on the cats and dogs dataset , and computed for each example its accessibility score and critical265 principal component . In Fig . 5b we see strong negative correlation between the two scores ( p=−0.93,266 r < 10−4 ) , suggesting that the PC-bias affects the order of learning as measured by accessibility.267 4.2 Neutralizing the PC-bias leads to diminishing LOC268 Whitening the data eliminates the PC-bias as shown in Fig . 3c , since all the singular values are now269 identical . Here we use this observation to further probe into the dependency of the Learning Order270 Constancy on the PC-bias . Starting with the linear case , we train 4 ensembles of N=10 2-layered271 linear networks on the cats and dogs dataset , 2 with and 2 without ZCA-whitening . We compute the272 accessibility score for each ensemble separately , and correlate the scores of the 2 ensembles in each273 test case . Each correlation captures the consistency of the LOC-effect for the respective condition.274 This correlation is expected to be very high for natural images . Low correlation implies that the275 LOC-effect is weak , as training the same network multiple times yields a different learning order.276 2As non-linear models achieve the accuracy of linear models within an epoch or 2 , low learning rate is used . Fig . 6a shows the results for deep linear networks . As expected , the correlation when using natural277 images is very high . However , when using whitened images , correlation plummets , indicating that278 the LOC-effect is highly dependent on the PC-bias . We note that the drop in the correlation is much279 higher when considering only the 20 % `` fastest learned '' examples , suggesting that the PC-bias affects280 learning order more evidently at earlier stages of learning.281 Fig . 6b shows the results when repeating this experiment with non-linear networks , training 2282 collections of N=10 VGG-19 networks on CIFAR-10 . We find that the elimination of the PC-bias283 in this case affects LOC much less , suggesting that the PC-bias can only partially account for the284 LOC-effect in the non-linear case . However , we note that at the beginning of learning , when the285 PC-bias is most pronounced , once again the drop is much larger and very significant ( half ) .286 4.3 Spectral bias , PC-bias and LOC287 The spectral bias ( Rahaman et al. , 2019 ) characterizes the dynamics of learning in neural networks288 differently , asserting that initially neural models can be described by low frequencies only . This may289 provide an alternative explanation to LOC . Recall that LOC is manifested in the consistency of the290 accessibility score across networks . To compare between the spectral bias and accessibility score,291 we first need to estimate for each example whether it can be correctly classified by a low frequency292 model . Accordingly , we define for each example a discriminability measure – the percentage out293 of its k neighbors that share with it class identity . Intuitively , an example has a low discriminability294 score when it is surrounded by examples from other classes , which forces the learned boundary to295 incorporate high frequencies . In §C.2 we show that in the 2D case analyzed by Rahaman et al . ( 2019 ) ,296 this measure strongly correlates ( r=−0.8 , p < 10−2 ) with the spectral bias.297 We trained several networks ( VGG-19 and st-VGG ) on several real datasets , including small-298 mammals , STL-10 , CIFAR-10/100 and a subset of ImageNet-20 . For each network and dataset,299 we computed the accessibility score as well as the discriminability of each example . The vector300 space , in which discriminability is evaluated , is either the raw data or the network ’ s perceptual space301 ( penultimate layer activation ) . The correlation between these scores is shown in Table 1.302 Using raw data , low correlation is still seen between the accessibility and discriminability scores303 when inspecting the smaller datasets ( small mammals , CIFAR-100 and STL10 ) . This correlation304 vanishes when considering the larger ImageNet-20 dataset . It would appear that on its own , the305 spectral bias can not adequately explain the LOC-effect . On the other hand , in the perceptual space,306 the correlation between discriminability and accessibility is quite significant for all datasets . Contrary307 to our supposition , it seems that networks learn a representation where the spectral bias is evident,308 but this bias does not necessarily govern its learning before the representation has been learned.309 5 PC-bias : further implications310 Early Stopping and the Generalization Gap . Considering natural images , it is often assumed that311 the least significant principal components of the data represent noise ( Torralba & Oliva , 2003 ) . In312 such cases , our analysis predicts that as noise dominates the components learned later in learning,313 early stopping is likely to be beneficial . To test this hypothesis directly , we manipulated CIFAR-10314 to amplify the signal in either the 1.5 % most significant ( higher ) or 1.5 % least significant ( lower ) 315 principal components ( see examples in Fig . 16 , Suppl . §D ) . Accuracy over the original test set,316 after training 10 st-VGG and linear st-VGG networks on these manipulated images , can be seen317 in Fig . 7 . Both in linear and non-linear networks , early stopping is more beneficial when lower318 principal components are amplified , and significantly less so when higher components are amplified,319 as predicted by the PC-bias.320 Slower Convergence with Random Labels . Deep neural models can learn any random label321 assignment to a given training set ( Zhang et al. , 2016 ) . However , when trained on randomly labeled322 data , convergence appears to be much slower ( Krueger et al. , 2017 ) . Assume , as before , that in natural323 images the lower principal components are dominated by noise . We argue that the PC-bias now324 predicts this empirical result , since learning randomly labeled examples requires signal present in325 lower principal components . To test this hypothesis directly , we trained 10 2-layered linear networks326 on datasets of natural images . Indeed , these networks converge slower with random labels ( see327 Fig . 8a ) . In Fig . 8b we repeat this experiment after having whitened the images , to neutralize the328 PC-bias . Now convergence rate is identical , whether the labels are original or shuffled . Clearly , in329 deep linear networks the PC-bias gives a full account of this phenomenon.330 To further check the relevance of this account to non-linear networks , we artificially generate datasets331 where only the first P principal components are discriminative , while the remaining components332 become noise by design . We constructed two such datasets : in one the labels are correlated with the333 original labels , in the other they are not . Specifically , PCA is used to reduce the dimensionality of a334 two-class dataset to P , and the optimal linear separator in the reduced representation is computed.335 Next , all the labels of points that are incorrectly classified by the optimal linear separator are switched,336 so that the train and test sets are linearly separable by this separator . Note that the modified labels337 are still highly correlated with the original labels ( for P = 500 : p = 0.82 , r < 10−10 ) . The338 second dataset is generated by repeating the process while starting from randomly shuffled labels.339 This dataset is likewise fully separable when projected to the first P components , but its labels are340 uncorrelated with the original labels ( for P = 500 : p = 0.06 , r < 10−10 ) .341 The mean training accuracy of 10 non-linear networks with P=10,50,500 is plotted in Fig . 9a ( first342 dataset ) and Fig . 9b ( second dataset ) . In both cases , the lower P is ( namely , only the first few principal343 components are discriminative ) , the faster the data is learned by the non-linear network . Whether the344 labels are real or shuffled makes little qualitative difference , as predicted by the PC-bias.345 6 Summary and discussion346 When trained with gradient descent , the convergence rate of the over-parameterized deep linear347 network model is provably governed by the eigendecomposition of the data , and specifically , pa-348 rameters corresponding to the most significant principal components converge faster than the least349 significant components . Empirical evidence is provided for the relevance of these results to more350 realistic non-linear networks . We term this effect PC-bias . This result provides a complementary351 account for some prevalent empirical observations , including the benefit of early stopping and the352 slower convergence rate with shuffled labels.353 We use the PC-bias to explicate the Learning Order Constancy ( LOC ) , showing that examples354 learned at earlier stages are more distinguishable by the higher principal components , demonstrating355 that networks ’ training relies more heavily on higher principal components early on . A causal link356 between the PC-bias and the LOC-effect is demonstrated , as the LOC-effect diminishes when the357 PC-bias is eliminated by whitening the images . We analyze these findings in view of a related358 phenomenon termed spectral bias . While the PC-bias may be more prominent early on , the spectral359 bias may be more important in later stages of learning.360 References361 Allen-Zhu , Z. , Li , Y. , and Liang , Y . Learning and generalization in overparameterized neural362 networks , going beyond two layers . arXiv preprint arXiv:1811.04918 , 2018.363 Arora , S. , Cohen , N. , and Hazan , E. On the optimization of deep networks : Implicit acceleration by364 overparameterization . In International Conference on Machine Learning , pp . 244–253 , 2018.365 Arora , S. , Du , S. , Hu , W. , Li , Z. , and Wang , R. Fine-grained analysis of optimization and generaliza-366 tion for overparameterized two-layer neural networks . In International Conference on Machine367 Learning , pp . 322–332 , 2019.368 Basri , R. , Jacobs , D. W. , Kasten , Y. , and Kritchman , S. The convergence rate of neural networks for369 learned functions of different frequencies . In Advances in Neural Information Processing Systems,370 pp . 4761–4771 , 2019.371 Basri , R. , Galun , M. , Geifman , A. , Jacobs , D. , Kasten , Y. , and Kritchman , S. Frequency bias in neural372 networks for input of non-uniform density . In International Conference on Machine Learning , pp.373 685–694 . PMLR , 2020.374 Cao , Y. , Fang , Z. , Wu , Y. , Zhou , D.-X. , and Gu , Q . Towards understanding the spectral bias of deep375 learning . arXiv preprint arXiv:1912.01198 , 2019.376 Dingle , K. , Camargo , C. Q. , and Louis , A . A. Input–output maps are strongly biased towards simple377 outputs . Nature communications , 9 ( 1 ) :1–7 , 2018.378 Fukumizu , K. Effect of batch learning in multilayer neural networks . Gen , 1 ( 04 ) :1E–03 , 1998.379 Gissin , D. , Shalev-Shwartz , S. , and Daniely , A . The implicit bias of depth : How incremental learning380 drives generalization . arXiv preprint arXiv:1909.12051 , 2019.381 Glorot , X. and Bengio , Y . Understanding the difficulty of training deep feedforward neural networks.382 In Proceedings of the thirteenth international conference on artificial intelligence and statistics,383 pp . 249–256 , 2010.384 Gunasekar , S. , Lee , J. , Soudry , D. , and Srebro , N. Implicit bias of gradient descent on linear385 convolutional networks . arXiv preprint arXiv:1806.00468 , 2018.386 Hacohen , G. , Choshen , L. , and Weinshall , D. Let ’ s agree to agree : Neural networks share classification387 order on real datasets . In International Conference on Machine Learning , pp . 3950–3960 . PMLR,388 2020.389 He , K. , Zhang , X. , Ren , S. , and Sun , J. Delving deep into rectifiers : Surpassing human-level390 performance on imagenet classification . In Proceedings of the IEEE International Conference on391 Computer Vision ( ICCV ) , December 2015.392 Heckel , R. and Soltanolkotabi , M. Denoising and regularization via exploiting the structural bias of393 convolutional generators . arXiv preprint arXiv:1910.14634 , 2019.394 Hu , W. , Xiao , L. , Adlam , B. , and Pennington , J . The surprising simplicity of the early-time learning395 dynamics of neural networks . arXiv preprint arXiv:2006.14599 , 2020.396 Jacot , A. , Gabriel , F. , and Hongler , C. Neural tangent kernel : Convergence and generalization in397 neural networks , 2020.398 Krueger , D. , Ballas , N. , Jastrzebski , S. , Arpit , D. , Kanwal , M. S. , Maharaj , T. , Bengio , E. , Fischer , A.,399 and Courville , A . Deep nets don ’ t learn via memorization . 2017.400 Le Cun , Y. , Kanter , I. , and Solla , A. S. Second order properties of error surfaces learning time and401 generalization . Advances in neural information processing systems , 3:918–924 , 1991.402 Li , M. , Soltanolkotabi , M. , and Oymak , S. Gradient descent with early stopping is provably robust403 to label noise for overparameterized neural networks . In International Conference on Artificial404 Intelligence and Statistics , pp . 4313–4324 . PMLR , 2020.405 Nakkiran , P. , Kaplun , G. , Kalimeris , D. , Yang , T. , Edelman , B. L. , Zhang , F. , and Barak , B. Sgd406 on neural networks learns functions of increasing complexity . arXiv preprint arXiv:1905.11604,407 2019.408 Pliushch , I. , Mundt , M. , Lupp , N. , and Ramesh , V. When deep classifiers agree : Analyzing409 correlations between learning order and image statistics . arXiv preprint arXiv:2105.08997 , 2021.410 Rahaman , N. , Baratin , A. , Arpit , D. , Draxler , F. , Lin , M. , Hamprecht , F. , Bengio , Y. , and Courville,411 A . On the spectral bias of neural networks . In International Conference on Machine Learning , pp.412 5301–5310 . PMLR , 2019.413 Saxe , A. M. , McClelland , J. L. , and Ganguli , S. Exact solutions to the nonlinear dynamics of learning414 in deep linear neural networks . arXiv preprint arXiv:1312.6120 , 2013.415 Saxe , A. M. , McClelland , J. L. , and Ganguli , S. A mathematical theory of semantic development in416 deep neural networks . Proceedings of the National Academy of Sciences , 116 ( 23 ) :11537–11546,417 2019.418 Shah , H. , Tamuly , K. , Raghunathan , A. , Jain , P. , and Netrapalli , P. The pitfalls of simplicity bias in419 neural networks . arXiv preprint arXiv:2006.07710 , 2020.420 Simoncelli , E. P. and Olshausen , B . A . Natural image statistics and neural representation . Annual421 review of neuroscience , 24 ( 1 ) :1193–1216 , 2001.422 Soudry , D. , Hoffer , E. , Nacson , M. S. , Gunasekar , S. , and Srebro , N. The implicit bias of gradient423 descent on separable data . The Journal of Machine Learning Research , 19 ( 1 ) :2822–2878 , 2018.424 Torralba , A. and Oliva , A . Statistics of natural image categories . Network : computation in neural425 systems , 14 ( 3 ) :391–412 , 2003.426 Ulyanov , D. , Vedaldi , A. , and Lempitsky , V. Deep image prior . In Proceedings of the IEEE conference427 on computer vision and pattern recognition , pp . 9446–9454 , 2018.428 Valle-Perez , G. , Camargo , C. Q. , and Louis , A . A . Deep learning generalizes because the parameter-429 function map is biased towards simple functions . arXiv preprint arXiv:1805.08522 , 2018.430 Zhang , C. , Bengio , S. , Hardt , M. , Recht , B. , and Vinyals , O . Understanding deep learning requires431 rethinking generalization . arXiv preprint arXiv:1611.03530 , 2016.432 Checklist433 1 . For all authors ... 434 ( a ) Do the main claims made in the abstract and introduction accurately reflect the paper ’ s435 contributions and scope ? [ Yes ] 436 ( b ) Did you describe the limitations of your work ? [ Yes ] 437 ( c ) Did you discuss any potential negative societal impacts of your work ? [ N/A ] 438 ( d ) Have you read the ethics review guidelines and ensured that your paper conforms to439 them ? [ Yes ] 440 2 . If you are including theoretical results ... 441 ( a ) Did you state the full set of assumptions of all theoretical results ? [ Yes ] See the442 `` Assumptions '' paragraph as Section 2443 ( b ) Did you include complete proofs of all theoretical results ? [ Yes ] Each theorem reference444 to its proof . Proofs can be found in Suppl . §A , B445 3 . If you ran experiments ... 446 ( a ) Did you include the code , data , and instructions needed to reproduce the main ex-447 perimental results ( either in the supplemental material or as a URL ) ? [ No ] All data,448 instructions and hyper-parameters are explictly written in the main paper and/or in the449 Suppl . ( see §D.4 , D.4 ) . The code itself will be provided once the anonymity will be450 lifted.451 ( b ) Did you specify all the training details ( e.g. , data splits , hyperparameters , how they452 were chosen ) ? [ Yes ] 453 ( c ) Did you report error bars ( e.g. , with respect to the random seed after running experi-454 ments multiple times ) ? [ Yes ] 455 ( d ) Did you include the total amount of compute and the type of resources used ( e.g. , type456 of GPUs , internal cluster , or cloud provider ) ? [ N/A ] 457 4 . If you are using existing assets ( e.g. , code , data , models ) or curating/releasing new assets ... 458 ( a ) If your work uses existing assets , did you cite the creators ? [ Yes ] 459 ( b ) Did you mention the license of the assets ? [ N/A ] 460 ( c ) Did you include any new assets either in the supplemental material or as a URL ? [ N/A ] 461 462 ( d ) Did you discuss whether and how consent was obtained from people whose data you ’ re463 using/curating ? [ N/A ] 464 ( e ) Did you discuss whether the data you are using/curating contains personally identifiable465 information or offensive content ? [ N/A ] 466 5 . If you used crowdsourcing or conducted research with human subjects ... 467 ( a ) Did you include the full text of instructions given to participants and screenshots , if468 applicable ? [ N/A ] 469 ( b ) Did you describe any potential participant risks , with links to Institutional Review470 Board ( IRB ) approvals , if applicable ? [ N/A ] 471 ( c ) Did you include the estimated hourly wage paid to participants and the total amount472 spent on participant compensation ? [ N/A ] 473

Summary: This work studies the training dynamics of over-parameterized deep linear networks. The authors propose the Principal Components bias (PC-bias) convergence pattern that characterizes the convergence behavior of deep linear networks training, which is supported by theoretical analysis and empirical results. This works also investigates the Learning Order Constancy effect (LOC-effect) and identified the connection between PC-bias and LOC-effect. Empirically, the authors also study several implications of PC-bias, including early stopping, convergence behavior under label noise, and provide interesting observations.