Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
Authors: Thomas George
Guillaume Lajoie
Aristide Baratin
Venue: Accepted by TMLR ()

Abstract: Among attempts at giving a theoretical account of the success of deep neural networks, a recent line of work has identified a so-called `lazy' training regime in which the network can be well approximated by its linearization around initialization. Here we investigate the comparative effect of the lazy (linear) and feature learning (non-linear) regimes on subgroups of examples based on their difficulty.  Specifically, we show that easier examples are given more weight in feature learning mode, resulting in faster training compared to more difficult ones. In other words, the non-linear dynamics tends to sequentialize the learning of examples of increasing difficulty. We illustrate this phenomenon across different ways to quantify example difficulty, including c-score, label noise, and in the presence of easy-to-learn spurious correlations. Our results reveal a new understanding of how deep networks prioritize resources across example difficulty.

Published in Transactions on Machine Learning Research (12/2022)
Lazy vs hasty: linearization in deep networks impacts learning schedule based on example difficulty
Thomas George
g Amiil. quebec
Mila, Université de Montréal
Guillaume Lajoic
g. g.lejoiie@umoontrral.cc
Mila, Université de Montréal
Canada CIFAR AI Chair
Aristide Baratin
Samsung, SAIT AI Lab, Montréal
Reviewed OpenReview: https: openreview. net/ forum? id= 1ukV
Abstract
Among attempts at giving a cheoretical account of the success of deep neural networks, recent line of work has identified so-called 'lazy' training regime in which the network can be well approximated by its linearization around initialization. Here investigate the comparative effect of the lazy (lincar) and feature learning (non-lincar) regimes on subgroups of examples based on their difficulty. Specifically, we show that easier examples are given more weight in feature learning mode, resulting in faster training compared to more difficult ones, In other words, the non-lincar dynamics tends to sequentialize the learning of examples of increasing difficulty. We illustrate this phenomenon across different ways to quantify example difficulty, inchuding c-score, label noise, and in the presence of easy-to-learn spurious correlations. Our results reveal a understanding of how deep networks prioritize resources across example difficulty.
1 Introduction
Understanding the performance of deep learning algorithms has been the subject of intense research efforts the past few years, driven in part by observed phenomena that seem to defy conventional statistical wisdom (Neyshabur et al., 2015; Zhang et al., 2017; Belkin et al., 2019). Notably, many such phenomena have been analyzed rigorously in simpler contexts of high dimensional linear random feature models (e.g., Hastie et al., 2022; Bartlett et al., 2021), which shed new light on the crucial role of overparametrization in the performance of such systems. These results also apply to the so-called loay training regime (Chizat et al., 2019), in which a deep network can be well approximated by its linearization at initialization, characterized by the neural tangent. kernel (NTK) (Jacot al., 2018; Du et al., 2019b; Allen-Zhu et al., 2019). Tn this regime, deep networks inherit the inductive bias and gencralization propertics of kernelized lincar models.
It is also clear, however, that deep models cannot be understood solely through their kernel approximation. Trained outside the lazy regime (e.g., Woodworth et al., 2020), they are able to learn adaptive representations, with time-varying tangent kernel (Fort et al., 2020) that specializes to the task during training (Kopitkov & Indelman, 2020; Baratin et al., 2021; Paccolat et al., 2021; Ortiz-Jiménez et al., 2021). Several results showed examples where their inductive bias cannot be characterized in terms of a kernel norm (Savarese et al., 2019; Williams et al., 2019), or where they provably outperform any lincar method (Malach et al., 2021). Yet, the specific mechanisms by which the two regimes differ, which could explain the performance gaps often observed in practice (Chizat et al., 2019; Geiger et al., 2020), are only partially understood.

---

Published in Transactions on Machine Learning Research (12/2022)
Our work contributes new qualitative insights into this problem, by investigating the comparative effect of the lazy and feature learning regimes on the training dynamics of varions groups of examples of increasing difficulty. We do so by means of a control parameter that modulates linearity of the parametrization and smoothly interpolates the two training regimes (Chizat et al., 2019; Woodworth et al., 2020). We provide empirical evidence and theoretical insights suggesting that the feature learning regime puts higher weight on casy examples at the beginning of training, which results in an increasod learning speed compared to more diffieult examples. This can be understood as an instance of the simplicity bias brought forward in recent work (Arpit et al., 2017; Rahaman et al., 2019; Kalimeris et al., 2019), which we show here to be much more pronounced in the feature learning regime. It also resonates with the old idea that generalization can benefit from some curriculum learning strategy (Elman, 1993; Bengio et al., 2009).
Contributions
We introduce and test the hypothesis of qualitatively different example importance between linear and non-linear regimes;
Using adequately normalized plots, we present a unified picture using 4 different ways to quantify example difficulty, where easy examples are prioritized in the non-linear regime. We illustrate empirically this phenomenon across different ways to quantify example difficulty, including c-score (Jiang et al., 2021), label noise, and spurious correlation to some easy-to-learn set of features.
We illustrate some of our insights in a simple quadratic model amenable to analytical treatment.
The general setup of our experiments is introduced in Scction 2. Scction 3 is our empirical study, which begins with an illustrative example on toy dataset (Section 3.1), followed by experiments on CIFAR 10 in two setups where example difficulty is quantified using respectively c-scores and label noise (Section 3.2). We also examine standard setups where easy examples are those with strong correlations between their labels and some spurious features (Section 3.3). Scction 4 illustrates our findings with theoretical analysis of a specific class of quadratic models, whose training dynamics is solvable in both regimes. We conclude in Section 5.
Related work The neural tangent kernel was initially introduced in the context of infinitely wide networks, for a specific parametrization that provably leads to the lazy regime (Jacot et al., 2018). Such a regime allows to cast. deep learning as a linear model using fixed kernel, enabling import of well-known results from linear models, such as guarantees of convergence to global optimum (Du et al., 2019b;a; Allen-Zhu et al., 2019). On the other hand, it is also clear that the kernel regime does not fully capture the behavior of deep models including, in fact, infinitely wide networks (Yang & Hu, 2021). For example, in the so-called mean field limit, training two-layer networks by gradient descont learns adaptive representations (Chizat & Bach, 2018; Mei et al., 2018) and it can be shown that the inductive bias cannot be characterized in terms of a RKHS norm (Savarese et al., 2019; Williams et al., 2019). Performance gaps between the two regimes are also often observed in practice (Chizat et al., 2019; Arora et al., 2019; Geiger al., 2020).
Subsequent work showed and analyzed how, for fixed ffnite-widtly network, the scaling of the model at initialization also controls the transition between the lazy regime, governed by the empirical neural tangent kernel, and the standard feature learning regime (Chizat et al., 2019; Woodworth et al., 2020; Agarwala et al., 2020). In line with this prior work, in our experiments below we use a scaling parameter a > 0 that modulates linearity of the parametrization and allows us to smoothly interpolates between the vanilla training (a = 1) where fcatures are Iearnod and the lazy (a + oo) regime where they are not. We compare training runs with various values of a and empirically assess linearity with several metrics described in Section 2 below.
Our results are in line with group of work showing how deep networks learn patterns and functions of increasing complexity during training (Arpit et al., 2017; Kalimeris et al., 2019). An instance of this is the so-called spectral bias empirically observed in Rahaman et al. (2019) where a sum of sinusoidal signals is incrementally learned from low frequencies to high frequencies, or in Zhang et al. (2021) that use Fourier decomposition to analyze the function learned by deep convolutional network on a vision task. The spectral bias is well understood in linear regression, where the gradient dynamics favours the large singular directions of the feature matrix. For neural networks in the lazy regime, spectral analysis of the neural tangent kernel
2

---

Published in Transactions on Machine Learning Research (12/2022)
have been investigated for architectures and data (e.g, uniform data on the sphere) allowing for explicit computations (e.g,, decompasition of the NTK in terms of spherical harmonics) (Bietti & Mairal, 2019; Basri et al., 2019; Yang & Salman, 2019). This is a setup where the spectral bias can be rigorously analyzed, i.e,, we can get explicit information about the type of functions that are learned quickly and generalize well. In this context, our work specifically focuses on comparing the lazy and the standard feature learning regimes.
Our theoretical model in Section 4 reproduces some of the key technical ingredients of known analytical results (Saxe et al., 2014; Gidel et al., 2019) deep linear networks. These results show how, in the context of multiclass classification matrix factorization, the principal components of the input-output correlation matrix are learned sequentially from the highest to the lowest mode. Note however that in the framework of these prior works, the number of modes is bounded by the output dimension which thus reduces to one in the context of regression or binary classification. By contrast, our theoretical analysis applies to the components of the vector of labels Y E IR. in the eigenbasis of the kernel XX" defined by the input matrix Thus, despite the tochnical similarities, our framework approaches the old problem of the relative
X E R learning speed of different modes in factorized models from novel angle. In particular, it allows us to frame the notion of example difficulty in this context.
Example difficulty is a loosely defined concept that has been the subject of intense research recently. Ways to quantify example difficulty for a model model/algorin /algorithmm to learn individual examples include e.g. self-influence (Koh & Liang, 2017), example forgetting (Toneva et al., 2019), TracIn (Pruthi et al., 2020), C-scores (Jiang et al., 2021), or prediction depth (Baldock et al., 2021). We believe that a comprehensive theory of generalization in deep learning will require to understand how neural networks articulate learning and memorization both fit the head (easy examples) and the tail (difficult examples) of the data distribution (Hooker al., 2020; Feldman & Zhang, 2020; Sagawa et al., 2020a;b).
2 Setup
We consider neural networks fo parametrized by 0 E IR (i.e. weights and biases for all layers), and trained by minimizing some task-dependent loss function (())= = Ei= ((fe(x))) computed on training dataset [x ,Xu], using variants of gradient descent,
( (t) vo(((()))
(1)
with some random initialization (()) and a chosen learning rate 17 > 0.
Linearization A Taylor expansion and the chain rule give the corresponding updates 1() ;= Sow for any network output, at first order in the learning rate,
(x)+)(()) 2 (x) Σ k (x,x,
(2)
which depend on the timo-varying tangent kernel (x,(x,,)) x)===
= V (x.). The lazy regime is one where this kernel remains nearly conslant throughout training. Training the network in this regime thus corresponds to training the linear predictor defined by
fo(x) = Sa: (x) (0 - ((0) Voloros (x).
(3)
Modulating linearity In our experiments, following Chizat et al. (2019), we modulate the level of "non- lincarity' during training of a deep network with a scalar parameter a 2 1, by replacing our prediction fo by
f8 (x) = foo (x) a afo (x) foro (x))
(4)
and by rescaling the learning rate as 7/ = n/a2. In this sctup, gradicnt descont steps in parameter space are rescaled by 1/ax while steps in function space (up to first order) are in 0(1) in a. 1 a can also be viewed as
Under some assumptions such as strong convexity of the loss, was shown (Chizat al., 2019, Thm 2.4) that as + oo, the gradient descent. trajectory of fo gets uniformly close to that of the linearization fgi--
3

---

Published in Transactions on Machine Learning Research (12/2022)
0.5
0.5
0.5
training loss
G
0.4
(d)
le]
0.0
linearized 100)
non-linear (ar
-0.5
-0.5
-0.5
0.0
1
(al task example dataset
102
10a
(c) AlossiXied training loss loss=0.5 -0.5
(d) Aloss(Xta)) at training lossm0.4
(e) 4loss( at training lossm0.3
iterations
Figure 1: 100 randomly initialized of 4 layers MLP trained the yin-yang dataset (a) using gradient descent in both the non-linear (a = 1) and linearized (a = 100) setting. The training losses (b) show a speed- up in the non-linear regime: in order to compare both regimes models extracted at equal training loss thresholds (c), (d) and (e). We visualize the differences Aloss (Frest) lnsn (Zeaa) finear (Fistt for test points paving the 2d square [-1, 1J using a color scale. We observe that these differences are not uniformly spread across examples: instead they suggest a comparative bias of the non- non-linear regime lowards correctly classifying easy examples (large areas the same class), whereas difficult examples (e.g. the small disks) are boosted in the linear regime.
equal progress,
normalize by comparing
controlling the level of feature adaplivily, where large values of result in linear training where features are not learned. We will also experiment with < 1 below, which enhances adaptivity compared the standard regime (a == 1). The goal of this procedure is two-fold: (i) to be able to smoothly interpolate between the standard regime at a 1 and the linearized one, (vi) to work with models that are nearly linearized yet. practically trainable with gradient descent.
Linearity measures In order assess linearity training runs for empirically chosen values of the re-scaling factor a, we track three different metrics during training (fig right):
Sign similarity counts the proportion of ReLUs in all layers that have kept the same activation status or > 0) from initialization.
Tangent kernel alignment measures similarity between the Gram matrix K k(()) (x;, xx) of the tangent kernel with its initial value ((0), using kernel alignment (Cristianini al., 2001)
Tr K 4
K
KA(K()) (()))=
(II || is the Froebenius norm)
1F
Representation alignment measures the similarity of the last non-softmax layer representation (t) (x) with its initial value (0) in terms of the kernel alignment (eq. 5) of the corresponding Gram matrices (K(1) (()))) 212 (x,) (x.)T
[0 (x;)-
3 Empirical Study
3.1 A motivating example on a toy dataset
We first explore the effect of modulating the training regime for binary classification task on toy datasct with 2d inputs, for which we get a visual intuition. We use fully-connected network with 4 layers and ReLU activations. For 100 independent initial parameter values, generate 100 training examples uniformly on the square 1] from the yin-yang dataset (fig. La). We perform 2 training runs for a 1 and a 100 using (full-batch) gradient descent with learning rate 0.01.
Global training speed-up and normalization After the very first few iterations, we observe a spoed-up in training progress of the non-linear regime (fig. 1.b). This is consistent with previously reported numerical experiments the lazy training regime (Chizat et. al., 2019, section 3). This raises the question of whether
4

---

Published in Transactions on Machine Learning Research (12/2022)
this acceleration comes from global scaling in all directions, or if it prioritizes certain particular groups of examples. We address this question by comparing the training dynamics at equal progress: we counteract. the difference in training speed by normalizing by the mean training loss, and we compare the linear and non-linear regimes at common thresholds ((c), (d) and (e) horizontal lines in fig. 1.b).
Comparing linear and non-linear regimes At every threshold value, we compute the predictions on test examples uniformly paving the 2d square. We compare both regimes on individual test examples by plotting (fig. I.c, I.d, L.e) the differences in loss values,
Aloss (Frest) = llsss nin-lneur (Ftest) fineeur (Fitest) )
(6)
Red (resp. blue) areas indicate lower test loss for the linearized (resp. non-linear) model. Remarkably, the resulting picture is not uniform: these plots suggest that compared to the linear regime, the non-linear training dynamics speeds up for specific groups of examples (the large top-right and bottom-left arcas) at the expense of examples in more intricate areas (both the disks and the areas between the disks and the horizontal boundary).
3.2 Hastening easy examples
We experiment with deeper convolutional networks CIFAR10, in two setups where the training examples are split into groups of varying difficulty. Additional experiments with various other choices of hyperparameters and initialization seed are reported in Appendix F.
3.2.1 Example difficulty using C-scores
In this section we quantify example difficulty using consistency scores (C-scores) (Jiang et al., 2021). Informally, C-scores measure how likely an example is to be well-classified by models trained on subsets of the dataset that do not contain it. Intuitively, examples with high C-score share strong regularities with large group of examples in the dataset. Formally, given a choice of model f, for each (input, label) pair (2r, y) in dataset. D, Jiang et al. (2021) defines its empirical consistency profile as:
ČD (x, v) = Er 1 DiD D)) =
where Èr is the empirical average over subsets D of size n uniformly sampled from D excluding (a, y), and f(, is the model trained on D. A scalar C-score is obtained by averaging the consistency profile over various values of n = 1, 1, [ - 1. For CIFARIO we use pre-computed scores available as https: //github. //ggithub..om//llakstttttttteeee sttpk//tassttuuuuaarrrreree
While training, compute the loss separately on 10 subsets of the training set ranked by inereasing C-scores deciles (e.g. examples in the last subset are top-10% C-scores), for both the training set and test set. We also train linearized copy of this network (so as to share the same initial conditions) with a 100. The 1 run is trained for 200 epochs (64 000 SGD iterations) whereas in order to converge the a = 100 run is trained for 1 100 epochs (320 000 SGD iterations). Similarly to fig. 1, we normalize training progress using the mean training loss in order to compare regimes at equal progress. We check (fig. top right) that the model with a 100 indeed stays the lincar regime during the whole training run, since all lincarity metries that we report essentially remain equal to 1. By contrast, in the non-linear regime (a (a=11 1), a steady decrease of linearity metrics training progresses indicates rotation of the NTK and the representation kernel, as well as lower sign similarity of the ReLUs.
The results are shown in fig. 2 (top left). As one might expect, examples with high C-scores are learned faster during training than examples with low C-scores in both regimes. Remarkably, this effect is amplified in the non-lincar regime compared to the lincar one, as we can observe by comparing e.g. the top (resp. bottom) decile in light green (resp dark blue). This illustrates a relative acceleration of the non-linear regime in the direction of easy examples.
5

---

Published in Transactions on Machine Learning Research (12/2022)
1.0
per c.core wws.reebinttoo wictrrbbnttiiinggg bintriiing Ioza
2.0
0.9
1.5
0.7
0.6
non-linear (or 1I 100)
0.5
10% ciccores
0.5
70-80% decile
0.4
sign simllaity alignment
non- inear la -1
20-30% decile
Inear 1001
0.0
bottom 10%
alignment
2.5
1.0
2.0
1.5
0.5
training
training
3.5
1.0
X
3.0
0.9
2.5
0.8
2.0
1.0
nor-linear (a 1] noley
0.5
non-linear test
Inear (a 100) noisy
0.3
sign smilanity alignment
non-lnear to 11
Incar (a- 1001
0.2
2.0
1.5
1.0
0.5
0.0
2.0
2.5
1.0
0.5
0.0
clean examples
Clcan lozs
Figure/ 2: Starting from the same initial parameters, we train 2 ResNet18 models with a 1 (standard training) and 100 (linearized training) CIFAR10 using SGD with momentum. (Top left) We compute the training loss separately on 10 subgroups of examples ranked by their C-scores. Training progress is normalized by the mean training loss on the æ-axis. Unsurprisingly, in both regimes examples with high C-scores learned faster. Remarkably, this ranking is more pronounced in the non-linear regime as can be observed by comparing dashed and solid lines of the same color. (Bottom left) We randomly flip the class of 15% of the training examples. At equal progress (measured by equal clean examples loss), the non-linear regime prioritizes learning clean examples and nearly ignores noisy examples compared to the linear regime since the solid curve remains higher for the non-linear regime. Concomitantly, the non-linear test loss reaches a lower value. (Right) On the same training run, as a sanity check we observe that the a 100 training run remains the linear regime throughout since all metrics stay close to whereas the run, the NTK and representation kernel rotate, and large part of ReLU signs are flipped. These experiments are completed in Appeudix with accuracy plots for the same experiments, and with other experiments with varying initial model parameters and mini-batch order.
3.2.2 Example difficulty using label noise
In this section we use label noise to define difficult examples. We train ResNet18 on CIFAR10 where 15% of the training examples are assigned a wrong (random) label. We compute the loss and accuracy independently on the regular examples with their true label, and the noisy examples whose label is flipped. In parallel, we train a copy of the initial model in the linearized regime with a 100. Fig. bottom right shows the 3 lincarity metrics during training both regimes.
The results are shown in fig. 2 (bottom left). In both regimes, the training process begins with phase where only examples with true labels are learned, causing the loss examples with wrong labels to increase. A second phase starts (in this run) at 1.25 clean examples loss for the non-linear regime and 1.5 clean loss for the linear regime, where random labels are getling memorized. We see that the first phase takes a larger part the training run the non-linear regime than in the linear regime. We interpret this as the fact that the non-lincar regime prioritizes learning casy examples. As a consoquence, the majority of the training process is dedicated to learning the clean examples in the non-linear regime, whereas in the linear regime both the clean and the noisy labels are learned simultaneously passed the 1.5 clean loss mark. Concomitantly,

---

Published in Transactions on Machine Learning Research (12/2022)
0.75
train acc. ex Wo spurious correl.
1.0
super adaptive
test acc. ex. w/o spurious correl.
1.0
super adaptive (a
0.70
non-linear (a 1)
non-linear (a - 1)
Coleb accuracy test
linear = 100)
linear (a 100)
0.65
0.8
0.8
0.60
0.7
0.7
super adaptive (a
0.55
non-linear (a-
0.6
0.6
linear (a 100)
0.5
0.5
1.00
accuracy train
train acc. ex. with spurious correl.
test acc. ex. with spurious correl.
0.80
super adaptive 0.5)
train acc. ex. w/o spurious correl.
1.0
super adaptive (a - 0.5)
test aCc. w/o spurious correl.
1.0
super adaptive (a - 0.5)
non-linear (ar-) 1)
0.9
0.9
non mon-linnerr (a=
non-linear (a=
linear (a 1100)
Waterbirds accuracy test
0.70
0.8
linear (a 100)
0.8
linear (ar 100)
0.7
0.7
0.6
0.6
0.60
0.4
0.4
0.50
0.3
0.3
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
0.5
0.6
0.7
0.8
0.9
1.0
accuracy train
acc. with spurious correl.
test acc. ex. spurious correl.
Figure 3: We visualize the trajectories of training runs spurious correlations setups, by computing the accuracy on 2 separate subsets: one with examples that contain the spurious feature (with spurious), the other one without spurious correlations (w/o spurious). On Celeb A (top row). the attribute 'blond' spuriously correlated with the gender 'woman', In the first phase of training we observe that (Ieft) the test accuracy is essentially higher for the linear run, which can further explained by observing that (middle) examples increases faster in the linear regime than in non-linear regimes at equal with spurious training accuracy. (right) A similar trend holds for test examples. In this first part the linear regime less sensitive to the spurious correlation (easy examples) thus gets better robustness. (bottom row) On Waterbirds, the background (e.g. lake) is spuriously correlated with the label (e.g. a water bird). (left) We observe the same hierarchy between the linear run and other runs. In the first training phase, the linear regime less prone to learning the spurious correlation: the w/o spurious accuracy stays higher while the with spurious examples are learned ((mdddle) and (right)). These experiments completed in fig. 12 Appendix with varying initial model parameters and mini-batch order.
training accuracy for w/o spurious
comparing the sweet spot (best test loss) of the two regimes indicates a higher robustness to label noise thus a clear advantage for generalization in the non-linear regime.
3.3 Spurious correlations
We now examine setups where easy examples are those with strong correlations between their labels and some spurious feature (Sagawa et al., 2020b). We experiment with CelebA (Liu et al., 2015) and Waterbirds (Wah al., 2011) datasets.
CelebA (Liu el. al., 2015) is a collection of photographs of celebrities' faces, each annolated with its attributes, such as hair color or gender. Similarly to Sagawa et al. (2020b), our task is to classify pictures based on whether the person blond or not. In this dataset, the attribute 'the person is a woman" is spurionsly correlated with the attribute 'the person is blond", since the attribute 'blond" is over-represented among women (24% blond) compared to men (2% are blond).
7

---

Published in Transactions on Machine Learning Research (12/2022)
We use 20000 examples of CelebA to train ResNet18 classifier on the task of predicting whether a person is blond or not, using SGD with learning rate 0.01, momentum 0.9 and batch size 100. We also extract a balanced dataset with 180 (i.e. the total number of blond men in the test set) examples in each of 4 categories: blond man, blond woman, non-blond man and non-blond woman. While training progresses, we mcasure the loss and accuracy on the subgroups man and woman. Starting from the same initial conditions, we train 3 different classifiers with a E 1,10).
Waterbirds (Wah et al., 2011) is a smaller dataset of pictures of birds. We reproduce the experiments of Sagawa et al. (2020a) where the task is to distinguish land birds and water birds. In this dataset, the background is spuriously correlated with the type of bird: water birds are typically photographed on a background such as a lake, and similarly there is generally no water in the background of land birds. There are exceptions: a small part of the dataset consists of land birds on water background and vice versa (e.g. a duck walking in the grass).
We use 4 795 training examples of the Waterbirds dataset. Since the dataset is smaller, we start from a pre-trained Res classifier from default Py'oorc models (pre-trained on ImageNet).. We replace the last layer with freshly initialized binary classification layer, and we set batch norm layers to evaluation mode? We train using SGD with learning rale 0.001, momentum 0.9 and minibatch size 100.
From the training set, we extract a balanced dataset with 180 examples in each of 4 groups: land birds on land background, land birds on water background, water bird on land background, and water bird on land background, which we group in two sets: with spurious when the type of bird and the background agree, and w/o spurious otherwise. While training, we mcasure the accuracy separately on these sets. We train different classifiers with (.5, 1003.
Looking at fig. 3 left, for both the Waterbirds and the CelebA experiments we identify two phases: in the first phase the test accuracy is higher for the linear a = 100 run than for other runs. In the second phase all 3 runs seem to converge, with a slight advantage for non-linear runs in Waterbirds. Taking a closer look at the first phase in fig. 3 middle and right, we understand this difference in test accuracy in light of spurious and non-spurious features: in the non-linear regime, the training dynamics learns the majority examples faster, at the cost of being more prone to spurious correlations. This can be seen both on the balanced training set and the balanced test set.
4 Theoretical Insights
The goal here is to illustrate some of our insights a simple setup amenable to analytical treatment.
4.1
A simple quadratic model
We consider a standard lincar regression analysis: given n input vectors x; E Rd with their corresponding labels y; E IR, S i is n, the goal is to a linear function (x( to the data by minimizing the least-squares loss ((0) == }/8 E, (x,) - y.).. We will focus on specific quadratic parametrization, which can be viewed subclass of two-layer networks with linear activations.
Notation We denote by X E Raxd the matrix of inputs and by y E R the vector of labels. We consider the singular value decomposition (SVD),
X = UM = Σ Mauuuu
(8)
where U Raxn V [adxd are orthogonal and M is rectangular diagonal. rx denotes the rank of X, M1 2 Mrx 0 are the non zero (squared) singular values; we also set μa = 0 for rx < < S max(n, d).
IIn order not to interfere with scaling (sce section 2), and since consider that batch norm whole different thcoretical challenge by itself, we chose turn off, and instead keep the mean and variance buffers their pre-trained value. See appendix D for further discussion of this point.
8

---

Published in Transactions on Machine Learning Research (12/2022)
Left and right singular vectors extend to orthonormal bases (u,,-.. un) and (#1.... va) of R" and R", respectively. In what follows we assume, without loss of generality?, that the vector of labels has positive components in the basis ux, Le., yx == ux y 2 for -....nn
Parametrization We consider the following class of functions
fo(x) 0 x, 0
wx Ux
(9)
In this setting, the least-squares loss is minimized by gradient descent over the vector parameter [wi,---wa wal" Given an initialization w", we want to compare the solution of the vanilla gradient (non linear) dynamics with the solution of the lazy regime, which corresponds to training the lincarized function (3), given in our setting by fo(x) = 0 x where 0 = =Ex uxux v2.
4.2 Gradient dynamics
For simplicity, we analyze the continuous-time counterpart of gradient descent,
w(0) w w(l) -
(10)
where the dot denotes the time-derivative. Making the gradients explicit and differentiating (9) yield
0(t) = (t)
E(t) = (Diag(uro(())... .. (aw2(t)
(11)
where 0r is the solution of the linear dynamics." Note how E(t) is obtained from the input correlation matrix X X by rescaling cach eigenvalue μll by the timc-varying factor w². By contrast, the lazy regime is described by the equation obtained from (11) by replacing Σ(l) the constant matrix 2()).
In the proposition below (proved in Appendix A), we consider the system (10, 11) initialized as 0 == $ Ex=n(u)) (2²)² yu where we assume that w * 0 for all x. We denote by ÿx the components of the input-label correlation vector XTy E IRd in the basis uy we have ÿ* = VPXMx for s X S rx and 0 when rx < A S d.
Proposition The solution of (10, 11) is given by,
0
d
if 0 0
0g
28 0g)
0(1) Σ O1(O)x A=1
0x(1)
(12)
gn
if 0 =
2ps t
By contrast, the solution in the linearized regime where E(t) E() is,
(1) 0%
-a0 '00"0 8
(13)
4.3 Discussion
While the gradient dynamics (12,13) converge to the same solution 0" we that the convergence rates of the various modes differ in the two regimes. To quantify this, for cach dynamical mode 1 < À rx such that 10 A 0, let (x((e) be the limes required for 0x to be e-close to convergence, iee 10, 0 07==,, in the two regimes. Substituting into (12) and (13) we find that, close to convergence e < 0% and for small
initialization, 0Q < 07
log & 14B0 E
tx(e) log
(*((c)
(14)
AA
cB
up to terms in 0C1/0/00//87 Two remarks are in order:
3 One can use the reflection invariance T = - of the SVD to flip the sign of yx- "Explicitly, 0* E x y P1 (09) where Σ is the pseudoinverse of the input correlation matrix Σ X X and Pi
projects onto the null space of X. It decomposes as 0" = Σi= vr in the basis vs where 0 = B//AA for 5 X S rx and 8 of for rx A AsA
9

---

Published in Transactions on Machine Learning Research (12/2022)
Cleanhoisy examples MSE
2.0
2.0
linear non-linear
per example squared
1.5
non-linear clean
Per group MSE
1.5
non-linear
spurious
noisy
spur.
0.5
0.5
0.0
0
50
100
150
200
250
100
200
100
200
time (arbitrary scale)
time (arbitrary scale)
time (arbitrary scale)
Figure 4: (left) Different input/label correlation (example 1): examples are learned in flipped order in the two regimes. (middle) Label noise (example 2): the non-linear dynamics prioritizes learning the clean labels (right) Spurious correlations (example 3): the non-linear dynamics prioritizes learning the spuriously correlated feature. These analytical curves are completed with numerical experiments on standard (dense) 2-layer MLP in figure 14 in appendix G, which shows similar qualitative behaviour.
Linearization impacts learning schedule. Specifically, while the learning speed each mode depends on the components of the label vector in the non linear regime, through jA = VIxMA does not in the linearized one. Thus, the ratio of e-convergence times for two given modes X, X is tx/tx i in the non-linear case and fxx/0xx px / in the linearized case, up to logarithmic factors.
Sequentialization of learning. Non-lincarity, together with a vanishingly small initialization, induces a sequentialization of learning for the various modes (see also Gidel et al., 2019, Thm 2). To see this, pick a mode and consider, for any other mode X, the value (tx) at the time tx = tx(e) where reaches e-convergence. Let us also write 03 = o0 where a small and 0% = 0(1) as + 0. Then elementary manipulations show the following:
0% 05.00
8 10
fx > Ux
(x
(15)
0-0
8Q.
/8x]
ūx < 0A
In words, for fixed and in the limit of small initialization, the mode À gets e-close to convergence before any of the subdominant mode deviates from their (vanishing) initial value: the modes are learned soquentially.
4.4 Mode vs example difficulty
We close this section by illustrating the link between mode and example difficulty on three concrete examples of structure for the training data (x In what follows, consider the overparametrized setting where d 1 n. We denote by e1.-..e, ea the canonical basis of Rd.
For each of the examples below, we consider an initialization w = vo[1 ... ... 1] with small a > 0 and the corresponding solutions in Prop. for both regimes.
We begin with rather trivial setup where each mode corresponds to training example. It will illustrate how non-linearity can reverse the learning order of the examples. Given sequence of strictly positive numbers in 2 ... Pn O, we consider the training data,
Example 1
x; V = fn-iit1 /
1 1sisn
(16)
In the lincarized regime, (x; converges to at the linear rate au:; the model learns faster the examples with higher Ne hence with lower index i. In the non linear regime, the examples are learned sequentially according to the value ÿi /KTi: = Pa-it1, hence from high to low index i. Thus in this setting, linearization flips the learning order of the training examples (see fig. 4 left).
In examples and below we aim at modelling situations where the labels depend on dov-dimesiional (in this case, one dimensional) projections on the inputs (e.g. image classification problem where the labels
10

---

Published in Transactions on Machine Learning Research (12/2022)
mainly depend on the low frequencies of the image). These two examples mirror the two sets of experiments in Sections 3.2.2 and 3.3, respectively.
Example 2 We consider a simple classification setup on lincarly separable data with label noisc. Here we assume d n. Conditioned on a set of binary labels yi = +1, the inputs are given by
x = K,y:e1 + ne;+1 1 S s n
(17)
where = =11 is some 'label flip' variable and > 0. We assumee we have 'noisy' examples with flipped labels, i.e. K2 -1, where 1 S < [n/2].
The SVD of the feature matrix X E Raxd defined by (17) can be made explicit. In particular, the top left. singular vector #1 = // n/ where y E R denotes the vector of noisy labels T. = R4gh- This singles out dominant mode y (u] y)u, of the label vector y that is learned first by the non-linear dynamics. Explicitly,
24
y1
n y
(18)
For a small noise ratio g/n < 1, this yields yıs s y for clean examples and -ys for noisy ones: fitting the dominant mode amounts to learning the clean examples while assigning the wrong label the noisy ones. This is illustrated in fig 4 (middle plot).
We consider simple spurious correlation setup (Sagawa et al., 2020b), obtained by adding to (17) a 'core' feature that separates all training points. Here we assume d > n + 1. Conditioned on set of binary labels y; = tl, the inputs are given by
Example 3
x; = K;y:e1 + Xy:e2 + 76142,
1
(19)
for some binary variable R2 =1, scaling factor X (0,1), 1), and 17 > 0. Given 1 S q < [n/2], we assume we have a majority group of n training examples with K; 1, 1, whose label is spuriously correlated with the spurious feature 62, and a minority group of examples with K; -1
The analysis similar as in Example 2. enhances the bias towards fitting first the majority group of ('casy") examples with spuriously correlated labels. This illustrates an increased sensitivity of the non linear regime the spurious feature at least in the first part of training. This is shown in fig (right plot).
For small noise ratio and scaling factor X, the non-linear dynamics
5 Conclusion
The recent emphasis on the lazy training regime, where deep networks behave as linear models amenable to analytical treatment, begs the question of the specific mechanisms and implicit biases which differentiates it from the fildefiedeed feature learning regime of the algorithms used in practice. In this paper, we investigated the comparative effect of the two regimes on subgroups of examples based on their difficulty. We provided experiments various sctups suggesting that casy examples are given more weight in non-lincar training mode (deep learning) than in linear training, resulting in a comparatively higher learning speed for these examples. We illustrated this phenomenon across various ways to quantify examples difficulty, through c-scores, label noise, and correlations to some casy-to-learn spurious features. We complemented these empirical observations with a theoretical analysis of quadratic model whose training dynamics is tractable in both regimes. We believe that our findings makes a step Lowards a better understanding of the underlying mechanisms that. drive the good generalization properties observed in practice.
Broader Impact Statement
This work proposes to improve our understanding of the mechanisms behind deep learning models training. There is no clear direct application of these theoretical observations to create harmful tools, however as for any technology, machine learning models can be used for malicious intentions. We also acknowledge that deep learning uses a significant amount of compute capacity when scaled to global tech companics, which in some places is powered by carbon emitting power plants, thus contributes to global warming.
11

---

Published in Transactions on Machine Learning Research (12/2022)
References
Atish Agarwala, Jeffrey Pennington, Yann Dauphin, and Sam Schocnholz. Temperature check: theory and practice for training models with softmax-eross-enrroyy losses, 2020. URL httpp:://axxivor//bbbb.... 07344.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- paramcterization. volume 97 of Proceedings of Machine Learning Research, pp. 242-252, Long Beach, Cali- fornia, USA, 09-15 Jun 2019. PMLR. URL http: http::////ccecingss.... ieapeeddiinnm........mmmmmmmmm mlr sarress/97/aalen-aae
html.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ Salakhutdiinov, and Ruosong Wang. On ex- act computation with an infinitely wide neural net. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Bue, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL sttps://proceediing
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 233-242. PMLR, 06-11 Aug 2017. URL https https////poceoe...../oo mlr. patigs.mlr.preseee prress/v7//ariittee
Robert John Nicholas Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreviiet httpp://oopenerrrrse.ee///rrtdtd
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. In Arindam Banerjee and Kenji Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, volume 130 of Proceedings of Machine Learning Research, pp. 2269 2277. PMLR, 13 15 Apr 2021. URL httos://proceedingg. mlr. www.sp/ere//baaaeeee.eepe hat.///bbaatinala..a html.
Peter Bartlett, Andrea Montanari, and Alexander Rakhlin. Deep learning: statistical viewpoint. Acta numerica, 30:87-201, 2021.
Ronen Basri, David Jacobs, Yoni Kasten, and Shira Kritchman. The convergence rate of neu- ral networks for learned functions of different frequencies. In Advances in Neural Infor- mation Processing Systems 32, pp. 4761-4771. 2019. URL http:// mips. caaps.cc/paperr. 8723-the-convert woonvergenneerateeeee pdf.
neeurll-netoon for- -eareeedddd--ee -ewarnedduunctionnsse -ifffrrent-------e--eeeeedee-eet freereent-reuuunciees..
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical bias variance trade-off. Proceedings of the National Academy of Sciences, 116(32): 15849 15854, 2019.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning, In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Alberto Bietti and Julien Mairal. On the inductive bias of neural tangent kernels. In Advances in Neu- ral Information Processing Systems 32, pp. 12893-12904. 2019. URL http://papers nips. cc/paper/
Lénaïc Chizat and Francis Bach. On the global convergence of gradient descent. for over-paramelerriedd models using optimal transport. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 31, pp. 3036-30 3046. Curran Associates, Inc., 2018. URL https: https://proceeednng: www.gus...rrrrroooooo... hcw.rppp.cc/ppprr///////ii .df..
12

---

Published in Transactions on Machine Learning Research (12/2022)
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32, 2019.
Nello Cristianini, John Shawe-Taylor, André Elisseeff, and Jaz Kandola. On kernel-target aligmment.
Tn T. Dietterich, S. Becker, and 7. Ghahramani (eds.), Advances in Neural Information Processing Systems, volume 14. MIT Press, 2001. URL https://proccediing.. wewpeediiinnnnnunis..r... 447/699b8001177776fffffff66
Simon S. Du, Jason D. Lee, Haochuan Li, Liwei Wang, and Xiyu Zhai. Gradient descent finds global minima of deep neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, TCML. 2019, 9- 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 1675 1685. PMLR, 2019a.
URL http://proceeding. www.iingsmrrr.....m www...rresss............. wwwppppwweeeeepe. htt///dl11cc.ct
Simon S. Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over- parameterized neural networks. In International Conference on Learning Representations, 2019b. URL https:://ppereete wtwppppewww.ee.eeettttttet//aaea hett/forummid=SSee331000
Jeffrey L. Elman, Learning and development in neural networks: The importance of starting small. Cognition, 48:71-99, 1993.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881-2891) 2020.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geome- try and the time evolution of the neural langent kernel. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advences in Neural Information Processing Systems, volume 33, pp. 5850-5861. Curran Associates, Inc., 2020. URL https: http:e//proceeeinggs neurips neurips.cc/papeereeee /c///////p///// 94575559000644581177777788888884 Parere.aapppppppppp pdf.
Mario Geiger, Stefano Spigler, Arthur Jacot, and Matthicu Wyart. Disentangling feature and lazy training in deep neural networks. Journal of Stalistical Mechanics: Theory and Experiment, 2020(11): nov 2020. doi:
URL
Thomas George. NNGeometry: Easy and Fast Fisher Information Matrices and Neural Tangent. Kernels in PyTorch, February 2021. URL attps://doi: http:://ooooo/////// ntrrg/10.5500 5221///eo000// 4532597. 45335599
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit regularization of discrete gradi- ent dynamics in linear neural networks. In Н. Wallach, Н. Larochelle, A. Beygelzimer, F. d' d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 32. Curran Associates, Inc., 2019. URL https://proceedingg. neurips. /c///// Paper. pdf.
Trevor Hastie, Andrea Montanari, Saharon Rosset, and Ryan J. Tibshirani. Surprises in high-dimensional ridgeless least squares interpolation. The Annals of Statistics, 50(2):949 986, 2022. doi: 10.1214/ 21-A0S2133. URL htttp:://do..or//1...
121142111102213.. org/10.1214/21-A000
Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do compressed deep neural networks forget? arXiv preprinl Xxi::1111.052. Les. 2020.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural langent kernel: Convergence and generalization in neural networks. In NIPS, pp. 8571-8580. 2018
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C. Mozer. Characterizing structural regularities labeled data in overparameterized models. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 5034-5044. PMLR, 2021. URL http:
//proceedings... mlr. wttpsi/smmmmmmmrrrs///11//1//// /eeess//v3//i/agllk jiang21k.. html.
13

---

Published in Transactions on Machine Learning Research (12/2022)
Dimitris Kalimeris, Gal Kaplun, Preetum Nakkiran, Benjamin Edelman, Tristan Yang, Boaz Barak, and Haofeng Zhang. Sgd on neural networks learns functions of increasing complexity. Advances in newral information processing systems, 32, 2019.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference machine learning, pp. 1885 1894. PMLR, 2017.
Dmitry Kopitkov and Vadim Indelman. Neural spectrum alignment: Empirical study. In International Conference on Artificial Neural Networks, pp. 168 179. Springer, 2020.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the rogularization effect of initial large lcarning rale in training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Compuler Vision (ICCV), December 2015.
Eran Malach, Pritish Kamath, Emmanuel Abbe, and Nathan Srebro. Quantifying the benefit of using differentiable learning over tangent kernels. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7379-7389. PMLR, 18-24 Jul 2021. URL https://ppeet /ppoceed/g/ wwr.m.rrreeeeeeee press/ v139/malach21a. html.
Song Mei, Andrea Montanari, and Phan-Minh Nguyen. A mean field view of the landscape of two-layer neural networks. Proceedings of the National Academy of Sciences, 115(33):E7663--7677, 2018. ISSN 0027-8424. doi: 10.1073/ (0007/p//s.111. 1tt.:/aaa...0079111111 1806579115 URL https://www wttp:/wwwwpppppppppt.apt.. pnas org/ wtagg//ontentt11//33///6666
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. TCL.R workshop track, 2015.
Guillermo Ortiz-Jiménez, Seyed- Mohsen Moosavi-Dezzooll and Pascal Frossard. What can linearized neural networks actually say about generalization? Advonces in Neural Information Processing Systems, 34, 2021.
Jonas Paccolat, Leonardo Petrini, Mario Geiger, Kevin Tyloo, and Matthieu Wyart. Geometric compression of invariant manifolds in neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2021 (4):044001, 2021.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advunces in Neural Information Processing Systems, 33:19920 19930, 2020.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning, pp. 5301-5311 PMLR, 2019.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020a. URL https://openrevies pee/fforumiid=ryx(ull
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang, An investigation of why overparameteri- zation exacerbates spurious correlations. In International Conference on Machine Learning, pp. 8346-8356. PMLR, 2020b.
Pedro Savarese, Itay Evron, Daniel Soudry, and Nathan Srebro. How do infinite width bounded norm networks look in function space? In Proceedings of the 32nd Annual Conference on Learning Theory (COLT), volume PMLR 99, pp. 2667-2690, 2019.
Andrew M. Saxe, James L. Mcclelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural network. In In International Conference Learning Representations, 2014.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An empirical study of example forgetting during deep neural network learning, In International Conference Learning Representations, 2019. URL https://openreview www..neetoooodddddaennamoa
14

---

Published in Transactions on Machine Learning Research (12/2022)
Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Francis Williams, Matthew Trager, Daniele Panozzo, Claudio Silva, Denis Zorin, and Joan Bruna. Gra- dient dynamics of shallow univariate relu networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (cds.), Advances in Neural Information Processing Systems, volume 32. Curran Associales, Inc., 2019. URL https://proceedii www.eediiinnnnuuiissss.oee CrA.A901.ccoTo71111100000ooooo0 Paper pdf.
Blake Woodworth, Suriya Gunasekar, Jason D. Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry; and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Jacob Abernethy and Shivani Agarwal (eds.), Proceedings of Thirty Third Conference Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 1633-3673. PMLR, 09-12 Jul 2020. URL https: / //proceedings.. Salr.presse wttpss/vv11//oooo/ ht2//////ooooor22a...... html.
Greg Yang and Edward J. Hu. Tensor programs IV: feature learning in infinite-width neural networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 11727-17735 PMLR, 2021. URL http://proceedingg mlr.
www.mmlrr...........m..m. wwwps//vs3s htt11/////gnn...aa html.
Greg Yang and Hadi Salman. A fine grained spectral perspective on neural networks. arriv preprint wtXfv.197.11059ffoo
2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. ICLR, 2017.
Xiao Zhang, Haoyi Xiong, and Dongrui Wu. Rethink the connections among generalization, memorization, and the spectral bias of dnns. In Zhi-Hua Zhou (ed.), Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pp. 3392 3398. International Joint Conferences on Artificial Intelligence Organization, 8 2021. doi: 10. 10.24963// /a/////jj.aa.// 0021/467. URL https://doi. org.:....../1................. 24963/ jjcai.iaii... 2021/467,
Main Track.
15


Reviews:
[{"review_id": "MF7sNm2yfh", "reviewer": "Reviewer_cZb7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\nThe studied problem is interesting and relevant; the theory of two-layer neural networks is usually divided into two different settings: the lazy (kernel) regime and the feature learning (mean-field) regime. Most previous works focused on proving optimization and generalization guarantees for neural networks in the two regimes, but to my knowledge, how this different parameterization affects the implicit bias of gradient descent (measured by example difficulty) has not been thoroughly studied. The experimental results demonstrate an interesting separation in the learning behavior, which may inspire future research towards understanding the limitation of the NTK description of neural networks. \n\n## Weakness\n\nI have the following concerns: \n\n1. The studied two-layer linear model is very idealized, and due to the quadratic parameterization, the sequentialization of learning is not surprising, as seen in many prior works on deep linear networks. \n\n2. The notion of learning order has already appeared in many prior papers that are not thoroughly discussed in this submission: \n- In idealized situations, gradient descent for kernel ridge regression already yields a \"simplicity bias\" where the complexity is defined by the polynomial degree, see [1][2].  \n- Similarly, the spectral bias of neural networks in the lazy (kernel) regime have been rigorously characterized in [3][4][5], from which we know that gradient descent learning favors large eigen-directions of the NTK, which often corresponds to low-complexity functions.  \n- For neural network in the feature learning regime, [6] identified a class of staircase functions where the learning of low- and high-degree components is \"coupled\". This allows the feature learning model to learn the difficult high-degree parts faster than the linearized kernel model. \n\n3. [minor] References to related works are not complete. \n- In the case of two-layer neural network, the feature learning regime is highly related to the mean-field regime. Please find the appropriate references for this. \n- The learning order in linear networks have been extensively studied but citations are missing. \n- Which theorem in [Frei et al. 2022] (cited on page 1) shows that neural network can \"provably outperform any linear method\"? \n\nI have the following questions: \n- In the experiments, how does the learning rate scaling affects the observed phenomena? It is known that large learning rate can alter the implicit bias and the feature learning dynamics [7] [8].  \n- For the theoretical analysis, would similar results holds for a vanilla two-layer linear network, where the different regimes are induced by changing the scale of initialization?  \n- Neural networks in the lazy regime should enjoy optimization benefits due to the exponential convergence in the training loss. In Figure 1b, why does the linearized model converge much slower (and apparently not reaching small error)? \n\n[1] Ghosh et al. 2021. The three stages of learning dynamics in high-dimensional kernel methods.  \n[2] Xiao 2022. Eigenspace restructuring: a principle of space and frequency in neural networks.  \n[3] Bietti and Mairal 2019. On the inductive bias of neural tangent kernels.  \n[4] Cao et al. 2020. Towards understanding the spectral bias of deep learning.  \n[5] Nitanda and Suzuki 2020. Optimal rates for averaged stochastic gradient descent under neural tangent kernel regime.  \n[6] Abbe et al. 2022. The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networks.  \n[7] Wu et al. 2021. Direction matters: on the implicit bias of stochastic gradient descent with moderate learning rate.   \n[8] Ba et al. 2022. High-dimensional asymptotics of feature learning: how one gradient step improves the representation. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission studies the \"simplicity bias\" of neural network training dynamics in different regimes defined by the model parameterization. Using various measures of example difficulty, the authors empirically show that the nonlinear feature learning regime has a stronger simplicity bias, meaning that easier examples are learned before difficult ones. The empirical findings are also supported by the theoretical analysis of a quadratic linear model, where the learning of different directions occurs at different time-scale (under sufficiently small initialization). ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the points in the Weaknesses section. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2HIIuKcWRB", "reviewer": "Reviewer_5RX7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The basic concept of the paper, and the subsequent experiments are very sound. In particular, the authors did a good job of considering multiple ways to define difficulty of examples, and attempted to properly normalize comparisons by, for example, grouping points according to their training loss rather than the number of GD iterations.\n\nOne question that arises is: do the accuracies show similar trends to the loss? Because loss and accuracy are only correlated, it's possible that even though non-linear models change loss more on easier examples, but the accuracy per-example is similar. This is somewhat answered in Figure 3 which does accuracy comparisons. However, in this scenario both the low accuracy (early time) and high accuracy (late time) are very similar across the models - making it harder to interpret the intermediate accuracy behavior.\n\nThe analysis of the theoretical model was very clean. However, it is unclear if the mechanism at play in the theoretical model is relevant for the phenomenology in real networks; the theoretical model only really allows for adjustment of the magnitude of features, not their directions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors examine the differences between linearized and non-linear models, with respect to which examples are learned first. In a variety of settings, they present evidence that easier examples are learned first in non-linear models - or perhaps more accurately, easier examples contribute to a larger fraction of the loss. They show this for various definitions of \"easiness\", including an intuitive picture in the yin-yang model, the  The authors also present a toy model which provably shows similar behavior.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "For Figures 1 and 2, it would also be informative to see plots in terms of classification accuracy. One of the interesting features about classification problems, especially those trained with cross-entropy loss is that the loss is only _correlated_ with the actual objective, the accuracy. It would be good to understand whether or not the effect is driven by non-linear models having more certainty in the easier datapoints, versus the classification being more correct on the easier datapoints. In particular it would be good to look at the accuracies as a function of c-score for the same number of GD steps, for optimal learning rates for the different models. I believe that this is a critical change.\n\nFigure 2, top left, is currently hard to parse. I don't feel confident looking at that plot and coming to the conclusions of the authors. Perhaps it would be useful to also have a plot of the differences between the models? Also, there should be a colorbar indicating the C-score.\n\nFor Figure 2, bottom, I didn't quite understand why the gap in the solid curves implied that easier examples were being learned first. Some more clarification on why \"increasing loss more on noisy data\" means \"easier examples are learned first\" would be quite helpful.\n\nFigure 3 shows some curves with $\\alpha <1$ - a setup which was described and analyzed in https://arxiv.org/abs/2010.07344, which should be added to the references. It would be interesting to included curves with other alpha values in Figures 1 and 2 (more specifically 2, I think figure 1 in its current form is simple and effective). This is especially interesting because, by the above reference, for $\\alpha <1$ the optimal learning rate goes as $1/\\alpha$ ($1/\\beta$ in the paper), which means that the change in loss slows down (in terms of GD steps). In this work my understanding is that the $\\alpha = 1$ setting always trains faster than the $\\alpha = 100$ setting.\n\nIt would also be useful to have an experiment on a dataset-model pair where the non-linear model performs significantly better than the linearized model after training - or at least shows accuracy differences according to example difficulty. The results of Figure 3 feel a bit weak because the curves diverge and then converge again, as it appears that the linearized model and the non-linear model learn to similar accuracies. Perhaps creating similar accuracy curves in the CIFAR problem, stratified by c-scores, would be sufficient. I think this is a critical change.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FLWfUIvJor", "reviewer": "Reviewer_93fy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\n\n* Using diverse datasets and metrics reduces the bias due to the metric or architecture used to calculate the image difficulty score. \n* Solid check with three metrics to make sure that the network stays in the linear regime \n\n*Weaknesses:*\n\n* The major weakness is that the difference between the linear and feature learning regimes is tiny. Even in the linear regime, I understand that sequentialization happens by looking at the learning curves. However from reading the text, it may sound like this only happens in the feature learning regime. \n* The title rhymes very well, but --hasty-- sounds like a negative property of feature learning, but I'm not sure if the authors made any such claim. \n* The datasets used in spurious correlation experiments are small: overall $180 \\times 4 = 720 $ samples. It is not hard to believe that the results would generalize to larger sample sizes given the CIFAR10 results, but the small dataset size weakens the results for spurious correlation experiments. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the difference between the lazy and feature learning regimes from the perspective of image difficulty. The authors empirically show that the network learns easy samples faster both in the linear and feature regimes; however, the difference between the learning speeds of the easy vs. hard groups is more pronounced in the feature learning regime. The contributions are:\n\n* The authors use three metrics to check whether the network stays in the linear regime despite being finite-width,\n* Learning curves are presented in both regimes for CIFAR-10, a small subset of CelebA and Waterbirds,\n* A quadratic model is introduced and solved analytically which is then used to study numerically learning speeds in groups for three toy example datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Discussion related to Figure 3 needs to be improved. Why are the learning curves in the middle for the Waterbirds dataset non-monotonic compared to CelebA? Also, the training accuracy starts at 0.7 which is very high for the first epoch due to the small dataset size. \n\n* How do the results compare with the gradient starvation paper from NeurIPS-2021 [1]?\n\n*Minor changes:*\n\n* Contributions point 2, the first sentence needs improvement. We present a unified picture, adequately normalized visualizations, etc. are too bombastic. If the toy dataset example in Fig. 1 is referred to here, please explicitly write it so. \n* Figure 2 right panel figures are hard to read. Using smaller markers should help. \n* Figure 4 middle figure can be plotted for different levels of label corruption $q$. \n* In Section 4.4., why only use small $\\sigma$ for initialization? A larger $\\sigma$ should remove the loss plateaus at the beginning of training in the feature regime, thus the reverse sequentialization phase should be more visible.   \n\n[1] Pezeshki, Mohammad, et al. \"Gradient starvation: A learning proclivity in neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 1256-1272. ", "broader_impact_concerns": "NA. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies differences between lazy and feature learning regimes from the perspective of example difficulty. Authors empirically show that in the feature learning regime (as opposed to the lazy learning regime) networks learn the easier examples before difficult ones. The Authors also analyze toy models to theoretically demonstrate similar behavior occurs.

Overall reviewers were convinced of the correctness of the claims. While the paper focuses on simple model / datasets for experiments, reviewers pointed out that these are done well and results are interesting and proposes phenomena meritting further study in the community thus of interest to the TMLR readership. 

After author response and discussions among reviewers, all reviewers recommended either accept or leaning accept. One reviewer raised that the submission is borderline. The AE agrees with reviewers recommendation and recommends accept as is. AE still strongly recommends authors to incorporate all the major/minor suggestions agreed upon with the reviewers. 

Decision: Accept (poster)