Deep Learning on a Data Diet: Finding Important Examples Early in Training
Authors: Mansheej Paul
Surya Ganguli
Gintare Karolina Dziugaite
Venue: NeurIPS 2021 Poster (2021)
Keywords: data pruning
neural network training dynamics
understanding deep learning
data compression
coresets

Abstract: Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. It is therefore natural to ask: how much of the data is superfluous, which examples are important for generalization, and how do we find them? In this work, we make the striking observation that, in standard vision datasets, simple scores averaged over several weight initializations can be used to identify important examples very early in training. We propose two such scores—the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores—and demonstrate their efficacy on a range of architectures and datasets by pruning significant fractions of training data without sacrificing test accuracy. In fact, using EL2N scores calculated a few epochs into training, we can prune half of the CIFAR10 training set while slightly improving test accuracy. Furthermore, for a given dataset, EL2N scores from one architecture or hyperparameter configuration generalize to other configurations. Compared to recent work that prunes data by discarding examples that are rarely forgotten over the course of training, our scores use only local information early in training. We also use our scores to detect noisy examples and study training dynamics through the lens of important examples—we investigate how the data distribution shapes the loss surface and identify subspaces of the model’s data representation that are relatively stable over training.

Deep Learning on a Data Diet: Finding Important Examples Early in Training
Mansheej Paul Stanford University mansheej@stante edu
Surya Ganguli
Stanford University; Facebook Al Research sganguli wwwpuuorsstaaoor..ooooooooo
Gintare Karolina Dziugaite
Mila gkdzogoogle.com com
Abstract
Recent success in deep learning has partially been driven by training increasingly overparametrized networks on ever larger datasets. therefore natural ask: how much of the data is superfluous, which examples are important for general- ization, and how do we find them? this work, we make the striking observation that, in standard vision datasets, simple scores averaged over several weight ini- tializations can be used to identify important examples very early in training. We propose two such scores- the Gradient Normed (GraNd) and the Error L2-Norm (EL2N) scores and datasets by pruning significant fractions of training data without sacrificing test accuracy. In fact, using EL.2N scores calculated a few epochs into training, we can prune half of the CIFAR training set while slightly improving test accuracy. Furthermore, for a given dataset, EL2N scores from one architecture or hyper- parameter configuration generalize other configurations. Compared recent work that prunes data by discarding examples that are rarely forgotten over the course of training, our scores only local information early in training. We also use our scores to detect noisy examples and study training dynamics through the lens of important examples- we investigate how the data distribution shapes the loss surface and identify subspaces of the model's data representation that are relatively stable over training.
and demonstrate their efficacy on range of architectures and
Introduction
Recently, deep learning has made remarkable progress driven, in part, by training over- parameterized models on ever larger datasets. This trend creates new challenges: the large com- putational resources required pose roadblock to the democratization of AI. Memory and resource constrained settings, such as on-device computing, require smaller models and datasets. Identifying important training data plays role in online and active learning. Finally, it is of theoretical interest to understand how individual examples and uub-populations of training examples influence learning.
To address these challenges, propose scoring method that can be used identify important and difficult examples early in training, and prune the training dataset without large sacrifices in test accuracy. We also investigate how different popplatiioons of the training data identified by our score affect the loss surface and training dynamics the model.
*This work was carried out while author was at ServiceNow. was finalized at Google Brain.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Recent work on pruning data [1, 2], can be placed in the broader context of identifying coresets examples that provably guarantee a small gap in training error on the full dataset [3-7]. However, due to the nonconvex nature of deep learning, coreset techniques make conservative estimates that lead to weak theoretical guarantees and are less effective in practice.
A different approach recently proposed by Toneva et al. [8] tracks the number of times through training an example transitions from being correctly classified misclassified, called "forget- ting event", and find that some examples are rarely forgotten, while others are forgotten repeatedly. Empirically, they observed that training accuracy is not affected by the rarely forgotten training ex- amples and large fraction of the training data can be removed without any impact on test accuracy. However, since this method relies on collecting forgetting statistics throughout training, the forget- ting score is typically calculated in the middle of or at the end of training. Toneva et al. [8] find that, in their example of a ResNet18 trained on CIFAR-10 for 200 epochs, the Spearman rank correlation between early and late scores is good after about 25 epochs and stabilizes after 75 epochs.
Broadly speaking, the ability to prune datasets raises number of questions: What is the nature of examples that can be removed from the training data without hurting accuracy? How early in training can we recognize such examples? How many examples do we need and how does this depend on the data distribution? These questions may have no generic answers and so, in this work, we begin to pursue them empirically in the context of several standard vision benchmarks and standard network architectures. Answers to these questions may both (1) lead to new methodologies that could dramatically reduce training times and memory requirements, and (2) offer important insights into the training dynamics of deep neural networks, and the role of data.
Our first finding is that very early in training (just few epochs), partial forgetting scores identify large fractions of data that can be pruned. Analyzing this puzzling result with a one gradient step analysis of training suggests very simple heuristic: use the loss gradient norm of individual ex- amples to identify important examples. While this approach does not work when the loss gradient norms are computed at the weights early in training of a single trajectory, we find that, surprisingly, averaging these norms over multiple weight initializations does produce a ranking that correlates strongly with forgetting scores and allows us to prune a significant fraction of examples early in training. Indeed, we can prune 50% of examples from CIFAR- 10 without affecting accuracy, while on the more challenging CIFAR-100 dataset, we can prune 25% of examples with only a 1 1% drop in accuracy.
Through series of empirical studies, have begun to tease apart the properties of important examples and how they can depend on the data distribution. In particular, we find that the examples with the very highest norms become superfluous as the amount of label noise increases. Indeed, even on clean data, we find that in the high pruning regime, the best population excludes the very highest-scoringg examples.
1.1 Contributions
We propose to score the importance of each training example Ui) by its expected loss gradient norm (GraNd score), which, up to a constant, bounds the expected change loss for an arbitrary example (Ir, y) caused by removing (Ia, Vi).
We show that pruming training samples with small GraNd scores at initialization allows one to train on much smaller subset of the training data without significant loss in accuracy. While the pruning levels are comparable to those provided by other methods [1, 8]. our score is the only one that is well-defined at initialization and early in training.
Our experimental findings suggest that, within the first few epochs of training, the GraNd score is well-approximated by the norm of the error vector (EI.2N score), where the error vector is the predicted class probabilities minus one-hot label encoding. fact, we find that the EL.2N score provides an even stronger signal for aata-pruning-fae for CIFAR10 we can prune 50% of the data, and for the harder CIFAR100, we can prune as much as 25% of the data without any loss in test accuracy.
We study the role of examples with the highest EL.2N scores, and find that excluding a small subset of the very highest scoring examples produces a boost in performance. This boost in performance is enhanced in corrupted label regime.
2

---

We introduce a method, based on linearly connected modes, for studying the empirical risk surface terms of the modes of subsets of data, allowing us to identify when, in training, the final performance on subpopulations is determined. We demonstrate that the linearly con- nected mode at-convergence of empirical risk surface computed on low E1.2N score examples is determined much earlier in training compared to high score examples.
Finally, we study how an example's EI.2N score connects to the network's training dynamics. We do so by tracking the data-dependent NTK submatrices corresponding to the low high score examples, and measuring the rate at which it evolves in scale-invariant way. We find that the NTK submatrix for the high score examples evolves faster throughout training, supporting our hypothesis that high-scoring examples are the ones driving the learning and the changes in the NTK feature space [9].
2
Which samples are important for learning?
2.1 Preliminaries
We consider supervised classification, where S ((ri,):)) v:) denotes the training set, drawn i.i.d. from an unknown data distribution D, with input vectors E R and one-hot vectors y E {0, 1j encoding labels. For a fixed neural network architecture, let fw( (x) E RK be the logit outputs of the neural network with weights w W S RRD on input x E Rd. Let a be the softmax function given by (2......; 2x)x = 2xp(2x// exp(2x). <xp(<>//E Let p(w, æ) = (f(w, denote the neural network output in the form of a probability vector. For any probability vector let Ep. v) E ((k) log p(k) denote cross-entropy loss.
Let Wo, W1, W2...., WT be the iterates of stochastic gradient descent (SGD), where, for some se- quence of minibatches So: , C S of size M, have
Wr W - 1) (x,))S, 1 - (x,,).
for gt-1 (x,, V Vw www-:lp(www-1:0 v). y), and 1....,7.
2.2 Gradient Norm Score and an infinitesimal analysis
Fix training set S. Due to training with SGD from a random initialization, the weight vector at time t >> O. w., is a random variable. The expected magnitude of the loss vector is our primary focus:
Definition 2.1. The GraNd score of training example (I, y) at time t is Xa(x, y) = Ew. (19,(1, 1)12
Here we describe conditions under which the GraNd score controls the contribution of training example to the change in the training loss. In order to simplify our analysis, we approximate the training dynamics as if they were in continuous time.
A key quantity in our analysis is the time derivative of the loss for generic labeled example (I; v): (e(( y), S.) d)(f(()).v)) (where fe(.) fw. (-)), i.e., the instantaneous rate of change in the loss on (#. y) at time t, where the gradient is computed on the minibatch S,. By the chain rule,
A,((,, v). Si) ge y) dw.
This relates our discrete time dynamics via dw: W - DI ge v/).
Our goal to understand how removing training point from minibatch S, affects Ae((2" y²), Se) for any (x" y²). ). If training point (z, y) is not in the minibatch S,, then the effect is trivial. We thus study Ae((=" 88 S) in order be able to rank all the training examples.
Lemma 2.2. Let S.,, S (1). Then for all (x", ,y²), ) there exists such that |a.((2" "o"""))),,,,,, )) Ae((x", "o""."")). yy), S-yil I c1l90(1), v,)ll
Proof. For given example (x*,²"), the chain rule yields De((x*,y)),)) d=(f)(=")))) dw., Since the weights are updated using SGD, we have dw.
d (x"))"))))) ).9")
dw - E (1SW)))s 90(1, v,). Letting m/ df(f(("") ||. the result follows.

---

At any given training step, given the current location We, the contribution of training example (a, 3)* to the decrease of loss on any other example, bounded by Eq. (3). Since the constant c does not depend on the training example (x, we only consider the gradient norm term, I19.(r; y)ll- The expected value of this gradient norm is exactly the GraNd score of (z, y)- In other words, examples with small GraNd score in expectation have bounded influence on learning how to classify the rest of the training data at a given training time". We therefore propose to rank training examples by their GraNd scores, larger norm meaning more important for preserving Az(z).
f(ae) (x)((()))))))))(())))) denote the kth logit gradient. Then GraNd (x) For an arbitrary input x E R. let 2/ (x) Wi can be written as
(x =
(K) Σ kk=1 V Jur ((F((()))))))) v) 1/2 (x)
Under the cross entropy loss, V orthogonal across logits, and are of a similar size across logits and training examples æ, then we can approximate GraNd by just the norm of the error vector.
p (w,, x; xz)()) yk. When (v% (x))
(z))x } are roughly
Definition 2.3. The EL2N score of training sample (x, y) is defined to be Blip(w,, yll2-
Our experimental results suggest that this approximation becomes accurate after a few epochs of training (see Section 3). These approximations are also in agreement with the empirical results reported in [9, 10]. Fort and Ganguli [10, See. 5.1] demonstrate that the mean logit gradients are nearly orthogonal among classes throughout training. The authors demonstrate that per-example gradients cluster around the mean logit gradient. Fort et al. [9, Figs. 12D- 14D] provide evidence that the mean logit gradient directions evolve rapidly early in training and then stabilize (as measured by the cosine distance between mean logit gradient vectors at different times in training).
2.3 Comparison to forgetting scores
Toneva et al. [8] define a "forgetting event" for a training sample to be a point in training when the classifier switches from making a correct classification decision to incorrect one. They define an approximate forgetting score for each training example as the number of times during training when it was included in minibatch and underwent forgetting event. Toneva al. demonstrate that examples with low forgetting score may be completely omitted during training without any noticeable effect on the accuracy of the learned predictor. In Fig. 1 and Appendix E.5, we make an empirical comparison of forgetting scores to our proposed GraNd and EL2N scores.
In Lemma 2.2, we bounded the contribution of a training example to the decrease of the loss of any other sample over a single gradient step. Due to (/(())'s 's being time-dependent, it is complicated to extend the analysis to multiple steps. However, it is interesting to consider a case when 1he (x.) = <xxz)) for all x4 in the training set, and K = 1. Then summing the bound in Eq. (3) on how much sample (z)) affects the logit output
arbitrary point each time E (1,.., T). obtain a score that depends on (1;) E, (x,))) - y. For two examples, (z,, y) and (a',y)) [z)( we that the example that is learned faster and maintains small error over training time will have a smaller GraNd score on average throughout training. Note that (pr(2;)) - vi)l. if rescaled, is an upper bound 0-1 loss, and therefore E, (pr()x)) - yj) upper bounds the number of forgetting events during training (after rescaling). In this simplified setting an example with high number of forgetting events will also have high GraNd score.
such that [|<(('))|
3 Empirical Evaluation of GraNd and EL2N Scores via Data Pruning
In the previous section, we motivated GraNd and EL2N scores by quantifying the influence of a training example on the loss of an arbitrary example after one optimization step. In this section,
HHere we drop the index j since we refer to an arbitrary training point.
"Note that e depends on (a" "=",y" yy") but for given (x".v"), vy"), e is fixed for all training inputs (=, v). allowing us to rank the training examples.
"The opposite is not necessarily true: examples with large scores may have gradients that cancel out and do not contribute much, meaning that this upper bound is loose.
'The score (z, y) is a function of (x, y). Thus (z,y) is non-random, and the expectation is taken over the remaining randomness (the weights at time which depend on random initialization, random minibatch sequence, GPU noise, etc.).

---

ResNet18
ResNe:18
0.955
Fihal Test Accuracy
0.950
0.80
0.945
Strategy
0.79
0.940
Pruring
Frune Randomiy
Forpet Score
0.88
0.70
EL2N Score
Grahi Score
0.930
0.87
0.77
12
16
20
4
8
12
16
20
4
12
16
20
Computation Epoch
Score Computation Epoch
Score Computation Epoch
0.96
Final Accuracy
0.95
0.90
0.80
0.94
Pruring Strabogy
0.78
0.93
Bandomly
Forget Scorc iEpoch 200)
0.88
0.76
0.92
EL2N Score iEpoch 20)
GraNd Score at zationzation
0.91
0.87
0.74
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Fraction Dataset Pruned
Fraction of Dataset Pruned
Froction of Dataset Pruned
Figure 1: Columns correspond to three different dataset and network combinations (labeled at the top). Each legend applies to all 3 figures in its row. First row: Final test accuracy achieved by training on subset of training data comprised of examples with maximum forgetting, EL2N and GraNd scores computed at different times early in training. Subsets of a fixed size are used: networks trained on 50% of training data for CIFAR-10, 60% for CINIC-10 and 75% for CIFAR- 100. Second row: Final test accuracy achieved by training after different fractions of the dataset are pruned. Here compare forgetting scores at the end of training, EL2N scores early in training (at epoch 20) and GraNd scores at initialization. In each case, examples with the lowest scores are pruned at initialization. In all experiments accuracies achieved by training on the full dataset and on a random subset of the corresponding size are used as baselines.
evaluate these scores empirically, and verify that they identify examples important for general- ization. Networks trained on subsets of the data with high scores achieve levels of test accuracy comparable training the full dataset and are competitive with other state of the art data pruning methods. Perhaps most remarkably, these scores are effective even when computed early in training and perform significantly better than a random baseline, even at initialization.
Data pruning experiments. We train convolutional neural networks of varying depth-ResNet and ResNet50 [11]-on standard vision datasets of varying difficult-CIFAR--DD CIFAR-100 [12], and CINIC- 10 [13]. All scores are calculated by averaging the scores from ten independent training runs. After calculating scores and selecting a training subset, final test accuracies are obtained by retraining networks from new random initializations on only the selected subset. Networks used for evaluating the scores are initialized with seeds that are different from those used to calculate the scores. For each experiment, we report the mean of four independent runs and represent variability across runs by shading the region which spans the 16th 84th percentile of obtained accuracies. See Appendix for more implementation details and Appendix for additional experiments.
In Fig. 1, we show the results of two sets of experiments (top and bottom) on three different network and dataset combinations. The first experiment asks, how early in training are forgetting, GraNd and EI.2N scores effective at identifying examples important for generalization? We compare the final test accuracy from training on subsets of fixed size but pruned based on scores computed at different times early in training. The second experiment compares how GraNd scores at initialization, EL2N scores carly in training and forgetting scores at the end of training negotiate the trade-off between generalization performance and training set size. The training sets are constructed by pruning differ- ent fractions of the lowest score examples. In all examples, training on the full dataset and a random subset of the corresponding size used baselines. We make the following observations.
Pruning at initialization. initialization that achieves test accuracy significantly better than random, and in some cases, com- petitive with training on all the data. This is remarkable because GraNd only contains information about the gradient norm at initializion, averaged over initializations. This suggests that the ge-
settings, GraNd scores can be used to select a training subset at
5

---

ometry of the training distribution induced by random network contains a surprising amount of information about the structure of the classification problem. EL2N scores, which only contain in- formation about errors, are not consistently effective at initialization and forgetting scores, which require counting forgetting events over training. are not defined at initialization.
Pruning early in training. We find that, after only a few epochs of training, EL2N scores are extremely effective at identifying important examples for generalization. For wide range of inter- mediate pruning levels, training on the highest scores performs on par with or better than training on the full dataset. Even at higher pruning levels, EL2N scores computed using local information early in training are competitive with forgetting scores which integrate information over the training tra- jectory. This suggests that the average error vector a few epochs into training can identify examples that the network heavily uses to shape the decision boundary throughout training.
Interestingly, extreme levels of pruning with either EL.2N GraNd scores, we observe sharp drop in performance. We hypothesize that this is because at high levels of pruning. using either GraNd or EL2N scores leads bad coverage of the data distribution. By only focusing on the highest error examples, it is likely that an entire subpopulation of significant size that is present in the test data is now excluded from the training set. We only fit a small number of very difficult examples and do not keep enough of a variety of examples for training models with good test error.
A property of the data. Our results suggest that the ranking of important examples induced by EL2N scores is a property of the dataset and not specific to a network. First, in Appendix E.2, we show that ResNer18 and ResNet50 trained on CIFAR-10 have similar performance curves and the same amount of data can be pruned, even though ResNer50 is a much deeper network with more parameters. Second, EL.2N scores calculated on one set of network architecture and hyperparam- eter configurations can be used to prune data for training with a different network architecture or hyperparameter configuration. The set of important examples generalizes across architectures and hyperparameters. See Appendix E.3 for the experiment on generalization across architectures and Appendix E.4 for the experiment on using scores calculated during hyperparameter optimization. Additionally, in an analysis of the sensitivity of the scoring methods to hyperparameters in Ap- pendix 1. we observe that scores calculated on a single network do not perform as well as those averaged across networks.
We hypothesize that averaging the gradient or error norms over multiple initializations or training trajectories removes dependence on specific weights, allowing a more accurate distillation of the properties of the dataset. EL2N scores can thus be used to probe and understand how the distribution of the training data impacts dynamics (as we show in the next sections). Additionally, it can also reduce the computational burden of training neural networks; once we compute the scores, future networks can be trained on the pruned dataset.
In the following experiments, focus EL2N scores computed early in training, as they appear to more accurately identify important examples.
4 Identifying noise examples
In the previous section, we studied the effect of keeping the highest-scoring examples,
0%
Labels Randemized
10% Labels Randomized
0.96
0.95
and found that we could train
Final Test Accuracy
0.94
0.85
on only the top 50% of exam- ples by score without drop in accuracy (CIFAR-10). What is
0.92
0.90
0.75-
0.88
Fruning Stratagy None
0.65
Pruning Strategy Nons
the nature of subpopulations of examples that allow us to reach high accuracy? One hypothe- sis is that the highest-scoring examples are the most impor- tant ones for achieving an ac- curate classifier. In this see- tion, we refute this hypothesis, and demonstrate the role of la- bel noise.
0.86
Random
Randem
EL2N Score
0.55
EL2N Score
0.84
0.0
0.1
0.2
0.3
0.4
0.5
0.6
0.0
0.1
0.2
0.3
0.4
0.5 0.6
Fraction Dataset Offset
Fraction Dataset Offset
Figure 2: ResNet18 trained on a 40% subset of CIFAR-10 with clean (left) and 10% randomized labels (right). The training subset contains the lowest scoring examples after examples with scores below the offset are discarded. Scores computed at epoch 10.

---

To test whether the scchist-scoring examples are most important for achieving high accuracy, we first sort the examples by increasing EL2N score computed after a small number of training epochs." Then we perform sliding window analysis by training on subset of examples with scores within a window from percentile f' to percentile f percentile, always keep P% of the data but sliding up f. As this window slides to higher percentiles, performance increases, except when the window includes examples with the very highest scores Fig. 2 (left), Indeed the the optimal sliding window actually excludes approximately 500 of the highest-scoring training examples. These effects are reduced in the low pruning regime (see Appendix F.I). In Appendix C, we visualize some of the images that are excluded from each class.
Before we analyze these results, we first place them into a wider context, where we also change the amount of noise in the underlying label distribution. We repeat the experiment outlined above, but corrupt a random K' % of labels, replacing them with a random label, mirroring the protocol popu- larized by Zhang et al. [14]. Fig. 2 reveals that with increased label corruption, the optimal window shifts and excludes higher number of examples. Therefore, the effect we see in the noiseless case appears to be magnified in the presence of label noise. Appendix F.2 examines how adding label noise influences the distribution of EL2N scores of examples.
These findings have several implications. The most obvious implication is that training with only the highest-scoring samples may not be optimal, especially when there label noise. When the population has low Bayes eπтоr rate, using only the highest scoring samples yields optimal re- sults. However, without a validation set, one should be cautious in excluding high-score examples. Feldman [15] discusses memorization in a noisy-label setup and gives conditions under which one should memorize in order to not misclassify singleton examples ( (examples in the training data that are the sole representatives of a subpopulation). For example, if the subpopulation appears with a frequency (1/NN,, ), memorizing such examples can improve generalization. In practice, we may not know whether our data fits these conditions. However, our analysis in Fig, suggests simple and powerful method to prune data for optimal performance by optimizing just two hyperparameters of sliding window using a validation set.
5
Optimization landscape and the training dynamics
5.1 Evolution of the data-dependent NTK
The dynamics of neural-networf training in the infinite-width limit are now well understood [16, 17]: for an appropriate scaling of the learning rate and initial weights, the neural network behaves like linear model in which the data is transformed by the Neural Tangent Kernel (NTK) at initialization, which is defined as the product of the Jacobians of the logits at initialization. In the limit, neural network training implements kernel regression with the fixed NTK as the kernel.
However, finite neural net- works outperform their infinite- width limits [18] and have dif- ferent dynamics early in train-
0% Labels Randomized
10% Labols Randomized
0.35
Kornel Velectty
0.30
0.5
ing [19]. In fact, rather
than being constant, the data-
0.25
0.4
dependent NTK, defined by
0.20
Fort et al. [9] as the Gram matrix of the logit Jacobian, evolves with high velocity in the initial phase of training. Then, around the time of on- set of linear mode connectiv- ity, the NTK velocity stabilizes at smaller value and remains nearly constant for the rest of the high learning rate training time.
0.15
0.2
0.10
0.1
10000
20000 30000 40000 50000
10000 20000 40000 50000 Example Index
Example Index
Figure 3: Kernel velocity for different subsets of images when ResNet18 is trained on CIFAR-10 with all true labels (left) and 10% label noise (right). Examples are sorted in ascending order by EL2N scores and each point corresponds to the kernel velocity of 100 contiguous images starting at example index. Both scores and velocities are computed at the same epoch indicated by color.
'I Appendix F.3, we repeat these experiments for the GraNd score.
7

---

Here we seek to understand which training samples contribute to the NTK gram matrix evolution. To empirically approximate the velocity of a NTK submatrix corresponding to a subset of images in a scale invariant way, we follow [9]. We compute the cosine distance between two NTK gram matrices on the given subset, one computed at epoch f, and another one at epoch 1+1, one epoch later (see Appendix B.3). We look at submatrices of fixed size, formed by examples with contiguous EL2N scores. Fig. shows that higher EL2N scores lead to higher velocities. This relationship is not affected by the time at which both are computed.
Interestingly, the kernel velocity drops off sharply for examples with the very highest scores when label noise is introduced. In Section 4, we showed that dropping these examples boosts the accuracy of the final predictor. We hypothesize that, while the kernel velocity is higher for harder examples that the model is actively trying to fit, the kemne velocity drops off for the very highest scoring examples that might be too difficult to learn, perhaps because they are unrepresentative samples or they have have label noise.
5.2 Connections to the Linear Mode Connectivity
We now examine how the ranking of the examples by EL2N connects the geometry of the loss surface. In particular, Frankle et al. [20] studied the effect of minibatch randomness on the training trajectory, focusing on identifying the point in training when two networks, starting from the same weights, but trained with independent minibatches, converge to the same "linearly connected" mode. They find that, for standard vision datasets, the onset of this "linear mode connectivity" (LMC) happens early in training.
More precisely, let W1, W2,..., Wr be the training trajectory of parent network, fix spawning time t", and let Up-, U1*+1,, 2r vy be an independent training trajectory (i.e., with independent minibatches). beginning at Gr- Wr. We call ur the child network and wee, Ut*41y the child trajectory. The (training) error barrier between two weights wo and w', denoted err(w, S)) is the maximum deviation of the training error surface fts()) above the line connecting the empirical risk at w and w'. That is,
- a err(w, w/; S) (0 (Rs( w + (1- w²) w))
- (()o))((()))
(5)
We then define the mean (training) error barrier, spawning at t", at time t, for t" St S T, denoted err. (S), to be the expected error barrier between Wi and e2 on the data S. That is,
err (S) tu |err(w,, U2; S)],
(6)
where the expectation is taken over the randommess in the trajectories of w and u after t" due to the choice of minibatches, conditional on the initial trajectories up through time 1" (Note that, at the end of training / = T, the supremum in err err(wr, UT; S) is often achieved near a = 1/2, and so this is cheap approximation used in practice.) The "onset" of linear mode connectivity is the earliest spawning time t" at which point erry (S) O, where S is the whole training set. In our work, we instead compute the error barrier on subsets of the training set, which allows us to compare the training dynamics and modes on subpopulations.
In Fig. 4, we measure the mean eтror barrier err;" (S') as function of the spawning time t", in the cases where S' are either 1) the training examples with the smallest scores, 2) the largest scores, or 3) random subset of training examples. We find that the error barrier falls close to zero very rapidly for examples that have low EL.2N scores, and stays high for high score examples. These findings suggest that the loss landscape derived from restricted subsets of examples with low and high EL2N behave very differently. The loss landscape derived from easy subsets of examples with low scores is quite flat, in the sense that error barriers between children as a function of spawn time rapidly diminish. On the other hand, the loss landscape derived from harder subsets of examples with higher scores is rougher, with higher error barriers that persist for longer in the spawn time. Further, this result is in agreement with the results presented in Section 5. showing that most of the learning happens in the high EL2N score examples.
6 Related Work
As discussed earlier, our work is closely related to an empirical study by Toneva et al. [8], which examines the frequency with which correct classification decisions are forgotten during training.
8

---

CIFAR1O ResNe:18
ResNet18
CIFAR100 ResNet50
1.0
1.0
1.0
Training Error Barfier
0.8
Suboel
Min E2N
0.8
Random
0.6
0.6
Subsot
0.6
Min E12N
0.4
0.4
Random
0.4
Max EDN
Sibset
0.2
0.2
0.2
0.0
0.0
0.0
H12
12
16
Spawn Epach
Spawn Epoch
Spawn Epach
Figure 4: The final training error barrier between children on subsets of 1000 highest (green) and lowest (orange) EL2N score examples, and randomly selected training subset (blue) as a function of the spawning time. Left to right: different dataset and network combinations.
The authors observe that examples that are rarely forgotten are also ones that do not contribute much the final accuracy of the predictor. In particular, if we retrain from initialization after having removed rarely forgotten examples from the training data, we achieve the same accuracy. Similar to our work, this work analyzes the dynamics of training in deep learning through the lens of training examples. However, unlike forgetting scores, our proposed methods use only local information early in training. This highlights two properties: example importance is reflected in the local properties of the loss landscape after a few epochs and the ordering of examples by importance is roughly preserved throughout training. We think that this is key contribution and hope it prompts future empirical and theoretical work exploring the early stage of learning.
small proxy network in combination with other training data selection methods to find a small subset of apportant-for-trinnee large state-of-the-art (SOTA) deep neural network. In their empirical study, they observe that most important examples selected via proxy model are also important for training a SOTA network. In addition, they study a proxy which reuses SOTA network's architecture but trained for shorter time. The authors observe that selecting the important examples after at least 50 epochs of training works better than selecting them at random, but not as well as after the full training run. They do not study shorter training times for proxies or relate it to the training dynamics in any other way.
Coleman et al. [1]
examples, that can then be used to train a
Another line of related work is on coresets (see, e.g., [4, 5, 7, 21-23], and many others). The term coresets generally refers to a possibly weighted subset of training data. Much of the work on coresets is focused identifying small training data subsets that provably yield e-approximate solution to the original objective (on all the training data). Most guarantees require the problem to have special structure, such as convexity. For nonconvex problems, like training deep neural networks, guarantees are provided for very conservative proxies, e.g., based on Lipschitz constants or smoothness. While coreset selection comes with nice theoretical guarantees, in our opinion, the utility of these methods is best considered an empirical question.
Coresets have also been studied in the active learning community. Here, the goal is to select a small set of examples to label at any given iteration of training (see, e.g.. [24-28], and references therein). Coreset selection has also been proposed as way to increase model robustness [29].
Pleiss et al. use similar method, the Area Under the Margin (AUM) statistic, but with a slightly different goal: identifying noisy and mislabeled examples. Their proposed method exploits differ- ences the training dynamics of clean and mislabeled samples by keeping track of statistics through the course of training. The AUM statistic is similar to forgetting scores in that it uses information from the whole training run. In contrast, we focus on instantaneous information in the carly phase of training. In addition to identifying noisy examples, we aim to rank points by importance and therefore also identify redundant/casy examples. Thus our approach is complimentary to the AUM statistic and can be used together to obtain higher levels of pruning on noisy datasets.
There have been a number of recent studies looking at the problem of estimating example difficulty. One such approach for identifying difficult examples within a given class is the Variance of Gradi- ents (VoG) score proposed by Agarwal, D'souza, and Hooker [31]. For each image, they calculate the gradient of the activations with respect to the pixels at K different checkpoints over training. The VoG score the average (over pixels) of the per-pixel variance across these K checkpoints. The authors conclude that the images that appear more difficult also have a higher VoG score. Un-

---

derstanding the connections between these two scores calculated in entirely different spaces is an interesting direction for future work.
Another quite different approach estimates example difficulty using prediction depth [32], which is defined as the first layer which a k-Nearest Neighbor classifier can correctly classify an example using the representation of the image in all subsequent layers. In additional to methodological dif- ferences, this method uses the final trained network. To our knowledge, we are the first to highlight the existence of a strong signal for estimating example difficulty early in training.
Informally, removing a training example from the training data and not hurting the generalization error suggests that the example has small "influence" on the test data. Influence of the training examples on test examples is studied in sample-baset basele explainability [33-35]. On the theory side, Feldman [15] recently proposed to model data as a mixture of populations and study the role of memorization when the data distribution is long-tailed. Feldman demonstrates conditions under which memorization is necessary for good generalization. In doing so, he proposes a definition of example memorization and influence, which can be interpreted as leave-one-out notion of stabil- ity. In an empirical study following this work, Feldman and Zhang [36] demonstrate that classifiers trained on computer vision benchmarks benefit from memorization. In particular, training without higg--memorizztioo--allll examples comes at a cost of accuracy of the learned neural network clas- sifier. Appendix G, we compare GraNd, EL2N, forgetting scores, and memorization values on CIFAR- 100--rained Resnet50 networks; memorization values do not correlate with the other scores.
7 Discussion
In summary, our work both (1) introduces methods to significantly prune data without sacrificing test accuracy using only local information very early in training (Fig. 1), sometimes even at initialization, and (2) uses the resulting methods obtain new scientific insights into how different subsets of training examples drive the dynamics of deep learning. We start from a principled approach by asking how much on average each training example influences the loss reduction of other examples, and from that starting point, we obtain 2 scores, namely gradient norm (GraNd) and error norm (EL2N) that bound or approximate this influence, with higher scores indicating higher potential influence. We find that examples with higher scores tend to be harder to learn, in the sense that they are forgotten more often over the entire course of training. We also find that the very highest scoring examples tend to be either unrepresentative outliers of class, have non standard backgrounds or odd angles, are subject to label noise, or are otherwise difficult. This observation yields a simple and powerful sliding window method (Fig. 2) to prune data by keeping examples within range of scores, where the start and the end of the range constitute just 2 hyperparmeters that can be tuned via a validation set. This tuning can be done using different hyperparameter settings different network saving computation time (Appendix E.3). Furthermore, we find that high-scoring examples primarily drive feature learning by maximally supporting the velocity of the NTK, whereas learning dynamics might actually give up on the very highest scoring examples that may correspond to unrepresentative examples or noise (Fig. 3). Finally we show that higher (lower) scoring subsets of examples contribute to a rougher (smoother) loss landscape (Fig. 4). Overall this decomposition of both loss landscape geometry and learning dynamics into differential contributions from different types of examples constitutes an exciting new methodology for analyzing deep learning. A deeper understanding of the differential role played by different subsets of examples could aid not only in data pruning, but also in curriculum design, active learning, federated learning with privacy, and analysis of fairness and bias. Our empirical findings raise a number of interesting theoretical questions, some of which we discuss in Appendix D.
Acknowledgements
The authors would like to thank Blair Bilodeau, Alex Drouin, Étienne Marcotte, and Daniel M. Roy for feedback on drafts, the Toolkit team at ServiceNow for providing the tools and computation resources that greatly accelerated our empirical work, and the NeurIPS reviewers for their thorough engagement and invaluable feedback on label-dependence and practical applications. S.G. thanks the Simons Foundation, NTT Research and an NSF Career award for funding while at Stanford.
10

---

References
[1] C. Coleman, C. Yeh, S. Mussmann, B. Mirzasoleiman, P. Bailis, P. Liang, J. Leskovee,, and M. Zaharia. Selection via proxy: Efficient data selection for deep learning, 2019. arXiv: 1906. 11829.
[2] M. Hwang, Y. Jeong, and W. Sung. "Data distribution search to select core-set for machine
learning". In: Proc. 9th Int. Conf. Smart Media & Appl. (SMA 2020), Jeju, Korea. 2020,
pp. 17-19.
[3] S. Har-Peled and A. Kushal. "Smaller coresets for k-median and k-means clustering". Dis- crete & Computational Geometry 37.1 (2007), pp. 3-19.
[4] J. H. Huggins, T. Campbell, and T. Broderick. "Coresets for scalable bayesian logistic regres-
sion" (2016). arXiv: 1605 06423.
[5] T. Campbell and T. Broderick. "Bayesian coreset construction via greedy iterative geodesic ascent". In: Int. Conf. Machine Learning. PMLR. 2018, pp. 698-706.
[6] I. W. Tsang, J. T. Kwok, P.-M. Cheung. and N. Cristianini. "Core vector machines: Fast SVM training on very large data sets." " Journal of Machine Learning Research 6.4 (2005).
[7] A. Munteanu, C. Schwiegelshohn, C. Sohler, and D. P. Woodruff. On coresets for logistic regression. 2018. arXiv: 1805. 08571.
[8] M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. 2018. arXiv:
1812. 05159.
[9] S. Fort, G. K. Dziugaite, M. Paul, S. Kharaghani, D. M. Roy, and S. Ganguli. "Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel". In: Advances in Neural Information Processing Systems. 2020.
arviv: 2010. 15110.
[10] S. Fort and S. Ganguli. "Emergent properties of the local geometry of neural loss landscapes". arXiv preprint arXiv:1910.05929 (2019).
[11] He, Zhang. S. Ren, and J. Sun. "Deep residual learning for image recognition" In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. 2016, pp. 770- 778.
[12] Krizhevsky, V. Nair, and G. Hinton. "Learning multiple layers of features from tiny im- ages" (2009).
[13] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, "CINIC-10 is not ImageNet or CIFAR-10". CoRR abs/1810.03505 (2018), arXiv: 1810. 03505.
[14] C. Zhang. S. Bengio, M. Hardt, B. Recht, and O. Vinyals. "Understanding deep learning re- quires rethinking generalization" In: Int. Conf. Representation Learning (ICLR). 2017. arXiv: 1611.03530v2
[15] V. Feldman. "Does learning require memorization? short tale about a long tail". In: Proc. 52nd Ann. ACM SIGACT Symp. Theory of Comput. (STOC). 2020, pp. 954-959. [16] A. Jacot, F. Gabriel, and C. Hongler. "Neural tangent kernel: Convergence and generalization in neural networks". In: Advances in Information Processing Systems (NeurIPS). 2018, arXiv:
1806. 07572.
[17] J. Lee, I.. Xiao, S. S. Schoenholz, Y. Bahri, R. Novak, J. Sohl-Dickstein, and J. Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. 2019. arXiv: 1902. 06720.
[18] S. Arora, S. S. Du, Z. Li, R. Salakhutdinov, R. Wang, and D. Yu. Harnessing the power of infinitely wide deep nets on small-data tasks. 2019, arXiv: 1910.0168 006633.
[19] A. Lewkowycz, Y. Bahri, E. Dyer, J. Sohl-Dickstein, and G. Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. 2020. arXiv: 2003.02218.
[20] J. Frankle,
Dziugaite, D. Roy, and M. Carbin. "Linear mode connectivity and the lottery ticket hypothesis". In: Int. Conf. Machine Learning (ICML). 2020, pp. 3259-3261 arXiv: 1912.05671
[21] P. K. Agarwal, Har-Peled, sets". Combinatorial and computational geometry 52 (2005), pp. 1-30.
Varadarajan, al. "Geometric approximation via core-
[22] D. Feldman, M. Schmidt, and C. Sohler. "Turning big data into tiny data: Constant-size core- sets for k-means, PCA, and projective clustering". SIAM Computing 49.3 (2020), pp. 601- 657.
11

---

[23] E. Tolochinsky and D. Feldman. Coresets for monotonic functions functions with applications to deep
learning. 2018. arXiv: 1802. , 07382.
[24] K. Wei, R. Iyer, and J. Bilmes. "Submodularity in data subset selection and active learning".
In: Int. Conf. Machine Learning. PMLR. 2015, pp. 1954-19633
[25] O. Sener and S. Savarese. Active learning for convolutional neural networks: A core-set ap-
proach. 2017. arXiv: 1708.00423 00489.
[26] K. Killamsetty, D. Sivasubramanian, G. Ramakrishnan, and R. Iyer. GLISTER: Generaliza-
tion based Data Subset Selection for Efficient and Robust Learning. 2020. arXiv: 2012.
10630.
[27] B. Mirzasoleiman, J. Bilmes, and J. Leskovec. "Coresets for data-efficient training of machine learning models". In: Int. Conf. Machine Learning (ICML). PMLR. 2020, pp. 6950-6960.
arXiv: 1906. 01827.
[28] Y. Shen, H. Yun, Z. C. Lipton. Y. Kronrod, and Anandkumar. Deep active learning for named entity recognition. 2017. arXiv: 1707. 05928.
[29] B. Mirzasoleiman, K. Cao, and J. Leskovec. "Coresets for Robust Training of Neural Net- works against Noisy Labels" (2020). Xiv: 2011.07451.
[30] G. Pleiss, T. Zhang, E. R. Elenberg, and K. Q. Weinberger. "Identifying mislabeled data using the area under the margin ranking". arXiv preprint arXiv:2001. 10528 (2020).
[31]
Agarwal, D. D'souza, and S. Hooker. "Estimating example difficulty using variance of gradients". arXiv preprint arXiv:2008. 1600 (2020).
[32] R. J. N. Baldock, H. Maennel, and B. Neyshabur. Deep Learning Through the Lens of Exam- ple Difficulty. 2021, arviv: 2106. 09647 [cs LL].
[33] P. W. Koh and P. Liang. "Understanding black-bor predictions via influence functions". In: Int. Conf. Machine Learning. PMLR. 2017, pp. 1885-1894.
[34] E. Barshan, M.-E. Brunet, and G. K. Dziugaite. "Relatif: Identifying explanatory training
samples via relative influence". In: Int. Conf. Artificial Intelligence and Statistics (AISTATS).
2020. pp. 1999-1909.
[35] G. Pruthi, F. Liu, S. Kale, and M. Sundararajan. "Estimating Training Data Influence by Tracing Gradient Descent". In: Advances in Neural Information Processing Systems. 2020.
[36] Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. 2020. arXiv: 2008. 03703.
[37] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, tions of Python+Num?? programs. Version 0.2.5. 2018.
Wandeeman-Milnn and Q. Zhang. JAX: composable transforma-
[38] J. Heek, Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee. Flax:
A neural network library and ecosystem for JAX. Version 0.3.4. 2020.
12


Reviews:
[{"review_id": "yys4QGI_sjG", "reviewer": "Reviewer_3e6T", "summary": "This paper proposes a heuristic to prune the dataset for training by using the norm of the gradient for an ensemble of random networks as a proxy score. They provide some intuition for the proposed method and conduct experiments to verify that the method works well in practice.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The problem is well-motivated with clear potential applications to speed up training and save computational costs\n- The method proposed is sound. It also poses an interesting question about how the initial random network 'essentially' determines characteristics of future training. The connection to NTK and forgetting score is also interesting. \n- The experimental set up is reasonable, and the empirical results suggest the method works well in practice", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have discussed these aspects well", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hrs"}, "extra_text": {}}, {"review_id": "qwy3HouKSX", "reviewer": "Reviewer_EkuL", "summary": "The paper proposes two metrics to rank training set samples for reducing the training set size:\n\n1. L2-norm of the error vector (EL2N).\n2. L2-norm of the gradient vector (GraNd).\n\nThese can be evaluated cheaply and relatively early in training (10-20 epochs) or even at initialization. For instance, by keeping only the top 50% CIFAR10 of training data according to this metric, one can achieve the same test accuracy as when using the whole training set.\n\nThe authors evaluate their method on several datasets and architectures, and run several experiments that relate these metrics to change in NTK during training or the onset of linear mode connectivity, and more.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Post-rebuttal update:\n\n\nI am raising my score from 5 to 6 due to good results on architecture/hyperparameter robustness, and conditioned on other changes discussed in the rebuttal: ablation against label-independent metrics + impact of number of evaluations on performance of GraNd/EL2N + rework presentation / theory based on the mismatch between empirical and theoretical GraNd. I believe these changes will not change the paper results/narrative by too much, but will improve its significance and address some of my prior concerns about motivation/ablations. I thank the authors for actively engaging in the rebuttal.\n\n-------\n## Initial Review:\n\nI find the key result of this paper very interesting and surprising (that one can trim the training set so significantly with such simple metrics). However, due to lacking motivation / insight / ablations, I do not feel like advocating for acceptance.\n\n## Originality\n\nTo my knowledge the idea is novel, although I am not well familiar with dataset pruning literature.\n\n## Quality\n\nOverall the quality is good, and experiments are thorough and exhaustive. However, certain aspects are lacking that make me lower the score:\n\n1. IMO section 2.2. is not at all clear and contains multiple errors. For instance:\n\n\t* Line 114: why does $\\Delta_t$ depend on  $S_t$? I don't see it on the RHS. $f_t$ does not depend on $S_t$ per equation (1). But even if it did, extra dependence on $S_t$ would be redundant due to dependence on $f_t$.\n    * Line 115: \"where the gradient is computed on the minibatch $S_t$\" - what does this mean exactly? Expression in line 114 is the derivative of the loss of outputs at $x$ after $t$ training steps, and as before, I don't see dependence on $S_t$.\n    * Line 116: should it be $w_t - w_{t - 1}$? Otherwise RHS contradicts the definition in Equation (1).\n    * Line 118: Here and after $\\Delta_t$ is evaluated at the whole $S$, but I still don't understand how exactly it depends on this argument.\n    * Line 120: Formula for $dw_t/dt$ now contradicts the one in line 116, where the sum is over $S_{t-1}$.\n    * Line 124: \"constant $c$ does not depend on $(x, y)$\" - isn't it clearly defined as a function of $(x, y)$ in line 121?\n    * Lines 131-133, 135-136: why do individual logit gradients become roughly orthogonal and similar in size _after_ training? I can see them being such at the beginning, but shouldn't logits start to differentiate as the network learns different classes?\n  \n  \nOverall, I found this section quite vague and confusing, and hope it can be improved in terms of mathematical rigor and clarity.\n\n2. By definition (i.e. in expectation over random initializations), for a fixed $x$, EL2N and GraNd at $t = 0$ are precisely the same for any $y$ among one-hot targets $[1, 0,\\dots, 0],\\dots, [0,\\dots, 0, 1]$ via a symmetry argument. Therefore, at initialization, they can't possibly depend on $y$, and yet the metrics are still helpful per Figure 1 and others. Hence,\n\n\t2.1. I think it's necessary to ablate the metrics against $y$-independent baselines such as (a) the L2 norm of the network outputs $f_t(x)$ (or L2 distance from $f_t(x)$ to e.g. a constant vector of $[1/C,\\dots,1/C]$), and (b) the Frobenius norm of the outputs-to-parameter Jacobian $\\partial f_t(x)/\\partial w_t$, or other alternatives. These could be simpler and more interpretable, and with more potential applications, since you don't need the label. \n    \n    2.2. Could you please confirm that the 10 random seeds used to compute the metrics do not intersect with 4 random seeds used to train and evaluate networks afterwards? Otherwise $y$ could indeed be useful at initialization, but only because it helps pick luckier random seeds.\n\n## Clarity\n\nApart from section 2.2, the paper is very clearly written. However, in my opinion the experimental design itself doesn't shed that much light into the nature of these metrics, and they remain a bit mysterious to me. For instance, why is one sometimes better than the other? Why can there be dips in performance, as in e.g. epoch 2, Figure 1? Do training samples get reordered frequently as training progresses? Do these metrics work for other loss functions? Can we predict the sharp dip in performance as we increase the fraction of pruned examples? Do rankings computed for one architecture work well for pruning the dataset for a different architecture?\n\n## Significance\n\nOn one hand, I think the findings are unexpected and can make follow-up work ask many interesting questions. On the other hand, I don't see a practical use case for compressing the dataset this way. Namely, there are quite a few hyper-parameters (EL2N vs GraNd; sliding window location; epoch used for ranking), and I think in the vast majority of cases it will be simpler to just train the network on the whole dataset.\n\nOne way to apply this could be to demonstrate that these metrics computed for one architecture (e.g. VGG) work well for other architectures (e.g. ResNet), in which case one could produce an architecture-agnostic ranking of images for each dataset, which would be very useful. If this kind of result could be added (or is already in the paper and I missed it?), I would rate this paper higher in terms of significance.\n\nAlternatively, significance could be improved if the authors used a label-independent metric as I mention in section \"Quality\", in which case I could imagine applications in semi-supervised / active learning (e.g. one could use these scores to request labels for highly-ranked images).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "All good.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "gFQxFn2RXgr", "reviewer": "Reviewer_qhFJ", "summary": "The authors propose GraNd score based on expected gradient loss to filter out low impact examples from the training. The proceed to empirically show that pruning examples with low GraNd score is competitive with state of the art methods with the benefit of being able to use the score early at training (as to save on compute). \n\nThe paper then discusses the relationship of GraNd with EL2N---the norm of the prediction vector minus the one hot of the true label, showing that EL2N can be used to improve the performance of models by removing the examples with high EL2N. \n\nThen, the authors go and compare different scoring ranges for pruning to find that the highest scores are not beneficial when there is label noise. Moreover, they experimentally show that high scoring examples drive learning velocities in NTK and contribute to rougher loss landscape.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While the score the authors suggest is interesting I have some reservations about the paper. First, the 'theoretical derivation' is very unclear (I mention a couple of examples below) and also I'm not completely sure whether it is correct. Therefore, this score could be just a random functional that might or might not be too interesting. \n\nMoreover, the results are a bit scattered and I'm not sure what is the main message of the paper. Do we want to find important examples? If so, I would first try to define what is an important example. Then for every experiment I would want to understand why this experiment was conducted, and why this demonstrates that the metric indeed finds examples that are important. To me it seems that the main experiment is the pruning one and it seems that the metric proposed by this paper is slightly worse than the forgetting score so the contribution is incremental. (Also, you should compare to Area-Under-Margin, see comment below).\n\nOther comments:\n\n* When the authors discuss time derivatives it is unclear what they mean. Unless doing gradient flow (which was not mentioned in the paper) time works in discrete steps, so I don't see why the time derivative is well defined. If the point was to discuss discrete derivatives it is unclear why the equation below eq 2 has \\approx because then it should be =. \n\n* In Lemma 2.2. I'm not sure why \\|d\\ell(f(x,y))  / dw_t \\| is finite and why does it not depend on (x,y) as mentioned in the discussion above? For the former, there could be example for which the loss is unbounded after some step, for the later, to me it seems that the derivative of the loss at example (x,y) has to be dependent on (x,y). Also, would appreciate an explanation to what does the chain rule mean (and why it holds) when the derivative discrete. \n\n* Not sure what 'maintaining \\Delta_t x` means in line 128.\n\n* I assume that in equation 4 the expectation is over initialization, this should be mentioned.\n\n* In figure 1. I would remove 'for pruning strategy see left' and have a large legend that spans the entire figure so it is clear that the legend is relevant to everything.\n\n* while forget score is calculated later in training it seems to be beating both proposed measures in the paper.\n\n* I think this paper is relevant and probably should be cited and compared against: https://arxiv.org/abs/2001.10528 \n\n+++++ \nPost rebuttal:\nI would like to thank the authors for their detailed review and alleviating some of my concerns and decided to change the score 5 -> 6. I think some minor edits along the lines of the rebuttal could help the paper's readability and motivation. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "aBLgQIyLo0X", "reviewer": "Reviewer_V5c3", "summary": "The paper focuses on identifying important examples for training neutral networks early in training with the goal of pruning a significant portion of training examples in later training stages. It analyses the influence of a training example on the loss reduction of other examples and defines two scores:  gradient norm (GraNd) and error norm (EL2N) that bound or approximate this influence. It shows that the  higher scores generally correlate with the higher influence. However, the paper also finds that the highest scoring examples tend to be outliers and are subject to label noise, or are otherwise difficult. The paper then proceeds with analyzing the high and low scoring examples (under the defined GraNd and EL2N metrics) in training dynamics. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "On one hand, the findings of this paper seem quite trivial, as examples that have larger gradients (in expectation) are likely to be more important. It reminds the heuristics of choosing neurons with higher weigh norm in neural network pruning in earlier works on model compression. On the other hand, the intuitive rule in this paper is motivated by a principled analysis (or at least an attempt to derive one). My other concern is the practical value of this paper. The empirical results show that when the pruned ratio becomes interesting (over 50%) then the proposed data pruning method is only marginally better than a random pruning. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new heuristic to prune the dataset for training a neural network by using the norm of the gradient for an ensemble of random networks as a proxy score (error norm: EL2N, gradient norm: GraNd).  Authors show that pruning examples with low GraNd score is competitive with state of the art methods with the benefit of being able to use the score early in the training. Using these metrics, they shed light on how the underlying data distribution shapes the training dynamics: they rank examples based on their importance for generalization, detect noisy examples and identify subspaces of the model’s data representation that are relatively stable over training.


Common weaknesses pointed out were some lack of clarity (`ghFJ`, `EkuL`), unclear practical value (`V5c3`, `EkuL` ) and efficacy(`V5c3`). The authors actively engaged in the discussion and provided explanations and results to address the reviewer's concerns. The AC believes analyses performed and explanations given during the discussion would improve the final version of the paper further and strongly encourages the authors to adopt them. Questions regarding architecture / hyperparameter robustness and lack of ablations, raised by reviewer `EkuL`, were also resolved. 

After the discussion, all reviewers recommended acceptance (1 clear accept, 3 weak accepts). While efficacy and practical applications are not fully developed, for this type of theory inspired work those should be criteria for higher distinction than acceptance. The AC believes strengths outweigh the weaknesses. The problem studied in the paper is well motivated and provides interesting insights for understanding deep neural networks and, as authors claim, have potential to inspire new methods and new theories of deep learning. The AC recommends acceptance. 

Decision: Accept (Poster)