Characterizing Datapoints via Second-Split Forgetting
Authors: Pratyush Maini
Saurabh Garg
Zachary Chase Lipton
J Zico Kolter
Venue: NeurIPS 2022 Accept (2022)
Keywords: example hardness
memorization
generalization
forgetting
unlearning

Abstract: Researchers investigating example hardness have increasingly focused on the dynamics by which neural networks learn and forget examples throughout training. Popular metrics derived from these dynamics include (i) the epoch at which examples are first correctly classified; (ii) the number of times their predictions flip during training; and (iii) whether their prediction flips if they are held out. However, these metrics do not distinguish among examples that are hard for distinct reasons, such as membership in a rare subpopulation, being mislabeled, or belonging to a complex subpopulation. In this paper, we propose *second-split forgetting time* (SSFT), a complementary metric that tracks the epoch (if any) after which an original training example is forgotten as the network is fine-tuned on a randomly held out partition of the data.  Across multiple benchmark datasets and modalities,  we demonstrate that *mislabeled* examples are forgotten quickly, and seemingly *rare* examples are forgotten comparatively slowly.  By contrast, metrics only considering the first split learning dynamics struggle to differentiate the two.  At large learning rates, SSFT tends to be robust across architectures,  optimizers, and random seeds. From a practical standpoint, the SSFT can (i) help to identify mislabeled samples, the removal of which improves generalization; and (ii) provide insights about failure modes.  Through theoretical analysis addressing overparameterized linear models, we provide insights into how the observed phenomena may arise.

Characterizing Datapoints via Second-Split Forgetting
Pratyush Maini'
Saurabh Garg'
Zachary C. Lipton'
J. Zico Kolter 1,2
Carnegie Mellon University' Bosch Center for A1
(pratyushmaini, zlipton) elipton)@cmm......... @cmu. edu; (sgarg2, zkolter]ecs cmu, edu
arXiv.2210.15051V [cs.LG] 26 Oct 2022
Abstract
Researchers investigating example hardness have increasingly focused on the dy- namics by which neural networks learn and forget examples throughout training. Popular metrics derived from these dynamics include (i) the epoch at which exam- ples are first correctly classified; (ii) the number of times their predictions flip dur- ing training; and (iii) whether their prediction flips they are held out. However, these metrics do not distinguish among examples that are hard for distinct reasons, such as membership in a rare subpopulation, being mislabeled, or belonging to complex subpopulation. In this paper, we propose second-split forgetting time (SSFT), complementary metric that tracks the epoch (if any) after which an origi- nal training example is forgotten the network is fine-tuned out partition of the data. Across multiple benchmark datasets and modalities, we demonstrate that mislabeled examples are forgotten quickly, and seemingly rare examples are forgotten comparatively slowly. By contrast, metrics only consider- ing the first split learning dynamics struggle to differentiate the two. At large learn- ing rates, SSFT tends to be robust across architectures, optimizers, and random seeds. From a practical standpoint, the SSFT can (i) help identify mislabeled samples, the removal of which improves generalization; and (ii) provide insights about failure modes. Through theoretical analysis addressing overparameterized linear models, we provide insights into how the observed phenomena may arise.'
randomly held
Introduction
growing literature has investigated metrics for characterizing the difficulty of training examples, driven by such diverse motivations as (i) deriving insights for how to reconcile the ability of deep neural networks to generalize [30] with their ability to memorize noise [15, 48]; (ii) identifying potentially mislabeled examples; and (iii) identifying notably challenging or rare sub-populations of examples. Some of these efforts have turned towards learning dynamics, with researchers noting that neural networks tend to learn cleanly labeled examples before mislabeled examples [17, 18, 33), and more generally tend to learn simpler patterns sooner- for several intuitive notions of simplicity [19, 35, 43]. Broadly, works in this area tend to characterize examples as belonging either to prototypical groups or memorized exceptions [7, 16, 25). Adapting these intuitions to real datasets, Feldman (15) propose rating the degree to which an example is memorized based on whether its predicted class flips when excluded from the training set. These, and other works [8, 21, 35, 43, 47] have proposed many metrics for characterizing example difficulty with Carlini et al. [7] comparing five such metrics. However, while many of these works distinguish some notion of easy versus hard samples, they seldom (i) offer tools for distinguishing among different types of hard examples; (ii) explain theoretically why these metrics might be useful for distinguishing easy versus hard samples. Moreover, existing metrics tend to give similar scores to examples that are difficult for distinct reasons, e.g, membership in rare, complex, or mislabeled sub- caaa-pouuatton..
Cooe for reproducing our experiments can be found at
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Typical Examples
Complex Examples
Never Forgotten
Forgetting Time
Rare Examples
Mislabeled Examples
Learning Time
Figure 1: Overview of example separation offered by the unified view of learning and forgetting time.
In this paper, we propose to additionally consider a new metric, Second-Split Forgetting Time (SSFT), calculated based on the forgetting dynamics that arise as training examples are forgotten when neural network continues to train on a second, randomly held out data partition. SSFT defined the fine-tuning epoch after which a first-split training example is no longer classified correctly. We find that SSFT identifies mislabeled examples remarkably well but does little to separate out under- versus over-represented subpopulations. Conversely, metrics based on the (first-split) training dynamics are more discriminative for separating these populations but less useful for detecting mislabeled examples. We leverage the complementarity of first- and second-split metrics, showing that by jointly visualizing the two, we can produce a richer characterization of the training examples.
In our experiments, we operationalize several notions of hard examples, namely: (i) mislabeled examples, for which the original label has been flipped a randomly chosen incorrect label; (ii) rare examples, which belong to underrepresented subpopulations; and (iii) complex examples, which belong subpopulations for which the classification task is more difficult (details in Section 3.2). We perform specific ablation studies with datasets complicated by just one type of hard example (Section 4.3), and show how SSFT can help to distinguish among these categories of examples. We observe that during second-split training, neural networks (i) first forget mislabeled examples from the first split; (ii) only slowly begin to forget rare examples (e.g., from underrepresented sub- populations) unique to the first training set; and (iii) do not forget complex examples.
This separation of hard example types has multiple practical applications. First, we can use the method to identify noisy labels: On CIFAR-10 10 with 10% added class noise, SSFT achieves 0.94 AUC for identifying mislabeled samples, while the first-split metrics range AUC between 0.58 0.90. Second, the method can also help improve generalization in noisy data settings: while the removal of hard examples according to first-split learning time degrades the performance of the classifier, the removal of hard examples according to SSFT can actually improve generalization. This is especially beneficial when e.g., training on synthetic data (produced by generative model) or mislabeled data. Third, we show how SSFT can identify failure modes of machine learning models. For example, simplified task classifying between horses and airplanes in the CIFAR-10 dataset, we find that training examples containing horses with sky backgrounds and airplanes with green backgrounds are among the carliest forgotten indicating that the model relies on the background as a spurious feature. Last, we also add that our metric is robust across multiple seeds, and the earliest forgotten examples are robust across architectures. Across multiple optimizers, SSFT distinguishes mislabeled samples, whereas first-split metrics appear more sensitive to the choice of optimizer.
Finally, we investigate second-split dynamics theoretically, analyzing overparametrized linear models [46]. We introduce notions of mislabeled, rare, and complex examples appropriate to this toy model. Our analysis shows that mislabeled examples from the first split are forgotten quickly during second- split training whereas rare examples are not. However, as we train for a long time, rare examples from the first split are eventually forgotten as the model converges to the minimum norm solution on the second split while predictions on complex examples remain accurate with high probability.

---

2 Related Work
Example Hardness. Several recent works quantify example hardness with various training-time
metrics. Many of these metrics are based on first-split learning dynamics [8, 25, 27, 35, 43]. Other works have resorted to properties of deep networks such as compression ability [21] and prediction depth [5]. Carlini et al. [7] study metrics centered around model training such as confidence, ensemble agreement, adversarial robustness, holdout retraining, and accuracy under
training. Closest in spirit to the SSFT studied in our paper are efforts in [7. 47J. Crucially, Carlini et al. [7] study the KL. divergence of the prediction vector after fine-tuning on held-out set at a low learning rate, and do not draw any direct inference of the separation offered by their metric. Focusing on (first- split) forgetting dynamics, Toneva et al. [47] defined metric based on the number of forgetting events during training and identified sets of unforgettable examples that are never misclassified once learned. In our work, we find complementary benefits of analysis based on first- and second-split dynamics.
Memorization of Data Points. In order to capture the memorization ability of deep networks. their ability to memorize noise (or randomly labeled samples) has been studied in recent work [3, 48). As opposed to the memorization of rare examples, the memorization of noisy samples hurts generalization and makes the classifier boundary more complex [15]. On the contrary, recent line of works has argued how memorization of (atypical) data points is important for achieving optimal generalization performance when data is sampled from long-tailed distributions [6. 11, 15].
Simplicity Bias. Another line of work argues that neural networks have bias toward learning simple features [43], and often do not learn complex features even when the complex feature is more predictive of the true label than the simple features. This suggests that models end up memorizing (through noise) the few samples in the dataset that contain the complex feature alone, and utilize the simple feature for correctly predicting the other training examples [1, 32].
Label Noise. Large-scale machine learning datasets are typically labeled with the help of human labelers [12] to facilitate supervised learning. It has been shown that significant fraction of these labels are erroneous in common machine learning datasets [39]. Learning under noisy labels is a long-studied problem [2, 26, 31, 37J. Various recent methods have also attempted to identify label noise [10, 23, 38, 40]. While the focus of our work is not to propose a new method in this long line of work, we show that the view of forgetting time naturally distills out examples with noisy labels. Future work may benefit by augmenting our metric with SOTA methods for label noise identification.
3 Method
The primary goal of our work is to characterize the hardness of different datapoints in a given dataset. Suppose we have dataset SA = yx;. such that yx) ~ D. For the purpose of characterization, we augment each datapoint y.) E SA with parameters (fslt,, ssft;) where fslt; quantifies the first-split learning time (FSLT), and ssft, quantifies the second-split forgetting time (SSFT) of the sample. To obtain these parameters, we next describe our proposed procedure.
Procedure We train model f on S to minimize the empirical risk: f( =E, (f(x)) vt)) We use fA to denote a model ff (initialized with random weights) trained on SA until convergence (100% accuracy on SA). We then train model initialized with FA on a held-out split SB Dn until convergence. We denote this model with fA-BB- To obtain parameters (fslt,, ssft,), ). we track per- example predictions (у.) at the end of every epoch (t") of training. Unless specified otherwise, train the model with cross-entropy loss using Stochastic Gradient Descent (SGD).
Definition 1 (First-Split Learning Time). For Ni) E SA, learning time is defined as the earliest epoch during the training of a classifier f on SA after which it is always classified correctly, i.e.,
fslt: = argmin (9 = y: 222
Ei E
(1)
Definition 2 (Second-Spli Forgetting Time). Let STAAABB) to denote the prediction of sample Yi) E SA after training f(A-B, for t epochs on Sg. Then, yi) E SA forgetting time is defined as the earliest epoch after which it is never classified correctly, i.e.
ssft;- argmii (F((A-B)) # y: Vt 2²) Vix.yi) E SA
(2)
3

---

First-split Training
Second-split Training
3
Akyp Sameim
7
Raze
Misiabbled
0 2
7
9
Complex
3
Feccue Sp4t Eoreetiing Time
7
Typical
Accuracy
0.6
0.6
0.4-
0.4
0.2
0.2
10a
100
10"
50
00
Number of Training Fpochs
(a)
(b)
Figure 2: Rate of Learning and Forgetring of examples for different groups in the synthetic dataset. While first-split training not able distinguish between rare and complex examples, second-spli training succeeds in distinguishing them. Additionally, second-split training separates mislabeled examples from the rest relatively better than first-split training. (b) Visualization of first-split learning and second-split forgetting times when training LeNet model the MNIST dataset.
3.1 Baseline Methods
We provide a brief description metrics for example hardness considered recent comparisons [25].
Number of Forgetting Events: (ny). An example y:) E S undergoes forgetting event when the accuracy on the example decreases between two consecutive updates. Toneva et al. [47] analyzed the total number of such events n during the training of neural network to identify hard examples.
Cumulative Learning Accuracy:
(acct). Jiang al. [25] suggest that rather than using the learning time (Definition 1), using the number of epochs during training when a machine learning model correctly classifies given sample is more stable metrie for predicting example hardness.
Cumulative Learning Confidence: (conf,). Similar to acc;, conf, measures the cumulative softmax confidence of the model towards the correct class over the course of training.
3.2 Example Characterization
We characterize example hardness via three sources of learning difficulty: (i) Mislabeled Examples: We refer to mislabeled examples those datapoints whose label has been flipped label uniformly at random. (fi) Rare Examples: We assume that rare examples belong to sub- populations of the original distribution that have low probability of occurrence. In particular, there exist ((1) examples from such sub- uub-populations in given dataset. In Section 4.3 we describe how we operationalize this notion in the case the CIFAR-ION -00 dataset. (iii) Complex Examples: These constitute samples that are drawn from sub-groups in the dataset that require either (1) a hypothesis class of high complexity; or (2) higher sample complexity be learnt relative to examples from rest of the dataset. We leave the definition of complex samples mathematically imprecise, but with the same intuitive sense prior work [3. 43]. For instance, a dataset composed of the union of MNIST and CIFAR-10 10 images, we would consider the subpopulation of CIFAR-10 images to be more complex.
incorrect
4 Empirical Investigation of First- and Second-Split Training Dynamics
4.1 Experimental Setup
Datasets We show results variety of image classification datasets-MMIsT" [13], CIFAR- 10 [29], and Imagenette [22]. For experiments in the language domain, we use the SST-2 dataset [45]. For each of the datasets, split the training set into two equal partitions (SA, SB). For experiments

---

Sentences in SST-2 datuset with smallest lorgetting time
Label
The director explores all three sides of his story with a sensitivity and inquisitiveness reminisecnt of Trutlaut Beneath the film's 's obvious determination to shock al any cost lies considerable skill and determination backed by sheer nerve This is fragmented film, The holiday message of the 37-minute Santa vs. the Snowman leaves a lot to be desired.
Nog
Neg
good idea that followed the had idlen turn it into movie
Pos
Pos
Epps has neither charisma the natural affahility that has made Tucker The hottom linc is the piece works brillianly
Pos
Neg
Alternative medicine obviously has its merits ... but Ayurveda does the field favors
Pos
What could have casily become cold. calculated exercise in postmodern pastiche winds up powerful and deeply moving example of melodramatic moviemaking
Nog
Lacks depth
Pos
Cortain to be distasteful to children and adults alike, Eight Crazy Nights is a total misfire
Pos
Table I:
First-split sentences that were forgotten by the 3rd epoch of second-split training of BERT- base model on the SST-2 dataset. Notice that all of these forgotten examples are mislabeled.
with mislabeled examples, we simulate mislabeled examples by randomly selecting a subset of 10% examples from both the partitions and changing their label to an incorrect class.
Training Details Unless otherwise specified, we train a ResNet-9 model [4] using SGD optimizer with weight decay 5e-4 and momentum 0.9. We use the cyclic learning rate schedule [44] with peak learning rate of 0.1 at the 10th epoch. We train for a maximum of 100 epochs or until we have epochs of 100% training accuracy. We first train on SΛ, and then using the pre-initialized weights from stage 1, train on Sg with the same learning parameters. All experiments can be performed on a single RTX2080 Ti. Complete hyperparameter details available in Appendix B.1.
4.2
Spectrum for various datasets
Synthetic Dataset We consider data (x, sampled from mixture of multiple distributions Dg s.t.
x E Rd. DD denotes the 9t group and has a sampling frequency of ny Each group D, = =Xy, {y i.e. the true label for all the samples drawn from given group is the same, and the examples in each group are non-overlapping. Each group is parametrized by a set of k & d unique indices I, C [d] such that I, nI, for + j. The discriminative characteristic of each group is the vector ug. such that, uulk = 1 if E Ig else 0 Vi E [d]. Then for any sample (x.y) E S:
P(x E Xg) Tg: x|X29 NNO, o I') + pgg.
For our simulation, we consider a 10 class-clasiification problem, with bg for typical groups, and My = 4 for complex groups (higher signal to noise ratio). For any sample drawn from a rare group, we have (1) samples from that group in the entire dataset (SA USB). Mislabeled samples are only generated from the majority typical groups. In Figure 2a, we show the rate of learning and forgetting of examples from each of these categories. We note that in the second-split training, the mislabeled examples are quickly forgotten, and the complex examples are never forgotten. The rare examples are forgotten slowly. In Section 5 we will theoretically justify the observations in the synthetic dataset and show that the rare examples are expected to be forgotten we train for an infinite time.
Image Domain In Figure 2b, we show representative examples in the four quadrants of the learning- forgetting spectrum. More specifically, we find that the examples forgotten fastest and learned last are mislabeled. And the ones learned early and never forgotten once learned are characteristic simple examples of the MNIST dataset. Examples in the first and third quadrant are seemingly atypical and ambiguous respectively. Similar visualizations for other image datasets can be found in Appendix B.2.
Other Modalities The forgetting and learning dynamics occur broadly across modalities apart from images. We repeat the same problem setup on the SST-2 [45] dataset for sentiment classification. We fine-tune pre-trained BERT-base model [14] successively In Table 1, we provide a list of the earliest forgotten samples when we train a BERT model on the second split of SST-2 dataset. The results suggest that SSFT is able to identify mislabeled samples.
two disjoint splits of the dataset.
4.3 Ablation Experiments
We design specific experimental setups to capture the three notions of hardness as defined Section 3.
5

---

CIPAR-101 Datane with Rarn Pepulattan
MNIRT
CIRANTO vi vililekal
Sccoed 544 Pospetiog Ream
CHARRD deen
18-
30-
GUNASE AUAidi
100
150
203
250
scceplos valggray
1O
16
118
1.S
15
Rergeting
(a) Mislabeled Examples
(b) Complex Examples
(c) Rare Examples
Figure 3: Ablation experiments to distinguish the learning and forgetting dynamics for specific types of hard examples. (a) Mislabeled samples may be learned as slowly as a high fraction of typical samples, but they are forgotten much faster. (b) FSLT distinguishes complex (CIFARIO 10 clean) and simple (MNIST clean) examples, but SSFT does not. On the contrary, FSLT can not distinguish clean and mislabeled examples of CIFARIO, while the SSFT can. (c) FSLT is able to distinguish examples based on the uub-grup frequency, however, SSFT has low correlation with the sub-group frequency.
Mislabeled Examples We sample 10% datapoints from both the first and second split of the CIFAR- 10 dataset, and randomly change their label to an incorrect label. Figure 3a shows the learning- forgetting spectrum for the dataset. In the adjoining density histograms, note that large fraction of the mislabeled and correctly labeled examples are learned at the same time. However, during second- split training, the mislabeled examples are forgotten quickly whereas a large fraction of the clean examples are never forgotten, allowing SSFT to succeed in distinguishing mislabeled samples.
Complex Examples We generate joint dataset that contains the union of both MNIST and CIFAR- 10 examples. This is motivated by work in simplicity bias [43] that argue that neural networks learn simpler features first. We also add 10% labeled noise to each of the datasets in the union to understand the learning and forgetting time relationship of a sample that is complex or mislabeled together. In Figure 3b, we show the FSLT and SSFT for MNIST and CIFAR-10 samples. We note that a high fraction of the CIFAR- 10 (complex) samples learn at the same speed as the mislabeled samples. However, when looking at the SSFT, we able draw strong separation between the mislabeled samples and complex samples. This indicates that the complexity of a sample has low correlation with its tendency be forgotten once learnt, but high correlation with being learned slowly.
The CIFAR-100 [29] dataset is a 100-class classification task. The dataset contains 20 superclasses, each containing 5 subclasses. We create 20-class classification dataset with long tails simulated through the 5 sub-classes within each superclass. More specifically, the number of examples in each subgroup for given superclass is given by (500, 250, 125, 64. 32) respectively (cxponentially decaying with a factor of 2). This is done to simulate the hypothesis of dataset subgroups following Zipf distribution [49] as argued for by Feldman [15]. This dataset is further divided into two equal splits to analyze the serming-forgetting dynamics. In order to remove any other effects of example hardness (either within subgroup, among subgroups), randomize both the chosen subset of examples and the ordering of the majority and minority groups between each superclass, by training the model 20 such random splits and aggregating learning and forgetting statistics over these runs. In Figure 3c, we show a scatter plot for the FSLT and SSFT, colored by the frequency of the group particular example belongs to. We observe that FSLT strongly correlates with the size of the subgroup, whereas the SSFT has very low correlation with the rareness of sample.
Rare Examples
We provide further ablations to show that FSLT is able to identify hard and rare examples, but SSFT shows nearly no discriminative power at finding the two in Appendix C.
4.4 Dataset Cleansing
Identifying Label Noise We present AUC scores for detection of label noise via various popular methods in example difficulty literature, across various datasets Table We note that (i) cumulative predictions over the course of training help stabilize both the learning time and forgetting time metrics;

---

Method
Islt
acc:
ssft (Ours)
acc, (Ours)
conf:
n
Joint (Ours)
Imagenette
0.834
0.912
0,931
0.941
0.786
0.781
0.957
CIFARIO
0.740
0.900
0.938
0,941
0.947
0.580
0.958
MNIST
0,973
0.998
0.997
0.998
0.965
0.377
0.998
CIFARIO0
0,700
0.899
0.865
0.885
0,860
0,300
0.926
EMNIST
0.987
0.990
0.987
0.989
0.984
0.386
0.997
Table 2: AUC for identification of label noise using various metrics for example hardness across different datasets. Across all datasets, ssft metric outperforms alternative baselines. We introduce acc; as the cumulative accuracy on the second-split training. inspired by previous work that suggests using cumulative accuracies helps make first-split learning time more stable [25]. All other notations are described in Section 3. In the case of the Joint method, we select new prediction ranks based on the combined learning and forgetting ranks, further improving over the ssft metric alone.
DCGAN gemerated data
DDPM gemerated data
CIFAR10: Clean
CIFARIO: 10% Nosse
85
81
Random
Accuracy on Test Set
80-
Learn 1ime
83
76
40
76
35
72-
81
68
10'
10
10a
10
10*
102
10
10
102
10a
104
Number Eicappes Removed
Number Examples Removed
(a) Synthetically generated CIFARIO-like data
(b) CIFARIO with and without label noise
Figure 4: Accuracy on CIFAR-IO test set after removing a varying number of samples from the training set based on (i) random choice, (ii) examples with the lowest SSFT, and (ii) the highest FSLT. Removing examples based on SSFT helps improve the generalization on the original test set.
(ii) for simple datasets such as MNIST with few ambiguous images, all of the baseline methods have very high AUC (greater than 0.99) in finding noisy inputs. However, in datasets such as CIFAR-10 and Imagenette, we find that second-split forgetting metrics do better than first-split training metrics. Finally, we also compare the use of both forgetting and learning time find noisy samples, and we find small improvement in the results of just using the forgetting time. While do not make explicit comparisons with other state of art methods dedicated to finding label noise, our results suggest that augmenting second split forgetting time information may help improve their results. As also observed in recent work [25], we find that the number of forgetting events (ny) [47] is an unreliable indicator of mislabeled samples. We hypothesize that this is because of the fact that mislabeled examples may often be (first) learnt very late, hence their count of total forgetting events is also low.
Cleaning synthetically generated datasets Generative models are capable of mimicking the distribution of a given dataset. We generate synthetic datasets of CIFAR 10-like samples using (i) DDPM (denoising diffusion model [20]); and (ii) DCGAN (Deep Convolutional GAN [41]). In
both cases, we assign pscudo-labels using the BiT model [28] as in prior work [36]. We collect a sample of 50,000 training examples and record the generalization performance
CIFAR-I0
remove 'hard' samples, as evaluated by various metrics. In Figure 4, we can see that removing the most easily forgotten examples can benefit by up 10% generalization accuracy on the clean test set of CIFAR- 10. In case the synthetic data generated using DDPM, the gains in generalization performance are under 2%. We hypothesize that this is because the samples generated by DDPM more representative of the typical distribution of CIFAR-10 than those generated by DCGAN.
Note: The ability to train on a second split allows SSFT the unique opportunity to train on a clean split of CIFAR- 10 in order to assess the alignment of the synthetic samples with the oracle samples. As result, the SSFT is much more effective in filtering out ambiguous first-split synthetic examples.
7

---

4.5 Evaluating Example Utility
Recent works [16, 47] have argued for removing a large fraction of the less memorized examples, and keeping the memorized ones. We will analyze the change in model generalization upon removing varying sizes of examples from the training set, as ranked by lowest SSFT and highest FSLT (Figure 4). In the presence of noisy examples, removing samples based on the SSFT helps improve generalization, whereas FSLT does not do much better than random. We draw the following inferences:
ESLT finds important samples As remove more samples from the dataset, the accuracy of the model trained after samples are removed based on the highest FSLT is significantly lower than random guessing. This suggests that the utility of these samples is higher than random samples. Put in line with the hypothesis of memorization of rare example as proposed in [15], we see that empirically, the examples that are slow learn are important for the model's test set generalization.
SSFT removes pathological samples On the contrary, removing examples based on the SSFT helps improve model generalization (especially when there is label noise). Even in the setting when there is no label noise, in contrast to FSLT, we find that removing examples that were easily forgotten has lower negative impact the model's generalization as opposed to removing random samples. This suggests that the examples that are forgotten in the early epochs of second-split training hurt a model's generalization, and may not be characteristic samples of their particular class.
Practitioner's view From the AUC numbers in Table 2, it may appear that removing examples via learning- based metrics such as learning time and cumulative learning accuracy also provides a high rate of removal of noisy samples. However, when we observe the example utility graphs in Figure 4, we draw the inference that the examples that are learned late, are often important examples (such as rare memorized examples). However, even when SSFT fails to capture the correct noisy examples, still removes unimportant samples and does not hurt generalization. Similar graphs for other metrics described in Table can be found in Appendix B.
4.6 Characterizing Potential Failure Modes
Recent works have attempted to train classifiers on datasets that contain spurious features [24, 42] (example Waterbirds, CelebA [34] dataset). However, a fundamental challenge is to first identify the spurious correlation that the classifier may be relying on. Only then can recent methods be trained to remove the reliance on spurious patterns. We train ResNet-9 model to classify CIFAR-10 images of horses and airplanes. In Figure 5, we observe that the model forgets planes with green backgrounds and horses with blue backgrounds. This suggests that the model relied on the background as spurious feature. By analyzing the forgotten examples we can further investigate the examples that the classifier fails to generalize to.
Figure 5: By inspecting the earliest- forgotten examples, can gain in- sights into potential failure modes. This model quickly forgets planes with green backgrounds and horses on blue backgrounds.
Stability of SSFT We note that SSFT is stable across multiple seeds (Pearson correlation of 0.8 1), and across architectures (Pearson correlation of 0.63). While the overall correlation for samples ranked by SSFT may be low across architectures, the top-ranked examples have a high correlation (0.85), suggesting the most forgotten examples are consistent across architectures. In contrast, FSLT has Pearson correlation of 0.52 across seeds. Most interestingly, the learning time metric is brittle to the choice of hyperparameters. As shown by Jiang et al. [25], when using Adam optimizer, examples of different hardness get learned together. In our experiments, we observe the same phenomenon during learning, however, SSFT is robust to the choice of the optimizer. Detailed results in Appendix C.
Limitations One limitation of the proposed metric that is brittle to the choice of the learning rate for the second-split training. If we use very small learning rate, then overparametrized deep models are capable of learning the new dataset without forgetting examples from the first split. Alternately, if we use very large learning rate, the model may diverge and undergo catastrophic forgetting. However, under 'reasonable' choices of learning rate (like that for first-split training), we find SSFT is robust. We provide detailed anaylsis of the same in Appendix C.
8

---

5 Theoretical Results
Through our theoretical analysis, we will characterize the forgetting dynamics of mislabeled, rare and complex examples in a simplified version of the framework used for our synthetic experiments in Figure 2a. Recall, our setup contains two dataset splits SA, SB. where we train on the first split until achieving perfect accuracy on all training points, and then with these weights train Sn for infinite time. In particular, we will prove that both mislabeled and rare examples are forgotten upon training for infinite time, with mislabeled examples being forgotten much faster. Further, will show that complex examples from the first split do not get forgotten if not continually trained on. We assume in our analysis that SB has no mislabeled or rare examples, and SA contains one example of each type.
We consider a dataset S = {x., such that (x). E X x y and x; = M4 + 2; where Z; N II)), and || ky² (as in Section 4.2). Let w R represent the weight vector of an overparametrized linear model. We analyze the learning and forgetting dynamics by minimizing the empirical risk: L(S; w) = =; f(w y;x;), where l is the exponential loss. Following Chatterji and Long [9], we make the following assumptions about the problem setup:
(A.1) The failure probability satisfies 0 S 5 S 1/c,
(A.2) The number of samples satisfies 2 C'log (1/6).
(A.3) The input dimension d 2 C'ax(nn n log(n/6), n2( (o2)) and k=///2 2 (l))))))).
where /=w represents the signal to noise ratio and is a large constant. Now we formalize the notions of rare, mislabeled and complex examples for theoretical analysis.
Definition 1 (Rare Examples, R [15]). Consider dataset S sampled from mixture of distributions (D,,.... Dxy } with frequency (T1,...,,T respectively. Let R. S. S be the set of rare examples. Then, for all (x) E R. if (z) ~ 'Dj. then there are ((1) samples from Dj in S.
Definition 2 (Mislabeled Examples, M). Consider a l class classification problem with y = (1.2....,A)) k). Let M C 8 be the set of mislabeled examples. Then for any (x DD a corre- sponding mislabeled example is given by (x E .M such that y E y (y).
Definition 3 (Complex Examples, C). Let C C 8 be the set of examples sampled from complex distributions. Let ys) E C such that (x,, Yi) ~ ~Dy (complex group), then jgg = us >> 1 where H4 is the coordinate mean for samples drawn from any simple distribution D, (Section 4.2).
Optimization
We perform gradient descent with fixed learning rate 17.
w(1 + 1 w(() (V())))((((())) = w(1) Ee yixi) y:x:
C(w"
Solution dynamics For sufficiently small learning rate 17. and (bounded) starting point w(0), Soudry et al. [46] showed that:
w t = w log + p(t),
(4)
where p(t) is bounded residual term, and is the solution to the hard margin SVM:
W argmin wERd
||w]2 s.t. w y 21
(5)
5.1 First-split Learning
For stage 1, we consider that we train the model for maximum of T epochs (until we achieve 100% accuracy on the first training dataset SA). This means that the learned weight vectors are close to, but have not converged to the max margin solution. The solution at the end of t epochs is given by w (I). At sufficiently large T, we have:
WT(T) WA log T PT(
(6)
WA(T)) yxx 2 v(x, E SA
For binary classification, y' (-1, +1}. The labels reversed for mislabeled examples.

---

5.2 Second-split Forgetting
We initialize the weights for second stage of training with W. (T) from first training stage, and then train on Sn. We provide the formal theorem statement and complete proofs in Appendix A, but provide informal theorem statements and an intuitive proof sketch below:
Theorem (Asymptotic Forgetting (informal)), For sufficiently small learning rate, given datasets SA. SB ~ D After training for T" + oo epochs, the following hold with high probability:
1. Mislabeled and Rare examples from SA are a forgotten.
2. Complex examples from S are not forgotten.
Proof Sketch. Sketch. We use the result from Soudry et al. [46] that for any bounded initialization, when trained on a separable data, the model converges to the same min-norm solution. As a result, we can generalization bounds from Chatterji
ignore the impact of SA at infinite time training. Then, and Long [9] to argue about the accuracy on mislabeled and complex examples. For the case of rare examples, we show that the probability of correct model prediction can be approximated by Gaussian CDF with mean O and (1/ vn) variance.
Theorem (Intermediate- Time Forgetting (informal)) For sufficiently small learning rate, given two datasets SA,Sp D For model initialized with weights, (B(0)= wT(T) and trained for T" f(T) epochs, the following hold with high probability:
1. Mislabeled examples from SA are no longer incorrectly predicted.
2. Rare examples from SA not forgotten.
Proof Sketch. Sg contains examples from the same majority distributions as SA. The mislabeled example also belongs to one of these distributions, but has the opposite label. However, Sg does not have samples from rare groups found in SA. Using representer theorem, we decompose the model updates into weighted sum of each training data point in Sg. Then, we analyze the change in prediction on rare and mislabeled examples, which is a dot product of the weight update with X.n or the other. As a result, the
X,. Per our assumptions, the the mean of each group Mg orthogonal rare example finds negligible coupling with any example in SB, and the variance of its prediction keeps increasing due to the noise term contributed in the model weights from each example in SB- On the contrary, the mislabeled examples have strong coupling with all the examples in its group. Due to its incorrect label, the mean of its predictions moves towards the correct label, with variance increasing at a similar rate. The final step is jointly analyze the rate of change of prediction of both the examples, and find an optimal time 1" when the prediction on the mislabeled example flipped and the rare example still retains its prediction with high probability.
6 Conclusion
While many prior works investigate training time dynamics to characterize the hardness of examples, we enrich this literature with a complementary lens focused on the second-split forgetting time. We demonstrate the potential of SSFT to distinguish among rare, mislabeled, and complex examples; and also show the differences in the example properties captured by first-split and second-spli metrics.
Our work opens new lines of inquiry in future work that can utilize the separation of hard examples. First, we expect state of art methods label noise identification to benefit by augmenting our approach. Further, we believe our ablations showing that complex, noisy, and mislabeled samples may all be learned slowly inspire future work that can unite different takes on the memorization- generalization research- learning, simplicity bias, and singleton memorization.
Acknowledgements
We would like to thank Aakash Lahoti and Jeremy Cohen for their insightful comments on this work. SG acknowledges Amazon Graduate Fellowship and JP Morgan PhD Fellowship for their support. ZL. acknowledges Amazon AI, Salesforce Research, Facebook, UPMC, Abridge, the PwC Center, the Block Center, the Center for Machine Learning and Health, and the CMU Software Engineering Institute (SED) via Department of Defense contract 1--772-11----000 for their generous support.
10

---

References
[1] Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint S0Ki::20/2.09816. 2020.
[2] Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(4):343-
370, 1988.
[3] Devansh Arpit, Stanistaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj. Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization deep networks. In International conference on machine learning.
pages 233-242. PMLR, 2017.
[4] Woonhyuk Baek. Torchskeleton. https://g/t thub. comm/bbaek//orrccskellettt. 2019.
[5] Robert John Nicholas Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL https: //openreview.. net/ htt/fformmmiii www.:ridduuunnimma..... d-wWda-HHH......
[6] Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memo- rization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Anmual ACM SIGACT Symposium on Theory of Computing, pages 123-132, 2021.
[7] Nicholas Carlini, Úlfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. ArXiv, abs/1990..3322,,
[8] Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization in gradient descent-based optimization. arXiv preprint 206.5 2020.
[9] Niladri S Chatterji and Philip M Long. Finite-sample analysis of interpolating linear classifiers in the overparameterized regime. J. Mach. Learn. Res., 22:129-1, 2021.
[10] Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In International Conference on Machine Learning, pages 1062-1070. PMLR, 2019.
[11] Chen Cheng, John Duchi, and Rohith Kuditipudi. Memorize to generalize: on the necessity of interpolation in high dimensional linear regression. arXiv preprint axXiv:220..09889, 2022.
[12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. Iece, 2009.
[13] Li Deng. The mnist datahase of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6)111-1422 2012.
[14] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1XXiv:/8/0.440 2018.
[15] Vitaly Feldman. Does learning require memorization? a short tale about a long tail. Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing. 2020.
[16] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33: 2881-2891, 2020.
[17] Jonathan Frankle, David J Schwab, and Ari S Morcos. The early phase of neural network training. arXiv preprint arXiv:2002.10365, 2020.
[18] Saurabh Garg, Sivaraman Balakrishnan, Zico Kolter, and Zachary Lipton. RATT: Leveraging unlabeled data to guarantee generalization. arXiv preprint arXiv:2 10:::0005 2021. URL. https:/ /arxivv.v.. org/ aba///bb/..../////////////////// 00303.

---

[19] Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural networks share classification order on real datasets. In International Conference on Machine Learning. pages 3950-3960. PMLR, 2020.
[20] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[21] Sara Hooker, Aaron Courville, Gregory Clark, Yann Dauphin, and Andrea Frome. What do compressed deep neural networks forget? arXiv preprint arXiv:1911.052:5 2019.
[22] Jeremy Howard. Imagenette. URL https: httpp::/ggitub..com/tttttt/e/ste http////attaiiiimmmaaa http:///ttttttaaaaammmtaete////.
[23] Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. O2u-net: A simple noisy label detection approach for deep neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3326-3334, 2019.
[24] Badr Youbi Idrissi, Martin Arjovsky, Mohammad Pezeshki, and David Lopez-Paz. Simple data balancing achieves competitive worst-group-acccuaayy arXiv preprint arXiv:21 14500 2021.
[25] Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206, 2020.
[26] Ishan Jindal, Matthew Nokleby, and Xuewen Chen. Learning deep networks from noisy labels with dropout regularization. In 2016 IEEE 16th International Conference on Data Mining (ICDM), pages 967-972. IEEE, 2016.
[27] Gal Kaplun, Nikhil Ghosh, Saurabh Garg, Boaz Barak, and Preetum Nakkiran. Deconstructing distributions: A pointwise framework of learning, arXiv preprint arXiv:2202.09933: 2022.
[28] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pages 491-507. Springer, 2020.
[29] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Citeseer, 2009.
[30] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[31] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020. URL hot..tttpss//ppereiiiie wwwpearwwwiweeeeetfeeeeeeeeeeeee www..ee..oormmmii-llllaaaaaa
[32] Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems, 32, 2019.
[33] Sheng Liu, Jonathan Niles-Weed, Narges Razavian, and Carlos Fermandez-Grant Granda. Early- learning regularization prevents memorization of noisy labels. Advances in neural information processing systems, 13:20331-20342, 2020.
[34] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, pages 3730-3738. IEER Computer Society, 2015. ISBN 978-1-4673-8891-- URL http://dblp.... uni ttrierrrr.. de/ hb//bb/cc/a wwt.///oo//cc///eeccceeeoee////e
htt1#LLLLWT11
[35] Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow learnable examples first? In ICML 2019 Workshop Identifying and Understanding Deep Learning Phenomena, 2019. URL /tttttps://ppeeeeeee
[36] Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and Iyya Sutskever. Deep double descent: Where bigger models and more data hurt, Journal of Statistical Mechanics: Theory and Experiment, 2021(12):124003 1021.
12

---

[37] Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. Advances neural information processing systems, 26, 2013.
[38] Curtis Northcutt, Lu Jiang, and Isaac Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 70:1373-1411, 2021.
[39] Curtis G Northeutt, Anish Athalye, and Jonas Mueller. Pervasive label errors in test sets destabilize machine learning benchmarks. arXiv preprint woXiv:2103.14749, 2021.
[40] Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. Identifying mislaheled data using the area under the margin ranking. Advances in Neural Information Processing Systems, 33:17044-17088 2020.
[41] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 1,11.66434 2015.
[42] Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint 191.0091 2019.
[43] Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33:9573-9585, 2020.
[44] Leslie N Smith. A disciplined approach to neural network hyper-parameters: Part l-learning rate, batch size, momentum, and weight decay. arXiv preprint arXiv:/803.00 09820, 2018.
[45] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages -661-, Seattle, Washington, USA, October 2013. Association for Computational Linguistics. URL https: : /www. aclweb. owg.wwwwbbrrraaee... www.wooggg.gggoooooooooooooooo/o DD13/11730111
[46] Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent separable data. The Journal of Machine Learning Research, 19(1):2222-288 2878, 2018
[47] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05133 2018.
[48] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3): 107-115, 2021.
[49] George Kingsley Zipf. The psycho-biology of language: An introduction dynamic philology. Routledge, 2013.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work attempts understand the phenomenon of training and forgetting of samples in deep learning architectures. We do not see any direct concerns with ethical or social impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes] In the Appendix of the paper.
(b) Did you include complete proofs of all theoretical results? [Yes] : In the Appendix of the paper.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]: In the Appendix
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Main paper and Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]: For experiments on generalization error after example removal.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We use standard publicly available
benchmark datasets.
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes] Code for Reproducing our experiments is provided in the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We use standard publicly available benchmark
datasets.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "d8Lw3niaRCv", "reviewer": "Reviewer_bwAK", "summary": "This paper studies the training time dynamics and the hardness of examples. It proposes a new metric to complement existing metrics called second-split forgetting time (SSFT). The paper studies three types of hard examples, including mislabeled examples, rare examples, and complex examples. The paper empirically shows that SSFT and FSLT together can be effective in identifying noisy labels, improving generalization in noisy data settings, identifying failure modes, and being robust to random seeds. In addition, the paper studies SSFT in theory based on a toy example.", "questions": "This paper proposes a new and interesting metric and the empirical evaluation is comprehensive. Overall, I think this is a solid paper. ", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. A new metric for characterizing the example hardness.\n2. Studied three types of hard examples and verify the effectiveness of the new metric with comprehensive experimental studies.\n3. Well-written paper.\n\nWeakness:\n\n1. The theoretical results are derived from a simplified setting.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ULKah_gL6DQ", "reviewer": "Reviewer_nsE6", "summary": "The paper proposes a new approach to analyse the learning and forgetting of examples in training deep neural networks, working on range of domains from vision to language. This approach shows promise in identifying *hard* examples, and differentiating rare classes mislabelled examples in particular.", "questions": "- It seems quite natural that neurons firing for classifying a rare class would not be overwritten, given that they are rarely seen. Whereas mislabelled examples (likely more common occurring classes) would overwrite. So the training curves as they are seem intuitively correct. Any comment on this statement? Is there something more complex going on?\n- Seems like it would be useful to have a number of synthetic examples to test the above? Say make classes (set a few to be mislabelled only) that are mislabelled as rare as the rare classes.", "limitations": "The authors have addressed the limitations of their approach, however not in the relation to societal impact. This does not seem necessary here. Although perhaps a statement on how this approach might impact minority groups (classes).", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is a strong paper, and I am quite fond of it. It has implications for the communities which makes specific use of hard examples, noisy labels and rare classes, and more broadly the community interested in learning dynamics. I do have a number of question and concerns which I've listed in the questions section.\n\nStrengths\n- Approach is simple yet effective.\n- Well written paper.\n- Experiments on a large array of tasks and domains.\n- Strong ablations to understand how, why and where this approach works.\n- Clear results.\n- Sections 4.5 and 4.6 are appreciated; drawing attention to utility of the method(s) and failure modes. This will be useful for practitioners using this. Limitations of the approach are clearly described in failure modes.\n\n\nWeaknesses\n- The approach requires a separate hold-out set to train on to describe the dynamics.\n- Retraining is computation and time consuming.\n- The description of the synthetic dataset in 4.2 could be more readable, clear and precise. \n\n\nOriginality\nNovel approach to classifying examples.\n\nQuality:\nTheoretically grounded paper, with good intuitions and strong results.\n\nClarity:\nWell written paper. It provides a good section of related work, ablations and explanations on method. Figures were clear and useful. \n\nSignificance:\nPaper builds on the work of the likes of Toneva et al. classifying hard examples, and has implications for communities interested in e.g. curriculum learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I501BTXLvFU", "reviewer": "Reviewer_AmTF", "summary": "This paper proposes a new (combination) of metrics to characterize datapoints in dataset, namely the First-Split Learning Time (FSLT) and the Second Split Forgetting Time (SSFT). These measure respectively for all the datapoints in the dataset used during training how fast they are learned, and how quickly they are forgotten, when retrained on a held out portion of the training set. The paper evaluates/investigates the relationship between FSLT and SSFT with respect to mislabeled samples, rare samples, and complex samples. This is done on multiple modalities (various (altered/syntetic) image datasets and a sentiment classification dataset. These metrics can effectively distinguish between these different examples, based on inspecting the effect on one/both metrics. \n\nMoreover, the paper also introduces some theoretical results to support their experiments", "questions": "A question I have regarding rare examples: what is the added benefit of using FSLT to find rare examples? Isn't it easy to just see how many datapoints there are per class?", "limitations": "-", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I particularly like this work, because the metrics introduced are simple (anyone familiar with some Deep Learning will understand them quickly), and are shown to be effective through an extensive set of experiments. In my opinion these types of simple metrics are easy to add to deep learning pipelines and can help investigate model behavior and dataset behavior, and guide the improvement of model training. \n\nMoreover, the paper is extremely well-written in my opinion, with very few spelling mistakes and good formalism and organisation of the paper. The narrative also reads nicely\n\nThe only experiment/addition that I could imagine which might be useful is the following: Since overparametrized deep models might not exhibit such nice curves, due to the effect mentioned in lines 228-292) it would be interesting to see if shallower architectures can be used to improve generalization for deeper models. For example, maybe repeat a similar type of experiment as in Figure 3, but where the data that is removed is chosen based on a shallower architecture, and the accuracy is reported on deeper architectures. However, this comment is not highly critical, so I don't encourage the authors to do this experiment during the rebuttal if there is no time.\n\nAdditional experiments could possibly be done on larger models to show the discrepancy with smaller models. However I don't consider this a major weakness.\n\nminor comments: \n- there is a space in front of the colon on line 109\n- on line 164, should $\\mathcal{I}_i \\cup \\mathcal{I}_j = \\phi$ also have a condition that $i\\neq j$?\n- Another suggestion is to increase the font inside the figures slightly, as they are a bit difficult to read.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5HFCMHdKIOZ", "reviewer": "Reviewer_9TS9", "summary": "The authors propose a new method for quantifying how \"hard\" training examples are. Previous works considered methods, such as based on the number of times an example is seen before it is classified correctly or the number of times its predicted label is flipped during training. In this work, the authors propose second-split forgetting time (SSFT), which is the time needed to forget a training example if the model is fine-tuned on a different subset of data. The authors argue that SSFT can distinguish between examples that are hard because the labels are noisy vs. examples that are hard because they come from a rare subpopulation. They also discuss other applications of SSFT. Finally, the authors argue that SSFT is robust to the choice of architecture, optimizer, and so on.  ", "questions": "1- When comparing the improvement on generalization after removing noisy examples, do you train the model on the entire set of data (both sets used in SSFT joined together) or one subset only? Can you please specify exactly how the comparison is done?\n\n2- Can you please generate a figure that shows that SSFT is indeed robust to changes in the learning rate? Please see my comment above. \n\n3- What is $\\rho(t)$ in Equation 5?\n\n4- ResNet-9 is a small architecture. I believe ResNet-18 should be used at minimum since it's a common baseline model used in the literature. Is there a reason deeper architectures are not used?\n\n5- I find it hard to understand why \"complex examples\" are defined to be those that have a high signal-to-noise ratio. I would expect the opposite. Is this a typo? If not, can you please explain the intuition behind this definition?\n", "limitations": "Please see my comments above for the limitations of the work. I don't see any potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1- A new method that can offer insight into datasets and can be useful in identifying and correcting noisy examples. \n\n2- The paper is well-executed and is a pleasure to read. \n\n3- The authors provide precise mathematical definitions in a simplified setting for the concepts they discuss in the paper; e.g. \"rare\", \"noisy\", and \"complex\" examples. However, these are not general definitions; they are specific to the construction they study analytically (i.e. separable examples with a linear decision boundary).  \n\n**Weaknesses**\n\n1- The experiments are conducted on a few datasets, even though it should be straightforward to conduct some of the experiments on a larger set of datasets, e.g. for the identification of label noise in Table 2. In particular, the authors used CIFAR100 in other experiments  but without reporting CIFAR100 results in Table 2. Since the claims made in the paper are empirical, the experiments should cover many datasets. \n\n2- The authors claim that SSFT can be used to improve generalization by identifying and removing noisy examples. But, in order for this to be indeed a useful application, the comparison should be against training the model on the *entire* data. In SSFT, the data is split into two subsets and I am not sure if the authors compare the impact on generalization by training on the entire set before and after removing the noisy examples. Please see my question below.\n\n3- As the authors acknowledge in their work, SSFT is sensitive to the choice of the learning rate. The authors claim that for \"reasonable\" choices, SSFT is robust but there is no evidence of this in the paper. At minimum, the authors can vary the learning rate and report the Pearson correlations (since they used it later to measure stability) in a 2D heat-map with the learning rate on the x and y axis. \n \n4- In some places, the mathematical notation is either imprecise or unclear (to me at least). For example, in Equation 3, $P(x\\sim X_g)$ should be $P(x\\in X_g)$. In Equation 5, $\\rho(t)$ is undefined. Also, when mentioning $O(1)$ in Definition 5.1, this means that the number of rare examples is bounded by a constant *as we vary another parameter*. What is the other parameter? Is it the number of mixtures $N$ or the total number of training examples (in which case $O(1)$ can be a function of $N$)? \n\n5- The experiments are done on a shallow architecture (ResNet-9). I think this is a big weakness of the work. At minimum, ResNet-18 should be used and (naturally) even deeper architectures would be more preferable. \n\n**Minor comments**\n\n- I think the authors should use the \"number of seen examples\" instead of the number of epochs because the dataset can be quite large (near-infinite data regime) in which a single epoch is used. This is especially important here because the model is *fine-tuned* so one would expect it to converge quickly.\n\n- In line 164, the condition $i\\neq j$ should be added to the statement $I_i\\cap I_j=\\emptyset$. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept