Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the training time dynamics and the hardness of examples. It proposes a new metric to complement existing metrics called second-split forgetting time (SSFT). The paper studies three types of hard examples, including mislabeled examples, rare examples, and complex examples. The paper empirically shows that SSFT and FSLT together can be effective in identifying noisy labels, improving generalization in noisy data settings, identifying failure modes, and being robust to random seeds. In addition, the paper studies SSFT in theory based on a toy example.
strengths_and_weaknesses: Strengths:

1. A new metric for characterizing the example hardness.
2. Studied three types of hard examples and verify the effectiveness of the new metric with comprehensive experimental studies.
3. Well-written paper.

Weakness:

1. The theoretical results are derived from a simplified setting.
questions: This paper proposes a new and interesting metric and the empirical evaluation is comprehensive. Overall, I think this is a solid paper. 
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a new approach to analyse the learning and forgetting of examples in training deep neural networks, working on range of domains from vision to language. This approach shows promise in identifying *hard* examples, and differentiating rare classes mislabelled examples in particular.
strengths_and_weaknesses: The paper is a strong paper, and I am quite fond of it. It has implications for the communities which makes specific use of hard examples, noisy labels and rare classes, and more broadly the community interested in learning dynamics. I do have a number of question and concerns which I've listed in the questions section.

Strengths
- Approach is simple yet effective.
- Well written paper.
- Experiments on a large array of tasks and domains.
- Strong ablations to understand how, why and where this approach works.
- Clear results.
- Sections 4.5 and 4.6 are appreciated; drawing attention to utility of the method(s) and failure modes. This will be useful for practitioners using this. Limitations of the approach are clearly described in failure modes.


Weaknesses
- The approach requires a separate hold-out set to train on to describe the dynamics.
- Retraining is computation and time consuming.
- The description of the synthetic dataset in 4.2 could be more readable, clear and precise. 


Originality
Novel approach to classifying examples.

Quality:
Theoretically grounded paper, with good intuitions and strong results.

Clarity:
Well written paper. It provides a good section of related work, ablations and explanations on method. Figures were clear and useful. 

Significance:
Paper builds on the work of the likes of Toneva et al. classifying hard examples, and has implications for communities interested in e.g. curriculum learning.
questions: - It seems quite natural that neurons firing for classifying a rare class would not be overwritten, given that they are rarely seen. Whereas mislabelled examples (likely more common occurring classes) would overwrite. So the training curves as they are seem intuitively correct. Any comment on this statement? Is there something more complex going on?
- Seems like it would be useful to have a number of synthetic examples to test the above? Say make classes (set a few to be mislabelled only) that are mislabelled as rare as the rare classes.
limitations: The authors have addressed the limitations of their approach, however not in the relation to societal impact. This does not seem necessary here. Although perhaps a statement on how this approach might impact minority groups (classes).
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a new method for quantifying how "hard" training examples are. Previous works considered methods, such as based on the number of times an example is seen before it is classified correctly or the number of times its predicted label is flipped during training. In this work, the authors propose second-split forgetting time (SSFT), which is the time needed to forget a training example if the model is fine-tuned on a different subset of data. The authors argue that SSFT can distinguish between examples that are hard because the labels are noisy vs. examples that are hard because they come from a rare subpopulation. They also discuss other applications of SSFT. Finally, the authors argue that SSFT is robust to the choice of architecture, optimizer, and so on.  
strengths_and_weaknesses: **Strengths**

1- A new method that can offer insight into datasets and can be useful in identifying and correcting noisy examples. 

2- The paper is well-executed and is a pleasure to read. 

3- The authors provide precise mathematical definitions in a simplified setting for the concepts they discuss in the paper; e.g. "rare", "noisy", and "complex" examples. However, these are not general definitions; they are specific to the construction they study analytically (i.e. separable examples with a linear decision boundary).  

**Weaknesses**

1- The experiments are conducted on a few datasets, even though it should be straightforward to conduct some of the experiments on a larger set of datasets, e.g. for the identification of label noise in Table 2. In particular, the authors used CIFAR100 in other experiments  but without reporting CIFAR100 results in Table 2. Since the claims made in the paper are empirical, the experiments should cover many datasets. 

2- The authors claim that SSFT can be used to improve generalization by identifying and removing noisy examples. But, in order for this to be indeed a useful application, the comparison should be against training the model on the *entire* data. In SSFT, the data is split into two subsets and I am not sure if the authors compare the impact on generalization by training on the entire set before and after removing the noisy examples. Please see my question below.

3- As the authors acknowledge in their work, SSFT is sensitive to the choice of the learning rate. The authors claim that for "reasonable" choices, SSFT is robust but there is no evidence of this in the paper. At minimum, the authors can vary the learning rate and report the Pearson correlations (since they used it later to measure stability) in a 2D heat-map with the learning rate on the x and y axis. 
 
4- In some places, the mathematical notation is either imprecise or unclear (to me at least). For example, in Equation 3, $P(x\sim X_g)$ should be $P(x\in X_g)$. In Equation 5, $\rho(t)$ is undefined. Also, when mentioning $O(1)$ in Definition 5.1, this means that the number of rare examples is bounded by a constant *as we vary another parameter*. What is the other parameter? Is it the number of mixtures $N$ or the total number of training examples (in which case $O(1)$ can be a function of $N$)? 

5- The experiments are done on a shallow architecture (ResNet-9). I think this is a big weakness of the work. At minimum, ResNet-18 should be used and (naturally) even deeper architectures would be more preferable. 

**Minor comments**

- I think the authors should use the "number of seen examples" instead of the number of epochs because the dataset can be quite large (near-infinite data regime) in which a single epoch is used. This is especially important here because the model is *fine-tuned* so one would expect it to converge quickly.

- In line 164, the condition $i\neq j$ should be added to the statement $I_i\cap I_j=\emptyset$. 

questions: 1- When comparing the improvement on generalization after removing noisy examples, do you train the model on the entire set of data (both sets used in SSFT joined together) or one subset only? Can you please specify exactly how the comparison is done?

2- Can you please generate a figure that shows that SSFT is indeed robust to changes in the learning rate? Please see my comment above. 

3- What is $\rho(t)$ in Equation 5?

4- ResNet-9 is a small architecture. I believe ResNet-18 should be used at minimum since it's a common baseline model used in the literature. Is there a reason deeper architectures are not used?

5- I find it hard to understand why "complex examples" are defined to be those that have a high signal-to-noise ratio. I would expect the opposite. Is this a typo? If not, can you please explain the intuition behind this definition?

limitations: Please see my comments above for the limitations of the work. I don't see any potential negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a new (combination) of metrics to characterize datapoints in dataset, namely the First-Split Learning Time (FSLT) and the Second Split Forgetting Time (SSFT). These measure respectively for all the datapoints in the dataset used during training how fast they are learned, and how quickly they are forgotten, when retrained on a held out portion of the training set. The paper evaluates/investigates the relationship between FSLT and SSFT with respect to mislabeled samples, rare samples, and complex samples. This is done on multiple modalities (various (altered/syntetic) image datasets and a sentiment classification dataset. These metrics can effectively distinguish between these different examples, based on inspecting the effect on one/both metrics. 

Moreover, the paper also introduces some theoretical results to support their experiments
strengths_and_weaknesses: I particularly like this work, because the metrics introduced are simple (anyone familiar with some Deep Learning will understand them quickly), and are shown to be effective through an extensive set of experiments. In my opinion these types of simple metrics are easy to add to deep learning pipelines and can help investigate model behavior and dataset behavior, and guide the improvement of model training. 

Moreover, the paper is extremely well-written in my opinion, with very few spelling mistakes and good formalism and organisation of the paper. The narrative also reads nicely

The only experiment/addition that I could imagine which might be useful is the following: Since overparametrized deep models might not exhibit such nice curves, due to the effect mentioned in lines 228-292) it would be interesting to see if shallower architectures can be used to improve generalization for deeper models. For example, maybe repeat a similar type of experiment as in Figure 3, but where the data that is removed is chosen based on a shallower architecture, and the accuracy is reported on deeper architectures. However, this comment is not highly critical, so I don't encourage the authors to do this experiment during the rebuttal if there is no time.

Additional experiments could possibly be done on larger models to show the discrepancy with smaller models. However I don't consider this a major weakness.

minor comments: 
- there is a space in front of the colon on line 109
- on line 164, should $\mathcal{I}_i \cup \mathcal{I}_j = \phi$ also have a condition that $i\neq j$?
- Another suggestion is to increase the font inside the figures slightly, as they are a bit difficult to read.
questions: A question I have regarding rare examples: what is the added benefit of using FSLT to find rare examples? Isn't it easy to just see how many datapoints there are per class?
limitations: -
review_title: null

Meta-review: metareview: The paper studies metric to characterize the difficulty of examples in terms of training dynamics: in addition to first-split learning time (FSLT), similar in principle to existing methods, they propose the novel second-split forgetting time (SSFT). There is strong empirical evidence that SSFT allows to discriminate mislabelled and rare examples, and is a robust metric. The paper also proposes a theoretical analysis in a simplified setting.

The reviewers were unanimous: the method is very well written and provides good intuition. FSLT looks simple, and is yet a very efficient method. A very nice job was done by the authors to characterize various informal notions of difficulty ("rare", "mislabeled" and "complex"). A few minor problems were discussed with the reviewers, and carefully addressed by the authors. We therefore all enthusiastically recommend to accept the paper. Some reviewers spontaneously proposed to highlight the paper (oral or spotlight presentation or award) and I support this.
confidence: Certain
Decision: Accept