Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends the idea of Flooding to a sample-specific level and call it iFlooding. The extension is intuitively important and the authors also offered several analytical discussions to show its importance beyond the intuition. The empirical results are fairly relevant and strong. 
main_review: **Strength**

The paper poses several interesting contributions in terms of training of models while limiting the models' overly attention to a subset of samples. 
 - The idea of extending Flooding to iFlooding is very intuitive, thus it's fairly clear that such extension is needed. 
 - Some additional discussions on the properties iFlooding in terms of robustness to noise labels and biased samples are offered. 

**Weakness**

On the other hand, I also have several concerns about the presentation of the idea, especially on the empirical end
 - It's not clear that in the experiments, whether each data are shuffled when every iteration ends. Intuitively speaking, SGD (and its variants) has a natural counter to overly attending specific samples as the gradient are calculated every batch and shuffling samples at every iteration will avoid the batch to stay the same, thus the gradient will always be different (even if the weights are the same), avoiding some overly attending to specific samples. Therefore, the lack specification of this setup posts a challenge to appreciate the effects of the empirical results. 

 - Related, it has been shown that when the model is being trained, the model pays attention to different levels of details of the data as the iteration goes (Figures 4, 5 in [1]), therefore, instead of presenting a fixed table at the end of all the iterations, showing the curves of accuracies along the training process will help better appreciate the idea of iFlooding. It might also be interesting to present accuries on different test copies as in (Figures 4, 5 in [1]). 

 - If I understand correctly, Table 1 is a comparison of generalization (accuracy) improvement ability of methods, instead of a comparison of the methods devoted to calibrating the overly attention, thus conventional generalization boosting methods such as dropout, batchnorm, or even mixup should probably also been compared. I don't think the authors need to worry whether BatchNorm will deliver a higher accuracy than what their method gets, since these methods are motivated by different problems. The authors could always discuss these, or present the numbers in different cells. However, my point is that for a table that serves as a comparison of accuracy-boosting techniques, it's probably not a great idea that these conventional methods are not compared. 

 - I do not fully follow why Sections 4.3 and 4.4 are devoted only to iFlooding and Flooding, why it is not interesting to compare other methods? Even if these methods do not explicitly account for stability, it's still interesting to see the curves, especially since it seems Figure 2(a)(b) can be computed simultaneously for the experiments used to generate Table 1 with no additional computing resources needed. 

I might have asked for too many empirical results, which may not be doable in the rebuttal phase, but I will suggest the authors at least touch each of these with the CIFAR10 experiment. 

[1] High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks 
summary_of_the_review: I think the paper is very interesting, intuitive, and a natural extension of the Flooding idea. The empirical section can be improved though.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a flooding loss function that encourages the training loss for each example to be a positive bias instead of zero. The paper has found that such training objective stabilizes the training compared to the regular Flooding, which regularizes the average loss instead. The paper also provides reasoning why such a loss function provides more robustness for training with noisy labels and biased label distributions. The empirical benefit on standard datasets seems marginal, but the benefit on noisy labels is significant.
main_review: Strengths:
- Idea is simple and novel.
- The empirical improvement compared to standard flooding is encouraging.
- Experiments on noisy labels show significant improvement.

Weaknesses:
- The formulation of noisy labels and biased sample distribution is unclear. The current formulation of biased sample seems like another form of incorrect labels, where P(z|x) and P(y|x) are different. These two forms of noisy labels could potentially be merged in a unified setting. On the other hand, I am wondering whether distribution shift in the input distribution P(x) could be covered as well?
- I believe the benefit of noisy labels and biased samples should be further investigated. Currently only 2 out of the 7 datasets ran a simple noisy label experiment. There is no empirical evaluation of biased samples since the noisy labels are just uniform (based on the description in Appendix E).
- It’s unclear whether the stated theoretical properties applies only to iFlood or both Flood and iFlood and I would appreciate it if the authors can clarify the relations here.
- On standard benchmarks the improvement seems marginal. This wouldn’t be a major problem if the other properties such as stability and noisy labels can be highlighted more in the experiments.
- Regarding to the claims around stability, the paper uses several measures to look at the loss distribution and the gradient norm. However, there isn't strong and direct evidence that Flooding leads to "unstable" training, i.e. the training loss diverges. This argument could be strengthened more.
summary_of_the_review: This paper presents an interesting loss function that provides more robustness and also performs slightly better in regular settings. I found the paper interesting and potentially have a strong impact. I think the experimental design and theoretical sections could be strengthened more and highlight more benefits dealing with noisy labels and dataset biases. My overall rating is “weak accept”.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduced a simple but effective method to mitigate overfitting, which modifies the existing Flooding scheme. The proposed iFlood can solve the potential problems of the Flooding algorithm and improve the stability. Furthermore, the authors gave some theoretical analyses of the proposed method. The experiments indicated the proposed method is better than the baselines.

main_review: This paper is well organized and easy to read. Although the method is simple, it has a specific theoretical basis. 
My primary concerns are listed as follows:

1. In Section 3.1, the authors gave some analyses and concluded that ''iFlood encourages the model to better ﬁt the under-ﬁtted instances while suppressing the conﬁdence of over-ﬁtted ones.'' However, according to the analysis, it is difficult to determine how this conclusion was reached. Since, sometimes, the individual loss cannot reflect the model confidence. I think more experiments about model calibration are needed to support this claim. 

2.  In Eq.7, why do the authors use the Laplace distribution to simulate the loss function? 

3. For Table 2, the authors provided the results of variances. I understand that the authors evaluate ten models trained with different methods. I would like to know whether these results were obtained under fixed random seeds? Or is the random seed selected randomly for each experiment evaluation?

4. In Figures 2 (a) and (b), I noticed some abrupt changes in the learning curve, especially near the second drop of the learning rate. What is the reason for this phenomenon?

5. I think the experiments of label noise are not very good compared to current methods (like self-adaptive training [1]). 

[1] Self-Adaptive Training: beyond Empirical Risk Minimization, NeurIPS 2020. 




summary_of_the_review: I think the proposed iFlood is simple but effective. The theoretical analysis is straightforward. However, there are still some problems, and the experimental results are not good. So, I recommend to ''marginally below the acceptance threshold''.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a stable and effective regularizer to prevent overfitting. Specifically, this paper proposes individual Flood. Different from Flood which constrains training loss on mini-batch level, iFlood gives instance-level constraints on training loss. This paper also theoretically shows that the design of iFlood can be intrinsically connected with removing the noise or bias in training data. 
main_review: Strengths:
+ This paper is well written and it is enjoyable to read.
+ The idea of iFlood is neat. 
+ It is theoretically showed that the design of iFlood can be intrinsically connected with removing the noise or bias in training data. 

My main concerns are on the effectiveness of iFlood:
+ There are many regularizations which can prevent overfitting. I think more powerful baselines should be added (e.g. mix up).
+ The improvements are very marginal. This makes me very doubt whether iFlood is really effective. 
+ The performance of baseline in GLUE is not convincing (SST-2 and MNLI). It is shown in https://github.com/huggingface/transformers/blob/master/examples/pytorch/text-classification/README.md that sst-2 can achieve 93.2 and mnli can achieve 83.9. 

summary_of_the_review: I think the idea of iFlood is neat and simple. Theoretical analysis is good enough. However, the effectiveness of iFlood can not be guaranteed in experiments. So I think this paper is marginally below the acceptance threshold.
review_title: null

Meta-review: The paper extends the original work on flooding to individual instance level to prevent overfitting. Even though the technique is a intuitive extension, the reviewers appreciate its simplicity and effectiveness, and consider the extension necessary. Most reviewers' concerns were addressed through rebuttal.
Decision: Accept (Poster)