Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors of the paper propose a new method, the CORES (COnfidence REgularized Sample Sieve), to tackle the important problem of learning under instance dependent label noise. The proposed method, in essence, involves the use of a confidence regularization term that encourages more confident predictions and a sieving process to remove the samples with large losses. Theoretical justification and empirical experiments were conducted to demonstrate the effectiveness of the proposed method. 

All in all, the paper is clearly written and easy to follow. The proposed method seems technically sound and the motivation for the proposal is explained clearly. One major complaint I have for the paper is the lack of novelty of the paper. The two important building blocks of the paper, the confidence regularizer, and the sample sieve are derived from previous papers. Specifically, in my opinion, the confidence regularizer is a marginal extension of the "peer loss" [1], and the sample sieve algorithm is essentially the same as that proposed in [2], the only difference being a different choice of loss function for training and sieving, to the best of my knowledge and understanding. I think it is worth commenting on this very relevant line of work in Section 2.2. In addition, it would be interesting if the authors of the paper could offer some insights on why the proposed sieving strategy works better than the one previously proposed [2] based on softmax probability. All in all, with the lack of novelty addressed above, I think the submission is marginally below the acceptance threshold.

Other comments:
1. I find the intuitive justification for confidence regularization in Section 2.1 to be quite unconvincing. Specifically, it was stated that "when model overfits to the noise, its predictions often become less confident". From my understanding, this is not necessarily true at all. In fact, it was previously demonstrated that deep NNs can even perfectly overfit to datasets with randomly assigned labels? From this perspective, wouldn't encouraging confidence make the model overfit harder to the noisy labels? I would appreciate if the authors of the paper could provide further insights and intuition on why the introduced confidence regularization improves noise robustness. 


[1] Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In Proceedings of the 37th International Conference on Machine Learning, ICML ’20, 2020.

[2] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems, pp. 8778–8788, 2018.

--------------------------------------------------------------------------------------------------------------
The authors of the paper addressed carefully the concerns I raised above. As such, I am raising my score to a 6, and would like to recommend accepting this paper. 
review_title: Review for "Learning with Instance-Dependent Label Noise: A Sample Sieve Approach"

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary

The paper introduces a noise-robust loss function CORES2, motivated by peer loss. The novel loss adds a regularization term that promotes confident prediction and pushes the model prediction away from the prior of the label. Using this loss function, the authors propose a dynamic sample sieve to separate the clean data and corrupted data on-the-fly, by the magnitude of CORES2 loss. The author's approach is to rule out samples whose losses are larger than an adaptive threshold. Importantly, the process of sieving successfully sieves out corrupted samples, both in a theory of 'better than random guess classifier' and in practice. The authors, then show that the proposed CORES2 can be decoupled under the instance-dependent noise setting. Then CORES2 is proved to be noise-robust, which means CORES2 is equivalent to minimizing the original cross-entropy loss. They also show a principle approach for finding the hyperparameters $\beta$. Further, a consistency loss is adopted after sample sieve on the corrupted samples. The author conducts extensive experiments, including CIFAR10, CIFAR100, and Clothing1M under different settings of noise. CORES2 achieves the SOTA results in all the experiments.

Contributions

i) Proposal of a novel confidence regularized loss for image classification and a novel algorithm that dynamically sieves out corrupted samples basing on their loss. 

ii) The proposed loss is proved to be noise-robust. This theoretical result is valuable since less the instance-based noise is not well studied. A principle way to choose the hyperparameter $\beta$. 

iii) Application to CIFAR10, CIFAR100, and Clothing1M (real-world dataset), with the extensive comparison. 

Issues:

i) The motivation behind the dynamic sample sieve is unclear to me. One hypothesis is that if we set CORES2 as the objective, the model will fit clean samples much faster than corrupted samples at the beginning of training.  Hence the model can sieve the corrupted samples out. However, Fig2 shows that the cross-entropy can also separate clean/corrupted samples at the early training stage. I am curiously about the performance of cross-entropy loss + dynamic sample sieve. The comparison between CORES2 and cross-entropy in Fig2 is unfair. 

Theoretically, the paper only shows that once the model is better than random guess, the dynamic sample sieve will not sieve clean samples out. And the assumption is $ f_{x_n}(y_n) > 1/K$. But $f_{x_n}(\tilde{y}_n) > 1/K$ would also frequently happen, for $y_n\not=\tilde{y}_n$, which means the dynamic sample sieve will also keep these corrupted samples. 
Overall, I think the high-level insight into the sieving process is unclear in the paper.

ii) The training stability of the confidence regularizer: The authors show that confidence regularizer can promote confidence prediction. So some entries in $f(x)$ is near 0, and will make the optimization process unstable since the confidence regularizer -> inf in this case.
There is another concern related to theorem 4. Since the Bayes optimal classifier is $f^*_x[y]=1$, its CORES2 loss would be -inf, which seems problematic.

iii) Selection for $ \beta$, according to theorem 4: theorem 4 provides a principal way for estimation of \beta in different datasets. However, to estimate $ \beta$, we have to estimate the noise transition matrix beforehand. And there may exist some adversary samples that make the rough estimation impossible. In another perspective, we can also give some meaningful upper/lower bound for $\beta$ in the instance-independent noise setting. So it's unclear to me what makes the difference in the instance-dependent setting if we do not go through all the samples but do a rough estimation.

iv) One motivation of Confidence Regularizer is that confident prediction counters the overfitting of noise labels. But if the model capacity is sufficiently large, I think it can both overfit CORES2, and overfit those noise samples. 

Minors: 

a) The assumption shows that CORES2 does not favor the non-diagonal dominant noise rate, which means the diagonal entries in the noise transition matrix T can be smaller than some entries in the same row. Does CORES2 fail in this setting in practice?


Overall, the paper's approach is novel and easy to implement. I am willing to raise my score if the authors can address my issues.

#### EDIT------------

I think the authors replied to some of my concerns in a convincing way, hence I raise the score to 6. 

Unfortunately, I think the theoretical analysis for the noise-robust loss is orthogonal to their sampling sieving approach. And the analysis for choosing $\beta$ does not dependent on their instance-dependent noise settings. We can get the same $\beta$ by very rough estimation(their approach) in instance-independent noise settings. In addition, I guess $f_x^*[y]=1$ is still problematic in the theoretical analysis since CORES2=\inf given the ideal classifier. 

Overall, following author's response, I am leaning towards acceptance, but will let the AC judge the importance of the points above for the final decision.
review_title: Novel noise-robust loss for image classification

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: ***quality***
This paper is quite well-written. The contribution is critical in instance-dependent label noise learning. Moreover, both the theoretical and empirical justifications are convincing.

***clarity***
Although this paper contains heavy mathematics, it is not difficult to understand. I can see that the authors have spent a lot of efforts in paper writing. 

***originality***
In this paper, the authors proposed a novel sample sieve approach for instance-dependent label noise learning. The proposed model is novel, and the theoretical contributions are also new to the community.

***significance***
The proposed algorithm is simple, but the theory behind is rich. I like this kind of work, so I feel that the significance of this paper is high for future research.

***pros and cons***

Pros:
1. The topic is very important for realistic machine learning problems, and is helpful for reducing the human annotation efforts.
2. The theoretical study of this paper is quite impressive.
3. The experimental results show that the proposed method achieves SOTA performance.

Cons:
1. The authors claim that their method does not need to estimate the label transition probability or noise rate, which I think is nice! However, it would be better if the authors can explain why the proposed method can avoid this, namely which “component” helps to avoid the estimation for noise rate?
2. Since this work is an extension of cross-entropy loss to dealing with label noise, I think the comparison with “Symmetric Cross Entropy for Robust Learning with Noisy Labels”(ICCV 2019) is necessary, as this paper also aims to design a robust loss via modifying cross-entropy loss.
3. I feel that the sample sieve/filtering process (i.e. Eqs. 1-4) looks like self-paced learning (SPL) (see “Self-Paced Robust Learning for Leveraging Clean Labels in Noisy Data”, AAAI 20), as SPL also selects some “important” data for training in each iteration. Maybe the authors can discuss the relationship between these two methods? 
4. The authors misuse the terms “sample” and “example”. Statistically, we say that we have a sample X={x_1,x_2,…,x_n} from some distribution, in which every x_i is an example. 
5. Some recent typical works on label noise learning can be cited, such as “Are Anchor Points Really Indispensable in Label-Noise Learning?”(NeurIPS 19) and “A Bi-level Formulation for Label Noise Learning with Spectral Cluster Discovery” (IJCAI 20).

review_title: Solid work! Nice contribution!

Meta-review: Dear Authors,

Thank you very much for your very detailed feedback to the reviewers. They have highly contributed to clarifying some of the concerns raised by the reviewers and improved their understanding of this paper.

Overall, all the reviewers acknowledge the merit of this paper and thus I suggest acceptance of this paper.
However, as Reviewer #4 pointed out, there are conceptual and theoretical issues that need to be more carefully addressed.
Please clarify these issues in the final version of the paper.
Decision: Accept (Poster)