Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary:
The paper proposed a self-ensemble label filtering (SELF) method to deal with the noisy label learning problem. They progressively filter out the wrong labels during training, i.e.,  filtered samples are removed entirely from the supervised
training loss, and are leveraged via semi-supervised learning in the unsupervised loss. The filtering is based on identification of inconsistent predictions throughout training. 

Strengths:
1. The motivation of the paper is very clear. 
2. Experiments are conducted on various dataset CIFAR10, CIFAR-100 and ImageNet. 

Weakness:
1. The contribution of SELF is not clear. Just a combining of several previously proposed components?   
2. For the experimental comparisons, the authors at least should report the acc on clean test set, which is useful for understanding the ideal case performance. 
3. The organization of the tables and figures are somehow hard to read. 
3. The comparisons are not fair. SELF incorporate semi-supervised techniques while baselines are not. 
4. The author missed some important baselines here. 
     1) Symmetric cross entropy for robust learning with noisy labels, ICCV2019 
     2) Joint Optimization Framework for Learning with Noisy Labels, CVPR2018 
     3) Dimensionality-driven learning with noisy labels, ICML2018
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: --- Overall ---

This paper proposes an algorithm for learning from data with noisy labels which alternates between updating the model and removing samples that look like they have noisy labels, thereby allowing the training procedure to focus on clean samples. Overall, I found the paper very well-written, the proposed approach reasonable, and the experiments convincing. I have some questions about what assumptions are required for such a procedure to work, but in general, I think this is a strong paper.

--- Major comments ---

1. I found it somewhat unclear how large the methodological contribution was. In particular, has the approach of filtering out samples based on disagreement with predictions from the model been tried before (i.e. the primary contribution is self-ensembling)? When the proposed method includes multiple pieces (Mean teacher + iteratively creating a filtered dataset + self-ensembling), I recommend being *very* explicit about which parts are new contributions. With that said, I greatly appreciated the ablation experiments which really highlight the importance of each piece.

2. I don't feel like I have a good sense for what assumptions need to be satisfied for the proposed method to work. For example, in section 2.2 the authors say "If the noise is sufficiently random, the set of correct labels will be representative to achieve high model performance". What is meant by "sufficiently random" here? Is there a formal version of this assumption? Do any independence or positivity assumptions need to be satisfied? Most importantly, what happens when the label noise does not look like what you expect? I would love to see some experiments examining the failure modes of the algorithm. For example, what happens when label errors are concentrated in a particular region of the feature space (or just generally depend on the features)? In this case, even if the filtering procedure work perfectly, the filtered dataset will have a different feature distribution than the data distribution leading to potential covariate shift problems. If I understood the experiments correctly, the method was only tested on label-depended noise models.

3. Along the same lines: what are the necessary conditions to guarantee that this procedure converges? While the authors suggest that self-ensembling prevents samples from oscillating in and out of training set, is this a guarantee or an empirical observation? More broadly, it is not totally clear what the filtering does to the objective function or whether this procedure is even formally optimizing a well specified objective function (potentially some temperature limit of a soft-weighted objective?). 

--- Minor comments ---

1. Figures 1 and 4 are not readable in black and grey-scale.

2. I would front-load the justification for using self-ensembling. In particular, I think the two sentences starting with "When learning under label noise,..." on page 2 could be moved much earlier. 

3. I'll be interested to see what the other reviewers say, but I found Figure 2 hard to follow.

4. The formatting of Section 4.2.4 makes it a bit hard to figure out where the text starts (as opposed to the table captions).
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposed "self-ensemble label filtering" for learning with noisy labels where the label noise is instance-independent (in fact, the noise model is the class-conditional noise). Among the existing directions in this area, it falls into the sample selection direction, but it also takes semi-supervised learning based on the likely noisy data into account.

Novelty: borderline. As other sample selection methods, the proposed one would like to identify the training data with correct labels. What's new is that the authors "form running averages of predictions over the entire training dataset using the network output at different training epochs" and show that "these ensemble estimates yield more accurate identification of inconsistent predictions throughout training than the single estimates of the network at the most recent training epoch". This is the major contribution of the paper. Furthermore, the data likely to have incorrect labels are not thrown away but used in a semi-supervised manner. This is a minor contribution, because semi-supervised learning is orthogonal to label-noise learning and everybody in this area knows the combination of them can work better in practice. Note that this is an academic/scientific paper, not an industrial product, so you don't need to combine all things that might work.

Significance: high. The proposed method significantly outperformed all baseline methods. However, it's not completely fair to compare a label-noise + semi-supervised method with other label-noise only methods... As a matter of fact, you don't need to apply perturbation consistency (or other semi-supervised) regularization after identifying the training data with incorrect labels. Semi-supervised regularization such as virtual adversarial training can even improve supervised learning.

Issues: It's known under class-conditional noise model, the backward loss correction is the unique way to estimate the classification risk (or equivalently, the classification accuracy) given noisy validation data. So how can the validation (i.e., hyperparameter tuning) be performed for the proposed and baseline methods in Table 1 given noisy validation data? 
review_title: Official Blind Review #1

Meta-review: The authors addressed the issues raised by the reviewers; I suggest to accept this paper.
Decision: Accept (Poster)