Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: Summary: 
This paper proposes a general method for eliminating noisy labels in supervised learning based on the combination of two ideas: outputs of noisy examples are less robust under noise, and noisy labels are less likely to have a low loss. The authors then propose 3 concrete instantiations of the idea, and do a thorough empirical study (including ablations) across multiple architectures, datasets, noise types, and comparing to multiple related methods. The results show pretty convincingly that one of the new methods (LTEC) that uses past networks outputs to build an ensemble performs really well.

Caveats: 
1) I’m an emergency reviewer and had less time to do an in-depth review.
2) While my research is sufficiently close to review the paper, I’m not an expert on label noise specifically, so I cannot comment much on novelty and related work questions.

Comments:
* The readability of the paper could be dramatically improved by reporting results visually (eg bar-plots) and moving all tables into the appendix.
* The authors state that peak performance is valid because it *could* have been found using a validation set -- then why not just do that, and report this early-stopping performance everywhere, instead of always two numbers (peak and final)?
* Please make the perturbation properties in section 3.2 more precise: do you mean “there exists a threshold and perturbation such that for some (x, y)”? Or “For any threshold and any perturbation then for all (x, y) it holds...”? Or something in-between?
* For the “competing methods” paragraph, please cite the relevant papers in the main text, not only in the appendix.
* “over 4 runs” did you randomize the data noise in each run (and in the same way for each method?), or only the network initialisation?
* Table 5 is cool, but it raises the question: is 1.1epsilon the best, or would performance keep going up?
* Looking at the actual implementation of LTEC (Algorithm 4), I cannot resist the thought that M=infinity could work even better (at no extra cost): just maintain a monotonically shrinking set of samples?

Minor comments:
- Define the threshold symbol in section 3.2
- Define M in Algorithm 1
- Define (initial) \mathcal{P}_0 in Algorithm 4
- Fig 2: can you clarify what green is, does it correspond to “self-training”? 
- “label precision … does not decrease” -- well, not a lot, but it does decrease!
- Fig 1, Fig 2: set max y to 100
- Table 4: Include M=1 (which is self-training) for comparison
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors proposed to identify noisy training examples using ensemble consensus. The authors argued and demonstrated through numeric studies that, to the contrary of some earlier work, training examples with low training loss are not necessarily mislabeled. Rather, the authors hypothesized that examples with high noise require memorization, which is sensitive to perturbations. Thus, the authors proposed to identify and subsequently remove those examples from training by looking at the loss after small perturbations to the model parameters. Examples with consistently low training loss are retained for training. The authors also provided several alternatives of perturbations, including examining the consensus between an ensemble of networks, between multiple stochastic predictions, or between predictions from prior training epochs. Finally, the authors demonstrated the performance of their procedures using numerical studies.

This paper is well motivated and clearly presented. The idea of identifying noisy examples through ensemble consensus is novel and plausible. The numerical studies are relatively comprehensive and in-depth. I think this is a solid conference paper.

Some questions:
1. The authors proposed to perturb model parameters in order to find noisy training examples. Is there any reason that the authors did not perturb feature values in order to find noisy training examples? I would suppose that memorized examples are sensitive to feature value perturbation.
2. The intersection of small-loss examples in Line 13 of Algorithm 1 could be much smaller than (1 - epsilon / 100) * Bb, and could vary in size throughout training. Are there computationally efficient methods that can guarantee the size of the intersection is stable and not too small? I suppose that we do not want the mini-batch size to vary too much throughout training.
3. How to distinguish hard-to-classify, high-loss examples from high-noise, low-loss examples in Line 11 of Algorithm 1? Using the proposed algorithm, in addition to the noisy low-loss examples, we will also remove those hard-to-classify examples, which is arguably undesirable.
3. The authors should clearly discuss the computational complexity and space complexity of their proposals.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Caveat: I admit that I am not incredibly familiar with this particular research area, so I am probably not the best person to review this paper.  I am not certain what part of the matching and bidding process ended up with this paper assigned to me.

This paper presents a method for filtering out noisy training examples, assuming that there is a known percentage of label noise present in the data.  The gist of the method is to repeatedly perturb the weights in the network slightly, and only accept as training data examples that consistently get small loss across all perturbations.  The intuition is that the loss for noisily labeled examples will not be stable under these perturbations, so using an ensemble of perturbed models should find the label noise.  This intuition seems reasonable enough, and the method seems straightforward.

The reason I am giving a "weak reject" score is largely because the experiments seem weak to me.  It seems pretty unrealistic to randomly corrupt 20% or more of your data.  In what scenario will you actually have data that has 20%+ label noise?  If you actually have one, why not use that to show the effectiveness of your method, instead of an artificial setting?  You have shown that your method achieves better performance in a contrived setting, but in order for it to actually be _useful_, it needs to work on real data, which hasn't been shown.

One example of a real scenario where you might have a very high degree of label noise is in weakly-supervised semantic parsing.  This paper (https://openreview.net/forum?id=ryxjnREFwH), for example, uses confidence thresholding (though without an ensemble) to filter the training data.  It might be fruitful for you to try demonstrating the benefits of your method on this kind of task.

Minor issues:

Section 3.2 presents math suggesting that the perturbations are additive, with deltas drawn from some distribution.  Section 3.3 then presents the actual perturbation methods, which seem difficult to characterize as drawing a delta from a distribution.

Section 4.3 - citations for the baseline methods should be in the main paper, not only in the appendix.

Some things weren't explained well enough for the paper to be self-contained.  For example, the description of LSEC says that there are "stochastic operations", but these are not described anywhere, even with a simple sentence.  Instead the reader must refer to another paper.
review_title: Official Blind Review #1

Meta-review: This paper proposes an ensemble method to identify noisy labels in the training data of supervised learning.  The underlying hypothesis is that examples with label noise require memorization.  The paper proposes methods to identify and remove bad training examples by retaining only the training data that maintains low losses after perturbations to the model parameters.  This idea is developed in several candidate ensemble algorithms.  One of the proposed ensemble methods exceeds the performance of state-of-the-art methods on MNIST, CIFAR-10 and CIFAR-100.

The reviewers found several strengths and a few weaknesses in the paper.  The paper was well motivated and clear.  The proposed solution was novel and plausible.  The experiments were comprehensive.  The reviewers identified several parts of the paper that could be more clear or where more detail could be provided, including a complexity analysis and 
extended experiments.  The author response addressed the reviewer questions directly and also in a revised document.  In the discussion phase, the reviewers were largely satisfied that their concerns were addressed.

This paper should be accepted for publication as the paper presents a clear problem and solution method along with convincing evidence of method's merits.

Decision: Accept (Poster)