Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper focuses on the topic of learning from noisy -- or as they call it "corrupted" -- labels. Specifically this focuses on an approach where data selection -- ideally of cleaner/less noisy examples --  can help the learn model overcome data noise, akin to the approaches this builds upon (i.e, the Co-Teaching and MentorNet approaches). The specific idea here is to take an AutoML style approach to the problem  in particular to determine how many examples are selected in each mini-batch. The proposed method is based upon natural gradient based updates to the hparams (which was really the only feasible way to tackle this problem given the complex dependence on the hparams and a good choice). The experimental results using synthetic noise corruption are indicative of improved performance compared to the baseline techniques.

Overall while I thought the paper made for a very interesting read and showed some great promise I had some significant concerns as well.

On the plus side:

+ The empirical results on the simulated noisy data are quite positive/
+ The proposed method makes sense as does the search algorithm in the hparam space.

My main concerns with the work stem from the empirical study and choices made there. While I understand that other existing techniques like the Co-teaching and MentorNet approaches have used simulated noise to study the impact of performance of these robustness techniques, at some point I question their validity on real datasets. Noise patterns in real datasets hardly follow some set pattern and thus I hesitate to read much into results derived solely on synthetic datasets. Given that the goal of these techniques is to improve performance when training with real, noisy labeled data why not actually demonstrate performance on such benchmarks? For example, there are numerous datasets from domains like crowdsourcing that allow you to get "noisy" ratings for datapoints. Wouldn't a more compelling argument be derived by showing improved performance on such datasets?

Thus to summarize: I worry that the results derived solely on simulated noise may not be very indicative of performance in more realistic settings and would request the authors to consider providing evidence on more realistic datasets.

I also wanted to note that the paper exposition is lacking in some aspects and I needed to reread certain sections to make sure I understood them correctly. I think the paper would benefit from a good proofread not just from the grammar/spelling perspective (which there are multiple instances which could be improved) but also from the overall presentation and legibility perspective.

All this said: I want to clarify that this topic is not my research focus and hence I am uncertain as to how much findings on these simulated noise patterns carry over to real datasets and their associated noise patterns. If there is existing evidence indicating strong correlation, then perhaps my review may have varied.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper develops a method for sample selection that exploits the memorization effect. In essence, the authors adopt the co-teaching  (Han et al. NeurIPS 2018) and MentorNet (Jiang et al., ICML 2018) framework, which selects some fraction of examples per minibatch that are hopefully "cleaner" than noisier examples to compute updates from. While in Han et al. the number of instances R selected depends on the number of epochs that have been completed, this paper instead seeks to learn R by approximating it as a linear combination of different types of basis functions  and using natural gradient as the search algorithm. The search space proposed by the authors seem comprehensive: it encompasses the search space of co-teaching, the prior state of the art. Results on synthetic tasks as well as MNIST/CIFAR appear to show the superiority of the proposed method over random search, co-teaching, and other baselines, although the results don't seem conclusive. Overall, I have concerns with some of the contributions, experiments, and presentation, which leaves me at a weak reject.

comments:
- Section 3.1 isn't very compelling to me. Experiments done on just CIFAR with two architectures and optimizers are certainly not sufficient to make any broad claims. I don't think this qualifies as a "contribution" of the paper. 
- The paper is difficult to understand, and much of this difficulty stems from poor writing / presentation. The plots depicting experimental results are especially hard to follow. 
- I'm a little confused with the setup here. Most practitioners use early stopping to halt training after performance on the validation set drops. As such, why should we care about the held-out curve after the maximum is reached? Shouldn't we care more about the training curve, as at some point during training the noisy labels will also be memorized? Isn't this the definition of the "memorization effect"? 
- What about standard baseline methods e.g., active learning to help with this problem? Active learning seems highly relevant yet is not mentioned anywhere in this paper. 
- Are all of the basis functions in Fig 2 necessary for the performance of the proposed method? How were they selected? Why is this motivated by the Taylor expansion?
- Figure 5 shows a bunch of R(t) curves learned by the proposed approach across a variety of datasets / noise levels. All of the curves look very similar! A reasonable baseline motivated by these results is to just apply a simple decay function to R(t) with a single hyperparameter controlling the rate of decay. I suspect this would also work better than the co-teaching approach, and perhaps render the more complex method here unnecessary. In fact, all of the gains associated with this method could just be due to co-teaching dropping far less examples as training progresses, as its decay rule isn't optimal. 



review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies the problem of learning from corrupted labels via picking up clean instances from training dataset. The sample selection mainly based on function R(t), which controls how many instances are kept. This paper proposes a unique curvature of R(t) based on intuition and presents how R(t) can be learned via combination of some existing functions. Natural gradient is presented to optimize the parameters in the autoML framework. Experimental results on both synthetic data and real-world data demonstrate the effectiveness of the proposed method. 

A few comments on this paper:
1. The paper is very verbose and hard to follow. It introduces too many basic concepts in autoML.
2. A key part of the paper is the curvature of R(t), which is based on intuition. Meanwhile, the learned curvature (in Fig 5) doesn't follow the curvature. Does this mean this paper is contradicting its self? The curvature of defined R(t) is not needed?
3. The major difference between this paper and (Han et al. 2018) is how R(t) is defined and learned. The technical contribution of this paper is limited. 

Minor comments:
1. For all the figures, it is difficult to view the y-axis (or the y-axis is missing). 

review_title: Official Blind Review #3

Meta-review: This paper develops a method for sample selection that exploits the memorization effect. While the paper has been substantially improved from its original form, the paper still does not meet the quality bar of ICLR in terms of presentation of the results and experimental validation. The paper will benefit from a revision and resubmission to another venue.
Decision: Reject