Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
Update after rebuttal:

The good:
The rebuttal and updated paper address many of my concerns. Most importantly, the updated paper demonstrates the three-stage phenomenon on Open Images and adds experiments on IMDB showing that the Gambler's loss with AES helps a lot. The LAES iteration introduced in the updated paper alleviates my concern about performance drop compared to the CT baseline at certain corruptions on CIFAR-10.

The bad:
- From Figure 12, it looks like the three-stage phenomenon doesn't hold on IMDB. Does AES provide additional benefit beyond the Gambler's loss on IMDB? This needs to be clarified with the way Figure 12 turned out.
- There is a serious missing citation [1] that should be included as a baseline. The proposed method in [1] is at least superficially similar to the Gambler's loss and also makes use of the fact that it is easier to fit clean labels than noisy labels. My apologies for not noticing this earlier.

Overall:
I would still suggest acceptance, because the three-stage phenomenon is an interesting find that the authors make good use of. In light of the missing citation, though, I cannot raise my score.


[1]: Zhilu Zhang, Mert R. Sabuncu. "Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels". NeurIPS 2018.

-----------------------------------------------------------------------
Summary:
This paper proposes a method to alleviate label noise. It opens with the observation of three distinct stages when training in the presence of label noise. Importantly, there is a ‘gap’ stage during which the network has not begun memorizing noisy labels and early stopping is ideal. The authors then observe that the Gambler’s loss (Ziyin et al., 2019) elongates the gap stage and propose an analytic early stopping (AES) criterion for identifying when to stop training.

The analysis of the AES criterion, e.g. in Figure 5, and the observation of a phase transition when tuning the o hyperparameter are quite interesting, and the latter observation is of practical value when using the AES criterion.

The AES criterion seems to be well-motivated, and the empirical evaluation of the Gambler’s loss with and without early stopping is good. The results are strong on MNIST but somewhat weak on CIFAR-10. Specifically, the improvements on CIFAR-10 only appear for large corruption rates (0.7+), and performance is lower than the baselines for other corruption rates. This is a worrying problem, because it calls into question the value of the method on larger problems. However, seeing as this is a distinct approach from the baselines and that it demonstrates some promise, I recommend borderline accept. The authors could raise my score by demonstrating more consistent gains on another larger-than-MNIST CV dataset or an NLP/speech dataset. Other points of concern that I have are listed below.

Major points:
At the top of page 3, the authors say that the idealized gap assumption “holds well for simple datasets such as MNIST and on datasets with very high corruption rate, where our method achieves best results, and less so on more complicated datasets such as CIFAR10”. The idealized gap assumption is behind the AES criterion, but Figure 5 suggests that the AES criterion works well on CIFAR-10, so what do the authors mean when they say the assumption doesn’t work as well on CIFAR-10? Is this just referring to the results?

Saying traditional label noise correction methods are “of no use when one is not aware of the existence of label noise” seems unfair. The FC method and others do not require foreknowledge of the corruption rate and do not harm performance in the absence of label noise, so they can also be said to automatically correct label noise.

“FC, however, requires knowing the whole transition matrix, and is outperformed significantly by our method.”
This is not quite true, because Patrini et al. propose an estimate of the transition matrix as part of the Forward correction. Did you use the estimated or true transition matrix for the FC method? It would be good to clarify this in the paper.

Minor points:
There are a few grammatical errors and typos in the paper:

“or explicit regularization, this is also what is suggested by Abiodun et al. (2018)” (run-on sentence)\

“CIFAR10” should be “CIFAR-10”
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Updated review: Thanks for your comments. I feel the latest version of the paper is better than the previous version. 
However, as stated by other reviewers as well, the claims of the paper are quite ambiguous. Another example from the author response is the point about how Chapter 6 of the Elements of Infomation Theory is related to Gambler's loss. This is not clear to me. I would not object to accepting the paper but I find it difficult to recommend accept for this paper. Perhaps the authors can be more clear in their claims. 

----------------------------------------------------------------------------------------------------------------------------------------------------------

Summary: The paper focusses on the problem of noisy labels in supervised learning with deep neural networks. The paper, in turn, proposes an early stopping criterion for handling label noise. The early stopping criterion is dependent on a new loss function that is defined as the log of true label + weight on a reservation option? The paper shows that when the labels are corrupted then the propose early stopping criterion does better than early stopping criteria obtained via the validation set. 

\The first section of the paper establishes that when label noise is present in the dataset, then there are three stages to training a deep neural network. 
The learning stage where the highest accuracy on the test set is achieved. 
The gap stage where test set accuracy goes down. 
Memorization stage corresponds to when a deep neural network memorizes corrupt labels and test accuracy goes completely down. 
I cannot understand figure 1(a). The y-label says accuracy but it seems that the plot is about loss. What dataset was this and what architecture of DNN was used? The plot shows that the DNN achieved a 100% accuracy in 5 epochs. Is this result meaningful? Before establishing a hypothesis based on this should the hypothesis not be tested on multiple datasets.
The paper says that these stages are persistent across multiple architectures and datasets and as proof the paper says ‘we verified that’. Why can’t the reader see the experiments? By across datasets does the paper mean MNIST and CIFAR? By across architecture does the paper mean the two architectures mentioned in the appendix one each for MNIST and CIFAR respectively? 

The paper makes the assumption that label noise is symmetrically corrupted. Why and where does such an assumption hold? What happens to the proposed method if that is not true. 

Assumption 2: During the gap stage the model has learned nothing about the corrupt data points. 
How is that even possible? 

Equation 1: So the loss function proposed is log(f(x)_y + (1/o) f(x)_m+1) .  What is y here? The true label? Why is y called a point mass? Is this different from the cross-entropy loss + log loss on m+1 ?

I do not understand equations 2 to 5. 

For figure 3 again what datasets were used?

“ Making random bet will help with making money and a skilled gambler will not make such bets” 
Why does making random bet help with making money? If random is good how can a skilled gambler exist in such a game? What is this skill?

k denotes the sum of probability of predicting anything that is not y or m+1 (it does not denote prediction). 

In the experiments section what was the symbol for the rate of corruption changed from epsilon to r. Are they different? 
 
What is nll? 

It seems that gamblers loss best shines when the corruption rate is as high as 80% . That is 80 percent of the data is corrupted. Does this mean that if I trained with only 20% of the non-corrupt data I would still get a 99% accuracy on MNIST (even without gamblers loss)? A comparison of this sort would have been useful.  
One astonishing result the paper presents is that with gambler’s loss even with 80% corrupt labels a 94% test accuracy is possible on MNIST dataset. I think this is significant, this raises the question that is it required to label all the data points ina dataset to achieve high accuracy or is it possible to achieve just as much with only 20% of the labels?
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new loss function for dealing with label noise, claiming that the loss function is helpful in preventing overfitting caused by noisy labels. Some experiments show the effectiveness.

The theory part is unclear to me. (1 Why the derivative of Eq. (1) is Eq. (2)? The notation of f, f_\theta, and f_w has been abused. The notation is confusing without explanation. It seems Eq. (2) is not correct and the following is not convincing. (2 Why a small gradient will slower the model to fit the data? This is not clear and maybe not true. (3 The assumptions \hat{p}+\hat{k}+\hat{l}=1 is very strong to me. The events should be dependent. This makes all the theoretical analyses pseudo and not convincing at all. The authors may spend more effort to make the part clear, reasonable, and convincing.

Many claims are very ambiguous. For example, "the key point is that it always holds on some degree", on which degree and why always holds? "In some cases, it even leads to a better memorization phenomenon", in which cases and why lead to better memorization phenomenon?

Some claims are even wrong. For example, "Traditional methods in label noise often involves introducing a surrogate loss function that is specialized for the corrupted dataset at hand and is of no use when one is not aware of the existence of label noise" This is just for some specific methods, not for the most of them.

Typo: "We verify that . Therefore,"

Overall, I cannot understand why the proposed loss function works and cannot recommend acceptance for the current version.

review_title: Official Blind Review #3

Meta-review: This paper focuses on mitigating the effect of label noise. They provide a new class of loss functions along with a new stopping criteria for this problem. The authors claim that these new losses improves the test accuracy in the presence of label corruption and helps avoid memorization. The reviewers raised concerns about (1) lack of proper comparison with many baselines (2) subpar literature review and (3) state that parts of the paper is vague. The authors partially addressed these concerns and have significantly updated the paper including comparison with some of the baselines. However, the reviewers were not fully satisfied with the new updates. I mostly agree with the reviewers. I think the paper has potential but requires a bit more work to be ready for publication and can not recommend acceptance at this time. I have to say that the authors really put a lot of effort in their response and significantly improved their submission during the discussion period. I recommend the authors follow the reviewers' suggestions to further improve the paper (e.g. comparing with other baselines) for future submissions
Decision: Reject