Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Summary:
mitigate inherent label noise
This paper studies the effect of applying SGD noise on mitigating the inherent label bias which is common in real-world datasets. It introduces stochastic label noise (SLN), a variant of SGD noise induced by controllable label noise. It formalizes connections between SLN and two other existing SGD noise variants (Proposition 1-3). With such propositions, it shows that SLN can help the model to avoid sharp minima and prevent overconfidence (Claim 1-2). The experiments show that SLN helps improve generalization than baseline methods and can be further used to boost robust training methods on CIFAR10, CIFAR100, CLOTH-1M under five different types of noise settings. Apart from vanilla SLN, it further proposes momentum model (MO) and label correction (LC). Combining them together with SLN can further boost test accuracy for label-correction.


################################################

Reasons for score:
The paper is overall very well-written and gives theoretical insight on the connections of different SGD noise variants. It further provides comprehensive experiments both qualitatively and quantitatively to validate the effectiveness of its proposed SGD noise variant, SLN. The results show that SLN can simplify parameter tuning, producing superior results on label-correction without additional computational overheads.

################################################

Pros:

+very well-written and easy to follow in general

+great connection with as well as comprehensive discussions of related work

+comprehensive experiments along with good visualization

+the proposed method gives better performance without overhead and can be used to enhance existing methods

Cons:

-might be better to also give empirical evidence to support the claim of helping escaping sharp minimums. e.g. a visualization of gradient landscapes for CE and SLN, respectively.


################################################

Questions:

-As I mentioned in the cons, is there any empirical evidence of escaping sharp minima you observed to further support the theoretical finding?

-I see that in Fig 5, you give a qualitative visualization of using different sigma. Did you also do any quantitative ablation study on the hyperparameter sigma? How sensitive the results would be by choosing different sigma?

################################################

Post Rebuttal Update: the authors have well addressed my concerns, in particular (1) the additional visualization gives a good qualitative empirical evidence supporting the claim that SLN helps escaping sharp minima. (2) the search process for the hyperparameter $\sigma$ is very reasonable and makes the usage of SLN practical. I will keep my initial assessment and vote for accepting this paper.

review_title: Nice formalization of SGD noises variants with intuitive theoretical justification and comprehensive empirical results

final_score: rating: 7: Good paper, accept
review_content: **Summary** The paper tackles the problem of training under noisy labels. It proposes adding random  zero-mean Gaussian noise to the labels during training. It is shown that such a noise induces a variable gradient noise which adaptively increases 1) when the learnt network function has higher curvature around training points and 2) when the output prediction has lower entropy (higher confidence) for the training points. It is claimed that the former property helps avoiding sharper minima which generally improves generalization. The second property avoids overfitting to noise in similar fashion to label smoothing. 

**Quality** The paper is well written. The “related works” section is quite thorough but concise when covering the field of learning under label noise. Certain parts such as the tSNE plot (Figure 5) and the time-vs-accuracy plot (Figure 6) do not seem to be central to the paper and are not informative. Instead, certain parts could have been discussed more thoroughly (see the detailed technical comments below).

**Clarity** The claims and contributions are generally clear from the paper. The reasoning behind certain claims could be better clarified. Also, details are missing on the hyperparameter optimization of the baselines. Details come below.

**Originality** The method is close to other works in the analysis of noisy gradients for better generalization as well as the usage of label smoothing and random label perturbation [a] for generalization and learning under label noise. However, to the knowledge of the reviewer, this work combines the ideas of the two directions in a coherent and original way.

**Significance** The experiments are done on several setups and using 5 different independent runs for the baselines and three variants of the proposed method. The results show significant and consistent improvements. However, certain experiments could be added to better support the detailed claims as opposed to merely reporting best final numbers.

**Major technical comments**

*Experiments*
1. Interesting and informative side experiments including 1) the separation of noisy and correctly labeled data when using SLN compared to label smoothing, and 2) the strength of SLN compared to unnoisy cross entropy loss indicating the suitability of SLN in identifying correctly labelled samples (low-loss regime) and correcting noisy labels (high-loss regime).  
2. 5 different synthetic noise types are used for the experiments on CIFAR10 and CIFAR100.
3. the paper has two clear claims regarding the sharpness of the found local minimum and on overconfident predictions. While the experiments show clear improvements of the results across various settings and compared to different baselines, the connection of the improvements to the claims remain largely unsubstantiated. As such, the paper is missing direct experiments supporting the claims and/or shedding some light on them. Some suggestions are as follows:

     3.1. implement the noise as in proposition 3 i.e., directly applying the noise to the gradient. Then, one can modify the noise to decouple the two components and demonstrate individual contributions.

     3.2. quantitatively analyze the jacobian of the learnt network function and/or the sharpness of the local minimum with and without the added noise and for the different kinds of noise to directly investigate the first claim.
4. the performance of some of the baselines are low. Are the baselines reimplemented? How are the general training hyperparameters (learning rate, weight decay, batchsize, etc) and method-specific hyperparameters optimized? Is there a different set of optimized hyperparameters per baseline? What are them?
5. reading the end part of section 3.3 it seems that in SLN+MO+LC the label correction starts only after the full convergence of a SLN-only training. In light of this, how is that the increase in time complexity of SLN+MO+LC is negligible in Table 1? What is the stopping criteria for the initial training and then retraining with LC?

*Theory*
1. The paper misses to cite a relevant paper [a] that also randomizes the labels for avoiding overfitting and demonstrates better generalization. Regarding this, the paper should clearly acknowledge [a] when it comes to claiming the novelty of the noisy-label approach and also when it discusses the advantages of perturbing labels -- [a] discusses similarities to ensemble approaches. That being said, I believe the paper has enough originality on top of [a]: for instance [a] replaces provided labels with independent noise, does not experiment on learning under noisy labels and, the gradient noise analysis of the paper is complementary to [a]. 
2. When correcting the labels using SLN, why is the weight of the given label increases as the sample loss increases? Shouldn’t it be the opposite based on figure 3? I found that there are some discussions provided in appendix C. However, as this goes against the previously published work it deserves more formal discussion and corresponding experiments in the main paper.
3. From what I understand proposition 3 shows that functions that are smoother at training points will receive lower variance. If so, formal discussions are missing to connect smooth functions and flat minima.
4. As the training continues the loss tends to get smaller by getting the function closer to the given one-hot labels at the training data points. From proposition 3 it is argued that as this happens the noise in the gradient increases. This raises a caveat regarding convergence. A theoretical discussion and/or empirical observation are needed to study the convergence. For instance, does the variance of the model increase towards the end of the training or does it actually converge to a solution that is robust to the gradient noise (remains approximately unchanged in the functional space)?

**Overall** In the reviewer’s view, the paper has clear merits in bridging between the theory of gradient noise and label smoothing for learning under label noise, both empirically and theoretically. However, it can benefit from more clarity and additional informative experiments to better understand the effect of the proposed noise.

[a] "DisturbLabel: Regularizing CNN on the Loss Layer", CVPR 2016

**Post Rebuttal Update** 
The authors address many of the concerns, 1) [a] is properly acknowledged in the revised version and novelty is not claimed on additional label noise in the text, 2) while quantitative studies are still absent for claims on sharp vs. flat minima, qualitative results are provided for convergence to "flatter" minima  3) connections between smooth functions and generalization is discussed 4) answers and updates regarding complexity and convergence are *somewhat* convincing. Thus, I am willing to increase the score from 6 to 7 and confidence from 3 to 4 as I believe the paper provides relevant and interesting theoretical arguments.
review_title: The paper is original and brings consistent improvements. However, the connection of the claims to the performance improvement remain empirically unsubstantiated. Furthermore, the paper can improve its clarity.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies learning robust models with noisy labels. The authors argue that a specific SGD noise induced by stochastic label noise (SLN) can mitigate the effect of label noise. But the common SGD noise cannot achieve this. Then they apply the proposed SLN induced SGD noise to the existing label-correction methods for noisy-label learning and provide some experimental results.

Pros:

-The paper provides an interesting view of SGD noise in the lens of noisy labels. They claim that common SGD noise does not endow much robustness against label noise, but using a variant SGD noise by label perturbations can improve the generalization and boost existing robust training methods.

Cons:

-Learning with noisy labels is a hot research area as reviewed in the related work section. It seems that the selected baselines in the experiments are not representative and state-of-the-art methods. For example, Yu et al. (2019) improves co-teaching Han et al. (2018) and should be compared instead. 

And some representative regularization based methods should also be compared:

-Mixup: Beyond Empirical Risk Minimization, Zhang et at., ICLR 2018

-Virtual adversarial training: a regularization method for supervised and semi-supervised learning, Miyato et al., TPAMI 2019,

-SIGUA: Forgetting May Make Learning with Noisy Labels More Robust, Han et al., ICML 2020

since they are more related to the essence of training networks as claimed in the paper. 

-In Claim 1 and 2, it is said that with SGD noise induced by SLN training is difficult to converge in some cases. How to guarantee convergence of the proposed algorithm? Some convergence analysis under reasonable assumptions may be helpful.

-It is claimed in the paper that training without SGD noise under label noise can converge to sharp minima, and SLN helps escape from the sharp minima. It is not very intuitive. Could the authors explain more on it, maybe adding some citations or experimental results could be helpful.

-It is still unclear to me how to tune the standard deviation sigma in practice, which should be an important factor that affects the performance.

-The clarity of the paper could be improved, for example, adding brief proof sketches to the theorems may help for better understanding.

Overall, the paper provides some interesting analysis of SGD noise and label noise, but many unclear parts need to be clarified.
review_title: Interesting work on SGD noise and label noise, but many unclear parts need to be clarified

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper studies the noisy stochastic gradient descent algorithm in noisy label learning. Concretely, the authors added Gaussian noise on the labels rather than the gradient itself. By comparing different noisy SGD algorithms, the authors demonstrate that the proposed SLN algorithm not only help the model to escape from sharp local minima, but also help it to be over confidential. Finally, the authors demonstrate that SLN outperforms some classical SGD methods.

Pros:
- The proposed idea is novel to me. Theoretically, the proposed SLN framework jointly enjoys the ability to escape from sharp saddle points and make the prediction smooth. The authors provide a new perspective to develop robust learning algorithms.
- It seems that SLN method can be integrated with many state-of-the-art noisy label learning models.
- The experimental results in Figure 3 is very promising. The small loss samples are generally clean, which may help improve the performance of many sample-selection based approaches.

Cons:
- I have one main concern. While Figure 3 shows very good results, I noticed that quantitative results are far away from state-of-the-art models. Compared to state-of-the-art models, such as DivideMix, SLN demonstrates far lower accuracy. For example, on CIFAR-10, Asymmetric noisy with 40% noise, the accuracy of DivideMix is 92~93.4% and SLN-MO-LC is 87.85%. Although state-of-the-art performance is not the most essential for me, I think the authors require more experimental exploration. Since SLN is a rather flexible method, it can be integrated with many state-of-the-art models and I believe the performance would be competitive or at least at the same level as the SOTA models. 

Minor comments:
- While Figure 3 shows the clean sample ratio of converged models w.r.t different loss intervals, what would it be after the first few epochs?
- From my point of view, SLN is actually a new strategy of label smoothing (or not?). May the authors explain the superiority of SLN to conventional label smoothing methods?

Overall, I think this paper brings an interesting idea to the community, but the experiments are not enough for me.

review_title: Very interest idea, but the empirical studies are not sufficient

Meta-review: The paper studies the effect of explicitly introducing stochastic label noise into SGD updates, showing both theoretically and empirically that this can improve model performance on datasets with "inherent" label noise. The intuition is that this helps the model escape sharp local minima, where predictions may be overconfident. 

Reviewers broadly found the work to be conceptually and theoretically interesting, and the empirical results are promising. The paper is thus well-posed to be of broad interest to the community.
Decision: Accept (Spotlight)