Noise against noise: stochastic label noise helps combat inherent label noise
Authors: Pengfei Chen
Guangyong Chen
Junjie Ye
jingwei zhao
Pheng-Ann Heng
Venue: ICLR 2021 Spotlight (2021)
Keywords: Noisy Labels
Robust Learning
SGD noise
Regularization

Abstract: The noise in stochastic gradient descent (SGD) provides a crucial implicit regularization effect, previously studied in optimization by analyzing the dynamics of parameter updates. In this paper, we are interested in learning with noisy labels, where we have a collection of samples with potential mislabeling. We show that a previously rarely discussed SGD noise, induced by stochastic label noise (SLN), mitigates the effects of inherent label noise. In contrast, the common SGD noise directly applied to model parameters does not. We formalize the differences and connections of SGD noise variants, showing that SLN induces SGD noise dependent on the sharpness of output landscape and the confidence of output probability, which may help escape from sharp minima and prevent overconfidence. SLN not only improves generalization in its simplest form but also boosts popular robust training methods, including sample selection and label correction. Specifically, we present an enhanced algorithm by applying SLN to label correction. Our code is released.

Published as a conference paper at ICLR 2021
NOISE AGAINST NOISE: STOCHASTIC LABEL NOISE HELPS COMBAT INHERENT LABEL NOISE
Pengfei Chen', Guangyong Chen²;
Junjie Ye Jingwei Zhao Pheng-Ann Heng'.?
The Chinese University of Hong Kong
Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences
VVVV Lab {pfchen, pheng) } wecse.cuhk.edu.. yee jingwei.zhao) @vivo.com, gy.chen siat.accn
ABSTRACT
The noise in stochastic gradient descent (SGD) provides a crucial implicit regu- larization effect, previously studied in optimization by analyzing the dynamics of parameter updates. In this paper, we are interested in learning with noisy labels, where have collection of samples with potential mislabeling. We show that previously rarely discussed SGD noise, induced by stochastic label noise (SLN), mitigates the effects of inherent label noise. In contrast, the common SGD noise directly applied to model parameters does not. We formalize the differences and connections of SGD noise variants, showing that SLN induces SGD noise depen- dent on the sharpness of output landscape and the confidence of output probability. which may help escape from sharp minima and prevent overconfidence. SLN not only improves generalization in its simplest form but also boosts popular robust training methods, including sample selection and label correction. Specifically, we present an enhanced algorithm by applying SLN to label correction. Our code is released
INTRODUCTION
The existence of label noise is common issue in classification since real-world samples unavoid- ably contain some noisy labels, resulting from annotation platforms such as crowdsourcing sys- tems (Yan et al., 2014). In the canonical setting of learning with noisy labels, collect samples with potential mislabeling, but we do not know which samples are mislabeled since true labels are unobservable.
troubling that overparameterized Deep Neural Networks (DNNs) can memo- rize noise in training, leading to poor generalization performance (Zhang et al., 2017; Chen et al., 2020b). Thus, we are urgent for robust training methods that can mitigate the effects of label noise.
The noise in stochastic gradient descent (SGD) (Wu et al., 2020) provides crucial implicit reg- ularization effect for training overparameterized models. SGD noise is previously studied in opti- mization by analyzing the dynamics of parameter updates, whereas its utility in learning with noisy labels has not been explored to the best of our knowledge. In this paper, we find that the com- mon SGD noise directly applied model parameters does not endow much robustness, whereas a variant induced by controllable label noise does. Interestingly, inherent label noise is harmful to generalization. while we mitigate its effects using additional controllable label noise. To prevent confusion, we use stochastic label noise (SLN) to indicate the label noise we introduce. Inherent la- bel noise is biased and unknown, fixed when the data is given. SLN is mean-zero and independently sampled for each instance in each training step. Our main contributions are as follows.
We formalize the differences and connections three SGD noise variants (Proposition 1- 3) and show that SLN induces SGD noise that is dependent the sharpness of output landscape and the confidence of output probability.
"Corresponding to: gy.chen@siat.ace jinvic..e@vvvvo.com.. 'https://g thub. com/ chempf 025/SI.
vivo.com.

---

Published as a conference paper at ICLR 2021
Test Accuracy (CE)
Test Accuracy (SLM)
(SLN)
3.0
3,0
80
80
2.5
75
70
70
1.5
1.5
65
65
1.0
1.0
55
55
0.5
0.5
50
0.0
S0
100
150 Epoch
200
250
300
50
100
150 Epoch
200
250
300
0
50
100
150 200 250 300 Epoch
50
100
150
200
250
300
Epoch
Symmetric
Asymmetric
Dependent
Open-Set
Lass on clean samples
....:: Loss on noley samples
Figure I: Test accuracy and training loss, averaged in 5 runs.
Based on the noise covariance, we analyze and illustrate two effects of SLN (Claim 1 and Claim 2): escaping from sharp minima" and preventing overconfidence".
We empirically show that SLN not only improves generalization in its simplest form but also boosts popular robust training methods, including sample selection and label correc- tion. We present an enhanced algorithm by applying SLN to label correction.
In Fig. I, we present a quick comparison between models trained with/without SLN on CIFAR- 10 with
label noise. Throughout this paper, use CE to indicate a model trained with standard cross-entropy (CE) loss without any robust learning techniques, while the standard CE loss is also used by default for methods like SLN. In Section 4, we will provide more experimental details and more results that comprehensively verify the robustness of SLN on different synthetic noise and real-world noise. Here, the test curves in Fig. show that SLN avoids the drop of test accuracy, with converged test accuracy even higher than the peak accuracy of the model trained with CE. The right two subplots in Fig. 1 show the average loss on clean and noisy samples. When trained with CE, the model eventually memorizes noise, indicated by the drop of average loss on noisy samples. In contrast, SLN largely avoids fitting noisy labels.
2 RELATED WORK
2.1 SGD NOISE AND THE REGULARIZATION EFFECT
The noise in SGD (Wu et al., 2020; Wen et al., 2019; Keskar et al., 2016) has long been studied in optimization. It is believed to provide a crucial implicit regularization effect (HaoChen et al., 2020: Arora et al., 2019; Soudry et al., 2018) for training overparameterized models. The most common SGD noise is spherical Gaussian noise on model parameters (Ge et al., 2015; Neelakan- tan al., 2015; Mou et al., 2018), while empirical studies (Wen et al., 2019; Shallue et al., 2019) demonstrate that parameter-dependent SGD noise is more effective, is shown that the noise covari- ance containing curvature information performs better for escaping from sharp minima (Zhu al., parameterized 2019: Daneshmand et al., 2018). On a quaddaticallyypprmmeeeezeedd model (Vaskevicius et al., 2019; Woodworth al., 2020), HaoChen et al. (2020) prove that in an over- over-parameterized regression set- ting, SGD with label perturbations recovers the sparse groundtruth, whereas SGD with Gaussian noise directly added on gradient descent overfits to dense solutions. In the deep learning scenario, HaoChen et al. (2020) present primary empirical results showing that SGD noise induced by Gaus- sian noise on the gradient of the loss w.r.t. the model's output- avoids performance degencration of large-batch training. Xie et al. (2016) discuss the implicit ensemble effect of random label perturba- tions and demonstrate better generalization performance. this paper, we provide new insights by analyzing SGD noise variants and the effects, and showing the utility in learning with noisy labels.
2 "Around sharp minima, the output changes rapidly (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017). 'The prediction probability on some class approaches 1.
2

---

Published as a conference paper at ICLR 2021
2.2 ROBUST TRAINING METHODS
Mitigating the effects of label noise is a vital topic in classification, which has a long history (Ekholm & Palmgren, 1982; Natarajan et al., 2013) and attracts much recent interest with several directions explored. 1) Malach & Shalev-Shwartz (2017); Han et al. (2018b); Yu et al. (2019); Chen et al. (2019a); Wei et al. (2020) propose sample selection methods that train on trusted samples, identified according to training loss, cross-validation or (dis)agreement between two models. 2) Liu & Tao (2015); Jiang et al. (2018); Ren et al. (2018); Shu et al. (2019); Li et (2019a) develop sample- weighting schemes that aim to add higher weights on clean samples. 3) Sukhbaatar et al. (2015); Patrini et al. (2017); Hendrycks et al. (2018); Han et al. (2018a) apply loss-correction based on an estimated noise transition matrix. 4) Reed et al. (2015); Tanaka et al. (2018); Arazo et al. (2019); Zheng et al. (2020); Chen et al. (2020a) propose label correction based on the model's 's predictions. 5) Ghosh et al. 2011);; Zhang & Sabuncu (2018); Xu et al. (2019); Wang al. (2019); Lyu & Tsang (2020); Ma et al. (2020) study robust loss functions that have a theoretical guarantee for noisy risk minimization, typically with the assumption that the noise is class-conditional (Scott et al., 2013; Natarajan et al., 2013). 6) Chen et al. (2019b); Menon et al. (2020); Hu et al. (2020); Harutyun- yan al. (2020); Lukasik et al. (2020) apply regularization techniques to improve generalization under label noise, including explicit regularizations such as manifold regularization (Belkin et al., 2006) and virtual adversarial training (Miyato et al., 2018), and implicit regularizations such as dropout (Srivastava et al., 2014), temporal ensembling (Laine & Aila, 2017), gradient clipping (Pas- canu et al., 2012; Zhang et al., 2019; Menon al., 2020) and label smoothing (Szegedy et al., 2016). 7) One can combat label noise with refined training strategies (Li et al., 2019b; 2020; Nguyen et al., 2020) that potentially incorporate several techniques, including sample weeegtin/weigtiingg label correction, meta-learning (Li et al., 2019b) and semi-supervised learning (Tarvainen & Valpola, 2017; Berthelot et al., 2019). Among these methods, regularization techniques are closely related to the essence of training networks, and studying robustness under label noise provides new lens of understanding the regularization apart from the optimization lens.
3 METHOD
3.1 THE DIFFERENCES AND CONNECTIONS OF SGD NOISE VARIANTS
Notations. Let ((x11) y ()) be dataset with noisy labels. For each sample (2r, y). its label may be incorrect and the true label is unobservable. Let fx; 0) be the neural network model with trainable parameter 0 E IR. For a c-class classification problem, we have the output [(;; 0) E R We use a softmax function ((f((;; 0)) E 10 to obtain the probability of each class. The loss on sample is denoted as E(f.y). y). For classification, we use the cross-entropy (CE) loss by default. In parameter updates, a sample contributes Vol(f,y) to the gradient descent. With SGD noise, the model is trained with a noisy gradient vo((s,)) Following the standard notation of the Jacobian matrix, we have Vol E RIxp, v 1xc, Vol E Rex,, Vorl E Re and Vofi E RIXp.
Gaussian noise on the gradient of loss w.r.t. parameters. The most common SGD noise is the spherical Gaussian noise directly added to the gradient to parameters (Ncelakantan et al., 2015) as follow,
Vol(F.v) Vol(f. + 0020,
where dg 0, 2g E 1xP and 2g (O00
Gaussian noise on the gradient of loss w.r.t. the model output. Taking a step further, HaoChen al. (2020) study SGD noise induced by label noise on quadratically paralley-prrmmeeiizeee regression model, whereas for classification, they add mean-zero noise to V y) as follow,
Vol(s.) (V (V)(()))))))))))))++)++)))) Vof,
where o >> 0, 2y E R. 1xc and 2, (0,
Noise induced by SLN. The label perturbation is a common technique (Xie et al., 2016), while we provide new insights by analyzing the effects from the lens of SGD noise. Our SLN adds mean-zero Gaussian noise to the one-hot labels, where the noise is independently sampled for each instance in each training step.
vol(s.v) V +g
(3)
3

---

Published as a conference paper at ICLR 2021
where By > 0, 2y E IR and by ~ (O(0,, Here, Jy is the SLN on the label y.
In above Eq. (1-3), the standard deviation o is a hyperparameter. Since the SGD noise we introduce is i.d for each sample, consider each single sample (z, y) independently in the following propo- sitions. For convenience, we use f. to indicate the model output f(z; 0) and the softmax output (f((;; 0)) on a sample. The proofs are presented in Appendix A.
Proposition Compared with Eq. (1), Eq. (2) induces noise z ~ o3 on VoE(J.y) s.l,,
M E IRXX and Mi,j (V ( Vo,f f. Vi,j E {1.....Ph Note that the standard deviation of noise on the i-th parameter Og is os v
where |1-112 denotes the L2 norm.
Proposition 2. For the cross-entropy loss, compared with Eq. (2), Eq. (3) induces noise a ~ NO of M) on V v((f.v). s.t., M E Rexe, Ma, = c(. 1/c)² 1)/c and Mr. c. S - S. Sj. if + j. Note that the standard deviation of noise on the i-th entry is oy Vols 1/c)² + (c-1)/c.
Proposition 3. For the cross-entropy loss, compared with Eq. (1), Eq. (3) induces noise a (0, M) on Vol(f,v). v). s.t., M E Rexp and Mi, = ( (? Sy VayS Vi,j E (1...., where is denotes the element-wise division. Note that the standard deviation of noise on the i-th parameter
v1 5
0g is γy
8
3.2 THE EFFECTS OF SGD NOISE
Xie et al. (2016) discuss the effect of label perturbations implicit ensemble. In this paper. based on Proposition 1-3, we show that SIN induces SGD noise of high variance when the oulput landscape sharp or the prediction confidence is high. In this way, SI.N helps escape from sharp minima and prevents overconfidence. It was discussed that flat minima generalize well (Hochreiter & Schmidhu- ber, 1997; Keskar al., 2017; Neyshabur et al., 2017). Specifically, Achille & Soatto (2018) show that flat minima have lower mutual information between model parameters and training data, which leads to better generalization. The finding motivates several robust learning methods (Harutyunyan et al., 2020; Xie et al., 2020) and also supports our method. Moreover, preventing overconfidence can mitigate overfitting on noisy labels (Menon et al., 2020; Lukasik et al., 2020).
The SGD noise perturbs 0 so that the training can not converge when the noise has high variance. Therefore, we derive Claim and Claim 2 with justifications as follows.
For the most common spherical Gaussian noise shown in Eq. (1), its standard deviation is a constant throughout training, independent of the landscape.
For Eq. (2), Proposition 1 shows that the standard deviation of noise is ee. 112- /Vo2/ll can be very large around the sharp landscape, which means the SGD noise has high variance. The high variance makes the training difficult to converge, which helps es- cape from sharp minima. Note that for training without SGD noise, it can converge to sharp minima because 0 always follows the direction of gradient descent, whereas the direction of noise is random.
For our SINN in Eq. (3), Proposition 3 shows that the standard deviation of the SGD noise is oy S, with S in the numerator. SLN similarly induces SGD noise with high variance around the sharp landscape. Hence we have Claim 1.
Moreover, Proposition 2 shows that SLN induces SGD noise dependent on the confidence of S. The standard deviation of noise on an entry of v y) is minimized S, 1/c, maximized if S; = 1. Proposition 3 directly characterizes the equivalent noise on 9 where S is in the denominator (element-wise division). If is confident, s.t., the entropy (S) - O, which means an entry of S approaches I and others approach O, then the variance will be high since there are small numbers in the denominator. Hence we have Claim 2.
Claim 1. With SGD noise induced by Eq. (2) or Eq. (3), the training is difficult to converge when the output (f or S) landscape is sharp.
Claim 2. With SGD noise induced by Eq. (3), the training is difficult to converge when the output (S) is overconfiden, s.t. (S) + 0, where H-) is the entropy.
Fig. 2 shows visualizations of loss landscapes. The model trained with SLN converges to a flat min- imum that has small SGD noise. More discussions the convergence are presented in Appendix E.
4

---

Published as a conference paper at ICLR 2021
(A)
(b) Noize on Val. (1).
(c) Noiz on Vd. Eq.
SIN. Eq. (3)
2.4
1.5
2.2
10
2.0
1.45
0.6
0
0.3
1.8
1.40
0.1
0.1
0.1
0.1
Figure 2: Loss landscapes around the local minima of converged models trained on CIFAR-IO with symmetric noise, visualized using the technique in Li et al. (2018). We show the z-axis the same scale to compare the sharpness; and draw color bars separately to show the loss distribution around each minimum. (a): The model trained with CE converges to sharp minimum. (b): Training with Eq. (1) yields a minimum with a higher loss, yet it is still sharp. (c)& Consistent with our analysis, the model trained with Eq. (2) Eq. (3) converges to flat mínimum.
CE
L5, 0 0.2
LS, 2 0.5
LS. 2 0.8
SUN, o=1
12500
Cliaa
6000
Clean Noisy
6000
Clean Noisy
6000
Clean Nokey
6000
Clean Noisy
ssmp
10000
Noisy
7500
4000
4000
4000
4000
5000
2000
2000
2000
2000
2500
0.00 0.25 0.50 0.75
0.00
0.50
1.00
0.00 0.55
0.75 1.00
0.00
0.50 Prediction probability
0 0.25 0.75 Prediction probabiltty
Prediction probability
Prediction probability
Prediction probability
Figure 3: Samples density w.r.t. the prediction probabilities (the softmax outputs on the labeled class). SLN reduces overconfidence mostly noisy samples, while clean samples are less affected.
In terms of escaping from sharp minima, there are previous works (Zhu et al., 2019; Daneshmand et al., 2018) that study the inheret noise in SGD, showing that the noise covariance containing cur- vature information performs better. They use the second-order approximation apply integral to training steps to characterize the ability of escaping from sharp minima. In this paper, we study the noise induced by SLN and draw more direct intuition between the noise variance and the ability of escaping from sharp minima. Moreover, the dependency between the noise induced by SLN and the confidence of output probability further provides an intuition on avoiding overfitting under noisy labels. In terms of preventing overconfidence, we shall discuss la- bel smoothing (LS) (Lukasik al., 2020). It smooths the given one-hot label y into a soft one ÿ. s.t., ÿ (1 - y + ae/c, where e is an all-one vector and a > 0. In this way, LS introduces a fixed and biased perturbation on the label, whereas SLN introduces dynamic and mean-zero perturbations. LS does not introduce noise in each training step, while SI.N adaptively perturbs the parameter once the landscape is sharp or the prediction overconfident. Hence, the robustness of SIN may not result from preventing overconfidence alone, but also escaping from sharp minima. In Fig. 3, we plot the sample density w.r.t predictions on the labeled class, using CIFAR- 10 with 40% symmetric noise as an example. It shows that SLN does reduce overconfidence, while it mostly affects noisy samples.
the minima and
3.3 A DISSECTION ON TRAINING SAMPLES
In this section, show that SLN boosts popular robust training methods, including sample selec- tion and label correction. Many methods, demonstrated to work well, select or add higher weight on small-loss samples (Han al., 2018b; Jiang et al., 2018; Li et al., 2020), or use the model's predictions to correct noisy labels (Tanaka et al.. 2018; Arazo et al., 2019). A warm-up phase is usually required to initialize the model before sample selection or label correction, yet the model will memorize noise if the warm-up phase is too long. With SLN, we can simply train the model until convergence. In Fig. 4, we compare converged models on CIFAR- where the model is trained with/without SLN. The detailed noise setting can be found in Section 4. We first sort train- ing samples in ascending order of loss, then uniformly divide them into 1000 samples per interval, and finally obtain the number of four types of samples in each interval based on the correctness of the given label and the prediction. When trained with CE, there are many small-loss samples with incorrect labels (the blue region), then selecting soall-loss samples is not reliable. The model
5

---

Published as a conference paper at ICLR 2021
Symmetric (CE)
Asyimmetric (CE)
Dependent (CE)
1000
1000
1000
prediction
prediction
x, prediction
800
label prediction
800
label prediction
800
label prediction
label 7, prediction
label predictionn x
label v. prediction x
comples
600
label 7. prediction
samples
600
label prediction
samples
600
label v, prediction
400
400
400
200
200
200
30 Intervals floss
Intervals (loss 1
Intervals (loss
Symmatric (SLN)
Asymmetric (SLN)
Dependent (SLN)
1000
1000
1000
label x. prediction
label x, prediction
label K, prediction x
800
label prediction
800
label prediction
800
prediction
label v. prediction
label prediction
label prediction x
samples
6.0
label 7, prodiction
samples
600
label v, prediction /
somatc
600
labal v, prediction v
400
400
400
200
200
200
30
Intervals (loss 1
Intervals (less 1
Intervals (leas 1)
Figure 4: Training samples are sorted in ascending order of loss, uniformly divided into 1000 sam- ples per interval, and dissected according the conrectness of the given label and the prediction.
trained with SLN largely addresses the issue. Moreover, SLN is suitable for label correction since yields correct predictions for many originally incorrect samples (the orange region).
As a concrete example, we present an enhanced algorithm by applying SLN to label correction. With SLN, we train the model for sufficient epochs until convergence, without the need of carefully tuning a warm-up phase. Then we start label correction using Ycorrection = w y + (1 - w) S, where S is the softmax prediction, w E [0, 1] is the weight obtained by normalizing the training loss, s.t., for the i-th training sample, Wi = (&: f (emaz Emin). ). More discussions on the label correction presented in Appendix D.
EXPERIMENTS
4.1 EXPERIMENT SETUP
We comprehensively verify the utility of SLN different types of label noise, including symmetric noise, asymmetric noise (Zhang & Sabuncu, 2018), instance-dependen noise (Chen et al., 2020a) and open-set noise (Wang et al., 2018) synthesized on CIFAR-10 and CIFAR-100 and real-world noise on Clothing M (Xiao et al., 2015).
Symmetric noise assumes each label has the same probability of flipping to any other class. We uniformly flip the label to other classes with an overall probability 40%.
Asymmetric noise contains noisy labels flipped between similar classes. Following Zhang Sabuncu (2018), on CIFAR-I0, we flip labels between TRUCK-AUTOMOBIIC BIRD-AIRPPAAE, DEER-HORSEE and CAT++DOG, with probability 40%; on C00, we flip each class into the next class circularly with probability 40%.
Instance-dependent noise is challenging since the mislabeling probability should be depen- dent each instance's 's input features (Xia al., 2020; Chen et al., 2020a). We use the instance-dependent noise from Chen et al. (2020a) with a noise ratio 40%, where the noise is synthetized based on the DNN prediction error.
Open-set noise contains samples that do not belong to any class considered in the classi- fication task. Following Wang et al. (2018), we yield CIFAR-10 10 with open-set noise by randomly replacing 40% of its training images with images from CIFAR-100.

---

Published as a conference paper at ICLR 2021
Symmetric
Asymmetric
Dependent
Open Set
80-
80
75-
80
80
70
70-
65-
65
65-
60-
60
60-
60-
55-
50-
45
50
200
150
200
50
100
150 Epoch
200
250
300
100
150 Epoch
200
250
300
50
100
200
250 300
CF
d- 1e:3
2g. 5e:3
07 1
d- 1e:3
Se:
1e:2
(50N)
Figure 5: Performance of SGD noise variants on CIFAR-10. The accuracy is averaged in 5 runs.
For real-world noise, we use the large-scale benchmark Clothing M, which contains 1M training images with noisy labels from online shops.
We use the backbone wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) on CIFAR-10 and CIFAR-100 and ResNet-50 on Clothing More training details can be found in Appendix B. Apart from SLN, we introduce a momentum (MO) model (Tarvainen & Valpola, 2017) and la- bel correction (LC), with which we get SLN-MO and SLN-MO-LC. Parameters of the momentum model (0') is updated as a moving average of the the training model (0). At 1-th training step, ge(t) = af + (1 a))(t), We use a = 0.999 in all experiments. For SLN-MO-LC, we have discussed in Section 3.3 that SLN is reliable when applied label correction.
4.2 COMPARING SGD NOISE VARIANTS
We first show that compared with other variants, SLN stands out in mitigating the effects of label noise. The SGD noise variants have been formalized in Eq. (1-3), including 28 directly added to Vol, 2 on yy and our SLN (zy). For SLN, the standard deviation is = under symmetric label noise and a = 0.5 in all other cases. For other SGD noise, the standard deviation is equally tuned and the performance under three different is separately shown in Fig. 5. SLN significant improves the generalization, while other SGD noise variants can not achieve such impressive performance even if heavily tuned. It is worth noting that for all these variants, when a is too small, the model overfits noise since the training is similar to merely using CE loss; when a is too large, the model fails to fit the training data since the SGD noise is too high.
4.3 CIFAR-10 AND C0F
On CIFAR-10 and CIFAR-100, we compare with the following baselines: 1) standard cross- entropy (CE) loss, 2) Generalized Cross-Entropy (GCE) (Zhang & Sabuncu, 2018) loss, 3) Co- Teaching (Han et al., 2018b) that uses co-training and sample selection, 4) PHuber-CE (Menon et al., 2020) that uses gradient clipping and 5) label-smoothing (LS) Lukasik et al. (2020) that clips the label to be less confident before training. We use 5k noisy samples as the validation hyperparameters, then train the model on the full training set and report the test accuracy at the last epoch. SLN simply requires tuning the standard deviation a, which is tuned in { 10.1, 0.2, 0.5, 1). On CIFAR-10, the best a is under symmetric noise and 0.5 otherwise; On CIFAR-100, it is 0. under instance-dependent noise and 0.2 otherwise. The label correction in SLN-MO-LC is applied in the last 50 epochs. The softmax prediction is converted into one-hot label in correction. We repeat each experiment times.
tune
The average test accuracy at the last epoch is reported in Table I and Table 2. To illustrate the influence of a, an ablation study is presented in Fig. 6. In the tables, we mark the top-3 results in bold and present the average training time of each method, evaluated on single V100 GPU. Without the momentum model and label correction, vanilla SLN achieves impressive test performance, which is consistent with results in Fig. 1. In Section 3.3, we have analyzed that SLN can boost popular robust training methods, based on a detailed dissection on DNNs' predictions. As expected, we obtain further improvement with SLN-MO and SLN-MO-LC. Notably, SLN. SLN-MO and SLN-MO-LC sweep the top-3 results in almost all cases and require low computational cost. We do not expect vanilla SLN to achieve ofate-of-the-art performance compared with many integrated methods. Still,

---

Published as a conference paper at ICLR 2021
Table 1: Test accuracy (mean=std in 5 runs) CIFAR-I0. The Open-Set noise is generated by randomly replacing 40% images of CIFAR-10 with images from CIFAR-100. In Appendix C.I, Table 4 shows that SLN can improve many robust learning methods.
Method
Symmetric
Asymmetric
Dependent
Open-Set
Training time
CE
53.16=1.64
72.72+1135
72.33=0.50
77.74
1.18#0.03
GCE
79.27=0.54
70.90+2.98
73.05::.24
81.56+0.41
1.1910.03h
To-Teaching
82.37-0.32
19.6
76.6910.66
85.21
2.22100066
PHuber-CE
80.58:0.34
73.01+1.28
72.70=:0.24
81.62:t0.52
1.20+:004h
LS
1.25
66.96+2202
72.79+0.227
77.06++0.37
1.20+0002h
SLN
80.00=0.661
80.6310.61
78.48:10.28
85.33+0.52
1.24+00055
SLN-MO
84.71=00443
84.80:0.27
80.56:10.16
89.16+0.31
1330=0.03h
SIN-MO-LC
0..00
87.85+0.41
0..76
88.47:+0.22
1.314-0.04h
Table 2: Test accuracy (mean-Estd in 5 runs) on CIFAR-100.
Method
Symmetric
Asymmetric
Dependent
Training time
CE
3..63=0.83
36.26=0.65
55.59:10.43
1.22+0.04h
GCE
44507=0.56
36.75+1.077
54.56
1.23+0005h
Co- Teaching
53.05 1.11
39.78:10.22
47.06:10.44
2.23:10.07h
PHuber-CE
48.76=1.28
35.1310.38
55.1110.29
1.29+0.04F 04h
LS
40.57=0.68
43.51=0.48
53.89-0.39
1.23+0.03h
SLN
50.24=0.41
44.43=0.19
57.47
1.28+0.04h
SLN-MO
5.38
50.5910.35
61.2411.27
1.32+0.02h
SLN-MO-LC
58.64-0.37
63.90-0.46
61.1441.39
1.35+0003h
SLN can be a promising option in the family robust learning methods. Appendix C.I, we show that SLN can improve many existing methods.
Ablation: the performance of SLN
Test Accuracy
80
O
Reported in Table
Symmetric Asymmetric
Dependent
Open-Set
50
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
2.0
a
Figure 6: Test accuracy (te in 5 runs) of SLN on CIFAR-10 w.r.t a. small a results overfitting while a large yields underfitting. In Appendix C.2, visualize the embedding of overfitting/underitinngg For results reported in Table and Table 2, following Zhang & Sabuncu (2018); Chen al. (2020b), 5k noisy samples the validation to (0.1, 0.2, 1).
4.4 CLOTHINGIM
Clothing (Xiao et al., 2015) is large-scale benchmark of clothing images from online shops with 14 classes, containing real-world label noise. It has million noisy samples for training, 14k and 10k clean samples for validation and test. The number of images labeled as each class is unbalanced, ranging from 18976 to 88588 in the noisy training set. In previous works, some experiments are conducted by sampling class-balanced training subset in each epoch (Li et al., 2020), while others directly train on the full training set (Patrini et al.. 2017). Since the balanced training sampling itself affects the test performance, it is difficult compare results across papers. Therefore,
8

---

Published as a conference paper at ICLR 2021
Table Test accuracy (meandistd in runs) on Clothing The star" marks results copied from Pa- trini et al. (2017). The result of DivideMix (Li et al., 2020) is reproduced from its official imple- mentation, which uses blss-bbllanced training sampling.
Training sampling
Standard
messy-clas--allance
Method
Test accuracy
Training time
Test accuracy
Training time
CE
68.942 94
71.12::0.2
2.61+0.08h
Forward
19.884
71.28++0.27
2.74+0.05h
Backward
69.13*
71.03++0.33
2.72+t0.0hh
Co-Teaching
0..19
15.06=0.33h
72.14:10.28
4.51:0.17h
DivideMix
73.81=0.41
18.78:00322
SLN
0..42:1:0.4
9.25
72.95-10.31
2.73+00055
SLN-MO
71.15::0.1
9.311:0.09h
12.98:10.15
2.78:10.07h
SIN-MO-LC
72.61 0.2.3
7..08
3.314:0.03h
make it clear here and conduct experiments in both setting, including the standard sampling and the noisy-class-balane balance sampling. For the latter, in each epoch, 18976 instances per class are randomly sampled from the noisy training set. Other training details strictly follow the standard benchmark setting (Patrini et al., 2017), presented in Appendix B. We set the standard deviation of SLN as a 0.2. For SLN-MO-LC, the label correction applied since the first epoch. Results are listed in Table 3, with the best result in bold and previous published results marked by star. DivideMix trains two models in each rum and we average their test accuracy, rather than using additional ensemble of models. Our SLN outperforms many baselines. The variants SLN-MO and SLN-MO-LC further achieve higher test accuracy. Specifically, SLN-MO-LC achieves the best test accuracy in both settings. Our methods also stand out for training efficiency.
CONCLUSION
In this paper, we establish that SLN induces SGD noise dependent the sharpness of output land- scape and the confidence of output probability and analyze the effects of escaping from sharp local minima and preventing overconfidence. This partially explains the robustness of SLN under noisy labels since various works show that flat minima typically generalize well and preventing overconfi- dence helps mitigate overfitting on noisy labels. We empirically verify the robustness of SLN under various synthetic label noise and real-world noise. Moreover, we show that SLN boosts popular ro- bust training methods, including sample selection and label correction. In particular, we justify that SLN can enhance existing methods based detailed dissection training samples, then present a practical algorithm by applying SLN to label correction.
ACKNOWLEDGMENTS
This work was supported by a grant from Research Grants Council of the Hong Kong Special Ad- ministrative Region (Project No. CUHK 14201620) and the National Natural Science Foundation of China (Project No. 62006219).

---

Published as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep rep- resentations. Journal of Machine Learning Research, 2018.
Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In International Conference on Machine Learning, 2019.
Sanjeev Arora, Naday Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems, 2019.
Mikhail Belkin, Partha Niyogi, and Vikas Sindhwani. Manifold regularization: A geometric frame- work for learning from labeled and unlabeled examples. Journal of Machine Learning Research, 2006.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, 2019.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. International Conference on Machine Learn- ing, 2019a.
Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. A meta approach to defend noisy labels by the manifold regularizer psdr. arXiv preprint 4rXiv:1906.05509, 2019b.
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Beyond class- conditional assumption: A primary attempt to combat instance-dependent label noise. arXiv preprint arXiv:20/2.054588 2020a.
Pengfei Chen, Junjie Ye, Guangyong Chen, Jingwei Zhao, and Pheng-Ann Heng. Robustness ac- curacy metric and its inspirations in learning with noisy labels. arXiv preprint arXi::0012.04193, 2020b.
Hadi Daneshmand, Jonas Kohler, Aurelien Lucchi, and Thomas Hofmann. Escaping saddles with stochastic gradients. In International Conference on Machine Learning, 2018
Anders Ekholm and Juni Palmgren. A model for binary response with misclassifications. In GLIM 82: Proceedings of the international conference on generalised linear models, pp. 128- 143. Springer, 1982.
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle gradient for tensor decomposition. In Conference Learning Theory, 2015.
online stochastic
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. AAAI Conference Artificial Intelligence, 2017.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. In Advances Neural Information Processing Systems, 2018a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems, 2018b.
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao. Miao Xu, Ivor W Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In International Conference on Machine Learning. 2020.
Jeff HaoChen, Colin Wei. Jason D Lee, and Tengyu Ma. Shape matters: Understanding the implicit bias of the noise covariance. arXiv preprint arXiv:2006. 08680, 2020.
10

---

Published as a conference paper at ICLR 2021
Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization by controlling label-noise information neural network weights. In International Conference on Machine Learning. 2020.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. Using trusted data train deep networks on labels corrupted by severe noise. In Advances in Neural Information Processing Systems, 2018.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 1997.
Wei Hu, Zhiyuan Li, and Dingli Yu. Simple and effective regularization methods for training on noisily labeled data with generalization guarantee. In International Conference Learning Rep- resentations, 2020.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels. In International Conference on Machine Learning. 2018.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 1609:v:60000000 04836, 2016.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2017.
Samuli Laine and Timo Aila, Temporal ensembling for semi-supervised learning. In International Conference Learning Representations, 2017.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. In Advances in Neural Information Processing Systems, 2018.
Jia Li, Yafei Song, Jianfeng Zhu, Lele Cheng, Ying Su, Lin Ye, Pengcheng Yuan, and Shumin Han. Learning from large-scale noisy web data with ubiquitous reweighting for image classification. IEEE Transactions Puttern Analysis and Machine Intelligence, 2019a.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S Kankanhalli.
Learning to learn from noisy labeled data. IEEE Conference on Computer Vision and Pattern Recognition, 2019b.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representationss 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions Pattern Analysis and Machine Intelligence, 2015.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, 2020.
Yueming Lyu and Ivor W Tsang. Curriculum loss: Robust learning and generalization against label corruption. In International Conference on Learning Representations, 2020.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey, Nor- malized loss functions for deep learning with noisy labels. In International Conference Ma- chine Learning, 2020.
Eran Malach and Shai Shalev-Shward Decoupling "when to update" from "how to update". In Advances in Neural Information Processing Systems, 2017.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, and Sanjiv Kumar. Can gradient clipping mitigate label noise? In International Conference Learning Representations, 2020.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and semi cemi-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.

---

Published as a conference paper at ICLR 2021
Wenlong Mou, Liwei Wang, Xiyu Zhai, and Kai Zheng. Generalization bounds of sgld for non- convex learning: Two theoretical viewpoints. In Conference on Learning Theory, 2018.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in Neural Information Processing Systems, 2013.
Arvind Neelakantan, Luke Vilnis, Quoc V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization deep learning. In Advances in Neural Information Processing Systems, 2017.
Tam Nguyen, C Mummadi, T Ngo, L Beggel, and Thomas Brox. Self: learning to filter noisy labels with self-ensembling. In International Conference on Learning Representations, 2020.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. Understanding the exploding gradient prob- lem. CoRR, abs/1211.5063, 2012.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust label noise: A loss correction approach. In IEEE Conference Computer Vision and Pattern Recognition, 2017.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. In International Conference Learning Representations, 2015.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, 2018.
Clayton Scott, Gilles Blanchard, and Gregory Handy. Classification with asymmetric label noise: Consistency and maximal denoising. Conference on Learning Theory, 2013.
Christopher Shallue, Jaehoon Lee, Joseph Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Mcasuring the effects of data parallelism on neural network training. Journal of Machine Learming Research, 2019.
Jun Shu, Qi Xie, Lixuan Yi. Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta-weight- net: Learning explicit mapping for sample weighting. In Advances in Neural Information Processing Systems, 2019.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The im- plicit bias of gradient descent on separable data. Journal of Machine Learning Research, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 2014.
Sainbayar Sukhbaatar, Joan Bruna Estrach, Manohar Paluri, Lubomir Bourdev, and Robert Fergus. Training convolutional networks with noisy labels. In International Conference on Learning Representations, 2015.
Christian Szegedy, Vincent Vanhoucke. Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In IEEE Conference on Computer Vision and Pattern Recog- nition, 2018
Antti Tarvainen and Harri Valpola. Mean teachers better role models: averag-d consis- tency targets improve sumi-superrised deep learning results. Advances in Neural Information Processing Systems, 2017.
12

---

Published as a conference paper at ICLR 2021
Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. In Advances in Neural Information Processing Systems, 2019.
Yisen Wang, Weiyang Liu, Xingjun Ma, James Bailey, Hongyuan Zha, Le Song, and Shu-Tao Xia. Iterative learning with open-set set noisy labels. In JEEE Conference on Computer Vision and Pattern Recognition, 2018.
Yisen Wang. Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In IEEE International Conference on Computer Vision, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: joint training method with co-regularization. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. Interplay between optimization and generalization of stochastic gradient descent with covariance noise. arXiv preprint arXiv:1902.0823: 2019.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. arXiv preprint ar arXiv:2002..99 09277, 2020.
Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning, 2020.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang, Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Parts-dependent label noise: Towards instance-dependent label noise. In Advances Neural Information Processing Systems, 2020.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. IEEE Conference on Computer Vision and Pattern Recog- nition, 2015.
Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, and Qi Tian. Disturblabel: Regularizing enn on the loss layer. In IEEE Conference Computer Vision and Pattern Recognition, 2016.
Zeke Xie, Fengxiang He, Shaopeng Fu, Issei Sato, Dacheng Tao, and Masashi Sugiyama. Artificial neural variability for deep learning-on overfitting, noise memorization, and catastrophic forget- ting. arXiv preprint urXiv:201 2020.
Yilun Xu, Peng Cao. Yuqing Kong, and Yizhou Wang. L.dmi: A novel information-thheoriicc loss function for training deep nets robust to label noise, In Advances in Neural Information Process- ing Systems, 2019.
Yan Yan, Rómer Rosales, Glenn Fung, Ramanathan Subramanian, and Jennifer Dy. Learning from multiple annotators with varying expertise. Machine learning, 2014.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International Conference on Machine Learning. 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt. Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Analysis of gradient clipping and adaptive scaling with relaxed smoothness condition. arXiv preprint arXiv:1905.1 1881, 2019.
13

---

Published as a conference paper at ICLR 2021
Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems, 2018
Songzhu Zheng, Pengxiang Wu, Aman Goswami, Mayank Goswami, Dimitris Metaxas, and Chao Chen. Error-bounded correction of noisy labels. In International Conference on Machine Learn- ing, 2020.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochas- tic gradient descent: Its behavior of escaping from sharp minima and regularization effects. In International Conference Machine Learning, 2019.
14

---

Published as a conference paper at ICLR 2021
A PROOFS
Proposition 1. Compared with Eq. (1), Eq. (2) induces noise = ~ NO N M) on Vel(f. v)) s.t., M E x and M,j (V forff""' fo,f. Vi,j E {1,....P) Note that the standard deviation of noise on the i-th parameter 0g is of Vo,/12, where |1-1|2 denotes the Lo2 norm.
Proof. Dimension: 2f E Rixe, Vol E RIxp, VRLERING Vol E RROP, Vo, ERC
For Eq. (2), the noisy gradient is
Vol(J,)) (V,((,y) + tzzz)) Vol Volls.))+ + tzzz Voʃ
(4)
The noise on Vol(S.v) is 2 - -zzg- Vol E RIxp, Note that 2y NNO, lexe), let 23 be the i-th entry of z, we have
0fk 30;
2j oj
(5)
km
Hence,
E|=?] - = B =
(6)
Proposition 2. For the cross-entropy loss, compared with Eq. (2), Eq. (3) induces noise z
(O of M) on V Vyl(f.)) s.t., M E Rexe Mi, c(S - -//c)² + (c 1)/c. and Ma, e. S S. Sj, if # j. Note that the standard deviation of noise on the i-th entry is o, ((s 1/c)² + 1c/1//c
Proof. Dimension: y E R R By E RC, S E RC, V E Rixe, Vsl E
V Rexe
Firstly, for the softmax function S = S(f()), we have the derivative matrix
V A(S) ST
where A(S) is diagonal matrix with S; on its i-th diagonal element and 0 otherwise.
For the cross-entropy loss, {(f.g) - Ei= yk log Sg. Let denote element-wise division, we have
T
+
JyZy
V + = se
+ +ozzy)
vyS -
v.S
S
5 T v,S
T
.V,S
S
= vy((fn) - (
J
T
(A(S)
S 5T)
(8)
S
T
vy((f,y) By byy
y S
Then it is equivalent to induce noise 2 = -oy(fz Σi= zyp S) on V vf((f,)) y). whose i-th entry
is 2; = -oy(Ey EE= Zyy. S:). Note that zy ~ N(0, Texe), then we can derive for i = j.
E o1 2S, s?? (c((: 1/c)² (c 1)/c),
(9)
and for i A
E E[2,,]] = (c StS₁ - S; 12-5) -S})-
(10)
Proposition 3. For the cross-entropy loss, compared with Eq. (1), Eq. (3) induces noise z (0, M) on Vol(f. y), s.t. M e Roxp and M., = ( -o vo,5 Vi,j E (1.... p), where i denotes the element-wise division. Note that the standard deviation of noise on the i-th parameter
V..5 S
0, is oy 8
2
15

---

Published as a conference paper at ICLR 2021
Proof. Dimension: by E R' Vol Rx VsleR VoSERoX Vo.S Se.EE E R.. For Eq. (3), the noisy gradient is
y + Jyzy
vol(f.y) Vol(f.v + +g22)) = vel(f. 0o2z)) VoS
Vos
(I1)
y --(s) VoS
(0 T
VAS Vol(f.u)
(7520)² VOS.
S
The noise on Vol(f.y) is = of 2, we have
C v g
VaS. Note that By ~ N(O0 Lexo), let 24 be the i-th entry
0Sk Zy,.
Sk
2, = γy
0y
Zye
(12)
08,- S₁
Ske
Hence,
Vo, S
Vo,5
Vo.
E = 00
Blazzy = =o
(13)
5
5
5
B MORE DETAILS ON EXPERIMENT SETUP
B. CIFAR-I0 AND CIFAR-100
The backbone and general training hyperparameters. In all experiments on CIFAR-10 and CIFAR- 100, we train wide ResNet-28-2 (Zagoruyko & Komodakis, 2016) for 300 epochs using the SGD optimizer with learning rate 0.001, momentum 0.9. weight decay 5 x 10-4, and batch- size of 128. Standard data augmentation is applied, including per-pixel normalization, horizontal random flip and 32 32 random crop after padding with 4 pixels each side. The criterion for setting the training hyperparameters includes 1) all methods should converge (the training accuracy converges), 2) all methods share the same general training hyperparameters for fair comparison.
Method-specific hyperparameters. The backbone is not unified in previous papers and we reim- plement all methods in the same backbone for a fair comparison. Regarding this, we may not directly follow the default hyperparameters. Following Zhang & Sabuncu (2018), we use 5k noisy samples (10% of the training data) as the validation set tune method-specific hyperparameters. We then train the model on the full training set and report the test accuracy at the last epoch.
(ours). We tune a E {0.1, 0.2, 0.5, 1}. On CIFAR-10, we 1 for symmetric noise and 0.5 otherwise; On CIFAR-100, for instance-dependent noise and a = 0.2 otherwise. The momentum model is introduced with hyperparemeter 0.999 without tuning. The label correction (LC) is applied after con- vergence of training with SLN. All models are trained for 300 epochs and we introduce LC at the 250th epoch without tuning beacuse the training accuracy does not increase much after the 250th epoch. In this way, we do not increase the computation cost.
SLNISLN-NOSSLN-MO-
0.1
GCE (Zhang & Sabuncu, 2018). GCE loss is applied as the training starts and there is a warm epoch after which truncated GCE loss applied every 10 epochs. We tune the warm epoch in {0, 50, 100, 150, 200} and use 50 for CIFAR- 10, 150 for CIFAR-100. There is hyperparameter q for the GCE loss. We set q = 0.7 since it is used in all experiments on CIFAR-I0 and CIFAR-100 in its original paper (Zhang & Sabuneu, 2018).
Co-Teaching (Han et al., 2018b). The rate of selecting small-loss samples is linearly de- creased from to 1 - 5 at the first 10 epochs, where E is the noise rate. This setting is used in all experiments in the original paper (Han et al., 2018b) and it works well in our setting.
PHuber-CE (Menon al., 2020). There is hyperparameter 7 that controls the gradient clipping. The original paper (Menon et al., 2020) uses T = on CIFAR-10 and T = 10 on 100, but the default setting does not work well in our experiments. Hence, we tune T E {2, 5, 10, 30, 50) and finally, on CIFAR-10, we use т = 10 for asymmetric noise and - 30.
7 2 otherwise; CIFAR-100,
16

---

Published as a conference paper at ICLR 2021
LS (Lukasik al., 2020). There is hyperparameter that controls how much the label is smoothed. We tune a E {0.2. 0.5, 0.8) and finally, on CIFAR-I0, 10, we use a = 0.5 for asymmetric noise and a = 0.8 otherwise; on CIFAR- 100,, we use 0.8.
SIGUA (Han et al., 2020). The hyperpramemeter y is a factor that is multiplied on the loss on 'bad' samples. We tune it in {0.01, 0.001, 0.0001} and finally use 7 0.001 in all experiments.
DivideMix (Li et al., 2020). We tune the warm-up epoch and X the weight for unsu- pervised loss. In the official implementation, the warm-up epoch is 10 CIFAR-10 and 30 on CIFAR-100. The default hyperparameters do not work well in our experiments (we observe decrease of test accuracy after warm-up). Hence tune the warm-up epoch in {10, 30, 50, 100} but the performance does not improve compared with the default set- tings. Hence we use the default warm-up and tune Au. E {0, 1, 10, 25). Finally, we obtain impressive results with X = 0 in all experiments.
B.2 CLOTHINGIM
The backbone and general training hyperparameters. On ClothingIM, following the common setting (Patrini et al., 2017), we train an Imagenett-rettrined ResNet-50 using the SGD optimizer with momentum 0.9, weight decay 10-3 and batchsize 32. The initial learning rate is 10-3 and decreased to 10-4 after 5 epochs. We use standard data augmentation with per-pixel normalization, horizontal random flip and 224 x 224 random crop. Note that DivideMix uses the same backbone ResNet-50 but a different training schedule, and we follow its official implementation released on the GitHub.
Method-specific hyperparameters. Since the backbone ResNet-50 is used by default for most pub- lished results (Patrini et al., 2017; Li et al., 2020), we can easily follow the default hyperparameters.
(ours). Following previous methods (Patrini al., 2017; Li et al., 2020), the validation set cobtaining 14k clean samples is adopted to tune our hyperparameters. We tune {0.1, 0.2, 0.5} and choose a 0.2. The momentum model is implemented with hyperparemeter 0.999 without tuning. We fix the overall training epoch as 10 and tune the epoch for applying label correction {1, 5, 9}. Finally, we apply label correction after the first epoch in all experiments.
SLNSS.N-MOSSL--OO-.
Forward/Backwort (Patrini et al., 2017). The results is reproduced by reimplementing the method exactly following hyperparameters in Patrini al. (2017).
Co-Teaching (Han et al., 2018b). On ClothingIM, the estimated noise rate is around 0.4 (Xiao et al., 2015). Hence, we linearly reduce the rate of selecting small-loss sam- ples from 1 to 0.6 in 10 epochs.
DivideMix (Li et al., 2020). The result is reproduced from its official implementation.
C MORE EMPIRICAL RESULTS AND DISCUSSIONS
C.I SLN ENHANCES EXISTING METHODS
With detailed dissection predictions of DNNs trained with SLN (Section 3.3), we have shown that SLN can boost popular robust training methods such as label correction and sample selection. In this section, we verify this by integrating SLN with the following methods.
Co-teaching (Han et al., 2018b). It uses co-training and sample selection. Two modes select small-loss samples train each other.
Stochastic integrated gradient underweighted ascent (SIGUA) (Han et al., 2020). It adopts gradient descent on good data usual, and learning-ras wwtee-rrttteeeeeeeee gradient ascent on bad data.
DivideMix (Li al., 2020). combines co-training of two models, sample selection based the loss, label correction//uessingg based semi-supervised learning MixMatch (Berth- elot et al., 2019), and other techniques including regularization, augmenting each image twice.
17

---

Published as a conference paper at ICLR 2021
Table 4: Testing accuracy (mean-tstd in 5 runs) CIFAR-10. SLN consistently improves many robust learning methods. All methods are fairly compared using the same backbone wide ResNet- 28-2 and training hyperparameters (Appendix B).
Method
Symmetric
Asymmetric
Dependent
Open-Set
Co-Teaching
02.
79.61
76.69:10.66
85.21
LLN-Co-Teachin
84.22:10.43
87.79:0.117
80.37
90.37:1.322
Improvement
+1.85
+8.18
+3.68
+5.16
SIGUA
83.76:10.67
78.24
76.67
86.70::0.62
SLN-SIGUA
84.27:10.41
87.65:10.94
80.09:0.68
90.38:10.24
Improvement
+0.51
+9.41
+3.42
+3.68
DivideMix
90.3
87.88
80.37
90.4910.62
SIN-DivideMix
90.87=10.28
89.3140.39
82.86+0.41
91.65++0.59
Improvement
+0.49
+1.43
+0.65
+1.16
(overtit)
(robusti
ariplane
bird
cat
deer
dog
frog
horse
ship
truck
Figure 7: The t-SNE visualization of features on the model's penultimate layer.
Specifically, we use models trained with SLN as the initialization of these methods. As shown in Table 4, all methods obtain consistent improvement when integrated with SLN. Moreover, with SLN as initialization. we do not need to tune the warm-up phase for methods like DivideMix, because we can train with SLN until convergence. In contrast, without SLN, need carefully warm up the model so that it learns enough correct patterns and does not memorize too much noise. Note that better results for DivideMix are reported in the original paper with different backbone and carefully scheduled learning rate. We focus on fairly comparing the robustness of all methods: in all experiments, we train the same backbone wide ResNet-28- 28-2 for 300 epochs without learning rate change. Detailed training settings presented in Appendix B.
C.2 T-SNE VISUALIZATION OF FEATURES ON THE MODEL'S 'S PENULTIMATE LAYER
In Fig. 7, we show the (-SNE visualization of features on the model's penultimate layer, taking all training samples as input. We visualize the embedding on CIFAR-I0 with symmetric noise since the noise yields the most severe damage to the generalization and SLN provides significant improvement. Fig. shows that the model trained with SLN can yield a better embedding. It also demonstrates overfitting and underfitting when is too small or too large.
C.3 TEST ACCURACY W.R.T. TRAINING EFFICIENCY
In Fig. 8, we visualize the training efficiency and test accuracy. The figure clearly illustrates the superior generalization performance and high efficiency of our methods.
D LABEL CORRECTION
The convergence issue in label correction. When using the model's prediction to correct noisy labels, we find a convergence issue such that the test accuracy decreases, as shown in Fig. 10. The convergence issue is also reported in Arazo al. (2019), but it has not been widely discussed. In
18

---

Published as a conference paper at ICLR 2021
Symmetric
Asymmetric
Dependent
Open-Set
SD LC
SLN-MO-LC
SLN-MOILC
Co-Teachinge
Prajber- CEE
SLN-MO
SUM-MO
Accuracy
80
7.
OCF
Accuracy
80
Co-Teachinge
SLN*
Test Accuracy
SUN +
Test Accurscy
%
86
Co-Teachinge
SIN
CoTecchinng
6.5
75
PHuber CL
76
PHübir-CF #CCE
60
PHube REE
78
0.4
0.8
1.0
100
1/time Uh
-4
1/time (h-
1/time (h-
1/iia (h (h-
Figure 8: Test accuracy w.r.t. training efficiency (1/time) on CIFAR-10.
Symmetric
Asymmetric (CL)
Dependent (CE)
1000
1000
1000
label prodiction
label prediction
label x, prediction x
8.00
label x, prediction
800
label x, prediction
800
label x, prediction /
prediction
prodiction
prodiction x
samples
label prediction
samples
label prediction
# samples
600
label prediction
200
10
20
30
40
0
10
20
30
40
0
10
20
30
40
Intervals (loss
Intervals floss
Intervals (loss
Symmetric
Asymmetric (SUN)
Dependent
1000
1000
1000
label prediction
label prediction
label prediction x label x, prediction
800
label prediction
800
label prediction
800
label 7. prediction
label 7, prediction
label x, prediction x
samples
6.00
label prediction
samples
600
label predictionn
samples
600
prediction
10 20
30
40
0
10 20 30
40
0
10 20
30
40
Intervals (loss
Intervals (less
Intervals (loas
Figure 9: Training samples are sorted in ascending order of loss, uniformly divided into 1000 sam- ples per interval, and dissected according to the correctness of the given label and the prediction. This figure exactly the same as Fig. 4 in the main paper. We present it here for your convenience since we refer to the figure in Section D.
this section, we provide an intuitive hypothesis. We can analyze the effect of label correction on the four types of samples, as shown in Fig. 9.
Label correction can correct these samples, yet may make the case worse because the prediction error should be easier for the model to overfit compared to given noisy labels.
label x, prediction
2) label prediction v. These samples benefit from label correction.
label v. prediction Label correction is harmful for these samples.
4) label v. prediction V. Label correction does not significantly impact these samples because both the prediction and the label are correct.
In word, there exist samples of case 1) and case 3) that are not desired in label correction, but they are unavoidable since we do not expect a model with 100% prediction accuracy. In label correction, when trained with modified labels obtained from wrong predictions, the model may accumulate its own error due to positive feedback: yielding worse predictions after training on prediction errors. Therefore, hypothesize that samples of case and case 3) result in the convergence issue.
19

---

Published as a conference paper at ICLR 2021
Symmetric
Acymmetric
Dependent
Open-Set
90
90
90
82
84-
84
84
76
10
78
78
78-
w.y wi
76
wry
60)
76
w
w
-5
76
200
220
240
260
200
220
240
260
70
200
220
260
300
200
220
240
260
300
Fpoch
Epoch
Fpoch
Fpach
Figure 10: The test accuracy on CIFAR-10. 10, averaged in 5 runs. We zoom in to focus on the effects of label correction applied in the last 50 epochs. The open-set noise involves samples that do not belong to any class considered in the task, for which label correction can not address.
How do we assign weights in label correction? We use Vcorrection y (1 w) S, as opposed to Fcorrection (1 ---) w) y + w. S, where w is an instance-dependent weight positive correlated with the loss. We observe that our scheme mitigates the convergence issue, shown in Fig. 10. Our intuition that samples that need correction have large losses is consistent with the method of Arazo et al. (2019), but this does not mean that the weight on S should be positive correlated with the loss. We can consider the following cases.
For soall-loss samples, we have R y, as illustrated by the red region Fig. 9. Label correction does not affect these samples much regardless of the weight. Hence, we simply need to consider the effects of label correction on large-loss samples.
Large-loss samples of case can benefit from label correction, as illustrated by the orange region in Fig. 9. However, in this case, higher loss does not mean that it requires higher weight on for label correction 4
There exist large-loss samples of case 1) and case 3) for which label correction can be harmful, as has been discussed in the above paragraph and illustrated by the blue and green regions in Fig.
Therefore, assign small weight of 1 case 2) still benefit from label correction, while we mitigate the undesired effects on other large-loss samples for which label correction can be harmful. The effectiveness of our scheme is verified in Fig. 10.
S for large-loss samples. In this way, samples of
E THE CONVERGENCE
In Fig. 2, the visualizations of loss landscapes show that the model trained with SI.N converges to a solution that has small SGD noise. The center point the visualized landscape (i.e., the loss of the given model) is a local minimum. From Fig. 2 (d), we observe that the minimum has the following properties.
The gradient around the minimum is small since it is flat.
The predictions do not approach one-hot labels because the loss at the local minimum is high. shown in Fig. the prediction probabilities are much lower than
With the above two properties, Proposition 3 implies that around the flat minimum illustrated in Fig. 2 (d), the noise on gradients is small. Therefore, the model can converge in the local flat minimum.
*For example, considering two samples with the wrong label y1 y2 = [1, 0, and latent true label [0, 1,0), Imaging the predictions are S₁ = [0.4, 0.6, 0J, S₂ = [0.3, 0.3). Then the cross-entropy loss is f(y1, S1) < {y2, S2). while for the weight on S, we want w1 > because S1 is more correct compared with S2 (the second entry 0.6 0.4). This example implies that for samples that can henefit from label correction, higher loss does
that it requires higher weight on S for label correction.
20


Reviews:
[{"review_id": "hrJCNdyhg-", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary** The paper tackles the problem of training under noisy labels. It proposes adding random  zero-mean Gaussian noise to the labels during training. It is shown that such a noise induces a variable gradient noise which adaptively increases 1) when the learnt network function has higher curvature around training points and 2) when the output prediction has lower entropy (higher confidence) for the training points. It is claimed that the former property helps avoiding sharper minima which generally improves generalization. The second property avoids overfitting to noise in similar fashion to label smoothing. \n\n**Quality** The paper is well written. The “related works” section is quite thorough but concise when covering the field of learning under label noise. Certain parts such as the tSNE plot (Figure 5) and the time-vs-accuracy plot (Figure 6) do not seem to be central to the paper and are not informative. Instead, certain parts could have been discussed more thoroughly (see the detailed technical comments below).\n\n**Clarity** The claims and contributions are generally clear from the paper. The reasoning behind certain claims could be better clarified. Also, details are missing on the hyperparameter optimization of the baselines. Details come below.\n\n**Originality** The method is close to other works in the analysis of noisy gradients for better generalization as well as the usage of label smoothing and random label perturbation [a] for generalization and learning under label noise. However, to the knowledge of the reviewer, this work combines the ideas of the two directions in a coherent and original way.\n\n**Significance** The experiments are done on several setups and using 5 different independent runs for the baselines and three variants of the proposed method. The results show significant and consistent improvements. However, certain experiments could be added to better support the detailed claims as opposed to merely reporting best final numbers.\n\n**Major technical comments**\n\n*Experiments*\n1. Interesting and informative side experiments including 1) the separation of noisy and correctly labeled data when using SLN compared to label smoothing, and 2) the strength of SLN compared to unnoisy cross entropy loss indicating the suitability of SLN in identifying correctly labelled samples (low-loss regime) and correcting noisy labels (high-loss regime).  \n2. 5 different synthetic noise types are used for the experiments on CIFAR10 and CIFAR100.\n3. the paper has two clear claims regarding the sharpness of the found local minimum and on overconfident predictions. While the experiments show clear improvements of the results across various settings and compared to different baselines, the connection of the improvements to the claims remain largely unsubstantiated. As such, the paper is missing direct experiments supporting the claims and/or shedding some light on them. Some suggestions are as follows:\n\n     3.1. implement the noise as in proposition 3 i.e., directly applying the noise to the gradient. Then, one can modify the noise to decouple the two components and demonstrate individual contributions.\n\n     3.2. quantitatively analyze the jacobian of the learnt network function and/or the sharpness of the local minimum with and without the added noise and for the different kinds of noise to directly investigate the first claim.\n4. the performance of some of the baselines are low. Are the baselines reimplemented? How are the general training hyperparameters (learning rate, weight decay, batchsize, etc) and method-specific hyperparameters optimized? Is there a different set of optimized hyperparameters per baseline? What are them?\n5. reading the end part of section 3.3 it seems that in SLN+MO+LC the label correction starts only after the full convergence of a SLN-only training. In light of this, how is that the increase in time complexity of SLN+MO+LC is negligible in Table 1? What is the stopping criteria for the initial training and then retraining with LC?\n\n*Theory*\n1. The paper misses to cite a relevant paper [a] that also randomizes the labels for avoiding overfitting and demonstrates better generalization. Regarding this, the paper should clearly acknowledge [a] when it comes to claiming the novelty of the noisy-label approach and also when it discusses the advantages of perturbing labels -- [a] discusses similarities to ensemble approaches. That being said, I believe the paper has enough originality on top of [a]: for instance [a] replaces provided labels with independent noise, does not experiment on learning under noisy labels and, the gradient noise analysis of the paper is complementary to [a]. \n2. When correcting the labels using SLN, why is the weight of the given label increases as the sample loss increases? Shouldn’t it be the opposite based on figure 3? I found that there are some discussions provided in appendix C. However, as this goes against the previously published work it deserves more formal discussion and corresponding experiments in the main paper.\n3. From what I understand proposition 3 shows that functions that are smoother at training points will receive lower variance. If so, formal discussions are missing to connect smooth functions and flat minima.\n4. As the training continues the loss tends to get smaller by getting the function closer to the given one-hot labels at the training data points. From proposition 3 it is argued that as this happens the noise in the gradient increases. This raises a caveat regarding convergence. A theoretical discussion and/or empirical observation are needed to study the convergence. For instance, does the variance of the model increase towards the end of the training or does it actually converge to a solution that is robust to the gradient noise (remains approximately unchanged in the functional space)?\n\n**Overall** In the reviewer’s view, the paper has clear merits in bridging between the theory of gradient noise and label smoothing for learning under label noise, both empirically and theoretically. However, it can benefit from more clarity and additional informative experiments to better understand the effect of the proposed noise.\n\n[a] \"DisturbLabel: Regularizing CNN on the Loss Layer\", CVPR 2016\n\n**Post Rebuttal Update** \nThe authors address many of the concerns, 1) [a] is properly acknowledged in the revised version and novelty is not claimed on additional label noise in the text, 2) while quantitative studies are still absent for claims on sharp vs. flat minima, qualitative results are provided for convergence to \"flatter\" minima  3) connections between smooth functions and generalization is discussed 4) answers and updates regarding complexity and convergence are *somewhat* convincing. Thus, I am willing to increase the score from 6 to 7 and confidence from 3 to 4 as I believe the paper provides relevant and interesting theoretical arguments.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hlsx5-pGMDY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies learning robust models with noisy labels. The authors argue that a specific SGD noise induced by stochastic label noise (SLN) can mitigate the effect of label noise. But the common SGD noise cannot achieve this. Then they apply the proposed SLN induced SGD noise to the existing label-correction methods for noisy-label learning and provide some experimental results.\n\nPros:\n\n-The paper provides an interesting view of SGD noise in the lens of noisy labels. They claim that common SGD noise does not endow much robustness against label noise, but using a variant SGD noise by label perturbations can improve the generalization and boost existing robust training methods.\n\nCons:\n\n-Learning with noisy labels is a hot research area as reviewed in the related work section. It seems that the selected baselines in the experiments are not representative and state-of-the-art methods. For example, Yu et al. (2019) improves co-teaching Han et al. (2018) and should be compared instead. \n\nAnd some representative regularization based methods should also be compared:\n\n-Mixup: Beyond Empirical Risk Minimization, Zhang et at., ICLR 2018\n\n-Virtual adversarial training: a regularization method for supervised and semi-supervised learning, Miyato et al., TPAMI 2019,\n\n-SIGUA: Forgetting May Make Learning with Noisy Labels More Robust, Han et al., ICML 2020\n\nsince they are more related to the essence of training networks as claimed in the paper. \n\n-In Claim 1 and 2, it is said that with SGD noise induced by SLN training is difficult to converge in some cases. How to guarantee convergence of the proposed algorithm? Some convergence analysis under reasonable assumptions may be helpful.\n\n-It is claimed in the paper that training without SGD noise under label noise can converge to sharp minima, and SLN helps escape from the sharp minima. It is not very intuitive. Could the authors explain more on it, maybe adding some citations or experimental results could be helpful.\n\n-It is still unclear to me how to tune the standard deviation sigma in practice, which should be an important factor that affects the performance.\n\n-The clarity of the paper could be improved, for example, adding brief proof sketches to the theorems may help for better understanding.\n\nOverall, the paper provides some interesting analysis of SGD noise and label noise, but many unclear parts need to be clarified.", "extra_scores": {}, "extra_text": {}}, {"review_id": "S6Kx7Z5-rQl", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nmitigate inherent label noise\nThis paper studies the effect of applying SGD noise on mitigating the inherent label bias which is common in real-world datasets. It introduces stochastic label noise (SLN), a variant of SGD noise induced by controllable label noise. It formalizes connections between SLN and two other existing SGD noise variants (Proposition 1-3). With such propositions, it shows that SLN can help the model to avoid sharp minima and prevent overconfidence (Claim 1-2). The experiments show that SLN helps improve generalization than baseline methods and can be further used to boost robust training methods on CIFAR10, CIFAR100, CLOTH-1M under five different types of noise settings. Apart from vanilla SLN, it further proposes momentum model (MO) and label correction (LC). Combining them together with SLN can further boost test accuracy for label-correction.\n\n\n################################################\n\nReasons for score:\nThe paper is overall very well-written and gives theoretical insight on the connections of different SGD noise variants. It further provides comprehensive experiments both qualitatively and quantitatively to validate the effectiveness of its proposed SGD noise variant, SLN. The results show that SLN can simplify parameter tuning, producing superior results on label-correction without additional computational overheads.\n\n################################################\n\nPros:\n\n+very well-written and easy to follow in general\n\n+great connection with as well as comprehensive discussions of related work\n\n+comprehensive experiments along with good visualization\n\n+the proposed method gives better performance without overhead and can be used to enhance existing methods\n\nCons:\n\n-might be better to also give empirical evidence to support the claim of helping escaping sharp minimums. e.g. a visualization of gradient landscapes for CE and SLN, respectively.\n\n\n################################################\n\nQuestions:\n\n-As I mentioned in the cons, is there any empirical evidence of escaping sharp minima you observed to further support the theoretical finding?\n\n-I see that in Fig 5, you give a qualitative visualization of using different sigma. Did you also do any quantitative ablation study on the hyperparameter sigma? How sensitive the results would be by choosing different sigma?\n\n################################################\n\nPost Rebuttal Update: the authors have well addressed my concerns, in particular (1) the additional visualization gives a good qualitative empirical evidence supporting the claim that SLN helps escaping sharp minima. (2) the search process for the hyperparameter $\\sigma$ is very reasonable and makes the usage of SLN practical. I will keep my initial assessment and vote for accepting this paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7rU7TtgTBAw", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the noisy stochastic gradient descent algorithm in noisy label learning. Concretely, the authors added Gaussian noise on the labels rather than the gradient itself. By comparing different noisy SGD algorithms, the authors demonstrate that the proposed SLN algorithm not only help the model to escape from sharp local minima, but also help it to be over confidential. Finally, the authors demonstrate that SLN outperforms some classical SGD methods.\n\nPros:\n- The proposed idea is novel to me. Theoretically, the proposed SLN framework jointly enjoys the ability to escape from sharp saddle points and make the prediction smooth. The authors provide a new perspective to develop robust learning algorithms.\n- It seems that SLN method can be integrated with many state-of-the-art noisy label learning models.\n- The experimental results in Figure 3 is very promising. The small loss samples are generally clean, which may help improve the performance of many sample-selection based approaches.\n\nCons:\n- I have one main concern. While Figure 3 shows very good results, I noticed that quantitative results are far away from state-of-the-art models. Compared to state-of-the-art models, such as DivideMix, SLN demonstrates far lower accuracy. For example, on CIFAR-10, Asymmetric noisy with 40% noise, the accuracy of DivideMix is 92~93.4% and SLN-MO-LC is 87.85%. Although state-of-the-art performance is not the most essential for me, I think the authors require more experimental exploration. Since SLN is a rather flexible method, it can be integrated with many state-of-the-art models and I believe the performance would be competitive or at least at the same level as the SOTA models. \n\nMinor comments:\n- While Figure 3 shows the clean sample ratio of converged models w.r.t different loss intervals, what would it be after the first few epochs?\n- From my point of view, SLN is actually a new strategy of label smoothing (or not?). May the authors explain the superiority of SLN to conventional label smoothing methods?\n\nOverall, I think this paper brings an interesting idea to the community, but the experiments are not enough for me.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the effect of explicitly introducing stochastic label noise into SGD updates, showing both theoretically and empirically that this can improve model performance on datasets with "inherent" label noise. The intuition is that this helps the model escape sharp local minima, where predictions may be overconfident. 

Reviewers broadly found the work to be conceptually and theoretically interesting, and the empirical results are promising. The paper is thus well-posed to be of broad interest to the community.
Decision: Accept (Spotlight)