Do We Always Need to Penalize Variance of Losses for Learning with Label Noise?
Authors: Yexiong Lin
Yu Yao
Yuxuan Du
Jun Yu
Bo Han
Mingming Gong
Tongliang Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: Algorithms which minimize the averaged loss have been widely designed for dealing with noisy labels. Intuitively, when there is a finite training sample, penalizing the variance of losses will improve the stability and generalization of the algorithms. Interestingly, we found that the variance of losses sometimes needs to be increased for the problem of learning with noisy labels. Specifically, increasing the variance of losses would boost the memorization effect and reduce the harmfulness of incorrect labels. Regularizers can be easily designed to increase the variance of losses and be plugged in many existing algorithms. Empirically, the proposed method by increasing the variance of losses could improve the generalization ability of baselines on both synthetic and real-world datasets.

Under review as a conference paper at ICLR 2023
Do WE ALWAYS NEED TO PENALIZE VARIANCE OF LOSSES FOR LEARNING WITH LABEL NOISE?
Anonymous authors Paper under double-blind review
ABSTRACT
Algorithms which minimize the averaged loss have been widely designed for deal- ing with noisy labels. Intuitively, when there is a finite training sample, penalizing the variance of losses will improve the stability and generalization of the algo- rithms. Interestingly, we found that the variance of losses sometimes needs to be increased for the problem of learning with noisy labels. Specifically, increasing the variance of losses would boost the memorization effect and reduce the harm- fulness of incorrect labels. Regularizers can easily designed to increase the variance of losses and be plugged in many existing algorithms. Empirically, the proposed method by increasing the variance of losses could improve the general- ization ability of baselines on both synthetic and real-world datasets.
INTRODUCTION
Learning with noisy labels can be dated back to 1980s (Angluin & Laird 1988). It has recently drawn a lot of attention (Liu & Tao 2015 Nguyen al. 2019 Li et al. 2020 2021 because large- scale datasets used in training modern deep learning models can easily contain label noise, egg. ImageNet (Deng et al., 2009) and Clothing (Xiao al. 2015). The reason that it is expensive and sometimes infeasible to accurately annotate large-scale datasets. Meanwhile, many cheap but imperfect surrogates such as crowdsoureing and web crawling are widely used to build large-scale datasets. Training with such data can lead to poor generalization abilities of modern deep learning models because they will overfit noisy labels (Han et al., 20186 Zhang et al.) 2021).
Generally, the algorithms of learning with noisy labels can be divided into two categories: statisti- cally inconsistent algorithms and statistically consistent algorithms. Methods in the first category are heuristic, such as selecting reliable examples to train model (Han et al.) [00 Malach & Shalev-
Shwartz 2017 Ren et al. 2018 Jiang et al. 2018), correcting labels (Ma et al.) [2018] Kremer et al. 2018 Tanaka al. [0118] Reed al.] 2014)) and adding regularization (Han et al. 2018a Guo al.
2018 Veit et al. 2017] Liu et al. 2020). Those methods empirically perform well. However, it is not guaranteed that the classifiers learned from noisy data statistically consistent and often need extensive hyper-paramette tuning on clean data.
To address this problem, many researchers explore algorithms in the second category. Those algo- rithms aim to learn statically consistent classifiers (Liu & Tao 2015) Patrini et al., [017]] [iu et al.; 2020 Xia al. 2020). Specifically, their objective functions are specially designed to ensure that minimizing their expected risks on the noise domain is equivalent to minimizing the expected risk on the clean domain. In practice, it is infeasible calculate the expected risk. To approximate the expected risk, existing methods minimize the empirical risks, iee. the averaged loss over the noisy training examples, which an unbiased estimator to the expected risk (Xia et al.) 2019 Li et al.) 2021) because their difference will vanish when the training sample size goes to infinity. However, when the number of examples limited, the variance of the empirical risk could be high, which leads to a large estimation error.
However, we report that penalizing the variance of losses is not always helpful for the problem of learning with noisy labels. By contrast, in most cases, need to increase the variance of losses, which will boost the memorization effect (Bai & Liu 2021 and reduce the harmfulness of incorrect labels. This is because deep neural networks tend to learn easy and majority patterns first due to the memorization effect (Bai & Liu 2021 Zhang et al., 202T). The incorrectly labeled data is of minority and has more complex relationship between instances and labels compared with correctly

---

Under review as a conference paper at ICLR 2023
Test accuracy: 86 63%
Test accuracy. 88.85%
Test accuracy. 90 04%
Traaking fou indonacciti comee
"ruiing hos irmmancc with insonet loul
100 0
20
80
80
100
20
80
100
Epcohs
Epoctt
Epachs
(a) Penalizing variance of losses
(b) Original loss
(c) Increasing variance of losses
Figure 1: We visualize the averaged training loss of instances with correct labels (blue dashed lines) and instances with incorrect labels (yellow solid lines) obtained by penalizing the variance of losses, employing original loss, and increasing the variance of losses in (a)-(c), respectively. The dataset is CIFAR-10 with conmetry--lipiing noise, and the noise rate is 0.2. The neural network ResNet- 18 and the baseline Forward (Patrini et al. 2017) are employed. The transition matrix T is given and does not need to be estimated.
labeled data, then incorrectly labeled data is harder for neural networks to remember. Therefore, the losses of instances with incorrect labels are likely to be larger than those of correct instances (Han) et al., 2018b). Penalizing the variance of losses could force the model to reduce the loss of the instances with incorrect labels because the correct labels are of majority and their losses are smaller, making it hard to distinguish correctly and incorrectly labeled data and will lead to performance degeneration. In contrast, increasing the variance of losses could efficiently prevent large losses from decreasing, then the model may not overfit instances with incorrect labels. In Section 3, further show that increasing the variance of losses can be seen as a weighting method that assigns small weights to the gradients of large losses and large weights which could reduce the effect of instances with incorrect labels when updating model's parameters. More discussions about the memorization effect can be found in the Appendix.
the gradients of small losses,
Intuitively, as illustrated in Fig. [ change of the variance of losses does not have much influence on the averaged training loss of instances with correct labels, but makes the averaged training loss of instances with incorrect labels very different. Specifically, pennaaii the variance of losses leads the averaged training loss of instances with incorrect labels decreasing fast, which will encourage the model to overfit instances with incorrect labels. On the contrary, increasing variance of losses can prevent the averaged training loss of instances with incorrect labels from decreasing as shown in Fig. Tc Therefore, the memorization effect are boosted. As a result, the test accuracy is improved significantly by encouraging the variance of losses.
From the empirical risk minimization perspective, are encouraged to reduce the variance of losses to increase algorithmic stability. However, to handle label noise, as explained, we may need to boost the variance of losses. This implies that the label noise issue should be carefully considered when designing the loss variance part of learning algorithms. We empirically report that the variance of losses should be boosted in most settings of learning with noisy labels studied in the literature.
The rest of this paper is organized as follows. In Section 2, we introduce related work. In Section 3, propose method and its advantages. Experimental results both synthetic and real-world datasets are provided in Section 4. Finally, we conclude the paper in Section 5.
2
RELATED WORK
Some methods proposed reduce the side-effect of noisy labels using heuristics, For example, many methods utilize the memorization effect to select reliable examples (Han et al. 2020; Yao
| Kremer al., 2018 Tanaka et al.) 2018 Reed et al. 2014). Those methods empirically perform well. However, most of them do not provide statistical guarantees for the learned classifiers on noisy data. Some methods treat incorrect labels as outliers and focus on designing bounded loss functions (Ghosh) ctt al., 2017 Gong et al., 2018 Wang et al.) 2019 Shu et al. 2020). For example, a symmetric cross- entropy loss has been proposed which has proven to be robust to label noise asymptotically (Wang et al. 2019). These methods focus on the numerical property of loss functions, and the designed loss function can be proved to be noise-toleran if the noise rate is not large.
al. 2020al Yu et al. 2019 Jiang et al. 2018) or to correct labels (Ma al. [0188]
The label noise transition matrix T (x) E [0, CxC (Patrini et al. 2017 Liu & Tao 2015 Li et al.
2021) where C' is the number of classes, has been widely employed to design statistically consistent
2

---

Under review as a conference paper at ICLR 2023
classifiers (Liu & Tao 2015 Patrini al. 2017 Xia al. 2020 Li al. 2021). Let the clean class
posterior P(Y X = x) = P(Y = 1 = 2)
= C = x]] can be inferred by utilizing the noisy class posterior P = x) and the transition matrix T(x), where Tū =
P(Y P
= z). Then, the expected risk of function ((X,Y) modeling P(Y IX can be formulated as the expected risk of functioo g(X,, Y) ( Y)). In practice, not be calculated, existing methods approximate the
P(Y = = x. Le.. P(Y = x) =
modeling P(Y|XX multiplied by T(X), i.e. R()(X,Y))
the expected risk R((T(x))='9(X,, Y)) expected risk with the averaged loss over the noisy training examples. Normally, when the number of examples is limited, the variance of the losses the empirical risk could be high, which could make the algorithm unstable and lead to large estimation error.
Different variance reduction methods have been developed in many fields. For example, Trun- cated Importance Sampling (lonides 2008) limits the maximum importance weight, which solves the problem of infinite variance and decreases the mean squared estimation error of the standard importance sampling. |Anschel et al. (2017) proposed to stabilize training procedure and improve performance by reducing approximation error variance in target rewards. (Achab et al. (2015) pro- posed to use stochastic gradient descent (SGD) with variance reduction for optimizing a finite aver- age of smooth convex functions, and linear rate of convergence under strong convexity is proved. Similarly. Ahllen-Zhu & Hazan (016) proved that fast convergence rate can be achieved by using variance reduction on the non-convex optimization problem. Although the definitions of variance different, those works motivate us explore the role of variance of losses in learning with noisy labels because it is natural to consider that penalizing the variance of losses will have some benefits similar previous works.
3 ENHANCING VARIANCE OF LOSSES FOR LEARNING WITH NOISY LABEL
In this section, we propose our method, i.e., losses Variance Regularization for label-Noise Learning (VRNL). We reveal how the proposed method reduces the negative effects of incorrect labels. We also illustrate the advantage of our regularizer when working with existing methods.
3.1 METHODOLOGY
We show that the proposed method can efficiently prevent the model from learning incorrect labels. Intuitively, encouraging the variance of losses can prevent losses of instances with incorrect labels from decreasing, and promote the reduction of losses of instances with correct labels. Theoretically, the gradients of large-loss examples will be assigned with small weights, and gradients of small-loss examples will be assigned with large weights. As a result, the model puts more trust on small-loss examples; and small-loss examples will contribute more to the update of parameters, which could reduce the harmfulness of instances with incorrect labels.
To analyze the regularization effect in our method, we have to define some notations here. Let Var[-] denote variance of a distribution. For a random variable X, Var[X] = E[X*] - [X]. Let C denote the number of classes. Let fo : X + C-1 be mapping parameterized by 0 (e.g., a neural network), where A C- denotes probability simplex. Generally, the expected risk w.r.t. noisy data is formulated as , (X), Y)), where e(.) is the loss function employed. Following
analysis can be applied to all methods which can be formulated as EE (e(ffo() Y)], where E(-), including cetissicalll-consistee methods and cross-entropy loss-based methods. We propose to add variance regularizer to the losses. Specifically, the objective function of our method is
Ro(fs) = Ex,, ((xx,,)(([(6X)) ( (X()
where Var (x. (x.x))((ffo (X(, Y)] is a regularization term, and a is an adjustable control the strength of the regularization effect. To encourage the variance of losses, a is chosen to be a positive value. Usually, the strength of the regularization effect should not be too large, and is much smaller than 1. A suitable a can be obtained by utilizing validation sets. More details and discussions can be found in Appendix.
to
3

---

Under review as a conference paper at ICLR 2023
To exploit the influence of our designed regularizer with respect to the update of parameter 8, first illustrate the derivative of the objective function to 0, i.e.
Ro(So) 00
aR, (x. v, (((fo(X).
8Var (x (x,8) (x,xx,,,1,,00000..... (x) VD]
(e(fo(X),
F(x,))) w
(2)
00
00
00
where W 1 2a
- For specific example (x, D). its 80
corresponding gradient is w (((f(()).)) where the weight w/ is w = 1 + (x (())))((())))))))))))))))
(x)( ū)). As aforementioned. a is chosen to be small such that w should be positive. The above equation shows that 1) if the loss of the example (I, 5) is smaller than the expectation of the losses, ( (X),, -
will positive, and the weight associated with its gradient larger than 1. Then the example contributes more to the update of the parameter 0. 2) If the loss of the example (xr, ûi) is larger than the expectation of the losses, (x, (X), Y)j - {(fo(z),
will be negative. The weight associated with its gradient is small. Then the example contributes less to the update of parameter 0.
Due to the memorization effect, deep neural networks tend to learn easy examples first and gradually learn hard examples (Han et al. 2018b; Arpit et al., 2017). In learning with noisy labels, large-loss examples are more likely to have incorrect labels and should not be trusted (Bai et al.. 2021). By employing the proposed method, the gradients of examples with incorrect labels are assigned with small weights. In such a way, incorrecty-llabeled examples would have less contribution to update the parameter 0,, which prevents the model from overfitting incorrect labels.
Additionally, compared with existing small-loss based methods, our method can sufficiently ex- ploit the information contained the whole training dataset. Existing learning with noisy labels methods (Han et al. 8b; Nguyen et al., 2019) usually divide the training sample into confi- dent examples and unconfident examples based on the small- trick (Jiang et al., 2018; Malach & Shalev-Shwartz 2017 ei al. 2020). To be specific, the examples with large losses are unconfident examples, and their labels are ignored. However, some of the unconfident examples are hard-clean examples that contain useful information for learning noise-robust classifiers (Bai & Liu (02T). In contrast, our method does not ignore unconfident examples but assign them with small weights such that all the label information of the training dataset has been carefully exploited.
3.2 PRACTICAL IMPLEMENTATION
In practice, the expected risk Ra(fe) in Eq. I can not be calculated, the empirical risk is employed as an approximation. Let n be the number of training examples, generally, the empirical risk of our method is as follows:
R(f)) = Σ ((f(Zz))) -
1 (solzz), 0z)² 1
R(fa)
((fo(zs),
(3)
We further illustrate specific forms and settings of our designed regularization working with exist- ing methods, i... Importance Reweighting (Liu & Tao 2015). Forward (Patrini et al. (007)). and VolMinNet (Li et al. 2021). Empirically, our method improves their classification accuracy.
Work with Forward. Forward correction (Patrini et al. 2017) exploits the noise transition matrix T to estimate the clean class posterior distribution. We use the same method in the original paper (Patrini et al., 2017) to estimate the transition matrix.
The objective loss function by combining our method with Forward can be formulated as follows:
1 Σ (CE(T (o(s))
Retrwaal (.T)
n
Lee (T
- tor tox
where lcg is the cross-entropy loss, f is the estimated transition matrix, fo models the clean class- posterior distribution, Í fo models the noisy class-posteric distribution.
4

---

Under review as a conference paper at ICLR 2023
Work with Importance Reweighting. Importance Reweighting uses the weighted empirical risk estimate the empirical risk with respect to clean class-posterior distribution (Liu & Tao 2015). To calculate the weight the empirical risk, both noisy class-posterior distribution and clean class- posterior distribution need to be estimated. The objective loss function by combining our method with Important Reweighting can be formulated as follows:
RIn(Fo) - Σ ((lc)((((2)).)))
Σ ((oe(((()))..))
1 Σ (l
where Ba = Po(u(Ze)) D is the clean distribution, Dp is the noisy distribution. The gradient of Pb, (M:lo))
RIR w.r.t. an example (xi, W1) is as follows:
Σ di (1os(t6(e)) 03. Hlce(s((a)).) 86)
VRin(fo, (z,
+ B, 00
where wi = =1 2a Bylre(/ll(f -Bf))))())))(((()))))))))))))))) (&Cr((((zs)...)
+ 3, (f(x()) is the gradient of the original Importance Reweighting loss. When the label 09
The (CE(fol)) () 08
y1 is incorrect, the reweighted loss (f ū.) is usually larger than the averaged loss - (f Mz)) Then their difference is negative, which lead the weight úē; be small because the hyper-parameter a is positive. As result, the instance with incorrect label has small contribution to the update of parameter 0, the proposed method can prevent the model from memorizing the incorrect labels.
In the implementation, the early stopping technique is used for the approximation of the clean class- posterior distribution. Specifically, the model So is trained on noisy data with 20 epochs, and we feed the model output to a softmax function, then use the output of the softmax function g(x) to approximate the clean class- posstroostrror distribution. The noise transition matrix T has also been esti- mated by using the same approach as in Forward correction. Then the model fe is further optimized by both weighted empirical risk and regularization for the variance losses as follows:
go(z;)
Rin(())
(cE(fo(z:).. 9.)
agg.
(T)))(us)
where
2
(a(
(a()))
0%
(cE(fo(z.))
(CE(fo(r))))))
(Tg)s(z.)
(Tg)y (x.)
Work with VolMinNet. VolMinNet is an end-to-end label-noise learning method that learns the transition matrix and the clean class-posterio distribution simultaneously (Li et al. 2021). It opti- mizes two objectives: 1) trainable diagonally dominant column stochastic matrix T by minimizing the determinate log det((T); ; 2) the parameter of the model by the cross-entropy loss between the noisy label and the predicted probability by the neural network. regularizes the parameter by calculating the variance of cross-entropy losses. The objective by combining our method with VolMinNet can be formulated as follows:
experiments, our VRNL only
Rewl (0, f)
Σ n
lee fola))) B.) + Alog det(())
- leE (CE(T)o(m))))
where À >> O is an adjustable hyper-parameter, we set À = 0.0001 in all experiments. The transition matrix T should be differentiable. diagonally dominant and column stochastic.
Our method could help the state-of-the-art method VolMinNet (Li et al. 202I) to better estimate the transition matrix and the clean class-posterior distribution. Specifically, VolMinNet requires the to be diverse, which called the sufficiently seattered assumption (Li et al. 2021). By increasing the variance of the loss, the diversity of the estimated noisy class posteriors is encouraged, so the estimated clean class- posteriors are also encouraged. Then transition matrix can be better learned, which leads to the clean class-posterior distribution being better estimated.
clean
5

---

Under review as a conference paper at ICLR 2023
17.5
15.0
15.0
10.0
7.5
7.5
255
25
0.0
0.0
40
100
20
40
60
80 100
0
20
40
60
80
100
(a)
(b)
(c)
(d)
Figure 2: The change of losses with the increasing training epochs for Reweighting. (a) and (b) illustrate CE losses of P(Y|X) without with increasing variance of losses, respectively. (c) and
(d) illustrate CE losses of (X) without with increasing variance of losses, respectively.
0.35
0.35
Thoing aocred
0.30
0.30
8
0.25
0.25
0.20
100
20
-0
100
(a)
(b)
(c)
(d)
Figure 3: The change of losses with the increasing of training epochs for VolMinNet. (a) and (b) illustrate CE losses of P(Y|X) without or with increasing variance of losses, respectively. (c) and
(d) illustrate CE losses of P(Y without or with increasing variance of losses, respectively.
EXPERIMENTS
In this section, we first illustrate the empirical results of VRNL and other baselines on both synthetic and real-world noisy datasets. What is more, will delve into the different effects of the proposed method on correct and incorrect examples to verify its effectiveness on Sec. 3 Finally, we will illustrate the robustness of the proposed method when the estimated transition matrix is biased.
Datasets. We verify the performance of proposed method on the manually corrupted version of three datasets, i.e., MNIST (LeCun al., 2010), CIFAR-10 (Krizhevsky et al. 2009) and CIFAR- 100 (Krizhevsky et al. 2009). and one real real-world noisy dataset, i.e., Clothing M (Xiao et al. 2015). We leave out 10% of training data as validation sets. The experiments are repeated five times on the synthetic noisy datasets. Clothing IM (Xiao et al. (015) contains 1M images with real-world noisy labels, it also contains 50k, 14k, and 10k images with clean labels for training. validation, and testing, respectively. Existing methods like Forward (Patrini et al., 2017) and T-revision (Xia et al.) 2019) use the 50k clean data to initialize the transition matrix and validate on 14k clean data. We assume that the clean data is not accessible, therefore, the clean data are not used for training and validation. We leave out 10% examples from IM noisy data for validation.
The baselines used in our experiments: 1). CE, standard cross-entropy loss; 2). Decou- pling (Malach & Shalev-Shwartz/ 2017) trains two models at the same time, and only the instances which have different predictions from two networks
Baselines.
used to update the parameter; 3). Men- torNet (Jiang et al. 2018) pre-trains an extra model which used to select clean examples for the main model training: 4). Co-teaching (Han et al.) 2018b) trains two networks simultaneously. and each network is used to select small-loss examples as trust examples to its peer network for further training: 5). Forward (Patrini al., 2017) estimates the transition matrix in advance, then uses to approximate the clean class posteriors; 6). T-Revision (Xia et al., 2019) proposes method to fine-tune the estimated transition matrix to improve the classification performance; 7). Dual (Yao) et al. 20206) improves the estimation of the transition matrix by introducing an intermediate class, and then factorizes the transition matrix into the product of two easy-to-estimate transition matrices; 8). VolMinNet (Li et al.) 2021 is an end-to-end label-noise learning method, which can learn the transition matrix and the classifier simultaneously; 9). Reweight (Liu & Tao 2015) uses the impor- tance reweighting technique to estimate the expected risk on the clean domain by using noisy data. oatt--o-thhe-rr the-art noisy-label learning algorithm, we want to explore whether the variance of losses should be always penalized when learning with noisy labels. Therefore, we do not make comparisons with some methods which use the sumi-supervieed methods, such as Miv (Li al. 2020) and PES (Bai et al. 2021
Note that the aim of this paper is not to design a state-of-

---

Under review as a conference paper at ICLR 2023
Table I: Means and standard deviations (percentage) of classification accuracy. mean that they are the highest accuracy.
Results with "**
MNIST
CIFAR-10
CIFAR- 100
Sym-20%
Sym-50%
Sym-20%
Sym-50%
Sym-20%
Sym-50%
Decoupling
97.04 + 0.06
94.58 + 0.08
77.32 = 0.35
54.07 + 0.46
41.92 + 0.49
22.63 + 0.44
MentorNet
97.21 0.06
95.56 + 0.15
81.35 = 0.23
73.47 = 0.15
42.88 = 0.41
32.66 1: 0.40
Co-teaching
97.07 0.10
95.20 1: 0.23
82.27 0.07
75.55 1: 0.07
48.48 1: 0.66
36.77 0.52
T-Revision
98.72 +: 0.10
98.23 + 0.10
87.95 = 0.36
80.01 + 0.62
62.72 - 0.69
49.12 1:12::0 0.22
Dual T
98.43 + 0.05
98.15 + 0.12
88.35 = 0.33
82.54 = 0.19
62.16 = 0.58
52.49 + 0.37
CE
98.65 + 0.05
97.94 + 0.17
86.86 + 0.26
78.93 + 0.47
60.15 + 0.46
45.66 + 0.68
CE-VRNL
98.71 0.06
97.96 + 0.06
87.49 0.25
79.31 = 0.52
60.38 - 0.61
47.79 0.52
Forward
97.47 0.15
97.93 + 0.22
87.29 0.63
77.58 + 1.05
59.71 + 0.40
44.53 1.11
Forward- VRNI.
98.89 + 0.04"
98.14 0.27
0.29 89.81
81.16 0.55
68.19 0.31
54.10 1.2
Reweight
98.20 0.24
97.93 + 0.20
86.42
0.18
82.13 0.56
60.52 0.52
47.69
0.78
Reweight- VRNI.
98.61 0.21
98.27 0.15*
89.68 0.24
83.99 + 0.28
66.52 0.25
50.26 0.14
VoIMinNet
98.66 1: 0.14
97.83 +: 0.15
89,27 29.27 0.30
82.17 : 0.19
65.65 65 + 0.62
54.40 0.
VolMinNet-VER VRRL
98.78 +: 0,08
97.93 + 0.20
89.42 = 0.12
82.92 = 0.24
66.40 = 0.66
55,94 = 0.64
Asym-20%
Asym-50%
Asym-20%
Asym-50%
Asym-20%%
Asym-50%
Decoupling
96.79 + 0.01
94.71 + 0.08
78.63 = 0.27
71.01 = 3.72
39.42 + 0.48
21.64 + 0.23
MentorNet
97.03 + 0.05
94.66 + 0.11
78.99 = 0.34
68.00 = 2.09
10.03 + 0.33
11.14 + 0.25
Co-teaching
97.02 0.03
95.15 + 0.09
83,96 0.28
76.58 + 0.84
13.36 + 0.44
13.10 0.66
T-Revision
98.90
98.35 0.13"
88.38 0.56
81.51 0.74
59.52 0.43
45.56 1.86
Dual T
95.46 0.14
91.46 + 0.29
70.31 1.06
53.04 + 2.76
05.80 + 0.78
02.38 + 0.94
CE
98.76 0.00
97.91 + 0.23
87,31
0.32
79.47 + 0.47
59.83 + 0.69
45.08 + 0.71
CE-VRNL
98.77 +: 0.12
97.98 1: 0.20
87,38 +: 0.31
79.61 1 0,40
60.69 + 0,51
45.44 1: 0.10
Forward
98.42 + 0.06
97.92 + 0.04
87.70 + 0.29
79.25 + 1.61
60.24 + 0.42
43.39 + 1.15
Forward- VRRL
98.92 1: 0.06
98.13 1: 0.21
89.98 0.11
82.35 1 0.88
67.89 1 0,30
53.67 1: 0.52
Reweight
98.50 +: 0.07
98. + 0.08
88.55 + 0.32
82.72 + 0.38 84.20
60.81 + 0.70
46.36 1: 0.18 49.71
Reweight-VRR VRRNL
98.77 0.21
98.10 1 0.16
89.80 0.11
0.25*
66.62 1: 0.45
VolMinNet
98.62 +: 0.10
98.03 + 0.12
89.50 = 0.18
83.15 + 0.56
66.02 + 0.73
55.17 0,46
VolMinNer- VRNL
98.76 0.13
98.08 0.08
89.64 0.19
0.32 83.65
66.24 0.95
55.85 + 0.73'
Pair
Pair-45%
Pair 20%
Pair-45%
Pair-20%
Pair-45%
Decoupling
96.93 0.07
94.34 + 0.54
77.12 0.30
53.71 0.99
40.12 + 0.26
27.97 0.12
MentorNet
96.89 + 0.04
91.98 + 0.46
77.42 + 0.23
61.03 + 0.20
39.22 + 0.47
26.48 + 0.37
Co-teaching
97.00 + 0.06
96.25 + 0.01
80.65 + 0.20
73.02 + 0.23
42.79 + 0.79
27.97 + 0.20
T-Revision
98.89 1 0.08
84.56 1: 8.18
90.33 = 0.52
78.94 1: 2.58
64.33 +: 0.49
41.55 1: 0.95
Dual T
98.86 0.04
96.71 +
89.77 0.25
76.53
67.21 0.43
47.60 0.43
CE
98.71 0.08
83.49 +: 3.77
88.63 0.26
66.32 1: 2.44
61.04 + 0.31
39.78 0.30
CE-VRNL
98,80 0.10
84,00 + 3.65
88.71 0.30
67.71 2.04
61.00 + 0.32
39.91 0.20
Forward
98.85 0.09
96.45 1:: 4.03
90.88 0.23*
83.27 1: 9.47
62.54 0.42
41.96 1.45
Forward-VRNL
98.88 1 0,08
96.55 +: 3.88
90.88 :: 0.29"
83,54 1 9.29
62.78 + 0.32
42.29 + 1.23
Reweight
98.64 + 0.07
95.52 # 3.58
89.68 = 0.30
76.03 = 5.02
61.35 + 0.66
40.10 0.46
Reweight- VRNL
98.68 + 0.14
95.97 +: 3.52
89.83 = 0.30
76.75 = 6.15
61.37 + 0.42
40.30 +: 0.57
VolMinNet
99.05 : 0.05'
99.08 + 0.06
90.73 = 0.23
88.47 - 0.61
69.96 + 1.18
61.85 + 1.41
VolMinNer-Ve VRNL.
99.02 0.08
99.10 0.08
90.86 = 0.27
88.77 + 0.51
70.18 0.50
63.38 + 1.72"
Table 2: ing and validation.
Classification accuracy(perccenage)) on Clothing 1 M. Only noisy data are exploited for train-
Decoupling
MentorNet
Co-teaching
T-Revision
Dual T
PTD
54.53
56.79
60.15
70.97
70.17
71.67
Forward 71.27
Forward- 72.43
Reweight 71.62
Reweight-VRNN 72.14
VolMinNet 72.29
SoMMinNet-VRNU
72.66
Noise Types. To generate noisy dataset, we corrupted the training and validation sets manually according to a special transition matrix T. Specifically, we conduct experiments synthetic noisy datasets with three widely used types of noise: 1). Symmetry flipping (Sym-e) (Patrini et al., 2017); 2). Asymmetry flipping (Asym-e); 3). Pair flipping (Pair-e) (Han et al. 2018b). We manually corrupt the labels of instances according to the transition matrix T.
Network structure and optimization. We implement the proposed methods and baseline using Pytorch 1.9.1 and train the models on TITAN Xp. The model structure and optimizer are as same as the state-of-th thethe---atttt-- method (Li et al. 2021). Specifically, we use a LeNet-5 network (LeCun ct al. 1998) for MNIST, a ResNet- network for CIFAR- -0, a ResNet-32 network (He et 2016) for C00, ResNet-50 pretrained on ImageNet for Clothing 1M. On synthetic noise datasets,
7

---

Under review as a conference paper at ICLR 2023
0.9
0.9
0.9
0.8
0.8
0.8
0.7
0.7
0.7
0.6
8 0.6
0.6
0.5
0.5
0.5
0.4
Test accuracy with VRNL
0.4
0.4
0.3
accuracy without
0.3
0.3
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8
Noise rate
Noisc rale
Noisc rate
(a) Forward with VRNL
(b) Reweight with VRNL
(c) VolMinNet with VRNL
Figure 4: Test accuracies of the models trained on CIFARIO with coametty--lippingg noise and increasing noise rate.
Oriine rngprtattionn
0.30
8
2.50
0.50
1.55
0.5
375
2.50
1.5
0.31
Blus
(a) Reweight MNIST (b) Forward on MNIST (c) Reweight on CIFARIO (d) Forward CIFARIO Figure 5: Test accuracies of the models trained on MNIST and CIFARIO by using biased transition matrices, We increase the error of transition matrices manually. The proposed VRNL is robust to the biased transition matrix.
SGD is used to train the neural network with batch size 128, momentum 0.9, weight decay 10-4, and an initial learning rate 10-2 The algorithm is trained for 80 epochs, and the learning rate is divided by 10 after the 30-th and 60-th epochs. For Forward and Reweight, we set the hyper- parameter a 0.1 on symmetry-flipping noise, asymmetry-fippinn noise, and we a 0.01 on pair-flipping noise. For VolMinNet, we set = 0.005 on MNIST and CIFAR- with pair-flipping noise. For other experiments on synthetic noisy datasets, a 0.05 is employed. When the dataset is Clothing- 1M, for Forward and Reweight, SGD with batch size 64, momentum 0.9, weight decay 10 is used train the model, and a is set to be 0.1; for VolMinNet. SGD with batch size 64, momentum 0.9, weight decay 10-3 is used, and is set to be 0.005. For Forward and Reweight, the transition matrix T has be estimated in advance. For the end-to-end method VolMinNet, the transition matrix T and the classifier are learned simultancously. To estimate the transition matrix, we follow the same experimental settings described in their original papers (Patrini et al., 2017 Li eet al. 202Ip. The parameters of the model used to estimate the transition matrix will be used to initialize the weights of the classifier.
4.1 CLASSIFICATION ACCURACY EVALUATION
We embed VRNL into the label-noise learning methods, egg., Forward, Reweight and VolMin- Net which are named Forward-VRNI, Reweight-VRNI and VolMinNet-VRNL, respectively. In memumtrr--flipinng asymmetry-
Tab. [ we illustrate classification accuracies on datasets containing flipping and pair-flipping noise. It shows that VRNL improves the classification accuracies of all the label-noise learning methods on different datasets and different types of noise. The performance of VolMinNet usually better than that of Forward and Reweight and the estimation error of transition matrix used in Forward and Reweight is larger than in VolMinNet (Li et al.) (021). However, by employing VRNL, the performance of Forward and Reweight are comparable to that of VolMinNet, which suggests that VRNL is robust to the biased transition matrix. The improvements of VRNL under mair-flipping noise are not large compared with cuuuntry--flipinng and seyymmtrr-flippiinn noise. More experiments analyse the reason can be found in the Appendix.
We also provide the performance of VRNL under various noise rates. The experiment results are visualized in Fig. 4 VRNL can improve the performance of existing methods both little noise and extreme noise. Detailed digits are posed in Tab. and Tab. 5 in Appendix.
In Tab. 2, illustrate the results the real-world dataset Clothing IM. VRNL improves the gener- alization ability of backbone methods. The performance of VolMinNet- VRNL outperforms all other baselines.
IM.
8

---

Under review as a conference paper at ICLR 2023
4.2 THE INFLUENCE ON CLEAN AND NOISY CLASS POSTERIORS
To analyze the influence of variance of losses increase on clean class posteriors and noisy class posteriors. In Fig. 2 and Fig. 3 we visualize the change of cross-entropy losses for instances with clean labels and instances with noisy labels during the model training, respectively. The average loss and the standard derivation of mislabeled examples and correctly labeled examples are visualized separately for better illustration. The methods used are Reweight and VolMinNet.
By comparing Reweight-VRNL with Reweight, the loss of noisy labels for mislabeled examples is larger but the loss of noisy labels for correctly labeled examples is almost unchanged as shown in Fig. 2c and Fig. 2d] It means that the proposed method prevents the model from memorizing incorrect labels and has little influence on learning correctly labeled examples. By comparing Fig. 26 with Fig. 2a the loss of clean labels for mislabeled examples becomes smaller when our method employed. It implies that our method helps learn clean class posteriors of mislabeled examples.
Similarly, the results also hold for VolMinNet. Specifically, the variance of noisy class posteriors in Fig. 3d increases compared with Fig. 3c which could help VolMinNet better estimate the transition matrix. It is because our method encourages the diversity of noisy class posteriors, which makes the sufficiietlly scattered assumption easier to be satisfied when the sample size is limited. Meanwhile, it can be seen that the empirical risk defined by clean training examples decreases after using our method, as shown Fig. 3a and Fig.
3b. It means that the model can classify the examples better.
4.3 PERFORMANCE WITH THE BIASED TRANSITION MATRIX
In practice, the noise transition matrix generally is not given and is required to be estimated. How- ever, the estimated transition matrix could contain a large estimation error. One reason that the transition matrix can be hard to accurately estimate when sample size is limited (Yao et al. (0206)- Another reason is that the assumptions (Patrini et al. 2017 [i et al. (02I) used to identify the tran- sition matrix may not hold. This motivates us to investigate the performance of our regularizer when the transition matrix contains bias.
To simulate the estimation error, we manually inject noise into the transition matrix, i.e., T" T + A/ where A E CCCC sampled from standard multivariate normal distribution, and 7 E [0.01, 0.15]. Then we normalize the column of the transition matrix Tμ sum up to I by Tip C T Σ T... The estimation error Er of a transition matrix is calculated by employing the entry- k=
wise matrix norm, i.e., eT = |T - TM /
The biased transition matrix 7NN is adopted to Reweight, Reweight-VRNN Forward and Forward- VRNL, respectively. Experimental results shown in Fig. [jillustrate that our method is more robust to the bias transition matrix. Specifically, for most experiments and different levels of bias er, the test accuracies of Reweight-VRee VRRL and Forward- BorNard-VRNL are higher than Reweight and Forward. Additionally, the test accuracy of Reweight- drops much slower than Reweight with the in- creasing of bias ET.
5 CONCLUSION
In this paper, we study whether we should always penalize the variance of losses for the problem of learning with noisy labels. Interestingly, we found that increasing the variance of losses could be helpful, which can boost the memorization effect and reduce the harmfulness of incorrect labels. Theoretically, we show that increasing variance of losses can reduce the weights of the gradient with respect to instances with incorrect labels, therefore these instances have a small contribution to the update of model parameters. A simple and effective method VRNI. is also proposed which can be easily integrated into existing label-noise learning methods to improve their robustness. The experimental results both synthetic and real-world noisy datasets demonstrate that VRNL can dramatically improve the performance of existing label-noise learning methods. Empirically, we have shown that the proposed method can help models better leam clean class posteriors. We have also illustrated that VRNL can improve the classification performance of existing methods when the transition matrix is poorly estimated, which makes our method be practically useful.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Massil Achab, Agathe Guilloux, Stéphane Gaiffas, and Emmanuel Bacry. Sgd with variance reduc- tion beyond empirical risk minimization, arXiv preprint arXiv:1510.04822 2015.
Zeyuan Allen-Zhu and Elad Hazan. Variance reduction for faster non-convex optimization. In International conference on machine learning, pp. 699-707. PMLR, 2016.
Dana Angluin and Philip Laird. Learning from noisy examples. Machine Learning, 2(())343-370, 1988.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning, pp. 176-185. PMLR, 2017.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. A closer look at memorization in deep networks. In International Conference on Machine Learning, pp. 233-242. PMLR, 2017.
Yingbin Bai and Tongliang Liu. Me-momentum: Extracting hard confident examples from noisily labeled data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp-
2021.
Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang. Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. Advances in Neural Information Processing Systems, 34, 2021.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. International Conference on Machine Learn- ing. pp. 1062-1070. PMLR, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Aritra Ghosh, Himanshu Kumar, and PS Sastry. Robust loss functions under label noise for deep neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Maoguo Gong, Hao Li, Deyu Meng, Qiguang Miao, and Jia Liu. Decomposition- evolutionary multiobjective optimization self-paced learning. IEEE Transactions on Evolutionary Compu- tation, 23(2):288-302, 2018.
Sheng Guo, Weilin Huang, Haozhi Zhang, Chenfan Zhuang, Dengke Dong, Matthew R Scott, and Dinglong Huang. Curriculumnet: Weakly supervised learning from large-scale web images. In Proceedings the European Conference on Computer Vision (ECCV), pp. 135-150, 2018.
Bo Han, Jiangchao Yao, Gang Niu, Mingyuan Zhou, Ivor Tsang, Ya Zhang, and Masashi Sugiyama. Masking: A new perspective of noisy supervision. arXiv preprint arXiv: 08193, 2018a.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. arXiv preprint arXiv: 1804.06872 2018b.
Bo Han, Gang Niu, Xingrui Yu, Quanming Yao, Miao Xu, Ivor Tsang, and Masashi Sugiyama. Sigua: Forgetting may make learning with noisy labels more robust. In International Conference on Machine Learning, pp. 4006-4016. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2023
Edward L Ionides. Truncated importance sampling. Journal of Computational and Graphical Statis- tics, 17(2):25-311, 2008.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentornet: Learning data- driven curriculum for very deep neural networks on corrupted labels, In International Conference on Machine Learning. pp.
PMLR, 2018.
Jan Kremer, Fei Sha, and Christian lgel. Robust active label correction. In International conference on artificial intelligence and statistics, pp. 308-316. PMLR, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(1 (6(11)2227-2224, 1998.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online). Available: http://yann. lecun. com/exdb/nmist, 2, 2010.
Junnan Li, Richard Socher, and Steven CH Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. arXiv preprint arXiv:2002.07394 2020.
Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint arXiv: 1708. 178.0027,,,,282,, 2017.
Xuefeng Li, Tongliang Liu, Bo Han, Gang Niu, and Masashi Sugiyama. Provably end-to-end label- noise learning without anchor points, arXiv preprint arXiv:2102.02400, 2021.
Sheng Liu, Jonathan Niles- Weed, Narges Razavian, and Carlos Fernandez-Grandee Farly-learning
regularization prevents memorization of noisy labels. Advances neural information processing systems, 33:20331-20344 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transactions on pattern analysis and machine intelligence, 38(3):447-4611 2015.
Xingjun Ma, Yisen Wang, Michael E Houle, Shuo Zhou, Sarah Erfani, Shutao Xia, Sudanthi Wijew- ickrema, and James Bailey. Dimensionallty-driven learning with noisy labels. In International Conference on Machine Learning, pp. 3355-3364. PMLR, 2018.
Eran Malach and Shai Sbalev-Shwartz Decoupling" when to update" from" how to update". arXiv preprint arXiv: 120.0063.., 2017.
Duc Tam Nguyen, Chaithanya Kumar Mummadi, Thi Phuong Nhung Ngo, Thi Hoai Phuong Nguyen, Laura Beggel, and Thomas Brox. Self: Learning to filter noisy labels with self- ensembling. arXiv preprint 1rXiv:1910.01842,, 2019.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu Making deep neural networks robust to label noise: A loss correction approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning, pp. 4334-4343, 2018.
4M34- PMLR,
Jun Shu, Qian Zhao, Keyu Chen, Zonghen Xu, and Deyu Meng. Learning adaptive loss for robust learning with noisy labels. arXiv preprint arXiv:2002. 06442, 2020.
Christian Szegedy, Sergey loffe, Vincent Vanhoucke, and Alexander A Alemi. Inception-v4, inception-resned and the impact of residual connections on learning. In Thirty-first AAAI con- ference artificial intelligence, 2017.
11

---

Under review as a conference paper at ICLR 2023
Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. Joint optimization frame- work for learning with noisy labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5552-5560, 2018.
Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. Learning from noisy large-scale datasets with minimal supervision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 839-847, 2017.
Yisen Wang. Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey, Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 222-330, 2019.
Xiaobo Xia, Tongliang Liu, Nannan Wang, Bo Han, Chen Gong, Gang Niu, and Masashi Sugiyama. Are anchor points really indispensable in label-noise learning? Advances in Neural Information Processing Systems, 32:6838 384499966.. 2019.
Xiaobo Xia, Tongliang Liu, Bo Han, Nannan Wang. Mingming Gong, Haifeng Liu, Gang Niu, Dacheng Tao, and Masashi Sugiyama. Part-dependent label noise: Towards instance-dependent label noise. Advances in Neural Information Processing Systems, 33, 2020.
Tong Xiao, Tian Xia, Yi Yang, Chang Huang, and Xiaogang Wang. Learning from massive noisy labeled data for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2015.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin- Yau Kwok. Searching to exploit memorization effect in learning with noisy labels. In International Conference on Machine Learn- ing. pp. 10789-10798, PMLR, 2020a.
Yu Yao, Tongliang Liu, Bo Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. arXiv preprint XXix:2006.07805 2020b.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does dis- agreement help generalization against label corruption? In International Conference on Machine Learning. pp. 7164-7173, PMLR, 2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107- 115, 2021.
A THE GRADIENTS OF VRNL
The full derivation of the gradients of VRNL.
Rc(fs)
OR, (X.Y) (E(fo(X), Y)]
a aVor ((ffo))))(
00
00
00
DE xx (x.9) [X),, 8)
f
DE (x, (L 8)]
DE
(Exx,,|1([a>>) 8I1
00
00
00
= =Baag)~D (e),)(X).
B 2X) v) (e(Ja(X).
00
00
[(E(fo(X), v)
-2E (((o(()) (
00
= EEn.D)-D, (1+ (1 ((f6XX)) -2ee)),)))00-)((, 200(fo(X)) 23 (E(fa(X),))
80
w (e(So(x),
= Exx,)
(4)
where
W = 1 2a ((xx,)) V0) -
(5)
12

---

Under review as a conference paper at ICLR 2023
B DISCUSSIONS ABOUT THE MEMORIZATION EFFECT
Original loss
VRNL
Original loss
VRNL
39
2 classes
6 classes
classes
classes
E3
15
15
40
classes 120
20
classes
20
13
35
classes] 100
9 classes 1
Epoto
Eacols
Exie
(a) Symmetry-flippinn noise
Original loss
VRNL
Original loss
VRNI
48
38
2 classes
20x
6 classes
18
Tpals
3 classes
7 classes
16
18
10
classes
50x
24
classes 5
48
43
5 classes 52x
20
9 classes E
$20
02
(b) Pair-fipppig noise
Figure 6: We increase the number of classes gradually, and the differences between the loss of instances with incorrect labels and the loss of instances with correct labels in gymmetry-flipping noise are larger than the differences in muir-fipping noise. The differences are increasing gradually with the increase in the number of classes.
To investigate why the improvement of VRNL on pair-flipping noise is smaller than on symmetry- flipping noise, we conduct a series of experiments. We train a ResNet-18 (He et al. 2016) using
13

---

Under review as a conference paper at ICLR 2023
(a) Symmetry-flippern noise
(b) Pair-flipping noise
Figure 7: The differences between the loss of instances with incorrect labels and the loss of instances with correct labels in symmetry-lippinn noise and pair-flipping noise.
Forward loss (Patrini al., 2017) under the symmetry-flpppin noise pair flipping noise. The noise rate 0.2. We reclassify 100 classes CIFAR100 (Krizhevsky et al., (009). e.g. when the number of classes is the classes of 0-49 are reclassified as and the classes of 50-100 are reclassified as 1. The experiment results are shown in Fig. 6
By comparing with the data containing pair flipping noise, we found that the memorization effect is stronger in the data containing symmetry-flipiing noise. Then the improvement of VRNL on pair-fliping noise is smaller than symmetry-flipingg noise because our method relies on the mem- orization effect. Specifically, the results show that the difference between the loss of instances with incorrect labels and the loss of instances with correct labels becomes larger in both symmetry- flipping noise and pair-flipping noise. However, by comparing with the average difference under the pair-flipping noise, the difference between the loss of instances with correct labels and the loss of instances with correct labels is larger under commmetry-flipiingg noise.
implies that under symmetry-flipingg noise, the memorization effect strong because is much easier memorize the easy examples with correct labels than the hard examples with incorrect labels. We think the reason that the memorization effect is strong on crmmetry-flippng noise is that in noisy class, the contained noisy examples are from different classes, then memorizing all these noisy examples can be hard. By contrast, for pair-fipping noise, in a noisy class, the contained noisy examples from class, then it is easy for the learning model to find the common features and memorize these examples.
C EXPERIMENTS ON WEBVISION
We also conduct experiments on WebVision dataset 1.0 [Li al. 2017).
Following previous work
(Chen al. 2019 train models on Google image subset and model on Validation set. We first resize images to make shorter size as 320, then randomly crop patch of image whose size is 299x299, Horizontal random flipping is used. The network structure inception-resse resnett v2 (Szegedy) et al. 2017). To estimate transition matrix for Forward and Reweight, we train the network for 20 epochs. We use SGD with momentum as 0.9, learning rate as 0.01, weight decay as 10-3 Then we follow the previous work (Liu & Tao 2015 Patrini al. (017) using anchor point assumption to estimate transition matrix. We train the classifier for 80 epochs, learning rate is divided by 10 after 30 and 60 epochs. We start increasing variance of losses 30th epoch. is set to 0.05 for Forward- VRNL and VolMinNet- NRNNinn, For Reweight- VRNL, a is set to 0.005. The experiment results provided at Tab.
Table 3: The experiment results Visvin.on.
Forward
Reweight
VolMinNet
Without VRNL
33.40
46.28 48.44
72.44
With VRNL
35.00
72.92
14

---

Under review as a conference paper at ICLR 2023
D EXPERIMENTS UNDER EXTREME NOISE
We also conduct the experiments under extreme noise, the noise rate is 80%, noise type is symmetry- flipping.
Table 4: The test accuracy under Sym-80% noise.
Forward
Furward- VRNL
Reweight
Reweight- VRNL
VolMinNet
VolMinNet-- VRNLL
MNIST
91.66 0.68
92.46 2.13
93.09
1.79
94.49 : 0.52
92.16 +
92.35 + 0.25
CIFARIO
28.16 1886
29.27 + 2.13
27.64 3.43
43.56 4.13
37.10 3.33
37.73 + 3,87
CIFAR100
16.84 1 1.31
19.87 0.67
6.64 1.84
12.26 1 0.89
22.56 0.39
23.84 1 1.17
The experiment results show that the proposed method still can improve the performance of Forward, Reweight and VolMinNet when the noise rate is large.
EXPERIMENTS UNDER LITTLE NOISE
We also conduct the experiments under little noise CIFAR-10, the noise rate is 10%, noise type is symmetry-flipinng The experiments results are shown in Table 5
Table The test accuracy under Sym-10% 10% noise.
Forward
Reweight
VolMinNet
Without VRNL
89.74 + 0.27
90.08 + 0.51
90.49 + 0.09
With VRNL
91.30 + 0.15
91.11 0.19
90.77 0.13
The experiment results show that the proposed method still can improve the performance of Forward, Reweight and VolMinNet when the noise rate is small.
F EXPERIMENTS ON CLEAN DATASETS
We also conduct the experiment on clean datasets using standard cross entropy loss with VRNI., & set to 0.01. The experiment results shown in Tab. 6
The VRNL has little negative influence when the dataset is clean.
G SENSITIVITY ANALYSIS
conduct the sensitivity analysis on one synthetic dataset, CIFAR-10, under symmetry-lipping noise, the noise rate is 50%. The increases from 0.001 to 0.3. The experiment results are shown in Fig. Overall, the curve is smooth, thus VRNL not sensitive. We also show corresponding validation accuracy. As shown in Fig. g the tendency of validation accuracy is same test accuracy, even though the validation set is noisy. Therefore, the user can use the validation set to determine the best
H THE INFLUENCE ON HARD EXAMPLES
If the loss of incorrectly-labclee. examples is larger than the loss of hard but correctly-labeled exam- ples (e.g., the number of hard but correctly-labeled examples more than the number of incorrectly- labeled examples), VRNL should not have large negative impact hard but correctly-labeld labeled examples because can still separate hard correctly-labes labeled examples from incorrectly-labelee ex- amples.
the loss of incorrectly-abbe labeled examples equals the loss of hard but correctly-labeled examples (i.e,, these examples entangled), it hard to separate hard but correctly-labeles examples from
15

---

Under review as a conference paper at ICLR 2023
Table 6: Means and standard deviations (percentage) of classification accuracy. Results with **" mean that they are the highest accuracy.
MNIST
CIFAR-10
CIFAR-100
CE CE- VRNL
99.16 + 0.03
92.23 + 0.09
71.30 + 0.16
99.13 + 0.08
92.02 0.17
71.40
0.32
0.84
0.85
0,8400
0.83
Tedt ecournt with VINL
Text sccunsc in
0,8375
0.830
0.81
0.8325
0.80
$ 0.8300
0.79
0.78
0.70
0.8275
0.77
0.8250
0.85
0.8225
0.75
0,8200
0.001
0.006
0.01
0.05
0.3
0.001 0.005
0,001 0.005 0.01
02
Aphaa
Nppha
(a) Forward (Sym-50%)
(b) Reweight (Sym-50%)
(c) VolMinNet (Sym-5500
Figure 8: Sensitivity analysis for a.
0.445
Valimiee anourny with VRM
Yrasatto soorsey with.u VVOL
0.42
0.440
0.40
0.435
0.41
8 0.430
0.40
0.455
0.38
0.39
0.420
0.34
0.38
0.001
0,415
0,415
Apha
Apha
(a) Forward (Sym-50%)
(b) Reweight (Sym-50%)
(c) VolMinNet (Sym-50%)
Figure 9: Validation accuracy for different a.
lncorrectly-labeled examples. In such case, all existing sample selection and reweighting methods would have the same problem.
We conduct an experiment on CIFAR-I0 and found that the loss of hard but correctly-labeles exam- ples is smaller than the loss of incorrectly-labele ResNet-18 model on the clean dataset for 50 epochs and sort the cross-entropy loss of all training examples. The 30% examples with the largest loss are defined to be hard examples. Then corrupt all training examples manually by using 50% symmetry- fiiippingipiiggggg noise and train a new ResNet- 18 model using Forward- VRNL. The losses of hard but Norrectly-labelas examples 1O Hard but correctly-labeled examples and incorrectly- examples can be separated very well, thus VRNL should not have large negative impact hard but correctly- examples.
examples. To find out hard examples, we train
shown in Fig.
I EXPERIMENTS OF PROGRESSIVE EARLY STOPPING
We also conduct the experiment on Progressive Early Stopping with VRNL (PES- VRNL). The experiment results are shown and Tab. 7 The VRNL can still boost the performance of models.
J WARMING UP
The initial weights of VolMinNet random (The weights of Forward and Reweight acquired through early stopping which is also used to estimate transition matrix), the loss of instances with incorrect labels might not be larger than correct ones. Therefore, the influence of VRNL should
16

---

Under review as a conference paper at ICLR 2023
Test accuracy: 83.55%
Training loss of all correctly-labeled examples
4.0
Training loss of incorrectly-abbeled examples
3.5
Training loss of hard but correctly-labeled examples
3.0
loss
2.5
2.0
1.5
1.0
0.5
0
20
40
60
80
100
Epochs
Figure 10: The influence of VRNL on hard examples.
Table 7: The test accuracy under Sym-50% noise on CIFAR-10.
Sym-20%
Sym-50%
PES
92.57 + 0.22
87.78 + 0.34
PES-VRNL
92.66 = 0.09
87.85 + 0.17
17

---

Under review as a conference paper at ICLR 2023
not be large first. We conduct the experiment CIFAR-10 using VolMinNet, is increased with time. Specifically, we increase a from to 0.1 linearly every
a peaks at 0.1 after epochs. The experiment results are shown in Tab. g As can be seen in the experiments, the warming up strategy can increase the test accuracy compared with keeping or a constant.
Table 8: The test accuracy under Sym-50% noise.
Sym-20%
Sym-50%
VolMinNet VolMinNet- VRNL (constant)
89.27 + 0.30
82.17 = 0.19
89.42 + 0.12
82.92 0.24
MolMiNeet-RRNI. (warming up)
89.51 + 0.16
83.51 + 0.40
K THE LOSSES DURING TRAINING
To influence of VRNL on the losses distribution through training, we plot the training losses of the model with/without VRNI. (correct/incorrect labeled examples together). The noise type is symmetry-flipping, and the noise rate is 0.5. The results are shown in Fig. I1
12
VIN
10
lose
10
Epoons
Epochs
Epachs
(a) MNIST (Sym-50%%
(b) CIFAR-10 (Sym-50%)
CIFAR-100 (Sym-50%)
Figure 11: The training losses of models with VRNL and without VRNL.
The experiment results imply that when the model uses VRNL, the variance of training losses will increase.
L THE LOSSES AND VARIANCE OF LOSSES DURING TRAINING ON CLEAN AND NOISY DOMAIN
We conduct experiments, we train a Forward model on clean dataset and a noisy dataset separately, the noise type is symmetry- cymmpiry-flipping and the noise rate is 0.5.
10
Thaning
Training
0
10
20
30
40 Epoohs
50
60
10
20
10
20
30
40
50
60
70
80
Epochs
Epochs
(a) MNIST (Sym-50%)
(b) CIFAR-10 (Sym-50%)
CIFAR-100 (Sym-50%)
Figure 12: The training losses of models on clean and noisy domain.
18

---

Under review as a conference paper at ICLR 2023
M THE GRADIENTS OF SIWEIGHT-VRN.
The full derivation of the gradients RRWNight-VRNI.
RRInff)) 1
(
(8 i.)
m
30
( Ni:
B(cce(f(())...)))
09
where Em (. (s) is constant, we note it as Eir. therefore:
VRin(fe)
n Σ
00
(folc)(f((z)).. ūs)
00
(f
2ε,
00
1
(f
00
2 (fo(s)) D.)
00
00
(1-2033co((oott)) 01) 2aB.r)
03.lce(fo(2s)
00
19


Reviews:
[{"review_id": "mEk_xH5csp", "reviewer": "Reviewer_VFMQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes the seemingly counterintuitive claim that the variance of losses should be rewarded, rather than penalized, in problems involving learning with noisy labels. Mathematically, the authors show that rewarding the variance of losses leads to parameter updates so that if the loss of the example is smaller than the average loss over the examples, the magnitude of the update is magnified, and vice-versa. This leads to an intuitive safeguard against the influence of points with noisy labels, which tend to have high loss. The authors present extensive empirical results that show that this type of variance regularization boosts the performance of existing methods for learning with noisy labels.", "summary_of_the_review": "This paper addresses a problem that is highly relevant to the ML community. It challenges the traditional wisdom that variance of losses should be penalized, and it instead makes a compelling argument for rewarding the variance. Even though a more direct comparison of VRNL would go a long way (see above), the authors present compelling empirical evidence on the benefits of the proposed method when used in conjunction with prior methods. In light of these considerations, I recommend acceptance.", "strength_and_weaknesses": "## Strengths\n* Learning with noisy labels is a topic of high importance and interest to the wider ML community.\n* The proposed technique is novel and contrary to the traditional advice of penalizing the variance. The authors make a compelling case for the benefit of rewarding the variance in terms of attenuating the influence of noisy labels on the model during training.\n* The authors present extensive empirical evidence on a variety of scenarios that demonstrate the improved effectiveness of the method in boosting the performance of existing label-noise learning approaches.\n* Sensitivity analysis of the method’s hyperparameter is provided in the supplementary that shows that it can be estimated using a validation data set\n\n## Weaknesses\n* Even though there are experiments with VRNL augmenting prior label-noise learning approaches, it would have been very compelling to include results with VRNL alone, without knowledge or reasoning about the transition matrix. For instance, a performance comparison of VRNL vs. no VRNL on a simple CIFAR10 example where, e.g., 10% of the labels are artificially made to be incorrect, would go a long way. \n* The method requires tuning of the hyperparameter $\\alpha$ and a wide range of values for $\\alpha$ is used for the various experiments on different scenarios in Sec. 4. From the plots in the appendix (Fig. 7), the method’s performance is highly sensitive to the precise value of $\\alpha$. The authors did address that a validation data set can be used in a reliable way for CIFAR10, however, it is not clear whether this is always the case.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "b_ppraBYYi7", "reviewer": "Reviewer_JfaU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- the paper proposes a simple idea for robustness to label noise. The idea is to increase the variance of the loss function across training samples. A high variance could generally make sense since ideally the loss remains high for noisy-labelled samples during training while on the other hand the loss for correctly-labelled data is gradually decreased.\n\n- it further derives a form for the derivative of the proposed total loss w.r.t. the model parameters where the proposed loss can be seen as a reweighting of the gradient induced by individual training samples. It proportionally upweights the gradient of those samples with lower losses than the average loss and downweights the gradient of the samples with higher losses than the average loss.\n\n- it applies the method on top of three noise robustness techniques. Two of them are based on noisy label transition matrices, namely Forward and VolMinNet as well as a loss reweighting method.\n\n- the experiments show consistent improvement (although sometimes small) of the modified version of those three methods on different datasets, MNIST, CIFAR, Clothing1M) and noise types and rates (real, synthetic, symmetric, and asymmetric, 30% and 50%).\n\n- some additional studies are done to shed light on the inner workings of the proposed method. Interestingly, it is shown that the proposed method is more robust to erroneous estimation of the noisy label transition matrix.", "summary_of_the_review": "The paper proposes a simple change of the loss function that encourages variance of the loss across training samples and shows this can bring robustness to training label noise in some scenarios. However, there are some important concerns with the empirical evidence that has been provided in support of the paper's proposal. ", "strength_and_weaknesses": "- the paper proposes a simple idea for robustness to label noise\n\n- some results are encouraging compared to the chosen baselines.\n\n- the method seems original in that it enforces high variance on the training data loss. However, quite a few methods that apply some form of selection/reweighting/distribution on the individual sample loss functions can be seen as increasing the variance of training losses. It is important to make such discussion in the paper and highlight the differences of the paper's proposal to such prior works both formally and empirically.\n\n- since the paper's claim on increasing variance for noise robustness is general, either the main experiments should be done on standard loss functions such as CE or the claims and theoretical motivation should be adjusted specifically to the case of noise transition matrices. \n\n- due to the simplicity of the approach (increasing loss variance) it is natural to assume complications might arise from the confusion of hard but correctly-labelled examples with noisily-labelled examples. Therefore, extensive discussions and experiments are required as to how such variance encouragement is theoretically or empirically unharmful for the difficult examples (e.g., in a scenario where the data is clean but quite complex to model).\n\n- different $\\alpha$s are used for different setups, what is the sensitivity of the method to $\\alpha$ (in each/some of those setups)? How is  optimized for each setup? Also, it seems like too-early or too-high enforcement of large variances can be detrimental to fitting the cleanly labelled data. Does the method require heavy hyperparameter tuning? If not, how robust is it to hyperparameters? Why is it robust?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XqQgBqG7lo", "reviewer": "Reviewer_zTmG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aimed to explore if the variance of losses should be always penalized when learning with noisy labels. The paper suggests adding a variance regularization to the loss, which will encourage the variance of losses. Consequently, when calculating the gradients of the proposed loss, if the loss of an example  is smaller than the expectation of the losses (which the authors assume is true in the case of examples with clean labels), the weight associated with the respective gradient will be increased and the example contributes more to the update of the parameter. If the loss of the example is larger than the expectation of the losses, the weight associated with its gradient is small and the example contributes less to the update of parameter. The assumption is that the latest will happen for noisy examples, because the network tends first to learn from clean examples and increasing the variance of losses would boost this memorization effect and reduce the harmfulness of incorrect labels. \nThe authors explain how to add the suggested regularization to several recent methods and show improvement. ", "summary_of_the_review": "I have mixed feelings about this paper. I would suggest the authors reorganize the paper in a way that it provides some insights beyond just illustrating improvements to some methods (and providing the implementation details). \n1) The authors ask the question -- do we always need to finalize variances?  -- I could not find a clear to this question. It seems that the authors actually the opposite:  that the variances should be encouraged. So is the answer - always encourage losses?\n2) I am wondering why the authors did not provide the evaluation results of their method on WebVision dataset. Given relatively minor improvement on Clothing1M (< 0.5%), how will the results look like for WebVision? \n3) In my opinion, it's not clear why the authors push the results with little noise and strong noise to the appendix. It will be interesting to see those together with the results of Table 1 and maybe as a graph that shows the improvements (deltas)  as a function of noise. \n\nI will definitely reconsider my rating if the paper is reorganized and the main learning are clearly stated. \n", "strength_and_weaknesses": "Strength:\nThe overall idea of adjusting the weights, such that noisy samples influence weight update less, sounds like a good idea. The approach suggested to achieve that with addition of variance component to the loss is relatively simple (can be applicable as the authors show to several techniques) and yet relatively effective (improved results compared to the original method -- Tables 1 & 2).\n\nWeakness:\nThe paper mostly proves the effectiveness of the approach empirically, improving the results by some margin in case of the synthetic noise. The improvement on real noise ( Clothing1M  dataset) is relatively minor (Table 2 -- <0.5% improvement) and since there is no real theoretical proof that the suggested approach should work and under what conditions, the paper overall is not very convincing (in other worlds, that would not be my choice of an approach when dealing with noisy labels).\n In the absence of theory and given that the paper is not aiming to provide a SOTA technique, but discuss the loss variance penalization / encouragement, reorganization of the paper, e.g. :\n(1) bringing results from the appendix to the main body (e.g., uniting Table 1 and Table 4&5); (2) writing a paragraph with concrete conclusions and answer to the question posed in the title; (3)  clear conclusion on what types of noise, datasets  (balanced / not balanced) benefit from the approach, can help to strengthen the empirical contribution of the paper.    ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject