Understanding Generalized Label Smoothing when Learning with Noisy Labels
Authors: Jiaheng Wei
Hangyu Liu
Tongliang Liu
Gang Niu
Yang Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Learning with noisy labels
label smoothing
model confidence

Abstract: Label smoothing (LS) is an arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels. It was shown that LS serves as a regularizer for training data with hard labels and therefore improves the generalization of the model. Later it was reported LS even helps with improving robustness when learning with noisy labels. However, we observe that the advantage of LS vanishes when we operate in a high label noise regime. Puzzled by the observation, we proceeded to discover that several proposed learning-with-noisy-labels solutions in the literature instead relate more closely to $\textit{negative label smoothing}$ (NLS), which defines as using a negative weight to combine the hard and soft labels! We show that NLS differs substantially from LS in their achieved model confidence. To differentiate the two cases, we will call LS the positive label smoothing (PLS), and this paper unifies PLS and NLS into $\textit{generalized label smoothing}$ (GLS). We provide understandings for the properties of GLS when learning with noisy labels. Among other established properties, we theoretically show NLS is considered more beneficial when the label noise rates are high. We provide extensive experimental results on multiple benchmarks to support our findings too.

Under review conference paper at ICLR 2022
UNDERSTANDING GENERALIZED LABEL SMOOTHING WHEN LEARNING WITH NOISY LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
Label smoothing (LS) is arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels. was shown that LS serves regularizer for training data with hard labels and therefore improves the generalization of the model. Later was reported LS even helps with improving robustness when learning with noisy labels. However, we observe that the advantage of LS vanishes when we operate in a high label noise regime. Puzzled by the observation, we proceeded to discover that several proposed learning wiith-noisy-labee solutions in the literature instead relate more closely to negative label smoothing (NLS), which defines as using negative weight to combine the hard and soft labels! We show that NL.S differs substantially from LS in their achieved model confidence. To differentiate the two cases, we will call LS the positive label smoothing (PLS), and this paper unifies PLS and NLS into generalized label smoothing (GLS). We provide understandings for the properties of GLS when learning with noisy labels. Among other established properties, we theoretically show NLS considered more beneficial when the label noise rates are high. We provide extensive experimental results on multiple benchmarks to support our findings too.
INTRODUCTION
Label smoothing (LS) (Szegedy et al, 2016) is an arising learning paradigm that uses positively weighted average of both the hard training labels and uniformly distributed soft label:
y (1-r) K
(1)
where we denote the one-hot vector form of hard label and all one vector as y. respectively. K is the number of label classes, and is the smooth rate in the range of [0. 1]. It was shown that LS serves as a regularizer for the hard training data and therefore improves generalization of the model. The regularizer role of LS prevents the model from fitting overly on the target class. Empirical studies have demonstrated the effectiveness of LS in improving the model performance across various benchmarks (Pereyra et al.. 2017) (such as image classification (Szegedy et al.) (016)), machine translation (Vaswani et al. 2017). language modelling (Chorowski & Jaitly 2017)) and model calibration (Müller et 2019).
Later it was reported LS even helps with improving robustness when learning with noisy labels (Lukasik et al. 2020). However, we observe that the advantage of LS vanishes when we operate a high label noise regime. In Figure I present set of experiments smooth rates (possible to have tied smooth rates) under each label noise rate. Indeed, non non-negative smooth rates (circles colored in red) outperform negative ones when the label noise rates low. Nonetheless, with the increasing of noise rates, negative smooth rates r < (Eqn. (T). diamonds colored in green) appear to be more competitive when learning with noisy labels. Puzzled by the observation, proceeded to discover
UCI datasets (Dua & Graff 2017). We highlight best two

---

Under review conference paper at ICLR 2022
that several proposed learning- weitiin--wth--ois--labbbee solutions the literature, including Loss Correction (Patrini et al. 2017), NLNL (Kim et al., 2019) and Peer Loss (Liu & Guo 2020), instead relate more closely y to negative label smoothing (NLS), which defines as using a negative weight to combine the hard and soft labels!
UCI Dataset Twonem
UCI Dataset Waveform
UCI Dataset Splice
1.0
0.0
Smooth Rate
0.5
-2.5
0.0
-5.0
-5.0
Optimal Smooth Rate (PLS)
-7.5
Optimal Smooth Rate (NLS)
-7.5
-1.0
0.0
0.2 Noise Rate
0.4
0.0
0.4
0.4
Nolse Rate
Nolse Rate
Figure 1: Optimal smooth rates on UCI datasets with different label noise rates.
This paper unifies label smoothing with either positive or negative smooth rate into a generalized label smoothing (GLS) framework. Our paper motivated by the above inconsistent observations. And we aim to provide more thorough understanding of GLS under the setting of learning with noisy labels, rather than proposing a new method to compete for state-of-the-an performances. We firstly show that negative label smoothing differs substantially from positive label smoothing in their achieved model confidence. With the presence of label noise, we then proceed to show that there exists phase transition behavior when finding the optimal label smoothing rate for GLS. Particularly, when label noise is low, positive label smoothing able to uncover the optimal model while negative label smoothing is considered more beneficial in a high label noise regime.
We provide extensive experimental evidences to support our findings. For instance, on multiple benchmark datasets, we present the clear transition of the optimal smoothing rate going from positive to negative when we keep increasing noise rates. On CIFAR-10 test data, we show negative smoothing rate elicits higher model confidence correct predictions and lower confidence on wrong predictions compared with the behavior of a positive one.
Our contributions summarize as follows:
We provide understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels, where the label smooth rate can go negative. (An understanding paper rather than a method paper)
We show that several robust loss functions in the noise learning literature correspond to learning with GLS, under certain noise rate models. (Section
We show that negative label smoothing improves the confidence of model prediction. With the presence of label noise, we demonstrate learning with negative smooth rate can more robust to label noise compared with positive rate when label noise rates are high. (Section 4 5)
Empirical experiments on multiple datasets and methods demonstrate that with the presence of label noise, NLS becomes competitively robust to label noise. We also empirically show how GLS results in trade-offs in model confidence, bias and variance of the generalization error. (Section Appendix D and
Extensive experiment results validate our main theoretical conclusions. Besides, we also discuss practical considerations and multi-class extensions of GLS to mitigate the impact of label noise. (Appendix B
We defer all proofs Appendix Our work primarily contributes to the literature of learning with noisy labels (Scott et al., 20133 Natarajan et al.) 20133] Liu & Tao [013]] Patrini et al., 2017: Liu & Guo 2020).
Our core results contingent recent works of understanding the effect of label smoothing when training
deep neural network models, i.e., label smoothing improves model calibration (Müller et al.] 2019). more
complexed forms of label smoothing (Li et al.) 2020 Yuan al. 2020), and in particular when label noise
2

---

Under review conference paper at ICLR 2022
We generalize concept called generalized label smoothing and provide new understandings for when, instead of setting a positive smoothing rate as the literature would normally do, a negative smoothing rate is considered a better option. Due to the space limit, we defer more detailed discussion of related works to Appendix A
presents (Lukasik al. 2020 Liu 2021
2
PRELIMINARIES
2.1 LEARNING WITH SMOOTHED LABELS
For a K-class classification task, denote by X X high-dimensional feature and y 2,.,, K} the corresponding label. Suppose (X, Y E X' x y are drawn from a joint distribution D. Let yj be the one-hot encoded vector form of y; which generates according Y. The random variable of smoothed label yLS, with smooth rate E [0, 1] generates yi L.Srr as (Szegedy al. 2016):
LS.r = (1- vi K 1
Yi
For example, when 0.3, the smoothed label of yi = [0, 1, becomes y; I.S.r=0.3 [0.1, 0.8, 0.1].
We consider a broader setting where the smoothed label might be negatively related to the corresponding feature. As supplementary to existed works on label smoothing, we explore the benefits of learning with generalized label smoothing (GLS), i.e., r E (-o0, 1] instead of a non-negative
GLS,r = (1 (1) yi
1
(2)
K
where Y1 G.S.Sr is given by the random variable of generalized smooth label YGLS,r, We name the scenario
r < as negative label smoothing (NLS). To clarify, we don't It assume a strict lower bound for r. r oc, normalizing y; ,GLSSrr by 1- returns yi y GLS,r = Yi K- We will show when imposing a negative In the main paper, we focus on the binary classification task where E {0. }} and K = 2. And we defer multi-class extensions Appendix C Denote
1
smoothing parameter will be considered beneficial
compared to a positive
deep neural network, fxx:) is the model prediction of x; E X with element f(x1),, y, == P(Y = yi XX = f). Given the sample E X and a hard label E V, binary CE loss is defined (f v) = log f( f()),) Throughout this paper, we shorthand l as fcE for a clean presentation.
2.2 LEARNING WITH NOISY LABELS
The noisy label literature considers the setting where we only have access to samples with noisy labels from (X, Y). Suppose random variables (x, Y) E X' x ÿ are drawn from a noisy joint distribution D. Statistically, the random variable of noisy labels Y can be characterized by noise transition matrix T, where each element Ti, represents the probability of flipping the clean label Y i to the noisy label Y j. i.e T; = P(Ý = j|Y i). In this paper, we are interested in the widely studied class-dependent label noise. We assume the label noise is conditionally independent of features, i.e.,
P( = -jy= = = P( = =|,,X = 1), Vi,j E [K]
For the binary classification setting, define eg := P c P 0 generality, assume E1 eo Es 0. We denote the binary noise transition matrix in the noisy label
P 1). Without loss of
1 -
setting as: T
el
e1
3

---

Under review conference paper at ICLR 2022
2.3 MODEL CONFIDENCE
We define key quantity, model confidence, that plays important role in later sections.
Definition 1. Model confidence of model f for sample (z, y). Given model f, sample x with its target label y E {0, }), the model confidence of f w.r.t. sample defined as MC(f;a,y) f(x), 1(x)
MC(f: x, y) in definition characterizze the difference of the predicted probability between target class and the other class. MC(f; z, V) = 0 simply means f has confident on its predictions since the model can not identify the target class of æ. MC(f; x. y) returns a negative value when f gives a wrong prediction and is not confident to predict the label of as the target label y. To dig into how GLS influences the model confidence on correct and wrong predictions in following sections, we separate the distribution D into
> === = ( ~ D: ( 0}, Dj = ((X,)) D: MC((; $
3 CONNECTION TO OTHER ROBUST METHODS
In this section, we aim to theoretically explore the connection between GLS and popular methods such as backward/forward loss correction (Natarajan et al.) 2013] Patrini et al., 2017). NLNL (Kim et al., (2019) and peer loss (Liu & Guo 2020). We defer the corresponding empirical validations to Appendix B
For <. 1, let y be the vector form of noisy label obtained from Y, we define the smoothed label of øj gGLS,r, where gGLS,r = (1 r) ÿ (r/K). and is generated by the random variable ý-GLS,r Risk minimization of the Generalized Label Smoothing (GLS) w.r.t. noisy labels becomes:
Risk Minimization Using GLS: min E (x.Y)= (((()), F(ls.)
(3)
The GLS framework covers three special methods: PLS (r E (0, IJ), Vanilla (CE) Loss (r 0) and NLS (r < 0). Besides, we observe that NLS connects to a special case of label smoothing regularization. We highlight this in Theorem [
Theorem 1. Vr E [0, 11, NLS with smooth rate -r is a special form of label smoothing regularization:
min E (x. WDD ((((x), yoas.))) -mn E E ((((())))))))))))))))))))))))))) E(f(X), - 5.Iss))
3.1 LOsS CORRECTION
Loss correction (Patrini et al. 2017) studies two robust loss designs which are based on the knowledge of non- singular noise transition matrix T. The backward correction e (f(X), Y) re-weights the loss ((f()), Y) by T with being the model predicted label, while the proposed forward correction (f(X), Y) multiplies the model predictions by T.
Proposition 1. For TLC == 200 correction (with the knowledge of noise rates) equivalent to the combination of NLS and an extra bias term Bias-LC
250 < O, drc cs
=s TT risk minimization of both backward and forward
PD e- (f(X), v7] min R Fx (
min E
-min E xx PND [f(()), FGLSmO) Aec- Ex 1) E(F(()))
Blax-ICC
4

---

Under review conference paper at ICLR 2022
The incurred Bias-LC controls the model confidence on (X,Y =) Dy. Note that when the noise rate not substantially high, i.e, en E [0, 2). AI. 0. Then, compared with loss correction, NLS with smooth rate "LC makes the model f to less confident on (X, = ~ D and more confident on (X, Y = ~ Dj (wrong predictions). However, the impact of term Bias-LC diminishing when either es + (symmetric noise rates) or co + 0 (low noise rates) as specified in Theorem 2
Theorem 2. Assume the noise transition matrix is symmetric, i.e, es 0, backward and forward loss correction are a special form of NLS with smooth rate TLC.
3.2 LEARNING FROM COMPLEMENTARY LABELS
Complementary label [ssiida al. 2017) was firstly introduced to mitigate the cost of collecting data. Rather than encouraging the model to fit directly on the target, learning from complementary labels trains the model not fit on the complementary label which differs from the target. Later, an indirect training method "Negative Learnin" (NL) (Kim et al. 2019) was proposed to reduce the risk of providing incorrect information with the presence noisy labels and robust label noise in multi-class classification tasks. A more generic unbiased risk estimator of learning with complementary labels was proposed (Ishida et al.)
2019) and is defined as: &cL (f(),, Y) == ((f(X), Y) ((f(X),1-X)
Theorem Learning from complementary labels with teu equivalent NLS with smooth rate TeL + -00
min E (x, (x la (f(X), + min E ((x,)))(((((()), yassa (000aaa-))))))) [))
3.3 PEER LOSS FUNCTIONS
Peer loss functions (Liu & Guo 2020 propose family of robust loss measures which do not require the knowledge of noise rates. The mathematical representation of peer loss functions is Epr (f((),))) == ((f(X), Y) {(f(X,)) Y2), where (X,, Y:) D. The second term of peer loss evaluates on randomly paired data samples and labels to punish f from overly fitting noisy labels.
Proposition 2. For := 2 IP(Ý" 1), APL := 1 - TeL risk minimization of peer loss is equivalent to negative label smoothing regularization with an extra term Bias-PL, i.e.,
min E (x (el (f))) = min E(X, ((((2)) 8) ((f(x)) yoLs.m }]
[
+APL E x ((F(()))) e(F(X),))
The incurred term Bias-PL controls the model confidence on (X, 1) ~ DD and has diminishing effect as P(ý = 1) + 1 1/2. Generally, peer loss relates to GL.S as the negatively weighted GLS term appears be a regularizer. Note that we have access to the P(Ý = 1), we can bridge the gap between GLS and peer loss by adding an estimable term Bias-PL. With some derivations, we further show in Theorem 4 when noisy priors are equal, peer loss has an exact GLS form.
Theorem 4. When the noisy labels have equal prior, i.e, P(Ý = 0) = p( = 1), peer loss a special form of NLS regularization with smooth rale TPL. Besides,
min E (x. B)DD ([)) - min (xPP)~ [(F(X), poL))
4 GLS AND MODEL CONFIDENCE
Now we show that NLS differs substantially from PLS in their achieved model confidence. This discussion sets the foundation for our discussion when learning with noisy labels in next section.
5

---

Under review conference paper at ICLR 2022
4.1 GLS ON CLEAN DATA
We firstly show that NLS differs substantially from PLS in their achieved model confidence. This discussion sets the foundation for our discussion when learning with noisy labels in proceeding subsections.
When the label is clean, i.e, eo el - 0. Eqn.
(3) reduces to: -
min E(x,) ( [(((X))))
+ T 2/2 -
(4)
Term (f:X))
To clarify, we are not restricting D to have infinite samples, i.c., for the discrete distribution D = fI;, Vi Vi
Eqn. 44 becomes: min N EiE[M] ((((x)), u)] + 2N EiEN, (<(f(x)), 1 - y1) ((((x))) Note:
X) log (f(X)y -(()))): MC(f:X,Y) 22 -2xf(X)y
Both log (x/(1 z)) and 2r are monotonically increasing for x E (0. 1), model f with a high MC,(f; X, Y) has high MC(J; X, Y). The difference between PLS and NLS lie in the weight of Term (C, X, Y): NLS encourages high MCr(f;X,Y) Y) and MC(J;X, while PLS has an opposite effect.
4.2 INSIGHTS FROM AN EMPIRICAL OBSERVATION
We adopt the generation of 2D (binary) synthetic dataset from Amid et al. 2019 by randomly sampling two circularly distributed classes. The inner annulus indicates one class (blue), while the outer annulus denotes the other class (red). We hold 20% data samples for performance comparison.
Side-effects of over-confident In Fig- ure 2 the colored bands depict the dif- ferent levels of prediction probabilities: light blue orange bands indicate sam- ples that satisfy MC < 0.4 (low model confidence). When learning with clean data, GLS with non non-posittee pos--postteeee smooth rate may yield over-confidence on the model prediction and relatively low test accu- racy.
Decbrim
Figure Model confidence visualization of GLS on synthetic data (Type 1) with the clean data. r* E [0,0.4]. (left: NI.S; middle: Vanilla Loss; right: PL.S).
Label noise reduces model confidence
L 1.-44 A.4 2.055
Fandor 24. Tee 0266
r-0% 2.35
Recent works (Liu 2021: Cheng et al.] 2020) have demonstrated that with the presence of label noise, learning with noisy labels directly will eventually result in unconfident model predictions. Con- tinuing the synthetic 2D dataset, we flip the clean labels according to symmetric noise transition matrix with noise rate e, for both classes. With the presence of label noise in Figure 3 GLS generally becomes less confident on its predictions. more samples are of uncertain predictions. Thus, a smaller/negative smooth rate is beneficial when the noise rate increases by encouraging more confident predictions.
Cectron Eor
Figure 3: Model confidence visualization of GLS on synthetic data (Type 1) with noise rate e = 0.4. Topt = 00.4 (left: NLS; middle: Vanilla Loss; right: PLS).
Besides, when the smooth rate increases from negative to positive,
6

---

Under review conference paper at ICLR 2022
5 GLS WITH LABEL NOISE
In this section, we target at the optimal candidates of in GLS when the label noise presents. In Section we have shown that NLS and PL. have the opposite functionality on the model confidence when training clean data. Given the unseen test data, learning with non-negative smooth rates may not always return the best outcome (Figure 3. Based on this observation, we delve into details to show when NLS is more favorable than PLS and Vanilla Loss. We start with stating Assumption [T
Assumption 1. We assume learning with clean data distribution 'D with smooth rate r" S 1 in GLS returns the best performance on the unseen clean test data distribution Drest-
Assumption [ simply offers an "anchor" point to initiate our analysis for the noisy label setting. To clarify, we don't rule out the possibility that other methods outperform GLS with optimal smooth rate r". Later in Section and Appendix we will empirically test what r" usually on various benchmarks. We define optimal
classifier on Y which follows r* smooth label distribution as: f == arg miny E ((f(X), (ex())))
With the introduction of r* and fz- our goal is then to recover the classifier f using the noisy training labels. To bridge learning with noisy labels and clean labels for GLS, we define λ1, A2 and offer Theorem
(eo _ー (ce)) ]T
A
== (1-r)
2
Theorem 5. The risk minimization of GLS (Eqn. clean data with two additional bias terms:
3 in the noisy setting relates to the risk defined on the
min B(xY)~ ( ((F(X),
+A1 E(XYY ((f((), 1- Y) e(f(()))))
True
M-Incl
+\2 Ex -
(5)
M4-Icc2
The True Risk is the risk minimization w.r.t. clean optimal label distribution Y Training GLS on noisy labels results in two extra bias terms which affect the model confidence. We defer an empirical validation of Theorem 5 to Appendix B Now we proceed to answer "what parameters are preferred in the noisy setting".
5.1 SYMMETRIC ERROR RATES WITH 0
Symmetric error rates e = 6g = e, indicates the probability of flipping to the other class is equal for both classes. In this case, X2 and Term M-Inc2 is cancelled and Eqn. (5) reduces to
min E EXXYY~ (((())) +YI BEX,Y)D ((F(X),111)) E(F(X),
(6)
True Risk
M-Incl
Noisy labels impairs model confidence Vanilla Loss In the GLS frame- work, define the optimal r that will cancel the impact of Term M-Inel as:
r" - 2e
Noise
when Topt 1 2e
M-Inel = 0
The threshold Togi in Eqn. Timplies:
Theorem 6. With [ssumption] GLS with smooth rate Tops yields fbb
Figure 4: Decision between
When error rate rr /2, Topt > 0 (PLS);
NLS, PLS given e, r*.
7

---

Under review conference paper at ICLR 2022
When erfor rate e = r-/"/, (Vanilla Loss);
When error rate
r" /2, Topt < 0 (NL.S).
In Theorem adopting NLS when noise rate e < /2 induces Å < 0, Term M-Inel makes f overly- confident on its predictions compared with Y*. Figure with the decreasing of rf, PL. is less tolerant of labels with high noise. Similarly, if e 2 3 with the decreasing r", NLS is more robust the high noise regime while PLS makes the model become less-confident on its predictions. Clearly, NLS outperforms PLS especially when noise rates are large and r" is small.
5.2 ASYMMETRIC ERROR RATES WITH CY #
reo removes the Term M-Inc1. However, when r < 1, Term M-Inc2 not In this case, adopting r = 1-260 negligible due to assymetric noise transition matrix. As a result, Term M-Inc2 becomes:
1 - r" Ex ((((())))) -((()))))))((()))11 with es 1 20g
1 1-r" 20
6a
1 2ep
Term M-Ine2 in the minimization increases the model confidence (X,Y 0) Dj. The model will then become cverly-confident with the class that has low noise rate en- Mcanwhile, Term M-Inc2 decreases the model confidence on (X, = 1) ~ D (less-confident to the class with a high noise rate e1).
6 EXPERIMENT RESULTS
this section, we present our empirical observations regarding the role of GLS under clean and noisy labels by using UCI datasets, CIFAR-10 and CIFAR-100.
6.1 WHAT THE PRACTICAL DISTRIBUTION OF r" AND ToPr?
r" and ropt on UCI datasets (Dua Graff 2017) As for UCI datasets, we pick Twonorm and Splice for illustration in the main paper. The noisy labels are generated by a symmetric noise transition matrix with noise rate e, = [0.1, 0.2, 0.3, 0.4]. As highlighted in Table Topt appears with positive values when the data is clean (same r*) of low noise rate. With the increasing of noise rates, the performance of PLS results in a much larger degradation compared with NLS. We color-code different noise regimes where either PLS (red-ish) or NL. (green-ish) outperforms the other. Clearly there is separation of the favored smoothing rate for different noise scenarios (upper left low noise for PLS, bottom right high noise for NLS).
Table 1: Test accuracies of GLS on clean and noisy UCI datasets with best two smooth rates (green: NLS; red: PLS). Results on more benchmark datasets are deferred to Appendix D
Smooth Rase
0
Twonora
0.2
0.3
Spiice
0.2
0.4
0.8
0.990
0.990
0.986
0.982
0.968
0,980
0.946
0.919
0.856
0.760
0.990
0.389
0.987
0.981
0:972
0.3778
0.939
0.913
0.869
0.77%
0.990
0.987
0.971
0.922
0.797
7
0.2
0.990
0.989
0.986
0.985
0.969
0.978
0.948
0.919
0.878
0,800
1.D
0.990
03989
03987
03985
0.973
0.976
0.948
0.926
0.87
03306
0.4
0.986
0986
0.984
0.74
0.961
0.566
0.926
=
-1.0
0.986
0.988
0.988
0.983
0.977
0.966
0.954
0.932
0.889
0.819
2,0
0.866
0.965
0.966
0,980
0.976
03922
0.918
0.935
0.895
0.330
0,986
0,986
8.0
0.966
0.986
0.966
0.983
0.986
0.943
0.946
0.939
0.915
0.845
Fage
[.O. 0.8
00.4. 0.8
.1.0
-0.4
4.00 -0.4
0.00
J0.6, -0.4
(8.00
When learning with a larger scale and more complex dataset, like CIFAR-10 and CIFAR- models prone converge local optimal solution
and Topt on CIFAR datasets (Krizhevsky et al., 2009)
8

---

Under review conference paper at ICLR 2022
rather than the global optimum. This phenomenon occurs frequently in NLS which ends up with performance degradation. Thus, in Table 2 when learning with noisy labels, we report the better performance of GLS between direct training and loading the same warm-up model. And we observe that the performance of NLS is more competitive than PLS when learning with clean data. Clearly, NLS outperforms PLS in CIFAR-10 and CIFAR- 100 under various synthetic noise settings. The gap is larger when the noise rates are high.
Table 2: Test accuracies (meanstd) of GLS on synthetic noisy CIFAR datasets. Best two smooth rates for each synthetic noise setting are highlighted for each (green: NLS; red: PLS).
Smooch Rate
CHAIR TO Symmeiin
CIPAR
Axymomernic
CHEAR
Symmetric
0.6
92.11=006
66
484991
73.16-0.116
90.45-00066
8783-0015
54.0-093
19
02.33=000
37
82114086
73.59-00155
9044
87
52.22=0.155
40.9:000
0.4
93.0510.044
$7.1310.07
50.144144
74.2110.19
90449.0.10
87.9010.13
51.26.0.07
41,5710005
r = 0.0
91 01:44:0.16
85
80-4242.29
75. 15.34-00.13
88 38-3240
06.27-0.32
48.03-0.29
58 38.11=0.11 14
-0.4
93.5520.06
$7.5510005
81 5810.19
75.95
87.271183
68.33
56.87:0088
13.0010.16
-088
02.74=0055
86462001
81. 56:0011
76:15:00144
16:40:1..2
47.96=0443
57.35-00088
44 10:0000
-.0
58.58100S 92.580005
$1.5100.0
76,20110.10
88. 88.1710.15
87,5010775
57,01
43.8510.19
2.0
93. 00.30
83.6410.15
76.11=0.00
88.0040.117
81.27=00701
58. 18.10=0.08
03.13=0004
34
77.22-00.9
56-466--11
87 07.29-00.59
58.35-0009
46. 05
060
93.11100.5
58.9110111
815510013
77.4230.16
79.0010.21
87.373042
$7.73-0.10
46.161009
6.2 COMPARISONS BETWEEN GLS AND MORE ROBUST METHODS
In Table 3. we compare GLS with several robust methods in synthetic noisy CIFAR datasets. We emphasize that we are not proposing a new method to compete with state-of-the-art methods. Instead, we hope to help readers understand how the generalized label smoothing fare when label noise presents. Clearly, GLS (especially NLS) can definitely be viewed as a competitive and efficient robust loss function which outperforms Cross Entropy, Bootsrap (Reed et al.) 2014), SCE (Wang al. 2019). APL (Ma et al. 2020) and Forward correction (Patrini et al. 2017) in most settings.
Table 3: Performance comparisons synthetic noisy CIFAR datasets: adopt the same model architecture for all methods (ResNet 34 (He et al. 2016)), best achieved test accuracy is reported.
Method
CIPAR-10, Symmetric
CIPAR-10, Asymmetric
CIFAR-JOO, Synmetric
0.2
0l.4
0.6
0.2
0.3
0..
0.6
Cross Entropy
86.45
82.72
74.04
88.59
86.14
48,20
38.27
Bootstrap (Reed al. 2014
86.06
81.65
75.26
87.69
85.51
47.28
35.81
Forward correction (Parrin ét
2017
84.85
84.98
73.97
89.42
$8.25
$3.04
41.59
SCB Wane S 2019
89.39
80.31
75.28
88.07
85.93
49,34
38.87
API. Mi éf Mig 20.00
88.42
81.27
76.62
88.75
87.41
$1.63
42.31
Peer Loss
90.21
86.40
53,72
EIR i1 10 ATMD
92.57
91.32
88.86
93.48
92.21
68.03
60.49
AUM (Plews el
2120
91.52
87.85
81.71
97.17
90.63
59.29
44.05
Positive Label Smeothing (PLS)
90.24
83.78
75.01
90.61
88.04
55.17
41.63
Negative Label Smoothing (NLS)
89.05
84.85
77.82
90.02
88.42
58.47
46.58
Additional results More extensive empirical results are included in Appendix: empirical validation of main theorems (Appendix BB; practical considerations of GLS (Appendix C); experiment results on additional datasets (Appendix D); bias and variance of the generalization error on the clean data (Appendix E..
Conclusion In this paper, we provide understandings for generalized notion of label smoothing where the label smoothing rate can go negative. We show that learning with negatively smoothed labels explicitly improves the confidence of model prediction. This key property acts significant role when the confidence of model prediction drops. We make connections between negative label smoothing and existing learning with noisy label solutions. In contrast to existing works that promote the use of positive label smoothing, we show both theoretically and empirically the ad advantage of a negative smooth rate when the label noise rate increases. Our observations provide new understanding for the effects of label smoothing, especially when the training labels are imperfect.
9

---

Under review conference paper at ICLR 2022
REFERENCES
Ehsan Amid, Manfred KK Warmuth, Rohan Anil, and Tomer Koren. Robust bi-tempered logistic loss based on bregman divergences. Advances in Neural Information Processing Systems, pp. 14987-14996, 2019.
Antonin Berthon, Bo Han, Gang Niu, Tongliang Liu, and Masashi Sugiyama. Confidence scores make instance-dependent label-noise learning possible. In International Conference on Machine Learning, pp. 825-836. PMLR, 2021.
Blair Chen, Liu Ziyin, Zihao Wang, and Paul Pu Liang. An investigation of how label smoothing affects generalization. arXiv preprint arXiv:2010./2666 2020.
Hao Cheng, Zhaowei Zhu, Xingyu Li, Yifei Gong, Xing Sun, and Yang Liu. Learning with mestance-dependee label noise: A sample sieve approach. In International Conference on Learning Representations, 2020.
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration sequence to sequence models. Proc. Interspeech 2017, pp. 523-527, 2017.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive..t
dici.edu/mo ml
Xin Geng. Label distribution learning. IEEE Transactions on Knowledge and Data Engineering, 28(7): 1734-1748, 2016.
Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Míao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in nearal information processing systems, pp. 8527-8537, 2018.
Ramaswamy Harish, Clayton Scott, and Ambuj Tewari. Mixture proportion estimation via kernel embeddings of distributions. In International conference on machine learning, pp. 2052-20660 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 1X03.0500.025533 023331 2015.
Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. Learning from complementary labels. In Proceedings of the 31st International Conference on Neural Information Prucessing Systems, pp. 5644-5654, 2017.
Takashi Ishida, Gang Niu, Aditya Menon, and Masashi Sugiyama. Complementary-labe learning for arbitrary losses and models. In International Conference on Machine Learning, pp. 2971-2980, PMLR, 2019.
Lu Jiang, Zhengyuan Zhou, Thomas Leung, Li-Jia Li, and Li Fei-Fei. Mentomet: Learning data-driven curriculum for very deep neural networks corrupted labels. In International Conference Machine Learning, pp. 2304-2313. PMLR, 2018.
Youngdong Kim, Junho Yim, Juseung Yun, and Junmo Kim. Ninl: Negative learning for noisy labels. In Proceedings of the IEEE/CV International Conference on Computer Vision, pp. 101-110, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xiv:1412.6990 2014.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. Technical report, Citeseer, 2009.
10

---

Under review conference paper at ICLR 2022
Weizhi Li, Gautam Dasarathy, and Visar Berisha. Regularization via structural label smoothing. In Interna- tional Conference on Artificial Intelligence and Statistics, pp. 1453-1463. PMLR, 2020.
Sheng Liu, Jonathan Niles-Weed Narges Razavian, and Carlos Fernandez-Granda Early-learning regular- ization prevents memorization of noisy labels. Advances in Neural Information Processing Systems, 33, 2020.
Tongliang Liu and Dacheng Tao. Classification with noisy labels by importance reweighting. IEEE Transac- fions on pattern analysis and machine intelligence, 38(3):447-4611 2015.
Yang Liu. The importance of understanding instance-level noisy labels. arXiv preprint arXiv:2102.05336, 2021.
Yang Liu and Hongyi Guo. Peer loss functions: Learning from noisy labels without knowing noise rates. In International Conference on Machine Learning, pp. 6226 6236. PMLR, 2020.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pp. 6448-6458, PMLR, 2020.
Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah Erfani, and James Bailey. Normalized loss functions for deep learning with noisy labels. In International Conference on Machine Learning, pp. 6543-6552 PMLR, 2020.
Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. Learning from corrupted binary labels via probabiroyayl estimation. In International Conference on Machine Learning, pp. 125-134, 2015.
Rafael Müller, Simon Kornblith, and Geoffrey Hinton. When does label smoothing help? In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 4696-4705, 2019.
Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. Learning with noisy labels. In Advances in neural information processing systems, pp. 1196-1204, 2013.
Curtis G Northcutt, Lu Jiang, and Isaac L Chuang. Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research, 2021.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. Making deep neural networks robust label noise: A loss correction approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1944-1952, 2017.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 17Xiv:1701.06545 2017.
Geoff Pleiss, Tianyi Zhang, Ethan R Elenberg, and Kilian Q Weinberger. Identifying mislabeled data using the area under the margin ranking. arXiv preprint urXiv:2001. 2020.
Scott Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6596, 2014.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pp. 400-407, 1951.
Clayton Scott, Gilles Blanchard, Gregory Handy, Sara Pozzi, and Marek Flaska. Classification with asymmet- ric label noise: Consistency and maximal denoising. In COLT, pp. 489-511,2011
11

---

Under review conference paper at ICLR 2022
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6000-6010, 2017.
Yisen Wang, Xingjun Ma, Zaiyi Chen, Yuan Luo, Jinfeng Yi, and James Bailey. Symmetric cross entropy for robust learning with noisy labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 322-330, 2019.
Hongxin Wei, Lei Feng, Xiangyu Chen, and Bo An. Combating noisy labels by agreement: A joint training method with co-regularization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13726-13735, 2020.
Jiaheng Wei and Yang Liu. When optimizing f-divergence is robust with label noise. In International Conference on Learning Representations, 2020.
Yi Xu, Yuanhong Xu, Qi Qian, Hao Li, and Rong Jin. Towards understanding label smoothing. arXiv preprint arXiv:2006. 1 11653, 2020.
Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. L_dmi: An information-theorreicc noise-robust loss function. NeurlPS, arXiv: 03388, 2019.
Zitong Yang, Yaodong Yu, Chong You, Jacob Steinhardt, and Yi Ma. Rethinking bias-variance trade-off for generalization of neural networks. In International Conference on Machine Learning, pp. 10767-10777.
PMLR, 2020.
Quanming Yao, Hansi Yang, Bo Han, Gang Niu, and James Tin- Yau Kwok. Searching to exploit memorization effect in learning with noisy labels. In International Conference on Machine Learning, pp. 10789-10798. PMLR, 2020a.
Yu Yao, Tongliang Liu, Han, Mingming Gong, Jiankang Deng, Gang Niu, and Masashi Sugiyama. Dual t: Reducing estimation error for transition matrix in label-noise learning. arXiv preprint arXiv:2006.07805, 2020b.
Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. How does disagreement help generalization against label coruuption? In International Conference Machine Learning, pp. 7164-7173. PMLR, 2019.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
Helong Zhou, Liangchen Song, Jiajie Chen, Ye Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Re- thinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2102.006500 2021.
Zhaowei Zhu, Yiwen Song, and Yang Liu. Clusterability as an alternative anchor points when learning with noisy labels. arXiv preprint arXiv:2102.0529) 2021.
12

---

Under review conference paper at ICLR 2022
APPENDIX
The Appendix organized as follows.
Section A presents the full version of related works.
Section B includes empirical validations of theoretical conclusions in Section and Theorem 5
Section C discusses practical considcrations and multi-class extensions of GLS.
Section D shows additional experiments on synthetic dataset and UCI datasets.
Section E illustrates the bias and variance trade-off when learning with GLS from clean data.
Section F includes omitted proofs for theoretical conclusions in the main paper.
A FULL VERSION OF RELATED WORKS
Our work supplements two lines of related works.
Learning with noisy labels Annotated labels from human labelers usually consists of an non-negligible amount of mis-labeled data samples. Making deep neural nets perform robust training on "noisily" labeled datasets remains a challenge. Classical approaches of learning with noisy labels assume the noisy labels are
independent features. They firstly estimate the noise transition matrix (Liu & Tao 2015 Menon et al.
2015; Harish et al. 2016 Patrini et al., (017),, then proceed with a loss correction (Natarajan et al. 2013
Patrini et 20177 Liu & Tao 2015) miligate label noise. Recent works propose robust loss fuunctions (Xu et al. 2019; Kim et al.] 2019 Liu & Guo 2020; Wei & Liu. (2200 to train deep neural nets directly without the knowledge of noise rates, or design pipeline which dynamically select and train on "clean" samples with small loss (Jiang al. 2018 Han et al. 2018 Yu et al. 2019) Yao et al.) 2020a). More recently, several approaches target at addressing more challenging noise settings, such as instance-dependent label
noise Cheng et al. 2020 Berthon et al. 2021).
Understanding the effect of label smoothing Learning with one-hot labels prone to over-fitting, soft label learning then naturally draws attentions of machine learning researchers. Successful applications of soft label learning include the label distribution Icarning (Geng 2016) which provides an instance with description degrees of all the labels. Label smoothing (LS) (Szegeyy et al. 2016) is another arising learning paradigm that uses positively weighted average of both the hard training labels and uniformly distributed soft
labels. Empirical studies have demonstrated the effectiveness of LS in improving the model performance
(Pereyra et al. 2017; Szegedy et al., 2016) Vaswani et al.] 2017]: [Chorowski & Jaitly| (017)) and model calibration (Müller | 2019). However, knowledge distilling teacher network (trained smoothed labels) into a student network is much less effective (Müller et al. 2019). Later, generalization effects of more advanced forms of label smoothing was studied, such as structural label smoothing (Li et al. 2020 and non- non-uniform label smoothing (Chen et al.) 2020). More recently, it was shown that an appropriate label smoothing regularizer with reduced label variance boosts the convergence (Xu et al. 2020). When label noise presents, (Liu 2021) gives theoretical justifications for the memorizing effects of label smoothing. And the effectiveness of label smoothing in mitigating label noise is investigated in (Lukasik et al. 2020).
B EMPIRICAL VALIDATIONS OF MAIN THEOREMS
this section, we empirically validate our main theoretical conclusions, i.e, the connection between GLS and popular methods, and the comparison between clean empirical risks and noisy ones.
13

---

Under review conference paper at ICLR 2022
B.1 CONNECTION TO OTHER ROBUST METHODS
We firstly compare GLS with backward correction (Natarajan et al. 2013), forward correction (Patrini et al. 2017) and peer loss (Liu & Guo 2020 on CIFAR- 10 dataset. To approximate the performance of backward/forward Loss Correction, we adopt GLS with smooth rate Tল- As for the approximation of peer
loss, choose (f) f(f(X), f=.... which is equivalent to GLS when -oo. Experiment results in Table 4 on CIFAR-10 under symmetric noise settings demonstrate that the equivalent forms of GLS are robust to label noise.
Table 4: Comparison of test accuracies on CIFAR-10 under symmetric label noise.
CIFAR-IO, Synmetric
Method
e 0.2
0.4
0.6
Backward T (Patrini et al.
2017
84.79
83.40
71.52
Forward T [Patrim et al. 2017)
84.85
83,98
73.97
GLS Torm
87.33
81.73
75.80
Peer Loss (Liu Guo GLS form
2020
90.21
86.40
79.64
88.98
85.05
76.66
Explanation of the performance gap In practice, we adopt the same hyper-parametee setting as used for all other smooth rates for GLS form. Loss corrections will firstly warm-up with the cross-entropy loss, estimate the noise transition matrix with this pre-trained model, and then proceed train with the backward/forwaar corrected loss. Peer loss functions adopt dynamical adjustment for learning rate. The warming up, estimation error of noise transition matrix as well as the special hyper-parameter settings explain performance gaps.
B.2 CLEAN EMPIRICAL RISK V.S. NOISY EMPIRICAL RISK
Now we empirically verify Theorem 8, which relates the risk of GLS in the noisy setting to the clean ones. Assume the the noise label is generated through the symmetric noise transition matrix. We restate the connection between two risks as below:
E (xx [(((()), VOLS,r)
BXXYY) [((((X))))) + A ( 1 -
-(((x),
Noisy Risk
(Cleean) True Risk
Gap of Risk
In practice, obtaining the exact value of r" is not trivial. Finding accurate for GLS to make ) be exactly 0 is pretty hard. Note that when M-Incl is equipped with the Cross-entropy loss and NLS encourages confident predictions, one mistake could make the value of M-Incl has huge difference. To be more specific, suppose the label of an image is a "cat", if the NLS-trained model makes a mistake on this image, it is very likely to predict this image "cat" with probability that p + since NLS may push the model become overly-confident. Then log(p) can be extremely small. Thus, the empirical risk (evaluated on the clean data) for NL. will change much more significantly than PLS ones.
We use UCI dataset (Waveform) for illustration. The value of r" is approximately O, and the estimated (clean) true risk in above is 0.98 When the noise rates are 0.1, 0.2, 0.3, 0.4, the optimal smooth rate should be -0.25, -0.67, - 11.5, -4 according to Eqn. 7. The estimated noisy risk of GLS on these noise settings can be summarized Table 1. Clearly, when 0.1, r = 0.25 is closest to the estimated clean true risk (also returns the best test accuracy among these smooth rates). Similarly observations reach to all other e. Although the risks evaluated on noisy settings with the corresponding Topt can not be exactly the same as the true risk (evaluates the clean data with 0.0), Fig already reaches the smallest rates and the non-zero Gap of Risk is best explained by the precision of r* as well as the estimation error of risks.
among these smooth
14

---

Under review conference paper at ICLR 2022
Table 5: The difference between the empirical true risk of Y on the clean data and empirical risk GLS on noisy labels
dataset): r", empirical true risk, and empirical noisy risks under various noise
levels are highlighted in purple.
Smooth rate
Risk (elcan)
Risk le 0.1)
Risk (e = 0.2)
Risk (e = 0.33
Risk = 0.4)
0.8
0.6773
0.6831
0.6873
0.6899
0.6923
0.6
0.6295
0.6521
0.6689
0.6833
0.6900
r 0.4
0.5437
0.5994
0.6408
0.6718
0,6873
0.2
0.0
0,1798
0.4057
0,5399
0.6314
U.6758
= 0.25
-36,8005
0.1983
0.4381
0.5957
0.6685
= 0.67
-353.1283
-28.3508
0.2167
0,5132
0.6503
-1.5
-97.4378
8047
-94,5099
0.1911
0.6003
== 4..0
270.6445
-179.0037
93.11557
-156,4539
0.1165
C PRACTICAL CONSIDERATION OF GLS
the main paper, we theoretically show when we should adopt NLS and PLS. In this section, we discuss more practical considerations of GLS, including the optimal smoothing parameter, how to reduce the impacts bias terms, and multi-class extensions.
C.I THE OPTIMAL SMOOTHING PARAMETER
In practice, we don't 't have access to noise rates 6,. Our work does not intend to particularly focus on the noise rate estimation. For readers interested in the noise rate estimation, please refer to (Liu & Tao 2015 Menon et al. 2015 Harish et al. 2016 Patrini et al.) 2017 Yao et al.] 2020b Zhu et al. 2021). To estimate Topt T 2e one can simply assume r" + 0. And the noise rate e is estimable by a large family noise estimation methods mentioned above. Our practical observations show that NLS with CE warm-up not sensitive to the negative smooth rate, for example, on CIFAR-10 and CIFAR- 100 synthetic noisy datasets, r < 1.0 frequently achieves best results (see Table the main paper). Our current contribution focuses understanding the generalized label smoothing, and prefer leaving the task of identifying the optimal smooth rate to future works.
C.2 MAKING GLS MORE ROBUST TO LABEL NOISE
There is a line of related works targeting at distinguishing clean labels from the noisy labels. Current literature in selecting clean samples from noisily labeled dataset is based on the empirical evidence that samples with noisy/wrong labels have a larger loss than clean ones. For interested readers, please refer to (Han et al. 2018 Jiang et al. 2018 Yu et al. 2019 Yao et al. 2020a Wei et al. 2020; Northcutt et al.) 2021). Compared with the risk minimization over the clean data distribution (X, ~ D, learning directly with GLS on the noisy
distribution (X, Y)- D will result in an extra term (el - eg) (1-r) ---rr)EE, =1),y= [f(f(X), 0) - ((f(X), 1)]
compared to the clean scenario. Empirically, we can estimate the bias term, perform a bias correction by subtracting the estimated bias term from the objective function in Eqn. (3).
Suppose we have access to a clean distribution Delean which consists of selected clean samples. Denote the estimated noise rates é, when e사 * 0. in order to make GLS be more robust to label noise and fit the optimal distribution Y*, we improve GLS by performing a model confidence correction on the dominating class through:
15

---

Under review conference paper at ICLR 2022
GLS-C:
min E (x (((r(x), yrass.)) ((
-éé -00) (1)r))--(.y=- B(xYY=l)>Dum ((f(().)) (((())))))
confidence correction
C.3 MULTI-CLASS EXTENSION
As an extension to the binary classification task, we next show how GLS extends to the multi-class setting under broad families of noise transition model. When learning with multi-class classification tasks, we assume Assumption 1 holds the multi-class setting. And for Y, E [K], we extend the definition of model confidence to multi-class classification tasks as:
Definition 2. Model confidence of sample (K-class elassification). Given model J, sample with is target label E [K) the model confidence score of f w.r.t. sample x: is defined as
MC(f;a,8) =((x2 K
f(x); kE
Sparse noise transition matrix Sparse noise model (Wei & Liu 2020) assumes K is an even number. For E [K/2], ie Je. sparse noise model specifies K/2 disjoint pairs of classes (fe: to simulate the scenario where particular pairs of classes are ambiguity and misleading for human annotators. The off-diagonal element of T reads T T...j. = co. T = e₁. Suppose eo + e 1, the diagonal entries become Tesi eI, Ticje 1 en. Clearly, our conclusions in Section 5 extends directly to the sparse noise transition matrix by simply splitting the K-clas, classification task into f disjoint binary ones.
Symmetrie noise transition matrix Symmetric noise model (Kim al. 2019) is a widely accepted synthetic noise model in the literature of learning with noisy labels. The symmetric noise model generates the noisy labels by randomly flipping the clean label the other possible classes with probability Vi # j.
Ti c/(K - 1). and the diagonal entry Ti; = 1 cc Define the optimal r for GLS in the multi-class setting
as Togt 1)-- -K-r Theorem can extended the multi-class setting as:
K-r
[K-
Theorem 7. Under Assumption 1
suppose the symmetric noise rate is not too large, i.e, e < K-1 K
GLS with
smooth rate r Yaps yields fz.
When enror rate e (K-1)--
Top > 0 (PLS);
K
When error rate E (K-1)-r"
= 0 (Vanilla Loss);
When error rate (K-1)r"
r Top < (NLS).
K
D ADDITIONAL EXPERIMENT RESULTS AND DETAILS
this section, we include more experiment results, observations and details for learning with GLS.
D.1 EXPERIMENT DETAILS ON CIFAR-I0 10, CIFAR-100
We firstly introduce experiment details CIFAR- 10 dataset adopted in experiment designs.
16

---

Under review conference paper at ICLR 2022
We adopted ResNet34 (He al. 2016), trained for 200 epochs with batch-size 128, SGD (Robbins & Monro T95T] optimizer with Nesterov momentum of 0.9 and weight decay le-4. The learning rate of first 100 epochs 0.1. Then multiples with 0.1 for every 50 epochs.
Training settings of clean CIEAR-10 dataset (Krizhevsky et al., 2009)
Generating noise labels on CIFAR datasets We adopt symmetric noise model which generates noisy labels by randomly flipping the clean label to the other possible classes with probability And we set e 0.2, 0.4, 0.6 for CIFAR-10, = 0.4, 0.6 for CIFAR-100. We also make use of asymmetric noise model. The asymmetrie noise generated by flipping the true label the next class with probability c. We set
< = 0.2, 0.3 for CIFAR-10.
Training settings of synthetic noisy CIFAR datasets The generation of symmetric noisy dataset is adopted from CCeng et al. 2020). The symmetric noise rates are 0.4, 0.6]. We choose two meth- ods to train GLS.
0.4,
Direct training: this setting is the same as training on clean CIFAR-10 dataset.
Warm-up: in this case, we firstly train a ResNet34 model with Cross-Entropy loss for 120 epochs. For this warm-up, the only difference in hyper-parameter setting is the learning rate, where the initial learning rate is 0.1 and multiplies 0.1 for every 40 epochs. After the warm-up, GL.S loads the same pre-trained model and trains for 100 epochs with learning rate le-6.
D.2 WHY NLS OVERLOOKED?
When learning from relative large scale dataset, NLS tends to push the model become overly confident early in the training. The poor performances of NLS (direct-train) in TableBJexplain why NLS is neglected. When there is no warm-up, training NLS directly without warming up will reach a 88% 92% test accuracy on the clean data. The performance will degrade much more significantly than PLS when the noise level is high or |r| is large. In Table 6 we provide the comparisons between direct-train and warm-up in several settings. The improvement bring by a warm-up procedure becomes much more significantly in the high noise regime. NLS makes the classifier be overly confident at the early training which results in converging to bad local optimum (without CE warm-up. NI.S frequently results in a worse performance in CIFAR-10 10 and CIFAR- 100). Since the model will usually fit the clean data first, then over-fits the noisy ones (Liu ct al. 2020). a large number of approaches (such as Loss corrections Patrini et al. 2017). Peer Loss (Liu & Guo 2020), etc) adopt CE warm-up firstly. Note that there is no difference in the computing costs between NLS (with CE warmup) and CE loss, proceeding with NLS to enhance the model confidence makes NLS much more competitive in the high noise regime, also gives practical insights on how to make NLS work better when learning with clean data.
Table 6: Test accuracies GLS on assymetric noisy CIFAR-10 and symmetric CIFAR-100 (left/right denotes direct train warm-up).
Cifur-I0 10 Asymmetric
CIFAR-100 Symmetric
Smooth Rate
$ = 0.2
5 0.3
== 0.4
5 0.6
0.8
87.89 / 90.51
86,38/87
54.78/
40.21 / 80
0.6
$9.14790.55
85,97 88.01
$2.83752.35
39.647 40.57
0.4
88.23/ /9.611 90.61
86.95
88.88.0 04
51.40754.55
38.29/41.66 4..66
= 0.4
19.71 / 89.60
21.86
88.42
50.30
31.35 / 43.91
== 0.8
.789.02
788828
57.43
26.75 76.75/44.19
1.0
8888.68
-788.29
-/57.53
-/44.59
F= -20
-7 -788.86
788.13
758.21
-745.47
4.0
-/ /9.80
-/ 88.20
-/ /58.47
-/46.86
-6.0
70.0.2
-/ 188811
-7 -757.87
-747.18
17

---

Under review conference paper at ICLR 2022
D.3 EXPERIMENT DETAILS ON SYNTHETIC DATASETS AND UCI
We introduce experiment details on synthetic datasets and UCI datasets adopted in our experiment designs.
Generation of synthetic dataset In the synthetic (Type 1) dataset, we generate 500 points for both classes. Class +1 distributes inside the circle with radius 0.25. Class -1 generates by randomly sampling 500 data points in the annulus with inner radius 0.28 and outer radius 0.45. As for synthetic (Type 2) dataset, we uniformly assign labels for 50% samples in the annulus (with inner radius 0.22, outer radius 0.31) based on Type 1 dataset.
Generating noise labels on synthetic datasets and UCI datasets Note that these datasets are all binary classification datasets, each label in the training and validation set is flipped to the other class with probability e, and we set 0.1, 0.4 for synthetic Type I dataset, e 0.1, 0.3 for synthetic Type 2 dataset.
Training settings of synthetic datasets For both types of synthetic datasets, we adopted a three- ReLU Multi-Layer Perceptron (MLP), trained for 200 epochs with batch-size 128 and Adam (Kingma & Ba 2014) optimizer. The initial learning rate is 0.1, and it multiplies 0.1 for every 40 epochs.
Training settings of UCI datasets Dua & Grail (017 We adopted (Liu & Guo 2020) two-layer ReLU Multi-Layer Perceptron (MLP) for classification tasks on multiple UCI datasets, trained for 1000 episodes with batch-size 64 and Adam (Kingma & Ba 2014) optimizer. We report the best performance for each smooth rate under a set of learning rate settings, [0.0007, 0.001, 0.005, 0.01, 0.05].
D.4 ADDITIONAL EXPERIMENT ON r" AND TOPT
r" and ropt on synthetic dataset We generate 2D (binary) synthetic dataset by randomly sampling two circularly distributed classes. The inner annulus indicates one class (blue), while the outer annulus denotes the other class (red). Clearly, the generated synthetic dataset is well-separable (Type 1) and we hold 20% data samples for performance comparison. The noise transition matrix takes symmetric form with noise rate e, for both classes. To simulate the scenario where the clean data may not be perfectly separated due to non-negligible amount of uncertainty samples clustering at the decision boundary, we flip the label of 50% samples near the intersection of two annulus to the other class (Type 2). As specified in Table 7 (0.1, 0.4) for Type data and f* [0.0, 0.2] for Type 2 data. With the presence of label noise, the distribution of Topt shifts from non-negative ones to negative values. Even though NLS fails to outperform PLS on clean data, we observe that NLS is less sensitive to noisy labels. Data with high level noise rates clearly favor NLS with low smooth rate!
Table Test accuracies of GLS on clean and noisy synthetic data. We report best test accuracy for each method. Topt and the corresponding test accuracy are highlighted (green: NLS; red: PLS).
Syntheric data (Type I)
Synthetic data (Type 2)
Method
4 0
0.2
0.4
e; 0.2
0.4
PLS
0,878
0.786
0.894
0.848
0.842
Vanilla Loss
0,889
0.882
0.806
0.894
0.875
0.868
NLS
0,893
0.885
0.825
0.883
0.884
0.875
Yige
0.41.
-0.2
-0.4
[..
-0.3
-0.5
r" and Nopt on more UCI datasets We further test the performance of generalized label smoothing on 7 more UCI datasets (Heart, Breast 1, Breast 2, Diabetes, German, Image and Waveform). Our observation remains unchanged: there exists a general trend that with the increasing of noise rates, NL.S becomes much more competitive than PI.S. Here, we attach the results of 4 additional UCI datasets for illustration.
The noisy labels are generated by a symmetric noise transition matrix with noise rate e; = [0.1, 0.3, 0.4). As highlighted in Table 8 Nopt appears with positive values when the data is clean (same r*) or of low
18

---

Under review conference paper at ICLR 2022
Table 8: Test accuracies of GLS on clean and noisy UCI datasets (Image, Waveform, Heart, Banana) with best two smooth rates (green: NLS; red: PLS).
Smooth
Inage
Woveform
Es 0.1
6 = 0.2
By
0.3
0.4
de
0
0.1
6,
0.2
6,= = 0.3
By 0.4
f 0.8
0,992
0.983
0.733
0.945
0.875
0.939
0.935
0.931
0.927
0.885
0.6
0,993
0.987
0,970
0.939
0.869
0,943
0,943
0.943
0,929
0.901
0.4
0.997
0,980
0.933
0.99
0.865
0.911
0.937
0,913
0931
0.905
0.2
0.903
0.993
0.96
0.9
0.75
0.941
0.33
0.33
0.31
0.913
0935
-0.2
0.912
0.96
0353
0919
0.822
0.937
0399
0,399
0.933
03907
0..4
0,882
0.23
0.%
0.822
0.37
0.33
0.917
0.8
0.842
0.882
0.26
0933
0.872
0.921
0925
0.939
0,931
0.923
-1.0
0.832
0.869
03099
0.929
0.882
0.921
0.923
0.933
0,929
0.907
y=
-2.0
0,818
0.815
0389
0,909
0.906
0.911
03913
0.921
0.927
0391
Smooth Rate
Heart
Bonome
0.1
0.2
0.3
0.4
0.2
0.3
0.4
0.8
0,885
0.853
0.836
0.300
0.738
0.896
0,893
0.876
0.847
0.790
0.6
0.902
0.836
0.20
0.836
0.738
0.903
0.881
0.876
0.855
0.811
0.4
0,885
0.853
0.866
0.820
0.771
0,900
0.887
0,874
0,839
0.807
0,902
0333
0.00
0.754
0.94
0.876
0.800
= 0.0
0,902
0.833
0820
0.800
0.771
0399
0.881
0871
0.349
0.833
0.836
0385
0
0.040
0.6
0,809
0.836
0320
0333
0.721
03455
0.864
0.861
0.899
0.377
-1.0
0,383
0.869
0,003
0383
0.754
0.796
0.812
0.852
0.34
0.811
T= -2.
0.383
0.869
0320
0,333
0.787
0.759
0.761
0.819
0.852
0.819
-4.0
0,885
0.869
0883
0,885
0.820
0.718
0.725
0.738
0.787
0.813
0,869
0.869
0.700
0.699
0.738
noise rate. With the increasing of noise rates, NLS becomes more competitive than PLS. We color-code different noise regimes where either PLS (red-ish) or NLS (green-ish) outperforms the other. Clearly, there is a separation of the favored smoothing rate for different noise scenarios (upper left low noise for PLS, bottom right & high noise for NLS).
D.5 ADDITIONAL EXPERIMENT RESULTS ON MODEL CONFIDENCE
NLS improves model confidence on Synthetic Type dataset In this case, the clean data that are close to decision boundary distributes randomly. In Figure the colored bands depict the different levels of prediction probabilities. When the smooth rate increases from negative to positive, more samples fall in the orange and light blue band which indicates uncertain predictions. When the smooth rate increases from negative to positive, learning with GLS will result in more uncertain predictions. With the increasing of noise rates (c; = 0- - 0.4), GLS with a fixed smooth rate becomes less confident on its predictions. Thus, a smaller smooth rate is required when the noise rate increases.
Clean 0w
Clean, nw0. ACC=O
AccwO 0.89
Boundaryy (0.5)
Pred Prob 0.7
Figure 5: Model confidence visualization of GLS on synthetic data (Type 2) with the clean data. r" [0, 0.2).
(left: NLS: middle: Vanilla Loss; right: PLS).
Model confidence of GLS on CIFAR-10 test dataset When GLS trained on symmetric 0.2 noisy CIFAR- 10 training dataset (see Figure 7 with the decreasing of smooth rates (from right to left), the model
19

---

Under review conference paper at ICLR 2022
Rendom noiee U.J, ro40.. U.S. bat AccoD.075
Rendom nanu 0.3, muo. best Accmn0.000
Rendom moine 3, roO.1 Test Accin 1.142
Fred Prof 0.3 Decision
Figure 6: Model confidence visualization of GLS on synthetic data (Type 2) with noise rate e; 0.3.
Topt -0.5. (left: NLS; middle: Vanilla Loss; right: PLS).
confidence on correct predictions gradually approach its maximum, while for wrong predictions, the model confidence converges to its minimum value. NLS makes the model prediction become over-confident on correct predictions and in-confident on wrong predictions.
Carrec direitatee
2320
Merng--laas Bcadon
1750
3750
1750
5300-
6300-
1250
1230
1250
4300
3000-
750
2010-
2300
250
250
250
Model Coniilancc
Mode Getieess
Vodd Cerfslanc
Vodel Gelieroo
Figure 7: Model confidence distribution of correct and wrong predictions on CIFAR-10 test data. (From left right: NLS (r -0.8, -0.4), Vanilla Loss, PLS (r 0.4), trained symmetric 0.2 noisy CIFAR-I0 dataset).
D.6 EFFECT GLS PRE-LOGITS
We visualise the pre-logits of ResNet-34 for three classes CIFAR-10. We adopt the method from (Müller let al. 2019) which illustrates how representations differ between penultimate layers of networks trained with different smooth rates in GLS. In Figure 8 NLS makes the model / be confident her predictions and the distances between three clusters are clearly larger than those appeared in Vanilla Loss and PLS.
Figure 8: Effect of GLS on pre-logits (left: NLS; middle: Vanilla Loss; right: PLS; trained with symmetric 0.2 noisy CIFAR-10 10 training dataset).
20

---

Under review conference paper at ICLR 2022
E BIAS AND VARIANCE TRADE-OFF OF GLS
Denote fu. fs as pre-trained models on the training dataset w.r.t hard labels and soft labels, respectively. The vector form of the prediction w.r.t. sample æ given by fu and fs are (z;(;; D) and (a; D). For ease of presentation, we relate notations with subscript H/S to hard/soft labels without further explanation. Given the sample æ and the one-hot label y, we denote the averaged model prediction by:
1 Zn
F cz; D) = exp ED log (Ir D)),, (s(:;)) = Zs
exp En log (x) D))]
where ZH, Zs are normalization constants. The bias of model prediction is defined as the KL divergence DRL between target distribution (one-hot encoded vector form) and the averaged model prediction.
Bias == Er.y y log
D) Biass == En, y log
D)
While the variance of model prediction measures the expectation of KL. divergence between the averaged model prediction and model prediction over D:
F (2; D) f (r; D }
[s D) is (x; D)
]
Vari = ED Eyy (2; D) log
Vars = Ep Ery [s(r; ; D) log
Empirical observation from (Zhou et al. 202 shows that the variance brought by learning with positive soft labels given by a teacher's model (Hinton al. extension, we are interested in how GLS interferes with the bias and variance of model prediction.
2015) less than the direct training w.r.t hard labels. As an
Bias and variance of GLS clean dataset We introduce empirical observation regarding the role of
GLS bias and variance trade-off in Figure 9 We select nine smooth rates of GLS for illustration. Each smooth rate setting of GLS trains on the CIFAR-10 dataset for times with different data augmentations. To estimate the variance and bias of pre-trained models, we adopt the implementation in (Yang et al.) 2020). Empirical results show that learning directly with larger positive smooth rate typically results in lower variance and higher bias. In Figure we can observe almost constant bias values and very low variance for NLS. This is best explained by the warm-up pre-trained models and the fact that NLS pushes the classifier to give confident predictions. As for PLS, with the increase of smooth rate, the overall bias has an increasing tendency while the variance the decreasing pattern. Especially when the smooth rate approaches to i.e., 0.9, the variance is close to 0.
Vortance
2
Accuraty
0.2
1.0
-3.0
0.00
0.4
Smooth
Figure 9: Bias and variance of pre-trained GLS models clean CIFAR-10 test dataset.
21

---

Under review conference paper at ICLR 2022
F OMITTED PROOFS
F.1 PROOF OF THEOREM
Before we prove Theorem 1 we first introduce Lemma]
1
Lemma 1. y(z, GlS.)) ( (1 -5) ((f((x).)) + ((((xx)) y).
Proof of Lemma 1
Proof. For CE loss, due to its linear property w.r.t. the label, we directly have:
5) [(f(x), (OS.,)) ((f(x),) (1-r)+ () = =(1- =(1-)) ((f(x). v) + (((((x), y)
Proof of Theorem 1
Proof. Based on Lemmal)
I with bit of math, for NLS, we have:
min E (x, (f(()) pas.. (f((X).Fas.TT)
=min E /~ (1+ (): ((f((),))) 1) - --- -(((x)),)11))) 8)]
-min E ( [11 (1++-) + (1- f) -((P(X), - [(11 -(1-- ()) ((((X),))) - + - -
min E, (x,
2. ((((()))))))))))))))))) P) ((((x), pGI..)) ((.s))
22

---

Under review conference paper at ICLR 2022
F.2 PROOF OF THEOREM S
Proof.
(1- 2) -(((())))) 8) + r ((1(
Equ
3
min E
:10
= -min P( = 0|Y = -)) (a ((((()))) + 02 10022e((f ((F(X),
+ +P( = 1|Y = 0) (a (((()).))) + 022 (...))))).(0000 ((((()))))
+ + |P(K = -0-=0) =) (a. (()(((())))) +
((f(x).1)
+P(0 = 1/2 1) (a.
+ -( 00)]
- =min [[11 co) + cv 02] ((((X)))) 0) + [(1 - c.) .2 t a] ((f(X).)) 1)]
+ +
[11 (1))))
c2] (f(()) [(1 [(1-a) (0aa))0222 +e aq (((()))))
=x,n
[[[1 co) teo 22 ((f(X)))) 0) +
a) (((()).))
[[1 (6)) t 0g) ((((X),)) ((1- (()-o)))))(()))))))))))))))))))) c2 tc a ((((x),)) 0]
+ +x,y==
+Exy=
cs (c22- - c)) (c))(())))e(((())) -es (
=min ( [[1 -e) 1 teo 02] (((()),)) Y) + [(1- ev) 02 + t0 ((((X),1-X)
-es (a 02) -xxx=1 -0)---0))Bxx--1 [(F(X), 1) - -((f(X). 0)]
=min Ex,, (a 02) ((f(X),))
+ (1 60) 02 + 00
.0. al 1Ex,Y ((((()), - x(P()))))
-es . (a1 02) Ex [(f(())))) - -((F(X),))
= =min (XY)) (0c + 02) {((()), rr]
x² + (1 - (1-eo) C2 + ai] EXXYND (e(xx)), - -((((x),))
+
2
-es . (a - 62) ExxY=I [(((()),) - ((F((),)))
= mmin (XYY) (f(X), +λ1 -Ec.Y)> ( ((f(x),1-Y) -(((x)))))
True Risk
M-Inel
+\2 Ex ((((()))) (((((),1) )
M-Ine2
23

---

Under review conference paper at ICLR 2022
F.3 PROOF PROPOSITION
Proof. The risk minimization of backward correction is equivalent to:
E (x. e (f(X), ( y)]
(By Theorem in (Patrnni et al. 2017)
The risk minimization of forward correction is equivalent to:
E (x. ( (((X), =B ((((()))))
[ v)]
(By Theorem 2 in (Patrini et al. 2017))
Theorem and in (Patrini et al. 2017) demonstrate that forward and backward corrected losses equal the original loss l computed on the clean data expectation. Thus, for rLC 2e0-1 by Theorem (adopt
2np
r" = 0), we have:
[(f(X), rrass + Arc Ex,v- 1 ((((()))))
min E (x,D)DD
Bias--C
-min BX ((((X), y) 1 (1 -2e) TLC
ExXYYYD ((f(())) Y) - -((((x).
+ (1 nc)) Rx ((F(())))
( - -e(f(X), 1) + ALC: Ex ((f(x)))) - -e(((x))0)
=min BXXYND ( (((()), tea
1 (1-200 2e0
-) 1Ex,Y= (((()))) -(((()).1)
1 2e0
= =min B(nxY (((()),))
Thus,
min E (x e- (P(X), = minn (x, vy [ (f(x)
= =min (x, ((f(X), FOaSS.))] + ALC Ex,y=1 ((((X), 1) {(F(X).))
Bias- LC
F.4 PROOF OF THEOREM 2
Proof. Based on Proposition when es = 0, ALC O, we directly have:
min E i (e (f(()))))) min E (x. (f(X),
min (x. ((((X), {OLSS,e))
F.5
PROOF OF THEOREM 3
Proof. Note that
min E Ex. (((x))))((((())))))
min E (x (xx)))=((((((X)) ((((())))) ((f(x), 1-
24

---

Under review as conference paper at ICLR 2022
We have:
min E (x, ((((x)) y(ls.m))
=min (x,))= (1 (1-22 e((((X),)) + raL, ((f(()),11))) })
2
min E (ND (f(X), f) + 2 - raL
TCL
([(()),1-1)
When rat - -00, we have
nm I-ra
+ -1. Thus,
min R (x,)D
(eanND/(a(fff la (f) min B [((((),, p(ssaa-)
F.6 PROOF OF PROPOSITION 2
Proof. Note that:
E (x [(((()))))
E E (x ( [(f(x),,ooss.)))
= (,,ND (1- () (f((x), F) - ((((()))1-))
r
B(X, ((P(X), -(((x),1- 87]
2
And we have:
E P: [(f(X))., 12]
=Ex P(C = 0) (()((((())))))))))(()) + = 0)) 0)) (I(
(1- =,, [P( = 0) (1( = 0))
((F()),))
+ +Ex -x Tp = 0) ())
= 0)) ((f(x))))
=Ex,v=o pP( 0) (0--)))((80)) -(((())))) + - P(
= 0]) P1-P(8-
1((())..)
+ + (1 - -P(Ý 0)) ((f(X).)) +P( =0) ())
+ (1- - 2. p(ÿ 0)) E Ex, [(((()). 1) - -e(F(x).0)
25

---

Under review conference paper at ICLR 2022
Thus,
min E ) Epr (f(X), 8)] min E (x, NND (((()).)) - e(((())), Pa)
E ) ((((X), 8) - -(x
( ((((())))
=min
-min E x.y - ((F(X).)) 0) + + ((F(X),))
-Ex, P PP(P = 0) (((()),)) 0) + (1- - P(8 -()= (((()),)) 1)]
-Ex v1 =1 (1 - P(( = 0)) (f())))) + PP( =0) ((f(X), 1]
(1 - 2. P( = =0] xx (((()).) - <(F(x).0) 0)
- -min 8=0 (1- - P(γ- = 0)) [((((X),0) - -e(r(x), 1)]]
+ +Ex -- (1 -P(p = 0)) ((((()),) 1) - -e(F(x).0]]
- -11-2-P() -0)) -xxy= (((()), 1) - -((((X).0)
-mi E (x D (1-P(8- = =0] ((((()), F) - -((((x),1-))) 8]]
8)
-11-2.P(* P( = =0)) -x [((F(x)) (AA--))--(FFFEE e(F(X))0)
Thus, for rr. P(V = 1), Am. AM
- rPL, we have:
B (X, (f(X), 87] F ((X.FI0(<((XX))
T (x,PDD [(F(X), {(L...))
(x, (I-P(Ý- = 0)) [[(((x),)) -((((x),1- 871]
-(1-2 21-2 === 0)) Ex
- - TP.. -E ( [(((X), 8) -(f(()).--)) ]}]]
-E (x (1 P(8- = 0) - IP(Y = =1) (e(P(x), 8) - -(P(x),
(2 P(Ý 1) 11) -x ((((()) ((((x))))))(( [(F(X).,)
=XPr. -pr.-Ex,y=1 ExP==I ((((())))) - -(((())))) 0]]
And can conclude that:
min Ex, (<m 8)] min E E( (((((x) 8) e(P(X), VOLS.m)
+XPI. Ex,8=1 ((((x), 1) -(((()),))
Bias. PL.
26

---

Under review conference paper at ICLR 2022
F.7 PROOF THEOREM 4
Proef. When PPY 0) -P(Y 1), according to Proposition
we have Apr. = 0 and:
min B P)D EpL. (f(X), {)
min I8
(x,Y D~
((f(X), ((F(X),, P.LS.M. (
-min E (x,5)~
TPL ((((())) 8) PPL
((f((), -P]
2
min (X,F)D [(f(X), - ((f())11-8)
When PPL. - -00, we further have:
min E (x,8) [F(()) p(
- min E (x P) ((((X), P) + 2- raL
rCL
((((x),1-)) V
min Px P (F(()), 8) ((f(X),1-))) 8)]
Thus, Theorem is proved.
F.8 PROOF OF THEOREM 6
Proof. Note that the optimal that will cancel the impact of Term M-Incl is:
22c
Topt ==
1 1 2
When e < r- 2 Topt > 0. In this case, learning PLS with smooth rate Topt results in:
min E (xx,))D [(((X), F(IS) = -iin Ex ((f(X),
which yields fh:
When e = _ Topt = 0. Learning with the Vanilla Loss yields fö since:
min E (X
min EXXYI~ (((())))))) (
Similarly, when 5. learning NLS with Topt < 0 yields fö.
27

---

Under review conference paper at ICLR 2022
F.9 PROOF OF THEOREM 7
Proof. Denote pi = PY = i) as the clean label distribution, D==P(P = i) as the clean label distribution.
K.e Let c' K-1' we have:
E [Xx (1Fx)))((())) + Ex
Σ iE[K]
r
(f(x).
K
Σ (E[K]
B bxx (1-)))))
e(F(()), }]]
Ex
E
_ お {(F(X))
iE|K]
(1 r)
Σ E (X. Σ
The (((x)){=))) =b]]
+Bx
+
Σ K
{((()),
iE|K]
jE[K]
iE|K]
(1 r)
Ex (1 ()) e((f(X), +
Σ sE[K]
é [(((x),0)
Exx
Σ
r (((())))))
K
K
kE[K]
LiE(K]
(1
Σ Ex (1- (((()) {)]
+ EEx
(1-r),
+
Σ ((F(X),))
K
K
iE[K]
JE(K]
(1-r) (I (
18 EX ( ((((()))))
+ +Ex
(1-r)-e
Σ ((F()),))
K
K
=03
C3
EXXYWD (((((),))) {
r Σ ((P(X),.))
C4 Ex Σ ((((X),))
+
K
SE[K]
jELK]
[ [F((X), x1) rr]
+ (cs
C3 r
Ex
((f(X),))
(1 r+)) K
SE[K]
True Risk
M-Ine1
Adopting Topt
r'=e' I-er with a bit of math, the weight of Term M-Inel becomes 0 and
E 1x, (F((X), y rL)s
= -x, - rop)) {(F(X), + Ex
Σ TE|K]
Topt e(f(())
K
c3
E(x,y~~
(F(2)) { x.'] (f(x)))+))
= E(x,v) [ ((F(X)))
1 p-
28


Reviews:
[{"review_id": "Zoqt9xSn3A", "reviewer": "Reviewer_Ww4B", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels and theoretically show that negative label smoothing improves the expected model confidence over the data distribution. The empirical experiments on multiple benchmark datasets demonstrate that with the presence of label noise, negative label smoothing (NLS) becomes competitively robust to label noise.\n\nHowever, to the best of my knowledge, the following points need to be addressed before it can be published:\n\n1)\tMy first concern is the originality and novelty of this work. In this work distributed soft label learning is used. To my knowledge, label distribution learning is a related research topic about soft label-based multi-label learning, which should be discussed.\n2)\tFor Figure 1, more detailed descriptions and explanations should be given.\n3)\tThe last paragraph in Section Introduction, please don't cite so many references at the same place. I suggest to delete some unimportant references and then dispersedly cite these references. \n4)\tTwo UCI datasets are used to evaluate the effectiveness of the proposed algorithm. My question is why the authors choose these datasets. To my knowledge, there are a large number of UCI datasets can be used for evaluating the effectiveness of the proposed algorithm. More datasets may be needed. At least, please explain the reason why these datasets are used in the current experiments.\n5)\tIt is more convinced if the authors can compare the proposed algorithm with some other state-of-the-art label distribution learning algorithms.\n6)\tWith regard to the comparison results, statistical tests are needed in the comparison results.\n7)\tThe presentation can be further polished. For example, some equations have punctuation at the end and some don't have.\n", "summary_of_the_paper": "This paper provides understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels and theoretically show that negative label smoothing improves the expected model confidence over the data distribution. ", "summary_of_the_review": "This paper provides understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels and theoretically show that negative label smoothing improves the expected model confidence over the data distribution. The empirical experiments on multiple benchmark datasets demonstrate that with the presence of label noise, negative label smoothing (NLS) becomes competitively robust to label noise.\n\nHowever, to the best of my knowledge, the following points need to be addressed before it can be published:\n\n1)\tMy first concern is the originality and novelty of this work. In this work distributed soft label learning is used. To my knowledge, label distribution learning is a related research topic about soft label-based multi-label learning, which should be discussed.\n2)\tFor Figure 1, more detailed descriptions and explanations should be given.\n3)\tThe last paragraph in Section Introduction, please don't cite so many references at the same place. I suggest to delete some unimportant references and then dispersedly cite these references. \n4)\tTwo UCI datasets are used to evaluate the effectiveness of the proposed algorithm. My question is why the authors choose these datasets. To my knowledge, there are a large number of UCI datasets can be used for evaluating the effectiveness of the proposed algorithm. More datasets may be needed. At least, please explain the reason why these datasets are used in the current experiments.\n5)\tIt is more convinced if the authors can compare the proposed algorithm with some other state-of-the-art label distribution learning algorithms.\n6)\tWith regard to the comparison results, statistical tests are needed in the comparison results.\n7)\tThe presentation can be further polished. For example, some equations have punctuation at the end and some don't have.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WPfs98iRAld", "reviewer": "Reviewer_nStq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors proposed a very simple yet interesting idea NLS. They theoretically shown NLS can improve the expected model confidence. When label noise rates are high, they shown NLS can be more robust to label noise compared with PLS. Then they demonstrated the theoretical findings by conducting experiments on UCI and CIFAR datasets. The paper is easy to understand and well-organized. It is very good to provide a section about how the proposed GLS connects to other robust methods. However, I have several questions/comments that need the authors to address. \n\n(a). Since the authors didn't provide any experiments to compare GLS with other loss adjusted methods of learning with noisy label such as [1,2], we have no idea about the practicality of GLS and its practical performance when learning with noisy label. It would be better if the authors could compare GLS with other methods such as [1,2]. Otherwise, its practicality is limited.\n\n(b). Although the authors have shown theoretical results, the unknown parameters (for example, $r^*$ and $e$ in Theorem 7) makes the proposed GLD less practical. That is to say, in general, we don't know how to use GLS in practice.\n\n(c). In the proof of theorem 5 (on page 28), why the inequality (see below) after \"Besides, we have ...\" holds? It is not obvious to me. \n\\begin{equation*}\n\\mathbb{E}   \\left[\\log\\left(\\frac{f^*_r(X)_Y}{1-f^*_r(X)_Y}\\right)^{r/2} - \\log\\left(f^*_r(X)_Y\\right)\\right] \\\\\n\\ge \\mathbb{E} \\left[\\log\\left(\\frac{f^*_s(X)_Y}{1-f^*_s(X)_Y}\\right)^{|r|/2} - \\log\\left(f^*_s(X)_Y\\right)\\right] \n\\end{equation*}\n\nwhere $\\mathbb{E}[\\cdot] := \\mathbb{E}_{(X, Y)\\sim \\mathcal D}[\\cdot], r:=r_N, s:=r_P$\n\n(d). Is the quantity model confidence commonly used in the literature? If so, please add a reference.\n\nReference\n\n1. Symmetric cross entropy for robust learning with noisy labels, ICCV 2019.\n\n2. Early-Learning Regularization Prevents Memorization of Noisy Labels, NeurIPS 2020.\n\n========== comments after rebuttal ============\n\nI would like to thank the authors for their response, which solves my concerns except for the proof of Theorem 5 (of the initial version). The result of original Theorem 5 is important to the contribution of this paper, which unfortunately was removed in current version. In addition, after reading other reviewers' comments, there are still some other problems. Therefore, I will keep my overall score.", "summary_of_the_paper": "This paper studies label smoothing when learning with noisy label. It proposed generalized label smoothing (GLS) containing positive label smoothing (PLS) and negative label smoothing (NLS), where NLS allows the smoothing parameter to be negative. The authors found that when the noisy rate is high, NLS is more beneficial than PLS, both theoretically and empirically.", "summary_of_the_review": "This is a theoretical understanding paper for lable smoothing when learning with noisy label. However, its practicality is limited. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "UNyqOYJG0Ag", "reviewer": "Reviewer_ENj2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The major problem with the paper is analyzing the expected loss. Label smoothing is a regularizer which has been shown to improve generalization and model calibration. However, in the infinite sample limit (minimizing the expected loss) it makes no sense to add any regularizer since directly minimizing the expected loss is optimal, e.g., In eq. 4 the authors analyze the expected loss (under the data distribution) subject to a label smoothing penalty. Why does this make sense ?\n- The authors claim that negative label smoothing is different from standard label smoothing and present it as a significant result. However, this is trivial. The label-smoothed cross entropy loss can be equivalently written in the constrained form as:\n    \n    $$\\min l(f(x),y) \\quad s.t.~ H(U, f(x)) \\leq c$$\n    \n    where U is the discrete uniform distribution and $l(.)$ is the log-likelihood. For negative label smoothing the inequality constrained is reversed. By virtue of the constraints, the minimizer of NLS objective will have greater expected margin (or model confidence as called in the paper) than standard LS which pushes the model predicted class distributions to be closer to uniform.\n    \n- The claim that negative label smoothing performs well in the high label noise regime is not surprising both from a theoretical and practical standpoint. The point of using label smoothing is to prevent the distribution learned by neural networks to become too peaked (low entropy or overconfident). But when the label noise is high, with labels flipped randomly, then the model trained without any label smoothing is not going to be overconfident (the expected loss of the model is at least the Bayes error and with label noise the Bayes error is already high).\n- The paper doesn't position itself clearly with respect to related work. Chen et al 2020 have already studied generalization performance under label smoothing where, unlike this paper, they study the generalization performance of the finite sample minimizer under label smoothing and label noise. While that paper is cited, the results of this paper are not put in context with their results.\n\n## Other comments\n\n- In Theorem 3, what does it mean for the smoothing rate $r_{CL}$ to approach infinity ? In this case the likelihood term is completely ignored.\n- High label noise is never clearly defined.", "summary_of_the_paper": "The paper considers generalized label smoothing (GLS) for binary classification tasks where the smoothing coefficient is allowed to be negative (negative label smoothing, NLS). The authors show the connection between GLS and several existing loss functions through elementary algebraic manipulations of the expected loss under (generalized) label smoothing. They also analyze GLS in the noisy label setting where the labels are randomly flipped and show that using negative label smoothing is desirable in the high label noise setting.", "summary_of_the_review": "The main problem with the paper is analyzing the effect of label smoothing on the minimizer of the expected loss of a classifier. There is no need for any regularization when minimizing the expected loss. This is a fundamental conceptual problem with the paper. Furthermore, the results are fairly elementary and do not significantly contribute towards theoretical understanding of label smoothing beyond existing literature (c.f. Chen et al 2020).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "NUlc7zkAN-5", "reviewer": "Reviewer_2zXX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe paper is very well motivated, and revealed new findings for the power of label smoothing. The paper provided empirical instructions for when a negative or a positive smoothing rate should be adopted.\n\nThe proposed generalized label smoothing scheme ties closely to several existing learning with noisy labels solutions. This new connection might reveal a deep connection between label smoothing and other regularization techniques in the literature of learning with noisy labels. This seems to provide a set of new understandings to the literature of learning with noisy labels. \n\nThe discovered phase transitioning behavior is supported both theoretically and empirically. This result quantifies the scenarios of using positive and negative label smoothing.\n\nThe presented experiment results are thorough and complete. The analysis is well presented. \n\nWeakness & question:\n\nThe discussion in Section 4 on the bias-variance tradeoff seems to disconnect a bit with Section 5. Could the authors clarify its use?\n\nThe characterization of the phase transitioning behavior seems to depend on the knowledge of r^*, the optimal smooth rate when data is clean. This knowledge is important to inform when one should adopt a negative smoothing rate and when a positive one. What will happen when only an imperfect r^* is known? \n\n", "summary_of_the_paper": "Label smoothing shows a promising bias-variance tradeoff when training deep neural networks, while it is unclear if label smoothing remains helpful when training labels are noisy. A recent work shows the benefits of using label smoothing. However, the current submission finds contradicting and appealing observations. In Figure 1, the authors presented a convincing set of results that clearly show a phase-transitioning behavior of using label smoothing: while label smoothing seems to work with a low label noise rate, the benefits disappear when the rate goes up high.\n\nThe above observation can imply that label smoothing is considered unfavorable with a high label noise rate. The paper then presented a generalized label smoothing scheme where the smoothing constant can instead go negative. Even though using a negative parameter sounds weird, the paper showed that a label smoothing with a negative (but different) smoothing parameter corresponds to several existing solutions to learning with noisy labels! This signals that the negative label smoothing scheme might turn out to be helpful when dealing with high label noise rate. I think this is a pretty interesting observation and connection to uncover.\n\nThe two cases are integrated as the generalized label smoothing, where the smoothing constant can be either negative to positive. In section 5, it is shown under what noise rate, a positive or a negative label smoothing is considered a better option. This characterization reveals a clear phase transitioning behavior corresponding to figure 1.", "summary_of_the_review": "I tend to accept this paper. It does explain some interesting things, and the experiment is also sufficient.\n\n========== post rebuttal ============\n\nThe rebuttal of this article solves my concerns to some extent, but as other reviewers pointed out, some problems still exist, so I changed my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies generalized label smoothing (GLS), which unifies positive label smoothing (PLS) and negative label smoothing (NLS), and studies its connections to existing loss functions. It also shows the benefit of NLS in the high noise regime.

Although the reviewers acknowledge that the idea of NLS in this paper is interesting, they also expressed the concerns that: the practicality of GLS is not thoroughly evaluated against prior works; the empirically best setting of parameter r is only verified in a limited number of datasets; the theoretical results' difference with prior works is limited. We encourage the authors to take the reviewers' feedback to strengthen the paper in the next iteration.
Decision: Reject