UNDERSTANDING GENERALIZED LABEL SMOOTHING
Year: 2021

Abstract:
Label smoothing (LS) is an arising learning paradigm that uses the positively weighted average of both the hard training labels and uniformly distributed soft labels. It was shown that LS serves as a regularizer for training data with hard labels and therefore improves the generalization of the model. Later it was reported LS even helps with improving robustness when learning with noisy labels. However, we observe that the advantage of LS vanishes when we operate in a high label noise regime. Puzzled by the observation, we proceeded to discover that several proposed learning-with-noisy-labels solutions in the literature instead relate more closely to negative label smoothing (NLS), which defines as using a negative weight to combine the hard and soft labels! We show that NLS differs substantially from LS in their achieved model confidence. To differentiate the two cases, we will call LS the positive label smoothing (PLS), and this paper unifies PLS and NLS into generalized label smoothing (GLS). We provide understandings for the properties of GLS when learning with noisy labels. Among other established properties, we theoretically show NLS is considered more beneficial when the label noise rates are high. We provide extensive experimental results on multiple benchmarks to support our findings too.

heading: 1 INTRODUCTION
text: Label smoothing (LS) (Szegedy et al., 2016) is an arising learning paradigm that uses positively weighted average of both the hard training labels and uniformly distributed soft label:
yLS,r = (1− r) · y + r K · 1 (1)
where we denote the one-hot vector form of hard label and an all one vector as y, 1 respectively. K is the number of label classes, and r is the smooth rate in the range of [0, 1]. It was shown that LS serves as a regularizer for the hard training data and therefore improves generalization of the model. The regularizer role of LS prevents the model from fitting overly on the target class. Empirical studies have demonstrated the effectiveness of LS in improving the model performance across various benchmarks (Pereyra et al., 2017) (such as image classification (Szegedy et al., 2016), machine translation (Vaswani et al., 2017), language modelling (Chorowski & Jaitly, 2017)) and model calibration (Müller et al., 2019).
Later it was reported LS even helps with improving robustness when learning with noisy labels (Lukasik et al., 2020). However, we observe that the advantage of LS vanishes when we operate in a high label noise regime. In Figure 1, we present a set of experiments on UCI datasets (Dua & Graff, 2017). We highlight best two smooth rates (possible to have tied smooth rates) under each label noise rate. Indeed, non-negative smooth rates (circles colored in red) outperform negative ones when the label noise rates are low. Nonetheless, with the increasing of noise rates, negative smooth rates r < 0 (Eqn. (1), diamonds colored in green) appear to be more competitive when learning with noisy labels. Puzzled by the observation, we proceeded to discover
that several proposed learning-with-noisy-labels solutions in the literature, including Loss Correction (Patrini et al., 2017), NLNL (Kim et al., 2019) and Peer Loss (Liu & Guo, 2020), instead relate more closely to negative label smoothing (NLS), which defines as using a negative weight to combine the hard and soft labels!
This paper unifies label smoothing with either positive or negative smooth rate into a generalized label smoothing (GLS) framework. Our paper is motivated by the above inconsistent observations. And we aim to provide a more thorough understanding of GLS under the setting of learning with noisy labels, rather than proposing a new method to compete for state-of-the-art performances. We firstly show that negative label smoothing differs substantially from positive label smoothing in their achieved model confidence. With the presence of label noise, we then proceed to show that there exists a phase transition behavior when finding the optimal label smoothing rate for GLS. Particularly, when label noise is low, positive label smoothing is able to uncover the optimal model while negative label smoothing is considered more beneficial in a high label noise regime.
We provide extensive experimental evidences to support our findings. For instance, on multiple benchmark datasets, we present the clear transition of the optimal smoothing rate going from positive to negative when we keep increasing noise rates. On CIFAR-10 test data, we show a negative smoothing rate elicits higher model confidence on correct predictions and lower confidence on wrong predictions compared with the behavior of a positive one.
Our contributions summarize as follows:
• We provide understandings for a generalized notion of label smoothing (GLS) when learning with noisy labels, where the label smooth rate can go negative. (An understanding paper rather than a method paper)
• We show that several robust loss functions in the noise learning literature correspond to learning with GLS, under certain noise rate models. (Section 3)
• We show that negative label smoothing improves the confidence of model prediction. With the presence of label noise, we demonstrate learning with a negative smooth rate can be more robust to label noise compared with a positive rate when label noise rates are high. (Section 4, 5)
• Empirical experiments on multiple datasets and methods demonstrate that with the presence of label noise, NLS becomes competitively robust to label noise. We also empirically show how GLS results in trade-offs in model confidence, bias and variance of the generalization error. (Section 6, Appendix D and E)
• Extensive experiment results validate our main theoretical conclusions. Besides, we also discuss practical considerations and multi-class extensions of GLS to mitigate the impact of label noise. (Appendix B, C)
We defer all proofs to Appendix F. Our work primarily contributes to the literature of learning with noisy labels (Scott et al., 2013; Natarajan et al., 2013; Liu & Tao, 2015; Patrini et al., 2017; Liu & Guo, 2020). Our core results are contingent on recent works of understanding the effect of label smoothing when training deep neural network models, i.e., label smoothing improves model calibration (Müller et al., 2019), more complexed forms of label smoothing (Li et al., 2020; Yuan et al., 2020), and in particular when label noise
presents (Lukasik et al., 2020; Liu, 2021). We generalize a concept called generalized label smoothing and provide new understandings for when, instead of setting a positive smoothing rate as the literature would normally do, a negative smoothing rate is considered a better option. Due to the space limit, we defer a more detailed discussion of related works to Appendix A.

heading: 2 PRELIMINARIES

heading: 2.1 LEARNING WITH SMOOTHED LABELS
text: For aK-class classification task, we denote byX ∈ X a high-dimensional feature and Y ∈ Y := {1, 2, ...,K} the corresponding label. Suppose (X,Y ) ∈ X × Y are drawn from a joint distribution D. Let yi be the one-hot encoded vector form of yi which generates according to Y . The random variable of smoothed label Y LS,r with smooth rate r ∈ [0, 1] generates yLS,ri as (Szegedy et al., 2016):
yLS,ri = (1− r) · yi + r
K · 1
For example, when r = 0.3, the smoothed label of yi = [0, 1, 0] becomes y LS,r=0.3 i = [0.1, 0.8, 0.1].
We consider a broader setting where the smoothed label might be negatively related to the corresponding feature. As a supplementary to existed works on label smoothing, we explore the benefits of learning with generalized label smoothing (GLS), i.e., r ∈ (−∞, 1] instead of a non-negative r.
yGLS,ri := (1− r) · yi + r
K · 1 (2)
where yGLS,ri is given by the random variable of generalized smooth label Y GLS,r. We name the scenario r < 0 as negative label smoothing (NLS). To clarify, we don’t assume a strict lower bound for r. If r → −∞, normalizing yGLS,ri by 1− r returns y GLS,r i = yi − 1 K . We will show when imposing a negative smoothing parameter will be considered beneficial as compared to a positive one. In the main paper, we focus on the binary classification task where yi ∈ {0, 1} and K = 2. And we defer multi-class extensions to Appendix C. Denote f as a deep neural network, f(xi) is the model prediction of xi ∈ X with element f(xi)yi := P(Y = yi|X = xi, f). Given the sample x ∈ X and a hard label y ∈ Y , binary CE loss is defined as `CE(f(x), y) := − log(f(x)y). Throughout this paper, we shorthand ` as `CE for a clean presentation.

heading: 2.2 LEARNING WITH NOISY LABELS
text: The noisy label literature considers the setting where we only have access to samples with noisy labels from (X, Ỹ ). Suppose random variables (X, Ỹ ) ∈ X × Ỹ are drawn from a noisy joint distribution D̃. Statistically, the random variable of noisy labels Ỹ can be characterized by a noise transition matrix T , where each element Ti,j represents the probability of flipping the clean label Y = i to the noisy label Ỹ = j, i.e, Tij = P(Ỹ = j|Y = i). In this paper, we are interested in the widely studied class-dependent label noise. We assume the label noise is conditionally independent of features, i.e.,
P(Ỹ = j|Y = i) = P(Ỹ = j|X,Y = i),∀i, j ∈ [K]
For the binary classification setting, define e0 := P(Ỹ = 1|Y = 0), e1 := P(Ỹ = 0|Y = 1). Without loss of generality, we assume e1 − e0 = e∆ ≥ 0. We denote the binary noise transition matrix in the noisy label setting as: T = (
1− e0 e0 e1 1− e1
) .

heading: 2.3 MODEL CONFIDENCE
text: We define a key quantity, model confidence, that plays an important role in later sections.
Definition 1. Model confidence of model f for sample (x, y). Given a model f , a sample x with its target label y ∈ {0, 1}, the model confidence of f w.r.t. sample x is defined as MC(f ;x, y) = f(x)y − f(x)1−y. MC(f ;x, y) in definition 1 characterizes the difference of the predicted probability between target class and the other class. MC(f ;x, y) = 0 simply means f has no confident on its predictions since the model can not identify the target class of x. MC(f ;x, y) returns a negative value when f gives a wrong prediction and is not confident to predict the label of x as the target label y. To dig into how GLS influences the model confidence on correct and wrong predictions in following sections, we separate the distribution D into
D+f := {(X,Y ) ∼ D : MC(f ;X,Y ) > 0}, D − f := {(X,Y ) ∼ D : MC(f ;X,Y ) ≤ 0}

heading: 3 CONNECTION TO OTHER ROBUST METHODS
text: In this section, we aim to theoretically explore the connection between GLS and popular methods such as backward/forward loss correction (Natarajan et al., 2013; Patrini et al., 2017), NLNL (Kim et al., 2019) and peer loss (Liu & Guo, 2020). We defer the corresponding empirical validations to Appendix B.
For r ≤ 1, let ỹ be the vector form of noisy label ỹ obtained from Ỹ , we define the r smoothed label of ỹ as ỹGLS,r, where ỹGLS,r := (1− r) · ỹ + (r/K) · 1 and is generated by the random variable Ỹ GLS,r. Risk minimization of the Generalized Label Smoothing (GLS) w.r.t. noisy labels becomes:
Risk Minimization Using GLS: minE(X,Ỹ )∼D̃ [ `(f(X), Ỹ GLS,r) ] (3)
The GLS framework covers three special methods: PLS (r ∈ (0, 1]), Vanilla (CE) Loss (r = 0) and NLS (r < 0). Besides, we observe that NLS connects to a special case of label smoothing regularization. We highlight this in Theorem 1.
Theorem 1. ∀r ∈ [0, 1], NLS with smooth rate −r is a special form of label smoothing regularization: minE(X,Ỹ )∼D̃ [ `(f(X), Ỹ GLS,−r) ] = minE(X,Ỹ )∼D̃ [ 2 · `(f(X), Ỹ )− `(f(X), Ỹ GLS,r) ]

heading: 3.1 LOSS CORRECTION
text: Loss correction (Patrini et al., 2017) studies two robust loss designs which are based on the knowledge of nonsingular noise transition matrix T . The backward correction `←(f(X), Ỹ ) re-weights the loss `(f(X), Ỹ ) by T−1 Ŷ ,Ỹ
with Ŷ being the model predicted label, while the proposed forward correction `→(f(X), Ỹ ) multiplies the model predictions by T .
Proposition 1. For rLC := 2e02e0−1 < 0, λLC := e∆ · 1 1−2e0 , risk minimization of both backward and forward correction (with the knowledge of noise rates) are equivalent to the combination of NLS and an extra bias term Bias-LC
minE(X,Ỹ )∼D̃ [ `←(f(X), Ỹ ) ] = minE(X,Ỹ )∼D̃ [ `→(f(X), Ỹ ) ] =minE(X,Ỹ )∼D̃ [ `(f(X), Ỹ GLS,rLC) ] + λLC · EX,Y=1 [ `(f(X), 1)− `(f(X), 0) ] ︸ ︷︷ ︸
Bias-LC
The incurred Bias-LC controls the model confidence on (X,Y = 1) ∼ Df . Note that when the noise rate is not substantially high, i.e, e0 ∈ [0, 12 ), λLC > 0. Then, compared with loss correction, NLS with smooth rate rLC makes the model f to be less confident on (X,Y = 1) ∼ D+f and more confident on (X,Y = 1) ∼ D − f (wrong predictions). However, the impact of term Bias-LC is diminishing when either e∆ → 0 (symmetric noise rates) or e0 → 0 (low noise rates) as specified in Theorem 2. Theorem 2. Assume the noise transition matrix is symmetric, i.e, e∆ = 0, backward and forward loss correction are a special form of NLS with smooth rate rLC.

heading: 3.2 LEARNING FROM COMPLEMENTARY LABELS
text: Complementary label (Ishida et al., 2017) was firstly introduced to mitigate the cost of collecting data. Rather than encouraging the model to fit directly on the target, learning from complementary labels trains the model to not fit on the complementary label which differs from the target. Later, an indirect training method “Negative Learnin” (NL) (Kim et al., 2019) was proposed to reduce the risk of providing incorrect information with the presence of noisy labels and is robust to label noise in multi-class classification tasks. A more generic unbiased risk estimator of learning with complementary labels was proposed (Ishida et al., 2019) and is defined as: `CL(f(X), Ỹ ) := `(f(X), Ỹ )− `(f(X), 1− Ỹ ). Theorem 3. Learning from complementary labels with `CL is equivalent to NLS with smooth rate rCL → −∞:
minE(X,Ỹ )∼D̃ [ `CL(f(X), Ỹ ) ] ⇐⇒ min E(X,Ỹ )∼D̃[`(f(X), Ỹ GLS,rCL→−∞)]

heading: 3.3 PEER LOSS FUNCTIONS
text: Peer loss functions (Liu & Guo, 2020) propose a family of robust loss measures which do not require the knowledge of noise rates. The mathematical representation of peer loss functions is `PL(f(X), Ỹ ) := `(f(X), Ỹ )− `(f(X1), Ỹ2), where (Xi, Ỹi) ∼ D̃. The second term of peer loss evaluates on randomly paired data samples and labels to punish f from overly fitting on noisy labels.
Proposition 2. For rPL := 2 · P(Ỹ = 1), λPL := 1 − rPL, risk minimization of peer loss is equivalent to negative label smoothing regularization with an extra term Bias-PL, i.e.,
minE(X,Ỹ )∼D̃ [ `PL(f(X), Ỹ ) ] = min E(X,Ỹ )∼D̃ [ `(f(X), Ỹ )− `(f(X), Ỹ GLS,rPL) ] +λPL · EX,Ỹ=1 [ `(f(X), 1)− `(f(X), 0) ] ︸ ︷︷ ︸
Bias-PL The incurred term Bias-PL controls the model confidence on (X, Ỹ = 1) ∼ D̃ and has a diminishing effect as P(Ỹ = 1)→ 1/2. Generally, peer loss relates to GLS as the negatively weighted GLS term appears to be a regularizer. Note that we have access to the P(Ỹ = 1), we can bridge the gap between GLS and peer loss by adding an estimable term Bias-PL. With some derivations, we further show in Theorem 4, when noisy priors are equal, peer loss has an exact GLS form.
Theorem 4. When the noisy labels have equal prior, i.e, P(Ỹ = 0) = P(Ỹ = 1), peer loss is a special form of NLS regularization with smooth rate rPL. Besides,
minE(X,Ỹ )∼D̃ [ `PL(f(X), Ỹ ) ] ⇐⇒ min E(X,Ỹ )∼D̃ [ `(f(X), Ỹ GLS,r→−∞) ]

heading: 4 GLS AND MODEL CONFIDENCE
text: Now we show that NLS differs substantially from PLS in their achieved model confidence. This discussion sets the foundation for our discussion when learning with noisy labels in next section.

heading: 4.1 GLS ON CLEAN DATA
text: We firstly show that NLS differs substantially from PLS in their achieved model confidence. This discussion sets the foundation for our discussion when learning with noisy labels in proceeding subsections.
When the label is clean, i.e, e0 = e1 = 0, Eqn. (3) reduces to: minE(X,Y )∼D [ `(f(X), Y ) ] + r
2 · E(X,Y )∼D
[ `(f(X), 1− Y )− `(f(X), Y ) ] ︸ ︷︷ ︸
Term MC`(f ;X,Y )
(4)
To clarify, we are not restricting D to have infinite samples, i.e., for the discrete distribution D = {xi, yi}Ni=1, Eqn. (4) becomes: min [ 1 N ∑ i∈[N ] `(f(xi), yi) ] + r2N [∑ i∈[N ] ( `(f(xi), 1− yi)− `(f(xi), yi) )] . Note:
MC`(f ;X,Y ) = log ( f(X)Y / ( 1− f(X)Y )) ; MC(f ;X,Y ) = 2 · f(X)Y − 1
Both log ( x/(1 − x) ) and 2x − 1 are monotonically increasing for x ∈ (0, 1), model f with a high
MC`(f ;X,Y ) has high MC(f ;X,Y ). The difference between PLS and NLS lie in the weight of Term MC`(f ;X,Y ): NLS encourages high MC`(f ;X,Y ) and MC(f ;X,Y ) while PLS has an opposite effect.

heading: 4.2 INSIGHTS FROM AN EMPIRICAL OBSERVATION
text: We adopt the generation of 2D (binary) synthetic dataset from (Amid et al., 2019) by randomly sampling two circularly distributed classes. The inner annulus indicates one class (blue), while the outer annulus denotes the other class (red). We hold 20% data samples for performance comparison.
Side-effects of over-confident In Figure 2, the colored bands depict the different levels of prediction probabilities: light blue + orange bands indicate samples that satisfy MC < 0.4 (low model confidence). When learning with clean data, GLS with non-positive smooth rate may yield over-confidence on the model prediction and a relatively low test accuracy.
Label noise reduces model confidence Recent works (Liu, 2021; Cheng et al., 2020) have demonstrated that with the presence of label noise, learning with noisy labels directly will eventually result in unconfident model predictions. Continuing the synthetic 2D dataset, we flip the clean labels according to a symmetric noise transition matrix with noise rate ei for both classes. With the presence of label noise in Figure 3, GLS generally
becomes less confident on its predictions. Besides, when the smooth rate increases from negative to positive, more samples are of uncertain predictions. Thus, a smaller/negative smooth rate is beneficial when the noise rate increases by encouraging more confident predictions.

heading: 5 GLS WITH LABEL NOISE
text: In this section, we target at the optimal candidates of r in GLS when the label noise presents. In Section 4, we have shown that NLS and PLS have the opposite functionality on the model confidence when training on clean data. Given the unseen test data, learning with non-negative smooth rates may not always return the best outcome (Figure 3). Based on this observation, we delve into details to show when NLS is more favorable than PLS and Vanilla Loss. We start with stating Assumption 1: Assumption 1. We assume learning with clean data distribution D with smooth rate r∗ ≤ 1 in GLS returns the best performance on the unseen clean test data distribution Dtest.
Assumption 1 simply offers an “anchor” point to initiate our analysis for the noisy label setting. To clarify, we don’t rule out the possibility that other methods outperform GLS with optimal smooth rate r∗. Later in Section 6.1 and Appendix D, we will empirically test what r∗ usually is on various benchmarks. We define optimal classifier on Y ∗ which follows r∗ smooth label distribution as: f∗D := argminf E(X,Y )∼D [ `(f(X), Y ∗) ] . With the introduction of r∗ and f∗D, our goal is then to recover the classifier f using the noisy training labels. To bridge learning with noisy labels and clean labels for GLS, we define λ1, λ2 and offer Theorem 5.
λ1 := [ (e0 − r∗
2 ) + (1− 2e0) ·
r
2
] , λ2 := e∆ · (1− r)
Theorem 5. The risk minimization of GLS (Eqn. (3)) in the noisy setting relates to the risk defined on the clean data with two additional bias terms:
min E(X,Y )∼D [ `(f(X), Y ∗) ] ︸ ︷︷ ︸
True Risk
+λ1 · E(X,Y )∼D [ `(f(X), 1− Y )− `(f(X), Y ) ] ︸ ︷︷ ︸
M-Inc1 +λ2 · EX,Y=1 [ `(f(X), 0)− `(f(X), 1) ] ︸ ︷︷ ︸
M-Inc2
(5)
The True Risk is the risk minimization w.r.t. clean optimal label distribution Y ∗. Training GLS on noisy labels results in two extra bias terms which affect the model confidence. We defer an empirical validation of Theorem 5 to Appendix B. Now we proceed to answer “what parameters are preferred in the noisy setting”.
5.1 SYMMETRIC ERROR RATES WITH e∆ = 0
Symmetric error rates e := e0 = e1 indicates the probability of flipping to the other class is equal for both classes. In this case, λ2 = 0 and Term M-Inc2 is cancelled and Eqn. (5) reduces to
min E(X,Y )∼D [ `(f(X), Y ∗) ] ︸ ︷︷ ︸
True Risk
+λ1 · E(X,Y )∼D [ `(f(X), 1− Y )− `(f(X), Y ) ] ︸ ︷︷ ︸
M-Inc1
(6)
Noisy labels impairs model confidence on Vanilla Loss In the GLS framework, define the optimal r that will cancel the impact of Term M-Inc1 as:
when ropt := r∗ − 2e 1− 2e , M-Inc1 = 0 (7)
The threshold ropt in Eqn. 7 implies: Theorem 6. With Assumption 1, GLS with smooth rate r = ropt yields f∗D. • When error rate e < r∗/2, r = ropt > 0 (PLS);
• When error rate e = r∗/2, r = 0 (Vanilla Loss); • When error rate e > r∗/2, r = ropt < 0 (NLS).
In Theorem 6, adopting NLS when noise rate e < r∗/2 induces λ1 < 0, Term M-Inc1 makes f overlyconfident on its predictions compared with Y ∗. In Figure 4, with the decreasing of r∗, PLS is less tolerant of labels with high noise. Similarly, if e ≥ r ∗
2 , with the decreasing of r ∗, NLS is more robust in the high noise
regime while PLS makes the model f become less-confident on its predictions. Clearly, NLS outperforms PLS especially when noise rates are large and r∗ is small.
5.2 ASYMMETRIC ERROR RATES WITH e∆ 6= 0
In this case, adopting r = r ∗−2e0 1−2e0 removes the Term M-Inc1. However, when r < 1, Term M-Inc2 is not negligible due to assymetric noise transition matrix. As a result, Term M-Inc2 becomes:
e∆ · 1− r∗
1− 2e0 · EX,Y=1
[ `(f(X), 0)− `(f(X), 1) ] , with e∆ · 1− r∗
1− 2e0 ≥ 0
Term M-Inc2 in the minimization increases the model confidence on (X,Y = 0) ∼ D+f . The model will then become overly-confident with the class that has a low noise rate e0. Meanwhile, Term M-Inc2 decreases the model confidence on (X,Y = 1) ∼ D+f (less-confident to the class with a high noise rate e1).

heading: 6 EXPERIMENT RESULTS
text: In this section, we present our empirical observations regarding the role of GLS under clean and noisy labels by using UCI datasets, CIFAR-10 and CIFAR-100.
6.1 WHAT IS THE PRACTICAL DISTRIBUTION OF r∗ AND rOPT? r∗ and ropt on UCI datasets (Dua & Graff, 2017) As for UCI datasets, we pick Twonorm and Splice for illustration in the main paper. The noisy labels are generated by a symmetric noise transition matrix with noise rate ei = [0.1, 0.2, 0.3, 0.4]. As highlighted in Table 1, ropt appears with positive values when the data is clean (same as r∗) or of a low noise rate. With the increasing of noise rates, the performance of PLS results in a much larger degradation compared with NLS. We color-code different noise regimes where either PLS (red-ish) or NLS (green-ish) outperforms the other. Clearly there is a separation of the favored smoothing rate for different noise scenarios (upper left & low noise for PLS, bottom right & high noise for NLS).
r∗ and ropt on CIFAR datasets (Krizhevsky et al., 2009) When learning with a larger scale and more complex dataset, like CIFAR-10 and CIFAR-100, models are prone to converge on a local optimal solution
rather than the global optimum. This phenomenon occurs frequently in NLS which ends up with performance degradation. Thus, in Table 2, when learning with noisy labels, we report the better performance of GLS between direct training and loading the same warm-up model. And we observe that the performance of NLS is more competitive than PLS when learning with clean data. Clearly, NLS outperforms PLS in CIFAR-10 and CIFAR-100 under various synthetic noise settings. The gap is larger when the noise rates are high.

heading: 6.2 COMPARISONS BETWEEN GLS AND MORE ROBUST METHODS
text: In Table 3, we compare GLS with several robust methods in synthetic noisy CIFAR datasets. We emphasize that we are not proposing a new method to compete with state-of-the-art methods. Instead, we hope to help readers understand how the generalized label smoothing fare when label noise presents. Clearly, GLS (especially NLS) can definitely be viewed as a competitive and efficient robust loss function which outperforms Cross Entropy, Bootsrap (Reed et al., 2014), SCE (Wang et al., 2019), APL (Ma et al., 2020) and Forward correction (Patrini et al., 2017) in most settings.
Additional results More extensive empirical results are included in Appendix: empirical validation of main theorems (Appendix B); practical considerations of GLS (Appendix C); experiment results on additional datasets (Appendix D); bias and variance of the generalization error on the clean data (Appendix E).
Conclusion In this paper, we provide understandings for a generalized notion of label smoothing where the label smoothing rate can go negative. We show that learning with negatively smoothed labels explicitly improves the confidence of model prediction. This key property acts as a significant role when the confidence of model prediction drops. We make connections between negative label smoothing and existing learning with noisy label solutions. In contrast to existing works that promote the use of positive label smoothing, we show both theoretically and empirically the advantage of a negative smooth rate when the label noise rate increases. Our observations provide new understanding for the effects of label smoothing, especially when the training labels are imperfect.

heading: APPENDIX
text: The Appendix is organized as follows.
• Section A presents the full version of related works. • Section B includes empirical validations of theoretical conclusions in Section 3 and Theorem 5. • Section C discusses practical considerations and multi-class extensions of GLS. • Section D shows additional experiments on synthetic dataset and UCI datasets. • Section E illustrates the bias and variance trade-off when learning with GLS from clean data. • Section F includes omitted proofs for theoretical conclusions in the main paper.

heading: A FULL VERSION OF RELATED WORKS
text: Our work supplements to two lines of related works.
Learning with noisy labels Annotated labels from human labelers usually consists of an non-negligible amount of mis-labeled data samples. Making deep neural nets perform robust training on “noisily" labeled datasets remains a challenge. Classical approaches of learning with noisy labels assume the noisy labels are independent to features. They firstly estimate the noise transition matrix (Liu & Tao, 2015; Menon et al., 2015; Harish et al., 2016; Patrini et al., 2017), then proceed with a loss correction (Natarajan et al., 2013; Patrini et al., 2017; Liu & Tao, 2015) to mitigate label noise. Recent works propose robust loss functions (Xu et al., 2019; Kim et al., 2019; Liu & Guo, 2020; Wei & Liu, 2020) to train deep neural nets directly without the knowledge of noise rates, or design a pipeline which dynamically select and train on “clean" samples with small loss (Jiang et al., 2018; Han et al., 2018; Yu et al., 2019; Yao et al., 2020a). More recently, several approaches target at addressing more challenging noise settings, such as instance-dependent label noise (Cheng et al., 2020; Berthon et al., 2021).
Understanding the effect of label smoothing Learning with one-hot labels is prone to over-fitting, soft label learning then naturally draws attentions of machine learning researchers. Successful applications of soft label learning include the label distribution learning (Geng, 2016) which provides an instance with description degrees of all the labels. Label smoothing (LS) (Szegedy et al., 2016) is another arising learning paradigm that uses positively weighted average of both the hard training labels and uniformly distributed soft labels. Empirical studies have demonstrated the effectiveness of LS in improving the model performance (Pereyra et al., 2017; Szegedy et al., 2016; Vaswani et al., 2017; Chorowski & Jaitly, 2017) and model calibration (Müller et al., 2019). However, knowledge distilling a teacher network (trained on smoothed labels) into a student network is much less effective (Müller et al., 2019). Later, generalization effects of more advanced forms of label smoothing was studied, such as structural label smoothing (Li et al., 2020) and non-uniform label smoothing (Chen et al., 2020). More recently, it was shown that an appropriate label smoothing regularizer with reduced label variance boosts the convergence (Xu et al., 2020). When label noise presents, (Liu, 2021) gives theoretical justifications for the memorizing effects of label smoothing. And the effectiveness of label smoothing in mitigating label noise is investigated in (Lukasik et al., 2020).

heading: B EMPIRICAL VALIDATIONS OF MAIN THEOREMS
text: In this section, we empirically validate our main theoretical conclusions, i.e, the connection between GLS and popular methods, and the comparison between clean empirical risks and noisy ones.

heading: B.1 CONNECTION TO OTHER ROBUST METHODS
text: We firstly compare GLS with backward correction (Natarajan et al., 2013), forward correction (Patrini et al., 2017) and peer loss (Liu & Guo, 2020) on CIFAR-10 dataset. To approximate the performance of backward/forward Loss Correction, we adopt GLS with smooth rate ( −1) . As for the approximation of peer loss, we choose `(f(X), Ỹ )− `(f(X), Ỹ GLS,r=0.5) which is equivalent to GLS when r → −∞. Experiment results in Table 4 on CIFAR-10 under symmetric noise settings demonstrate that the equivalent forms of GLS are robust to label noise.
Explanation of the performance gap In practice, we adopt the same hyper-parameter setting as used for all other smooth rates for GLS form. Loss corrections will firstly warm-up with the cross-entropy loss, estimate the noise transition matrix with this pre-trained model, and then proceed to train with the backward/forward corrected loss. Peer loss functions adopt a dynamical adjustment for learning rate. The warming up, estimation error of noise transition matrix as well as the special hyper-parameter settings explain performance gaps.

heading: B.2 CLEAN EMPIRICAL RISK V.S. NOISY EMPIRICAL RISK
text: Now we empirically verify Theorem 8, which relates the risk of GLS in the noisy setting to the clean ones. Assume the the noise label is generated through the symmetric noise transition matrix. We restate the connection between two risks as below:
E(X,Ỹ )∼D̃ [ `(f(X), Ỹ GLS,r) ] ︸ ︷︷ ︸
Noisy Risk
=E(X,Y )∼D [ `(f(X), Y ∗) ] ︸ ︷︷ ︸
(Clean) True Risk
+λ1 · E(X,Y )∼D [ `(f(X), 1− Y )− `(f(X), Y ) ] ︸ ︷︷ ︸
Gap of Risk
In practice, obtaining the exact value of r∗ is not trivial. Finding an accurate r for GLS to make λ1 be exactly 0 is pretty hard. Note that when M-Inc1 is equipped with the Cross-entropy loss and NLS encourages confident predictions, one mistake could make the value of M-Inc1 has a huge difference. To be more specific, suppose the label of an image is a “cat”, if the NLS-trained model makes a mistake on this image, it is very likely to predict this image “cat” with a probability that p → 0 since NLS may push the model become overly-confident. Then log(p) can be extremely small. Thus, the empirical risk (evaluated on the clean data) for NLS will change much more significantly than PLS ones.
We use a UCI dataset (Waveform) for illustration. The value of r∗ is approximately 0, and the estimated (clean) true risk in above is 0.1798. When the noise rates are 0.1, 0.2, 0.3, 0.4, the optimal smooth rate should be −0.25,−0.67,−1.5,−4 according to Eqn. (7). The estimated noisy risk of GLS on these noise settings can be summarized in Table 1. Clearly, when e = 0.1, r = −0.25 is closest to the estimated clean true risk (also returns the best test accuracy among these smooth rates). Similarly observations reach to all other e. Although the risks evaluated on noisy settings with the corresponding ropt can not be exactly the same as the true risk (evaluates on the clean data with r = 0.0), ropt already reaches the smallest one among these smooth rates and the non-zero Gap of Risk is best explained by the precision of r∗ as well as the estimation error of risks.

heading: C PRACTICAL CONSIDERATION OF GLS
text: In the main paper, we theoretically show when we should adopt NLS and PLS. In this section, we discuss more practical considerations of GLS, including the optimal smoothing parameter, how to reduce the impacts of bias terms, and multi-class extensions.

heading: C.1 THE OPTIMAL SMOOTHING PARAMETER
text: In practice, we don’t have access to noise rates ei. Our work does not intend to particularly focus on the noise rate estimation. For readers interested in the noise rate estimation, please refer to (Liu & Tao, 2015; Menon et al., 2015; Harish et al., 2016; Patrini et al., 2017; Yao et al., 2020b; Zhu et al., 2021). To estimate ropt = r∗−2e 1−2e , one can simply assume r
∗ → 0. And the noise rate e is estimable by a large family of noise estimation methods mentioned above. Our practical observations show that NLS with a CE warm-up is not sensitive to the negative smooth rate, for example, on CIFAR-10 and CIFAR-100 synthetic noisy datasets, r < −1.0 frequently achieves best results (see Table 2 in the main paper). Our current contribution focuses on understanding the generalized label smoothing, and we prefer leaving the task of identifying the optimal smooth rate to future works.

heading: C.2 MAKING GLS MORE ROBUST TO LABEL NOISE
text: There is a line of related works targeting at distinguishing clean labels from the noisy labels. Current literature in selecting clean samples from noisily labeled dataset is based on the empirical evidence that samples with noisy/wrong labels have a larger loss than clean ones. For interested readers, please refer to (Han et al., 2018; Jiang et al., 2018; Yu et al., 2019; Yao et al., 2020a; Wei et al., 2020; Northcutt et al., 2021). Compared with the risk minimization over the clean data distribution (X,Y ) ∼ D, learning directly with GLS on the noisy distribution (X, Ỹ ) ∼ D̃ will result in an extra term (e1−e0) ·(1−r) ·E(X,Y=1)∼D[`(f(X), 0)−`(f(X), 1)] compared to the clean scenario. Empirically, we can estimate the bias term, perform a bias correction by subtracting the estimated bias term from the objective function in Eqn. (3).
Suppose we have access to a clean distribution Dclean which consists of selected clean samples. Denote the estimated noise rates as êi, when e∆ 6= 0, in order to make GLS be more robust to label noise and fit on the optimal distribution Y ∗, we improve GLS by performing a model confidence correction on the dominating class through:
GLS-C: min E(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,r )] −(ê1 − ê0) · (1− r) · E(X,Y=1)∼Dclean [ ` ( f(X), 0 ) − ` ( f(X), 1 )] ︸ ︷︷ ︸
confidence correction

heading: C.3 MULTI-CLASS EXTENSION
text: As an extension to the binary classification task, we next show how GLS extends to the multi-class setting under two broad families of noise transition model. When learning with multi-class classification tasks, we assume Assumption 1 holds in the multi-class setting. And for Y, Ỹ ∈ [K], we extend the definition of model confidence to multi-class classification tasks as:
Definition 2. Model confidence of sample x (K-class classification). Given a model f , a sample x with its target label y ∈ [K], the model confidence score of f w.r.t. sample x is defined as
MC(f ;x, y) = f(x)y − 1 K − 1 ∑ i6=y f(x)i
Sparse noise transition matrix Sparse noise model (Wei & Liu, 2020) assumes K is an even number. For c ∈ [K/2], ic < jc, sparse noise model specifies K/2 disjoint pairs of classes (ic, jc) to simulate the scenario where particular pairs of classes are ambiguity and misleading for human annotators. The off-diagonal element of T reads Tic,jc = e0, Tjc,ic = e1. Suppose e0 + e1 < 1, the diagonal entries become Tic,ic = 1− e1, Tjc,jc = 1− e0. Clearly, our conclusions in Section 5 extends directly to the sparse noise transition matrix by simply splitting the K-class classification task into K2 disjoint binary ones.
Symmetric noise transition matrix Symmetric noise model (Kim et al., 2019) is a widely accepted synthetic noise model in the literature of learning with noisy labels. The symmetric noise model generates the noisy labels by randomly flipping the clean label to the other possible classes with probability . ∀i 6= j, Ti,j = /(K− 1), and the diagonal entry Ti,i = 1− . Define the optimal r for GLS in the multi-class setting as ropt := (K−1)·r∗−K· (K−1)−K· , Theorem 6 can be extended to the multi-class setting as:
Theorem 7. Under Assumption 1, suppose the symmetric noise rate is not too large, i.e, < K−1K , GLS with smooth rate r = ropt yields f∗D. • When error rate < (K−1)·r ∗
K , r = ropt > 0 (PLS);
• When error rate = (K−1)·r ∗
K , r = 0 (Vanilla Loss);
• When error rate > (K−1)·r ∗
K , r = ropt < 0 (NLS).

heading: D ADDITIONAL EXPERIMENT RESULTS AND DETAILS
text: In this section, we include more experiment results, observations and details for learning with GLS.

heading: D.1 EXPERIMENT DETAILS ON CIFAR-10, CIFAR-100
text: We firstly introduce experiment details on CIFAR-10 dataset adopted in our experiment designs.
Training settings of clean CIFAR-10 dataset (Krizhevsky et al., 2009) We adopted ResNet34 (He et al., 2016), trained for 200 epochs with batch-size 128, SGD (Robbins & Monro, 1951) optimizer with Nesterov momentum of 0.9 and weight decay 1e-4. The learning rate of first 100 epochs is 0.1. Then it multiples with 0.1 for every 50 epochs.
Generating noise labels on CIFAR datasets We adopt symmetric noise model which generates noisy labels by randomly flipping the clean label to the other possible classes with probability . And we set = 0.2, 0.4, 0.6 for CIFAR-10, = 0.4, 0.6 for CIFAR-100. We also make use of asymmetric noise model. The asymmetric noise is generated by flipping the true label to the next class with probability . We set = 0.2, 0.3 for CIFAR-10.
Training settings of synthetic noisy CIFAR datasets The generation of symmetric noisy dataset is adopted from (Cheng et al., 2020). The symmetric noise rates are [0.2, 0.4, 0.6]. We choose two methods to train GLS.
• Direct training: this setting is the same as training on clean CIFAR-10 dataset. • Warm-up: in this case, we firstly train a ResNet34 model with Cross-Entropy loss for 120 epochs. For
this warm-up, the only difference in hyper-parameter setting is the learning rate, where the initial learning rate is 0.1 and it multiplies 0.1 for every 40 epochs. After the warm-up, GLS loads the same pre-trained model and trains for 100 epochs with learning rate 1e-6.

heading: D.2 WHY NLS IS OVERLOOKED?
text: When learning from a relative large scale dataset, NLS tends to push the model become overly confident early in the training. The poor performances of NLS (direct-train) in Table 6 explain why NLS is neglected. When there is no warm-up, training NLS directly without warming up will reach a 88%− 92% test accuracy on the clean data. The performance will degrade much more significantly than PLS when the noise level is high or |r| is large. In Table 6, we provide the comparisons between direct-train and warm-up in several settings. The improvement bring by a warm-up procedure becomes much more significantly in the high noise regime. NLS makes the classifier be overly confident at the early training which results in converging to a bad local optimum (without CE warm-up, NLS frequently results in a worse performance in CIFAR-10 and CIFAR-100). Since the model will usually fit on the clean data first, then over-fits on the noisy ones (Liu et al., 2020), a large number of approaches (such as Loss corrections (Patrini et al., 2017), Peer Loss (Liu & Guo, 2020), etc) adopt a CE warm-up firstly. Note that there is no difference in the computing costs between NLS (with CE warmup) and CE loss, proceeding with NLS to enhance the model confidence makes NLS much more competitive in the high noise regime, also gives practical insights on how to make NLS work better when learning with clean data.

heading: D.3 EXPERIMENT DETAILS ON SYNTHETIC DATASETS AND UCI
text: We introduce experiment details on synthetic datasets and UCI datasets adopted in our experiment designs.
Generation of synthetic dataset In the synthetic (Type 1) dataset, we generate 500 points for both classes. Class +1 distributes inside the circle with radius 0.25. Class -1 generates by randomly sampling 500 data points in the annulus with inner radius 0.28 and outer radius 0.45. As for synthetic (Type 2) dataset, we uniformly assign labels for 50% samples in the annulus (with inner radius 0.22, outer radius 0.31) based on Type 1 dataset.
Generating noise labels on synthetic datasets and UCI datasets Note that these datasets are all binary classification datasets, each label in the training and validation set is flipped to the other class with probability e, and we set e = 0.1, 0.4 for synthetic Type 1 dataset, e = 0.1, 0.3 for synthetic Type 2 dataset.
Training settings of synthetic datasets For both types of synthetic datasets, we adopted a three-layer ReLU Multi-Layer Perceptron (MLP), trained for 200 epochs with batch-size 128 and Adam (Kingma & Ba, 2014) optimizer. The initial learning rate is 0.1, and it multiplies 0.1 for every 40 epochs.
Training settings of UCI datasets Dua & Graff (2017) We adopted (Liu & Guo, 2020) a two-layer ReLU Multi-Layer Perceptron (MLP) for classification tasks on multiple UCI datasets, trained for 1000 episodes with batch-size 64 and Adam (Kingma & Ba, 2014) optimizer. We report the best performance for each smooth rate under a set of learning rate settings, [0.0007, 0.001, 0.005, 0.01, 0.05].
D.4 ADDITIONAL EXPERIMENT ON r∗ AND rOPT
r∗ and ropt on synthetic dataset We generate 2D (binary) synthetic dataset by randomly sampling two circularly distributed classes. The inner annulus indicates one class (blue), while the outer annulus denotes the other class (red). Clearly, the generated synthetic dataset is well-separable (Type 1) and we hold 20% data samples for performance comparison. The noise transition matrix takes a symmetric form with noise rate ei for both classes. To simulate the scenario where the clean data may not be perfectly separated due to a non-negligible amount of uncertainty samples clustering at the decision boundary, we flip the label of 50% samples near the intersection of two annulus to the other class (Type 2). As specified in Table 7, r∗ = [0.1, 0.4] for Type 1 data and r∗ = [0.0, 0.2] for Type 2 data. With the presence of label noise, the distribution of ropt shifts from non-negative ones to negative values. Even though NLS fails to outperform PLS on clean data, we observe that NLS is less sensitive to noisy labels. Data with high level noise rates clearly favor NLS with a low smooth rate!
r∗ and ropt on more UCI datasets We further test the performance of generalized label smoothing on 7 more UCI datasets (Heart, Breast 1, Breast 2, Diabetes, German, Image and Waveform). Our observation remains unchanged: there exists a general trend that with the increasing of noise rates, NLS becomes much more competitive than PLS. Here, we attach the results of 4 additional UCI datasets for illustration.
The noisy labels are generated by a symmetric noise transition matrix with noise rate ei = [0.1, 0.2, 0.3, 0.4]. As highlighted in Table 8, ropt appears with positive values when the data is clean (same as r∗) or of a low
noise rate. With the increasing of noise rates, NLS becomes more competitive than PLS. We color-code different noise regimes where either PLS (red-ish) or NLS (green-ish) outperforms the other. Clearly, there is a separation of the favored smoothing rate for different noise scenarios (upper left & low noise for PLS, bottom right & high noise for NLS).

heading: D.5 ADDITIONAL EXPERIMENT RESULTS ON MODEL CONFIDENCE
text: NLS improves model confidence on Synthetic Type 2 dataset In this case, the clean data that are close to decision boundary distributes randomly. In Figure 5-6, the colored bands depict the different levels of prediction probabilities. When the smooth rate increases from negative to positive, more samples fall in the orange and light blue band which indicates uncertain predictions. When the smooth rate increases from negative to positive, learning with GLS will result in more uncertain predictions. With the increasing of noise rates (ei = 0→ 0.4), GLS with a fixed smooth rate becomes less confident on its predictions. Thus, a smaller smooth rate is required when the noise rate increases.
Model confidence of GLS on CIFAR-10 test dataset When GLS trained on symmetric 0.2 noisy CIFAR10 training dataset (see Figure 7), with the decreasing of smooth rates (from right to left), the model
confidence on correct predictions gradually approach to its maximum, while for wrong predictions, the model confidence converges to its minimum value. NLS makes the model prediction become over-confident on correct predictions and in-confident on wrong predictions.

heading: D.6 EFFECT OF GLS ON PRE-LOGITS
text: We visualise the pre-logits of a ResNet-34 for three classes on CIFAR-10. We adopt the method from (Müller et al., 2019) which illustrates how representations differ between penultimate layers of networks trained with different smooth rates in GLS. In Figure 8, NLS makes the model f be confident on her predictions and the distances between three clusters are clearly larger than those appeared in Vanilla Loss and PLS.

heading: E BIAS AND VARIANCE TRADE-OFF OF GLS
text: Denote f̂H, f̂S as pre-trained models on the training dataset D w.r.t. hard labels and soft labels, respectively. The vector form of the prediction w.r.t. sample x given by f̂H and f̂S are f̂H(x;D) and f̂S(x;D). For the ease of presentation, we relate notations with subscript H/S to hard/soft labels without further explanation. Given the sample x and the one-hot label y, we denote the averaged model prediction by:
f̄H(x;D) := 1
ZH exp [ED log(f̂H(x;D))], f̄S(x;D) :=
1
ZS exp [ED log(f̂S(x;D))]
where ZH, ZS are normalization constants. The bias of model prediction is defined as the KL divergence DKL between target distribution (one-hot encoded vector form) y and the averaged model prediction.
BiasH := Ex,y [ y log
y
f̄H(x;D)
] , BiasS := Ex,y [ y log
y
f̄S(x;D) ] While the variance of model prediction measures the expectation of KL divergence between the averaged model prediction and model prediction over D:
VarH := ED [ Ex,y [ f̄H(x;D) log ( f̄H(x;D) f̂H(x;D) )]] , VarS := ED [ Ex,y [ f̄S(x;D) log ( f̄S(x;D) f̂S(x;D) )]]
Empirical observation from (Zhou et al., 2021) shows that the variance brought by learning with positive soft labels given by a teacher’s model (Hinton et al., 2015) is less than the direct training w.r.t hard labels. As an extension, we are interested in how GLS interferes with the bias and variance of model prediction.
Bias and variance of GLS on clean dataset We introduce our empirical observation regarding the role of GLS in bias and variance trade-off in Figure 9. We select nine smooth rates of GLS for illustration. Each smooth rate setting of GLS trains on the CIFAR-10 dataset for 5 times with different data augmentations. To estimate the variance and bias of pre-trained models, we adopt the implementation in (Yang et al., 2020). Empirical results show that learning directly with a larger positive smooth rate typically results in lower variance and higher bias. In Figure 9, we can observe almost constant bias values and very low variance for NLS. This is best explained by the warm-up of pre-trained models and the fact that NLS pushes the classifier to give confident predictions. As for PLS, with the increase of smooth rate, the overall bias has an increasing tendency while the variance has the decreasing pattern. Especially when the smooth rate approaches to 1, i.e., r = 0.9, the variance is close to 0.

heading: F OMITTED PROOFS

heading: F.1 PROOF OF THEOREM 1
text: Before we prove Theorem 1, we first introduce Lemma 1. Lemma 1. ∀(x,yGLS,r), ` ( f(x),yGLS,r ) = ( 1− r2 ) · ` ( f(x), y ) + r2 · ` ( f(x), 1− y ) .

heading: Proof of Lemma 1
text: Proof. For CE loss, due to its linear property w.r.t. the label, we directly have:
`(f(x),yGLS,r) = ` ( f(x),y · (1− r) + r
2
) = ( 1− r
2
) · ` ( f(x), y ) + r 2 · ` ( f(x), 1− y )

heading: Proof of Theorem 1
text: Proof. Based on Lemma 1, with a bit of math, for NLS, we have:
minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,−r )] =minE(X,Ỹ )∼D̃ [( 1 + r
2
) · ` ( f(X), Ỹ ) − r 2 · ` ( f(X), 1− Ỹ )] =minE(X,Ỹ )∼D̃ [[( 1 + r
2
) + ( 1− r
2
)] · ` ( f(X), Ỹ ) − [( 1− r
2
) · ` ( f(X), Ỹ ) + r 2 · ` ( f(X), 1− Ỹ )]] =minE(X,Ỹ )∼D̃ [ 2 · ` ( f(X), Ỹ ) − ` ( f(X), Ỹ GLS,r )]

heading: F.2 PROOF OF THEOREM 5

heading: Proof.
text: Eqn.3 =minE(X,Ỹ )∼D̃ [ ( 1− r 2 ) ︸ ︷︷ ︸
:=c1
·` ( f(X), Ỹ ) + r
2︸︷︷︸ :=c2
·` ( f(X), 1− Ỹ )]
=minEX,Y=0 [ P(Ỹ = 0|Y = 0) · ( c1 · ` ( f(X), 0 ) + c2 · ` ( f(X), 1 )) + P(Ỹ = 1|Y = 0) · ( c1 · ` ( f(X), 1 ) + c2 · ` ( f(X), 0
))] + EX,Y=1 [ P(Ỹ = 0|Y = 1) · ( c1 · ` ( f(X), 0 ) + c2 · ` ( f(X), 1
)) + P(Ỹ = 1|Y = 1) · ( c1 · ` ( f(X), 1 ) + c2 · ` ( f(X), 0
))] =minEX,Y=0 [[ (1− e0) · c1 + e0 · c2 ] · ` ( f(X), 0 ) + [ (1− e0) · c2 + e0 · c1 ] · ` ( f(X), 1
)] + EX,Y=1 [[ (1− e1) · c1 + e1 · c2 ] · ` ( f(X), 1 ) + [ (1− e1) · c2 + e1 · c1 ] · ` ( f(X), 0
)] =minEX,Y=0 [[ (1− e0) · c1 + e0 · c2 ] · ` ( f(X), 0 ) + [ (1− e0) · c2 + e0 · c1 ] · ` ( f(X), 1
)] + EX,Y=1 [[ (1− e0) · c1 + e0 · c2 ] · ` ( f(X), 1 ) + [ (1− e0) · c2 + e0 · c1 ] · ` ( f(X), 0
)] + EX,Y=1 [ e∆ · (c2 − c1) · ` ( f(X), 1 ) − e∆ · (c2 − c1) · ` ( f(X), 0
)] =minE(X,Y )∼D [[ (1− e0) · c1 + e0 · c2 ] · ` ( f(X), Y ) + [ (1− e0) · c2 + e0 · c1 ] · ` ( f(X), 1− Y
)] − e∆ · (c1 − c2) · EX,Y=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] =minE(X,Y )∼D [ (c1 + c2) · ` ( f(X), Y
)] + [ (1− e0) · c2 + e0 · c1 ] · E(X,Y )∼D [ ` ( f(X), 1− Y ) − ` ( f(X), Y
)] − e∆ · (c1 − c2) · EX,Y=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] =minE(X,Y )∼D [ (c1 + c2) · ` ( f(X), Y ∗
)] + [ − r ∗
2 + (1− e0) · c2 + e0 · c1
] · E(X,Y )∼D [ ` ( f(X), 1− Y ) − ` ( f(X), Y )] − e∆ · (c1 − c2) · EX,Y=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] =minE(X,Y )∼D [ ` ( f(X), Y ∗
)] ︸ ︷︷ ︸
True Risk
+λ1 · E(X,Y )∼D [ ` ( f(X), 1− Y ) − ` ( f(X), Y )] ︸ ︷︷ ︸
M-Inc1 +λ2 · EX,Y=1 [ ` ( f(X), 0 ) − ` ( f(X), 1 )] ︸ ︷︷ ︸
M-Inc2

heading: F.3 PROOF OF PROPOSITION 1
text: Proof. The risk minimization of backward correction is equivalent to: E(X,Ỹ )∼D̃ [ `← ( f(X), Ỹ )] =E(X,Y )∼D [ ` ( f(X), Y )] (By Theorem 1 in (Patrini et al., 2017))
The risk minimization of forward correction is equivalent to: E(X,Ỹ )∼D̃ [ `→(f(X), Ỹ ) ] =E(X,Y )∼D [ `(f(X), Y ) ] (By Theorem 2 in (Patrini et al., 2017))
Theorem 1 and 2 in (Patrini et al., 2017) demonstrate that forward and backward corrected losses equal the original loss ` computed on the clean data in expectation. Thus, for rLC = 2e02e0−1 , by Theorem 5 (adopt r∗ = 0), we have:
minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,rLC )] + λLC · EX,Y=1 [ ` ( f(X), 1 ) − ` ( f(X), 0 )] ︸ ︷︷ ︸
Bias-LC =minE(X,Y )∼D [ ` ( f(X), Y )] + [ e0 + (1− 2e0) ·
rLC 2
] · E(X,Y )∼D [ ` ( f(X), 1− Y ) − ` ( f(X), Y )] + e∆ · (1− rLC) · EX,Y=1 [ ` ( f(X), 0 ) − ` ( f(X), 1 )] + λLC · EX,Y=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] =minE(X,Y )∼D [ ` ( f(X), Y )] + e∆ · ( 1 1− 2e0 − 1 1− 2e0 ) · EX,Y=1 [ ` ( f(X), 0 ) − ` ( f(X), 1
)] =minE(X,Y )∼D [ ` ( f(X), Y
)] Thus,
minE(X,Ỹ )∼D̃ [ `← ( f(X), Ỹ )] = minE(X,Ỹ )∼D̃ [ `→ ( f(X), Ỹ )] =minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,rLC )] + λLC · EX,Y=1 [ ` ( f(X), 1 ) − ` ( f(X), 0 )] ︸ ︷︷ ︸
Bias-LC

heading: F.4 PROOF OF THEOREM 2
text: Proof. Based on Proposition 1, when e∆ = 0, λLC = 0, we directly have: minE(X,Ỹ )∼D̃ [ `← ( f(X), Ỹ )] = minE(X,Ỹ )∼D̃ [ `→ ( f(X), Ỹ )] =minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,rLC )]

heading: F.5 PROOF OF THEOREM 3
text: Proof. Note that minE(X,Ỹ )∼D̃ [ `CL ( f(X), Ỹ )] = minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ ) − ` ( f(X), 1− Ỹ )]
We have:
minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,rCL )] =minE(X,Ỹ )∼D̃ [( 1− rCL
2
) · ` ( f(X), Ỹ ) + rCL 2 · ` ( f(X), 1− Ỹ )] ⇐⇒minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ ) +
rCL 2− rCL
· ` ( f(X), 1− Ỹ )]
When rCL → −∞, we have rCL2−rCL → −1. Thus,
minE(X,Ỹ )∼D̃ [ `CL ( f(X), Ỹ )] = min E(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,rCL→−∞ )]

heading: F.6 PROOF OF PROPOSITION 2
text: Proof. Note that:
E(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ )] − E(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,r )] =E(X,Ỹ )∼D̃ [ 1− ( 1− r
2
) · ` ( f(X), Ỹ ) − r 2 · ` ( f(X), 1− Ỹ )] = r
2 · E(X,Ỹ )∼D̃
[ ` ( f(X), Ỹ ) − ` ( f(X), 1− Ỹ )]
And we have:
E(Xi,Ỹi)∼D̃ [ ` ( f(X1), Ỹ2 )] =EX [ P(Ỹ = 0) · ` ( f(X), 0 ) + ( 1− P(Ỹ = 0) ) · ` ( f(X), 1
)] =EX,Ỹ=0 [ P(Ỹ = 0) · ` ( f(X), 0 ) + ( 1− P(Ỹ = 0) ) · ` ( f(X), 1
)] + EX,Ỹ=1 [ P(Ỹ = 0) · ` ( f(X), 0 ) + ( 1− P(Ỹ = 0) ) · ` ( f(X), 1
)] =EX,Ỹ=0 [ P(Ỹ = 0) · ` ( f(X), 0 ) + ( 1− P(Ỹ = 0) ) · ` ( f(X), 1
)] + EX,Ỹ=1 [( 1− P(Ỹ = 0) ) · ` ( f(X), 0 ) + P(Ỹ = 0) · ` ( f(X), 1
)] + ( 1− 2 · P(Ỹ = 0) ) · EX,Ỹ=1 [ ` ( f(X), 1 ) − ` ( f(X), 0 )]
Thus,
min f
E(X,Ỹ )∼D̃ [ `PL ( f(X), Ỹ )] = min
f E(X,Ỹ )∼D̃
[ ` ( f(X), Ỹ ) − ` ( f(X1), Ỹ2 )] =min
f E(X,Ỹ )∼D̃
[ ` ( f(X), Ỹ )] − E(Xi,Ỹi)∼D̃ [ ` ( f(X1), Ỹ2 )] =min
f EX,Ỹ=0
[ ` ( f(X), 0 )] + EX,Ỹ=1 [ ` ( f(X), 1 )] − EX,Ỹ=0 [ P(Ỹ = 0) · ` ( f(X), 0 ) + ( 1− P(Ỹ = 0) ) · ` ( f(X), 1
)] − EX,Ỹ=1 [( 1− P(Ỹ = 0) ) · ` ( f(X), 0 ) + P(Ỹ = 0) · ` ( f(X), 1
)] − ( 1− 2 · P(Ỹ = 0) ) · EX,Ỹ=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] =min
f EX,Ỹ=0
[( 1− P(Ỹ = 0) ) · [ ` ( f(X), 0 ) − ` ( f(X), 1 )]] + EX,Ỹ=1 [( 1− P(Ỹ = 0) ) · [ ` ( f(X), 1 ) − ` ( f(X), 0
)]] − ( 1− 2 · P(Ỹ = 0) ) · EX,Ỹ=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] =min
f E(X,Ỹ )∼D̃
[( 1− P(Ỹ = 0) ) · [ ` ( f(X), Ỹ ) − ` ( f(X), 1− Ỹ )]] − ( 1− 2 · P(Ỹ = 0) ) · EX,Ỹ=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] Thus, for rPL = 2 · P(Ỹ = 1), λPL = 1− rPL, we have:
E(X,Ỹ )∼D̃ [ `PL(f(X), Ỹ )] − [ E(X,Ỹ )∼D̃ [ `(f(X), Ỹ )] − E(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,rPL )]] =E(X,Ỹ )∼D̃ [( 1− P(Ỹ = 0) ) · [ ` ( f(X), Ỹ ) − ` ( f(X), 1− Ỹ
)]] − ( 1− 2 · P(Ỹ = 0) ) · EX,Ỹ=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] − rPL
2 · E(X,Ỹ )∼D̃
[ ` ( f(X), Ỹ ) − ` ( f(X), 1− Ỹ )] =E(X,Ỹ )∼D̃ [( 1− P(Ỹ = 0)− P(Ỹ = 1) ) · [ ` ( f(X), Ỹ ) − ` ( f(X), 1− Ỹ
)]] − ( 2 · P(Ỹ = 1)− 1 ) · EX,Ỹ=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] =λPL · EX,Ỹ=1 [ ` ( f(X), 1 ) − ` ( f(X), 0
)] And we can conclude that:
minE(X,Ỹ )∼D̃ [ `PL(f(X), Ỹ ) ] = min E(X,Ỹ )∼D̃ [ `(f(X), Ỹ )− `(f(X), Ỹ GLS,rPL) ] +λPL · EX,Ỹ=1 [ `(f(X), 1)− `(f(X), 0) ] ︸ ︷︷ ︸
Bias-PL

heading: F.7 PROOF OF THEOREM 4
text: Proof. When P(Ỹ = 0) = P(Ỹ = 1), according to Proposition 2, we have λPL = 0 and:
minE(X,Ỹ )∼D̃ [ `PL ( f(X), Ỹ )] =minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ ) − ` ( f(X), Ỹ GLS,rPL )] =minE(X,Ỹ )∼D̃ [rPL 2 · ` ( f(X), Ỹ ) − rPL 2 · ` ( f(X), 1− Ỹ
)] ⇐⇒minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ ) − ` ( f(X), 1− Ỹ
)] When rPL → −∞, we further have:
minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ GLS,rPL )] ⇐⇒minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ ) +
rCL 2− rCL
· ` ( f(X), 1− Ỹ )] ⇐⇒minE(X,Ỹ )∼D̃ [ ` ( f(X), Ỹ ) − ` ( f(X), 1− Ỹ
)] Thus, Theorem 4 is proved.

heading: F.8 PROOF OF THEOREM 6
text: Proof. Note that the optimal r that will cancel the impact of Term M-Inc1 is:
ropt := r∗ − 2e 1− 2e
• When e < r ∗
2 , ropt > 0. In this case, learning PLS with smooth rate ropt results in:
minE(X,Ỹ )∼D̃ [ `(f(X), Ỹ GLS,r=ropt) ] = minE(X,Y )∼D [ ` ( f(X), Y ∗ )] which yields f∗D;
• When e = r ∗ 2 , ropt = 0. Learning with the Vanilla Loss yields f ∗ D since:
minE(X,Ỹ )∼D̃ [ `(f(X), Ỹ ) ] = minE(X,Y )∼D [ ` ( f(X), Y ∗ )]
• Similarly, when e > r ∗ 2 , learning NLS with r = ropt < 0 yields f ∗ D.

heading: F.9 PROOF OF THEOREM 7
text: Proof. Denote pi = P(Y = i) as the clean label distribution, p̃i = P(Ỹ = i) as the clean label distribution. Let ′ = K· K−1 , we have:
E(X,Ỹ )∼D̃ [ (1− r) · ` ( f(X), Ỹ )] + EX [ ∑ i∈[K] r K · ` ( f(X), i )]
= [ ∑ i∈[K] E(X,Ỹ )∼D̃,Y=i [ (1− r) · ` ( f(X), Ỹ )]] + EX [ ∑ i∈[K] r K · `(f(X), i) ]
= [ (1− r) ·
∑ i∈[K] E(X,Ỹ )∼D̃,Y=i [ ∑ j∈[K] Ti,j · ` ( f(X), Ỹ = j )]] + EX [ ∑ i∈[K] r K · ` ( f(X), i )]
= [ (1− r) ·
∑ i∈[K] EX,Y=i [ (1− ′) · ` ( f(X), i ) + ∑ j∈[K] ′ K · ` ( f(X), j )]] + EX [ ∑ i∈[K] r K · ` ( f(X), i )]
= [ (1− r) ·
∑ i∈[K] EX,Y=i [( 1− ′ ) · ` ( f(X), i )]] + EX [[ (1− r) · ′ K + r K ] ∑ j∈[K] ` ( f(X), j )]
= [ (1− r) · ( 1− ′ ) ︸ ︷︷ ︸
:=c3
E(X,Y )∼D [ ` ( f(X), Y )]] + EX [ [ (1− r) · ′ K + r K ] ︸ ︷︷ ︸
:=c4
∑ j∈[K] ` ( f(X), j
)]
=
[ c3
1− r∗ · E(X,Y )∼D
[ ` ( f(X), Y ∗ ) − r ∗
K ∑ j∈[K] ` ( f(X), j
)]] + [ c4 · EX [ ∑ j∈[K] ` ( f(X), j )]]
=
[ c3
1− r∗ · E(X,Y )∼D
[ ` ( f(X), Y ∗ )]] ︸ ︷︷ ︸
True Risk
+ [( c4 −
c3 · r∗
(1− r∗) ·K
) · EX [ ∑ j∈[K] ` ( f(X), j )]] ︸ ︷︷ ︸
M-Inc1
Adopting ropt = r ∗− ′ 1− ′ , with a bit of math, the weight of Term M-Inc1 becomes 0 and
E(X,Ỹ )∼D̃ [ ` ( f(X), Y GLS,ropt )] =E(X,Ỹ )∼D̃ [ (1− ropt) · ` ( f(X), Ỹ )] + EX
[ ∑ i∈[K] ropt K · ` ( f(X), i )]
=
[ c3
1− r∗ · E(X,Y )∼D
[ ` ( f(X), Y ∗ )]] ⇐⇒ E(X,Y )∼D [ ` ( f(X), Y ∗ )]

Review summaries:
This paper studies label smoothing when learning with noisy label. It proposed generalized label smoothing (GLS) containing positive label smoothing (PLS) and negative label smoothing (NLS), where NLS allows the smoothing parameter to be negative. The authors found that when the noisy rate is high, NLS is more beneficial than PLS, both theoretically and empirically.

Label smoothing shows a promising bias-variance tradeoff when training deep neural networks, while it is unclear if label smoothing remains helpful when training labels are noisy. A recent work shows the benefits of using label smoothing. However, the current submission finds contradicting and appealing observations. In Figure 1, the authors presented a convincing set of results that clearly show a phase-transitioning behavior of using label smoothing: while label smoothing seems to work with a low label noise rate, the benefits disappear when the rate goes up high.  The above observation can imply that label smoothing is considered unfavorable with a high label noise rate. The paper then presented a generalized label smoothing scheme where the smoothing constant can instead go negative. Even though using a negative parameter sounds weird, the paper showed that a label smoothing with a negative (but different) smoothing parameter corresponds to several existing solutions to learning with noisy labels! This signals that the negative label smoothing scheme might turn out to be helpful when dealing with high label noise rate. I think this is a pretty interesting observation and connection to uncover.  The two cases are integrated as the generalized label smoothing, where the smoothing constant can be either negative to positive. In section 5, it is shown under what noise rate, a positive or a negative label smoothing is considered a better option. This characterization reveals a clear phase transitioning behavior corresponding to figure 1.

The paper considers generalized label smoothing (GLS) for binary classification tasks where the smoothing coefficient is allowed to be negative (negative label smoothing, NLS). The authors show the connection between GLS and several existing loss functions through elementary algebraic manipulations of the expected loss under (generalized) label smoothing. They also analyze GLS in the noisy label setting where the labels are randomly flipped and show that using negative label smoothing is desirable in the high label noise setting.