Understanding Generalized Label Smoothing when Learning with Noisy Labels

1 INTRODUCTION . Label smoothing ( LS ) ( Szegedy et al. , 2016 ) is an arising learning paradigm that uses positively weighted average of both the hard training labels and uniformly distributed soft label : yLS , r = ( 1− r ) · y + r K · 1 ( 1 ) where we denote the one-hot vector form of hard label and an all one vector as y , 1 respectively . K is the number of label classes , and r is the smooth rate in the range of [ 0 , 1 ] . It was shown that LS serves as a regularizer for the hard training data and therefore improves generalization of the model . The regularizer role of LS prevents the model from fitting overly on the target class . Empirical studies have demonstrated the effectiveness of LS in improving the model performance across various benchmarks ( Pereyra et al. , 2017 ) ( such as image classification ( Szegedy et al. , 2016 ) , machine translation ( Vaswani et al. , 2017 ) , language modelling ( Chorowski & Jaitly , 2017 ) ) and model calibration ( Müller et al. , 2019 ) . Later it was reported LS even helps with improving robustness when learning with noisy labels ( Lukasik et al. , 2020 ) . However , we observe that the advantage of LS vanishes when we operate in a high label noise regime . In Figure 1 , we present a set of experiments on UCI datasets ( Dua & Graff , 2017 ) . We highlight best two smooth rates ( possible to have tied smooth rates ) under each label noise rate . Indeed , non-negative smooth rates ( circles colored in red ) outperform negative ones when the label noise rates are low . Nonetheless , with the increasing of noise rates , negative smooth rates r < 0 ( Eqn . ( 1 ) , diamonds colored in green ) appear to be more competitive when learning with noisy labels . Puzzled by the observation , we proceeded to discover that several proposed learning-with-noisy-labels solutions in the literature , including Loss Correction ( Patrini et al. , 2017 ) , NLNL ( Kim et al. , 2019 ) and Peer Loss ( Liu & Guo , 2020 ) , instead relate more closely to negative label smoothing ( NLS ) , which defines as using a negative weight to combine the hard and soft labels ! This paper unifies label smoothing with either positive or negative smooth rate into a generalized label smoothing ( GLS ) framework . Our paper is motivated by the above inconsistent observations . And we aim to provide a more thorough understanding of GLS under the setting of learning with noisy labels , rather than proposing a new method to compete for state-of-the-art performances . We firstly show that negative label smoothing differs substantially from positive label smoothing in their achieved model confidence . With the presence of label noise , we then proceed to show that there exists a phase transition behavior when finding the optimal label smoothing rate for GLS . Particularly , when label noise is low , positive label smoothing is able to uncover the optimal model while negative label smoothing is considered more beneficial in a high label noise regime . We provide extensive experimental evidences to support our findings . For instance , on multiple benchmark datasets , we present the clear transition of the optimal smoothing rate going from positive to negative when we keep increasing noise rates . On CIFAR-10 test data , we show a negative smoothing rate elicits higher model confidence on correct predictions and lower confidence on wrong predictions compared with the behavior of a positive one . Our contributions summarize as follows : • We provide understandings for a generalized notion of label smoothing ( GLS ) when learning with noisy labels , where the label smooth rate can go negative . ( An understanding paper rather than a method paper ) • We show that several robust loss functions in the noise learning literature correspond to learning with GLS , under certain noise rate models . ( Section 3 ) • We show that negative label smoothing improves the confidence of model prediction . With the presence of label noise , we demonstrate learning with a negative smooth rate can be more robust to label noise compared with a positive rate when label noise rates are high . ( Section 4 , 5 ) • Empirical experiments on multiple datasets and methods demonstrate that with the presence of label noise , NLS becomes competitively robust to label noise . We also empirically show how GLS results in trade-offs in model confidence , bias and variance of the generalization error . ( Section 6 , Appendix D and E ) • Extensive experiment results validate our main theoretical conclusions . Besides , we also discuss practical considerations and multi-class extensions of GLS to mitigate the impact of label noise . ( Appendix B , C ) We defer all proofs to Appendix F. Our work primarily contributes to the literature of learning with noisy labels ( Scott et al. , 2013 ; Natarajan et al. , 2013 ; Liu & Tao , 2015 ; Patrini et al. , 2017 ; Liu & Guo , 2020 ) . Our core results are contingent on recent works of understanding the effect of label smoothing when training deep neural network models , i.e. , label smoothing improves model calibration ( Müller et al. , 2019 ) , more complexed forms of label smoothing ( Li et al. , 2020 ; Yuan et al. , 2020 ) , and in particular when label noise presents ( Lukasik et al. , 2020 ; Liu , 2021 ) . We generalize a concept called generalized label smoothing and provide new understandings for when , instead of setting a positive smoothing rate as the literature would normally do , a negative smoothing rate is considered a better option . Due to the space limit , we defer a more detailed discussion of related works to Appendix A . 2 PRELIMINARIES . 2.1 LEARNING WITH SMOOTHED LABELS . For aK-class classification task , we denote byX ∈ X a high-dimensional feature and Y ∈ Y : = { 1 , 2 , ... , K } the corresponding label . Suppose ( X , Y ) ∈ X × Y are drawn from a joint distribution D. Let yi be the one-hot encoded vector form of yi which generates according to Y . The random variable of smoothed label Y LS , r with smooth rate r ∈ [ 0 , 1 ] generates yLS , ri as ( Szegedy et al. , 2016 ) : yLS , ri = ( 1− r ) · yi + r K · 1 For example , when r = 0.3 , the smoothed label of yi = [ 0 , 1 , 0 ] becomes y LS , r=0.3 i = [ 0.1 , 0.8 , 0.1 ] . We consider a broader setting where the smoothed label might be negatively related to the corresponding feature . As a supplementary to existed works on label smoothing , we explore the benefits of learning with generalized label smoothing ( GLS ) , i.e. , r ∈ ( −∞ , 1 ] instead of a non-negative r. yGLS , ri : = ( 1− r ) · yi + r K · 1 ( 2 ) where yGLS , ri is given by the random variable of generalized smooth label Y GLS , r . We name the scenario r < 0 as negative label smoothing ( NLS ) . To clarify , we don ’ t assume a strict lower bound for r. If r → −∞ , normalizing yGLS , ri by 1− r returns y GLS , r i = yi − 1 K . We will show when imposing a negative smoothing parameter will be considered beneficial as compared to a positive one . In the main paper , we focus on the binary classification task where yi ∈ { 0 , 1 } and K = 2 . And we defer multi-class extensions to Appendix C. Denote f as a deep neural network , f ( xi ) is the model prediction of xi ∈ X with element f ( xi ) yi : = P ( Y = yi|X = xi , f ) . Given the sample x ∈ X and a hard label y ∈ Y , binary CE loss is defined as ` CE ( f ( x ) , y ) : = − log ( f ( x ) y ) . Throughout this paper , we shorthand ` as ` CE for a clean presentation . 2.2 LEARNING WITH NOISY LABELS . The noisy label literature considers the setting where we only have access to samples with noisy labels from ( X , Ỹ ) . Suppose random variables ( X , Ỹ ) ∈ X × Ỹ are drawn from a noisy joint distribution D̃ . Statistically , the random variable of noisy labels Ỹ can be characterized by a noise transition matrix T , where each element Ti , j represents the probability of flipping the clean label Y = i to the noisy label Ỹ = j , i.e , Tij = P ( Ỹ = j|Y = i ) . In this paper , we are interested in the widely studied class-dependent label noise . We assume the label noise is conditionally independent of features , i.e. , P ( Ỹ = j|Y = i ) = P ( Ỹ = j|X , Y = i ) , ∀i , j ∈ [ K ] For the binary classification setting , define e0 : = P ( Ỹ = 1|Y = 0 ) , e1 : = P ( Ỹ = 0|Y = 1 ) . Without loss of generality , we assume e1 − e0 = e∆ ≥ 0 . We denote the binary noise transition matrix in the noisy label setting as : T = ( 1− e0 e0 e1 1− e1 ) . 2.3 MODEL CONFIDENCE . We define a key quantity , model confidence , that plays an important role in later sections . Definition 1 . Model confidence of model f for sample ( x , y ) . Given a model f , a sample x with its target label y ∈ { 0 , 1 } , the model confidence of f w.r.t . sample x is defined as MC ( f ; x , y ) = f ( x ) y − f ( x ) 1−y . MC ( f ; x , y ) in definition 1 characterizes the difference of the predicted probability between target class and the other class . MC ( f ; x , y ) = 0 simply means f has no confident on its predictions since the model can not identify the target class of x. MC ( f ; x , y ) returns a negative value when f gives a wrong prediction and is not confident to predict the label of x as the target label y . To dig into how GLS influences the model confidence on correct and wrong predictions in following sections , we separate the distribution D into D+f : = { ( X , Y ) ∼ D : MC ( f ; X , Y ) > 0 } , D − f : = { ( X , Y ) ∼ D : MC ( f ; X , Y ) ≤ 0 } 3 CONNECTION TO OTHER ROBUST METHODS . In this section , we aim to theoretically explore the connection between GLS and popular methods such as backward/forward loss correction ( Natarajan et al. , 2013 ; Patrini et al. , 2017 ) , NLNL ( Kim et al. , 2019 ) and peer loss ( Liu & Guo , 2020 ) . We defer the corresponding empirical validations to Appendix B . For r ≤ 1 , let ỹ be the vector form of noisy label ỹ obtained from Ỹ , we define the r smoothed label of ỹ as ỹGLS , r , where ỹGLS , r : = ( 1− r ) · ỹ + ( r/K ) · 1 and is generated by the random variable Ỹ GLS , r . Risk minimization of the Generalized Label Smoothing ( GLS ) w.r.t . noisy labels becomes : Risk Minimization Using GLS : minE ( X , Ỹ ) ∼D̃ [ ` ( f ( X ) , Ỹ GLS , r ) ] ( 3 ) The GLS framework covers three special methods : PLS ( r ∈ ( 0 , 1 ] ) , Vanilla ( CE ) Loss ( r = 0 ) and NLS ( r < 0 ) . Besides , we observe that NLS connects to a special case of label smoothing regularization . We highlight this in Theorem 1 . Theorem 1 . ∀r ∈ [ 0 , 1 ] , NLS with smooth rate −r is a special form of label smoothing regularization : minE ( X , Ỹ ) ∼D̃ [ ` ( f ( X ) , Ỹ GLS , −r ) ] = minE ( X , Ỹ ) ∼D̃ [ 2 · ` ( f ( X ) , Ỹ ) − ` ( f ( X ) , Ỹ GLS , r ) ]

Summary: Label smoothing shows a promising bias-variance tradeoff when training deep neural networks, while it is unclear if label smoothing remains helpful when training labels are noisy. A recent work shows the benefits of using label smoothing. However, the current submission finds contradicting and appealing observations. In Figure 1, the authors presented a convincing set of results that clearly show a phase-transitioning behavior of using label smoothing: while label smoothing seems to work with a low label noise rate, the benefits disappear when the rate goes up high.  The above observation can imply that label smoothing is considered unfavorable with a high label noise rate. The paper then presented a generalized label smoothing scheme where the smoothing constant can instead go negative. Even though using a negative parameter sounds weird, the paper showed that a label smoothing with a negative (but different) smoothing parameter corresponds to several existing solutions to learning with noisy labels! This signals that the negative label smoothing scheme might turn out to be helpful when dealing with high label noise rate. I think this is a pretty interesting observation and connection to uncover.  The two cases are integrated as the generalized label smoothing, where the smoothing constant can be either negative to positive. In section 5, it is shown under what noise rate, a positive or a negative label smoothing is considered a better option. This characterization reveals a clear phase transitioning behavior corresponding to figure 1.