Towards Understanding Label Smoothing

1 INTRODUCTION . Due to the massive number of parameters , over-fitting becomes one of the major challenges in training effective deep neural networks ( Müller et al. , 2019 ) . Various strategies such as weight decay , dropout ( Hinton et al. , 2012b ) , batch normalization ( Ioffe & Szegedy , 2015 ) , data augmentation ( Simard et al. , 1998 ) , etc . have been proposed to alleviate the problem . Unlike regularization on input data or model weights , label smoothing regularization ( LSR ) ( Szegedy et al. , 2016 ) perturbs labels of data for better generalization . Concretely , for a K-class classification problem , the one-hot label in the prevalent cross entropy loss is replaced by a soft label as yLS = ( 1− θ ) y+ θŷ , where y is the one-hot label and ŷ = 1K is a uniform distribution for all labels . By optimizing the soft label instead , LSR helps to improve the generalization of deep neural networks in diverse applications including image classification ( Zoph et al. , 2018 ; He et al. , 2019 ) , speech recognition ( Chorowski & Jaitly , 2017 ; Zeyer et al. , 2018 ) , and language translation ( Vaswani et al. , 2017 ; Nguyen & Salazar , 2019 ) . Besides , it is complementary to other regularization methods and can work with others to make learned neural networks applicable ( He et al. , 2019 ) . Despite the success of LSR , efforts toward theoretical understanding are still limited . Müller et al . ( 2019 ) have empirically shown that LSR can help improve model calibration . However , they also found that LSR could impair knowledge distillation when the learned model is applied as the teacher network . Yuan et al . ( 2019a ) have proved that LSR provides a virtual teacher model for knowledge distillation . As a widely used trick , Lukasik et al . ( 2020 ) have shown that LSR works since it can successfully mitigate label noise . Most of existing works try to study LSR as regularizing label space while little work investigates the direct impact of LSR on training deep neural networks . In this work , we analyze the benefits of LSR for the learning process from the view of optimization . Concretely , our theory shows that an appropriate LSR can essentially reduce the variance of stochastic gradient in the assigned class labels and thus it can speed up the convergence of stochastic gradient descent , which is a standard optimizer for deep learning . Besides , the analysis implies that LSR may incur slow convergence at the end of optimization . Inspired by the result , a novel algorithm is developed to accelerate the convergence by a two-stage training paradigm . The effectiveness of the proposed method is verified both theoretically and empirically . We summarize the main contributions of this paper as follows . • To the best of our knowledge , it is the first work that establishes improved iteration complexities of stochastic gradient descent ( SGD ) ( Robbins & Monro , 1951 ) with LSR for finding an -approximate stationary point ( Definition 1 ) in solving a smooth non-convex problem in the presence of an appropriate label smoothing . The results theoretically explain why an appropriate LSR can help speed up the convergence . ( Subsection 4.1 ) • According to the analysis , we propose a simple yet effective method to accelerate the convergence of LSR ( Subsection 4.2 ) . It adopts a two-stage training paradigm that applies LSR in the first stage and then runs on one-hot labels in the second stage . • The theoretical analysis demonstrates that TSLA has an improved iteration complexity ( Subsection 4.3 ) and better generalization ( Subsection 4.4 ) compared to the conventional LSR . Extensive experiments on benchmark data sets confirm the effectiveness of our proposed method . 2 RELATED WORK . In this section , we introduce some related work . A closely related idea to LSR is confidence penalty proposed by Pereyra et al . ( 2017 ) , an output regularizer that penalizes confident output distributions by adding its negative entropy to the negative log-likelihood during the training process . The authors ( Pereyra et al. , 2017 ) presented extensive experimental results in training deep neural networks to demonstrate better generalization comparing to baselines with only focusing on the existing hyper-parameters . They have shown that LSR is equivalent to confidence penalty with a reversing direction of KL divergence between uniform distributions and the output distributions . DisturbLabel introduced by Xie et al . ( 2016 ) imposes the regularization within the loss layer , where it randomly replaces some of the ground truth labels as incorrect values at each training iteration . Its effect is quite similar to LSR that can help to prevent the neural network training from overfitting . The authors have verified the effectiveness of DisturbLabel via several experiments on training image classification tasks . Recently , many works ( Zhang et al. , 2018 ; Bagherinezhad et al. , 2018 ; Goibert & Dohmatob , 2019 ; Shen et al. , 2019 ; Li et al. , 2020b ) explored the idea of LSR technique . Ding et al . ( 2019 ) extended an adaptive label regularization method , which enables the neural network to use both correctness and incorrectness during training . Pang et al . ( 2018 ) used the reverse cross-entropy loss to smooth the classifier ’ s gradients . Wang et al . ( 2020 ) proposed a graduated label smoothing method that uses the higher smoothing penalty for high-confidence predictions than that for low-confidence predictions . They found that the proposed method can improve both inference calibration and translation performance for neural machine translation models . By contrast , we will try to understand the power of LSR from an optimization perspective and try to study how and when to use LSR . 3 PRELIMINARIES AND NOTATIONS . We first present some notations . Let∇wFS ( w ) denote the gradient of a function FS ( w ) . When the variable to be taken a gradient is obvious , we use∇FS ( w ) for simplicity . We use ‖ · ‖ to denote the Euclidean norm . Let 〈· , ·〉 be the inner product . In classification problem , we aim to seek a classifier to map an example x ∈ X onto one of K labels y ∈ Y ⊂ RK , where y = ( y1 , y2 , . . . , yK ) is a one-hot label , meaning that yi is “ 1 ” for the correct class and “ 0 ” for the rest . Suppose the example-label pairs are drawn from a distribution P , i.e. , ( x , y ) ∼ P = ( Px , Py ) . Let S = { ( x1 , y1 ) , . . . , ( xn , yn ) } denotes a set of n examples drawn from P. We denote by E ( x , y ) [ · ] the expectation that takes over a random variable ( x , y ) . When the randomness is obvious , we write E [ · ] for simplicity . Our goal is to learn a prediction function f ( w ; x ) : W ×X → RK that is as close as possible to y , where w ∈ W is the parameter andW is a closed convex set . To this end , we want to minimize the following expected loss under P : min w∈W FS ( w ) : = 1 n n∑ i=1 ` ( yi , f ( w ; xi ) ) ( 1 ) where ` : Y × RK → R+ is a cross-entropy loss function given by ` ( y , f ( w ; x ) ) = K∑ i=1 −yi log ( exp ( fi ( w ; x ) ) ∑K j=1 exp ( fj ( w ; x ) ) ) . ( 2 ) The objective function FS ( w ) is not convex since f ( w ; x ) is non-convex in terms of w. To solve the problem ( 1 ) , one can simply use some iterative methods such as stochastic gradient descent ( SGD ) . Specifically , at each training iteration t , SGD updates solutions iteratively by wt+1 = wt − η∇w ` ( yt , f ( wt ; xt ) ) , where η > 0 is a learning rate . Next , we present some notations and assumptions that will be used in the convergence analysis . Throughout this paper , we also make the following assumptions for solving the problem ( 1 ) . Assumption 1 . Assume the following conditions hold : ( i ) The stochastic gradient of FS ( w ) is unbiased , i.e. , E ( x , y ) [ ∇ ` ( y , f ( w ; x ) ) ] = ∇FS ( w ) , and the variance of stochastic gradient is bounded , i.e. , there exists a constant σ2 > 0 , such that E ( x , y ) [ ‖∇ ` ( y , f ( w ; x ) ) −∇FS ( w ) ‖2 ] = σ2 . ( ii ) FS ( w ) is smooth with an L-Lipchitz continuous gradient , i.e. , it is differentiable and there exists a constant L > 0 such that ‖∇FS ( w ) −∇FS ( u ) ‖ ≤ L‖w − u‖ , ∀w , u ∈ W. Remark . Assumption 1 ( i ) and ( ii ) are commonly used assumptions in the literature of non-convex optimization ( Ghadimi & Lan , 2013 ; Yan et al. , 2018 ; Yuan et al. , 2019b ; Wang et al. , 2019 ; Li et al. , 2020a ) . Assumption 1 ( ii ) says the objective function is L-smooth , and it has an equivalent expression ( Nesterov , 2004 ) which is FS ( w ) −FS ( u ) ≤ 〈∇FS ( u ) , w−u〉+L2 ‖w−u‖ 2 , ∀w , u ∈ W. For a classification problem , the smoothed label yLS is given by yLS = ( 1− θ ) y + θŷ , ( 3 ) where θ ∈ ( 0 , 1 ) is the smoothing strength , y is the one-hot label , ŷ is an introduced label . For example , one can simply use ŷ = 1K ( Szegedy et al. , 2016 ) for K-class problems . Similar to label y , we suppose the label ŷ is drawn from a distribution Pŷ . We introduce the variance of stochastic gradient using label ŷ as follows . E ( x , ŷ ) [ ‖∇ ` ( ŷ , f ( w ; x ) ) −∇FS ( w ) ‖2 ] = σ̂2 : = δσ2 . ( 4 ) where δ > 0 is a constant and σ2 is defined in Assumption 1 ( i ) . We make several remarks for ( 4 ) . Remark . ( a ) We do not require that the stochastic gradient∇ ` ( ŷ , f ( w ; x ) ) is unbiased , i.e. , it could be E [ ∇ ` ( ŷ , f ( w ; x ) ) ] 6= ∇FS ( w ) . ( b ) The variance σ̂2 is defined based on the label ŷ rather than the smoothed label yLS . ( c ) We do not assume the variance σ̂2 is bounded since δ could be an arbitrary value , however , we will discuss the different cases of δ in our analysis . If δ ≥ 1 , then σ̂2 ≥ σ2 ; while if 0 < δ < 1 , then σ̂2 < σ2 . It is worth mentioning that δ could be small when an appropriate label is used in the label smoothing . For example , one can smooth labels by using a teacher model ( Hinton et al. , 2014 ) or the model ’ s own distribution ( Reed et al. , 2014 ) . In the first paper of label smoothing ( Szegedy et al. , 2016 ) and the following related studies ( Müller et al. , 2019 ; Yuan et al. , 2019a ) , researchers consider a uniform distribution over all K classes of labels as the label ŷ , i.e. , set ŷ = 1K . Due to the space limitation , we include more discussions and the evaluations of δ in real-world applications in Appendix A . For example , we show that δ < 1 for CIFAR-100 in practice . We now introduce an important property regarding FS ( w ) , i.e . Polyak-Łojasiewicz ( PL ) condition ( Polyak , 1963 ) . More specifically , the following assumption holds . Assumption 2 . There exists a constant µ > 0 such that 2µ ( FS ( w ) − FS ( w∗S ) ) ≤ ‖∇FS ( w ) ‖2 , ∀w ∈ W , where w∗S ∈ minw∈W FS ( w ) is a optimal solution . Remark . This property has been theoretically and empirically observed in training deep neural networks ( Allen-Zhu et al. , 2019 ; Yuan et al. , 2019b ) . This condition is widely used to establish convergence in the literature of non-convex optimization , please see ( Yuan et al. , 2019b ; Wang et al. , 2019 ; Karimi et al. , 2016 ; Li & Li , 2018 ; Charles & Papailiopoulos , 2018 ; Li et al. , 2020a ) and references therein . To measure the convergence of non-convex and smooth optimization problems as in ( Nesterov , 1998 ; Ghadimi & Lan , 2013 ; Yan et al. , 2018 ) , we need the following definition of the first-order stationary point . Definition 1 ( First-order stationary point ) . For the problem of minw∈W FS ( w ) , a point w ∈ W is called a first-order stationary point if ‖∇FS ( w ) ‖ = 0 . Moreover , if ‖∇FS ( w ) ‖ ≤ , then the point w is said to be an -stationary point , where ∈ ( 0 , 1 ) is a small positive value . Algorithm 1 SGD with Label Smoothing Regularization 1 : Initialize : w0 ∈ W , θ ∈ ( 0 , 1 ) , set η as the value in Theorem 3 . 2 : for t = 0 , 1 , . . . , T − 1 do 3 : sample ( xt , yt ) , set yLSt = ( 1− θ ) yt + θŷt 4 : update wt+1 = wt − η∇w ` ( yLSt , f ( wt ; xt ) ) 5 : end for 6 : Output : wR , where R is uniformly sampled from { 0 , 1 , . . . , T − 1 } .

Summary: This paper studies the power of Label smoothing regularization (LSR) in training deep neural networks by analyzing SGD with LSR in different non-convex optimization settings. The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence. Besides, this paper proposes a simple yet efficient two-stage label smooth (TSLA) whose basic idea is to switch the training from smoothed label to one-hot label. Integrating TSLA with SGD can produce the improved convergence result that TSLA benefits from LSR in the first stage and essentially converges faster in the second stage. Extensive experiments show that TSLA improves the generalization accuracy of deep models on several benchmark datasets.