Year: 2021

Abstract:
Label smoothing regularization (LSR) is a prevalent component for training deep neural networks and can improve the generalization of models effectively. Although it achieves empirical success, the theoretical understanding about the power of label smoothing, especially about its influence on optimization, is still limited. In this work, we, for the first time, theoretically analyze the convergence behaviors of stochastic gradient descent with label smoothing in deep learning. Our analysis indicates that an appropriate LSR can speed up the convergence by reducing the variance in gradient, which provides a theoretical interpretation on the effectiveness of LSR. Besides, the analysis implies that LSR may slow down the convergence at the end of optimization. Therefore, a novel algorithm, namely Two-Stage LAbel smoothing (TSLA), is proposed to further improve the convergence. With the extensive analysis and experiments on benchmark data sets, the effectiveness of TSLA is verified both theoretically and empirically.

heading: 1 INTRODUCTION
text: Due to the massive number of parameters, over-fitting becomes one of the major challenges in training effective deep neural networks (Müller et al., 2019). Various strategies such as weight decay, dropout (Hinton et al., 2012b), batch normalization (Ioffe & Szegedy, 2015), data augmentation (Simard et al., 1998), etc. have been proposed to alleviate the problem. Unlike regularization on input data or model weights, label smoothing regularization (LSR) (Szegedy et al., 2016) perturbs labels of data for better generalization. Concretely, for a K-class classification problem, the one-hot label in the prevalent cross entropy loss is replaced by a soft label as yLS = (1− θ)y+ θŷ, where y is the one-hot label and ŷ = 1K is a uniform distribution for all labels. By optimizing the soft label instead, LSR helps to improve the generalization of deep neural networks in diverse applications including image classification (Zoph et al., 2018; He et al., 2019), speech recognition (Chorowski & Jaitly, 2017; Zeyer et al., 2018), and language translation (Vaswani et al., 2017; Nguyen & Salazar, 2019). Besides, it is complementary to other regularization methods and can work with others to make learned neural networks applicable (He et al., 2019).
Despite the success of LSR, efforts toward theoretical understanding are still limited. Müller et al. (2019) have empirically shown that LSR can help improve model calibration. However, they also found that LSR could impair knowledge distillation when the learned model is applied as the teacher network. Yuan et al. (2019a) have proved that LSR provides a virtual teacher model for knowledge distillation. As a widely used trick, Lukasik et al. (2020) have shown that LSR works since it can successfully mitigate label noise. Most of existing works try to study LSR as regularizing label space while little work investigates the direct impact of LSR on training deep neural networks.
In this work, we analyze the benefits of LSR for the learning process from the view of optimization. Concretely, our theory shows that an appropriate LSR can essentially reduce the variance of stochastic gradient in the assigned class labels and thus it can speed up the convergence of stochastic gradient descent, which is a standard optimizer for deep learning. Besides, the analysis implies that LSR may incur slow convergence at the end of optimization. Inspired by the result, a novel algorithm is developed to accelerate the convergence by a two-stage training paradigm. The effectiveness of the proposed method is verified both theoretically and empirically. We summarize the main contributions of this paper as follows.
• To the best of our knowledge, it is the first work that establishes improved iteration complexities of stochastic gradient descent (SGD) (Robbins & Monro, 1951) with LSR for
finding an -approximate stationary point (Definition 1) in solving a smooth non-convex problem in the presence of an appropriate label smoothing. The results theoretically explain why an appropriate LSR can help speed up the convergence. (Subsection 4.1)
• According to the analysis, we propose a simple yet effective method to accelerate the convergence of LSR (Subsection 4.2). It adopts a two-stage training paradigm that applies LSR in the first stage and then runs on one-hot labels in the second stage.
• The theoretical analysis demonstrates that TSLA has an improved iteration complexity (Subsection 4.3) and better generalization (Subsection 4.4) compared to the conventional LSR. Extensive experiments on benchmark data sets confirm the effectiveness of our proposed method.

heading: 2 RELATED WORK
text: In this section, we introduce some related work. A closely related idea to LSR is confidence penalty proposed by Pereyra et al. (2017), an output regularizer that penalizes confident output distributions by adding its negative entropy to the negative log-likelihood during the training process. The authors (Pereyra et al., 2017) presented extensive experimental results in training deep neural networks to demonstrate better generalization comparing to baselines with only focusing on the existing hyper-parameters. They have shown that LSR is equivalent to confidence penalty with a reversing direction of KL divergence between uniform distributions and the output distributions.
DisturbLabel introduced by Xie et al. (2016) imposes the regularization within the loss layer, where it randomly replaces some of the ground truth labels as incorrect values at each training iteration. Its effect is quite similar to LSR that can help to prevent the neural network training from overfitting. The authors have verified the effectiveness of DisturbLabel via several experiments on training image classification tasks.
Recently, many works (Zhang et al., 2018; Bagherinezhad et al., 2018; Goibert & Dohmatob, 2019; Shen et al., 2019; Li et al., 2020b) explored the idea of LSR technique. Ding et al. (2019) extended an adaptive label regularization method, which enables the neural network to use both correctness and incorrectness during training. Pang et al. (2018) used the reverse cross-entropy loss to smooth the classifier’s gradients. Wang et al. (2020) proposed a graduated label smoothing method that uses the higher smoothing penalty for high-confidence predictions than that for low-confidence predictions. They found that the proposed method can improve both inference calibration and translation performance for neural machine translation models. By contrast, we will try to understand the power of LSR from an optimization perspective and try to study how and when to use LSR.

heading: 3 PRELIMINARIES AND NOTATIONS
text: We first present some notations. Let∇wFS(w) denote the gradient of a function FS(w). When the variable to be taken a gradient is obvious, we use∇FS(w) for simplicity. We use ‖ · ‖ to denote the Euclidean norm. Let 〈·, ·〉 be the inner product. In classification problem, we aim to seek a classifier to map an example x ∈ X onto one of K labels y ∈ Y ⊂ RK , where y = (y1, y2, . . . , yK) is a one-hot label, meaning that yi is “1” for the correct class and “0” for the rest. Suppose the example-label pairs are drawn from a distribution P, i.e., (x,y) ∼ P = (Px,Py). Let S = {(x1,y1), . . . , (xn,yn)} denotes a set of n examples drawn from P. We denote by E(x,y)[·] the expectation that takes over a random variable (x,y). When the randomness is obvious, we write E[·] for simplicity. Our goal is to learn a prediction function f(w;x) :W ×X → RK that is as close as possible to y, where w ∈ W is the parameter andW is a closed convex set. To this end, we want to minimize the following expected loss under P:
min w∈W
FS(w) := 1
n n∑ i=1 `(yi, f(w;xi)) (1)
where ` : Y × RK → R+ is a cross-entropy loss function given by
`(y, f(w;x)) = K∑ i=1 −yi log ( exp(fi(w;x))∑K j=1 exp(fj(w;x)) ) . (2)
The objective function FS(w) is not convex since f(w;x) is non-convex in terms of w. To solve the problem (1), one can simply use some iterative methods such as stochastic gradient descent (SGD). Specifically, at each training iteration t, SGD updates solutions iteratively by wt+1 = wt − η∇w`(yt, f(wt;xt)), where η > 0 is a learning rate. Next, we present some notations and assumptions that will be used in the convergence analysis. Throughout this paper, we also make the following assumptions for solving the problem (1). Assumption 1. Assume the following conditions hold:
(i) The stochastic gradient of FS(w) is unbiased, i.e., E(x,y)[∇`(y, f(w;x))] = ∇FS(w), and the variance of stochastic gradient is bounded, i.e., there exists a constant σ2 > 0, such that E(x,y) [ ‖∇`(y, f(w;x))−∇FS(w)‖2 ] = σ2.
(ii) FS(w) is smooth with an L-Lipchitz continuous gradient, i.e., it is differentiable and there exists a constant L > 0 such that ‖∇FS(w)−∇FS(u)‖ ≤ L‖w − u‖,∀w,u ∈ W.
Remark. Assumption 1 (i) and (ii) are commonly used assumptions in the literature of non-convex optimization (Ghadimi & Lan, 2013; Yan et al., 2018; Yuan et al., 2019b; Wang et al., 2019; Li et al., 2020a). Assumption 1 (ii) says the objective function is L-smooth, and it has an equivalent expression (Nesterov, 2004) which is FS(w)−FS(u) ≤ 〈∇FS(u),w−u〉+L2 ‖w−u‖
2, ∀w,u ∈ W. For a classification problem, the smoothed label yLS is given by
yLS = (1− θ)y + θŷ, (3) where θ ∈ (0, 1) is the smoothing strength, y is the one-hot label, ŷ is an introduced label. For example, one can simply use ŷ = 1K (Szegedy et al., 2016) for K-class problems. Similar to label y, we suppose the label ŷ is drawn from a distribution Pŷ. We introduce the variance of stochastic gradient using label ŷ as follows.
E(x,ŷ) [ ‖∇`(ŷ, f(w;x))−∇FS(w)‖2 ] = σ̂2 := δσ2. (4)
where δ > 0 is a constant and σ2 is defined in Assumption 1 (i). We make several remarks for (4). Remark. (a) We do not require that the stochastic gradient∇`(ŷ, f(w;x)) is unbiased, i.e., it could be E[∇`(ŷ, f(w;x))] 6= ∇FS(w). (b) The variance σ̂2 is defined based on the label ŷ rather than the smoothed label yLS. (c) We do not assume the variance σ̂2 is bounded since δ could be an arbitrary value, however, we will discuss the different cases of δ in our analysis. If δ ≥ 1, then σ̂2 ≥ σ2; while if 0 < δ < 1, then σ̂2 < σ2. It is worth mentioning that δ could be small when an appropriate label is used in the label smoothing. For example, one can smooth labels by using a teacher model (Hinton et al., 2014) or the model’s own distribution (Reed et al., 2014). In the first paper of label smoothing (Szegedy et al., 2016) and the following related studies (Müller et al., 2019; Yuan et al., 2019a), researchers consider a uniform distribution over all K classes of labels as the label ŷ, i.e., set ŷ = 1K . Due to the space limitation, we include more discussions and the evaluations of δ in real-world applications in Appendix A. For example, we show that δ < 1 for CIFAR-100 in practice.
We now introduce an important property regarding FS(w), i.e. Polyak-Łojasiewicz (PL) condition (Polyak, 1963). More specifically, the following assumption holds. Assumption 2. There exists a constant µ > 0 such that 2µ(FS(w) − FS(w∗S)) ≤ ‖∇FS(w)‖2,∀w ∈ W, where w∗S ∈ minw∈W FS(w) is a optimal solution.
Remark. This property has been theoretically and empirically observed in training deep neural networks (Allen-Zhu et al., 2019; Yuan et al., 2019b). This condition is widely used to establish convergence in the literature of non-convex optimization, please see (Yuan et al., 2019b; Wang et al., 2019; Karimi et al., 2016; Li & Li, 2018; Charles & Papailiopoulos, 2018; Li et al., 2020a) and references therein.
To measure the convergence of non-convex and smooth optimization problems as in (Nesterov, 1998; Ghadimi & Lan, 2013; Yan et al., 2018), we need the following definition of the first-order stationary point. Definition 1 (First-order stationary point). For the problem of minw∈W FS(w), a point w ∈ W is called a first-order stationary point if ‖∇FS(w)‖ = 0. Moreover, if ‖∇FS(w)‖ ≤ , then the point w is said to be an -stationary point, where ∈ (0, 1) is a small positive value.
Algorithm 1 SGD with Label Smoothing Regularization 1: Initialize: w0 ∈ W , θ ∈ (0, 1), set η as the value in Theorem 3. 2: for t = 0, 1, . . . , T − 1 do 3: sample (xt,yt), set yLSt = (1− θ)yt + θŷt 4: update wt+1 = wt − η∇w`(yLSt , f(wt;xt)) 5: end for 6: Output: wR, where R is uniformly sampled from {0, 1, . . . , T − 1}.

heading: 4 TSLA: A TWO-STAGE LABEL SMOOTHING ALGORITHM
text: In this section, we present our main method with its convergence analysis. As a warm-up, we first show the convergence analysis of SGD with LSR to understand LS from theoretical side. Then we will introduce the proposed TSLA and study its convergence results both in training error (optimization) and testing error (generalization).

heading: 4.1 CONVERGENCE ANALYSIS OF SGD WITH LSR
text: To understand LSR from the optimization perspective, we consider SGD with LSR in Algorithm 1 for the sake of simplicity. The only difference between Algorithm 1 and standard SGD is the use of the output label for constructing a stochastic gradient. The following theorem shows that Algorithm 1 converges to an approximate stationary point in expectation under some conditions. We include its proof in Appendix C.
Theorem 3. Under Assumption 1, run Algorithm 1 with η = 1L and θ = 1 1+δ , then ER[‖∇FS(wR)‖2] ≤ 2FS(w0)ηT + 2δσ 2, where R is uniformly sampled from {0, 1, . . . , T − 1}. Furthermore, given a target accuracy level , we have the following two results. (1) when δ ≤ 2
4σ2 , if we set T = 4FS(w0) η 2 , then Algorithm 1 converges to an -stationary point in expectation, i.e., ER[‖∇FS(wR)‖2] ≤ 2. The total sample complexity is T = O ( 1 2 ) . (2) when δ > 2
4σ2 , if we set T = FS(w0) ηδσ2 , then Algorithm 1 does not converge to an -stationary
point, but we have ER[‖∇FS(wR)‖2] ≤ 4δσ2 ≤ O(δ).
Remark. We observe that the variance term is 2δσ2, instead of ηLσ2 for standard analysis of SGD without LSR (i.e., θ = 0, please see the detailed analysis of Theorem 6 in Appendix D). For the convergence analysis, the difference between SGD with LSR and SGD without LSR is that ∇`(ŷ, f(w;x)) is not an unbiased estimator of ∇FS(w) when using LSR. The convergence behavior of Algorithm 1 heavily depends on the parameter δ. When δ is small enough, say δ ≤ O( 2) with a small positive value ∈ (0, 1), then Algorithm 1 converges to an -stationary point with the total sample complexity of O ( 1 2 ) . Recall that the total sample complexity of standard
SGD without LSR for finding an -stationary point is O ( 1 4 ) ((Ghadimi & Lan, 2016; Ghadimi et al., 2016), please also see the detailed analysis of Theorem 6 in Appendix D). The convergence result shows that if we could find a label ŷ that has a reasonably small amount of δ, we will be able to reduce sample complexity for training a machine learning model from O ( 1 4 ) to O ( 1 2 ) . Thus, the reduction in variance will happen when an appropriate label smoothing with δ ∈ (0, 1) is introduced. We will find in the empirical evaluations that different label ŷ lead to different performances and an appropriate selection of label ŷ has a better performance (see the performances of LSR and LSR-pre in Table 2). On the other hand, when the parameter δ is large such that δ > Ω( 2), that is to say, if an inappropriate label smoothing is used, then Algorithm 1 does not converge to an -stationary point, but it converges to a worse level of O(δ), indicating that it may slow down the convergence at the end of optimization.

heading: 4.2 THE TSLA ALGORITHM
text: Previous subsection have shown that LSR could not find an −stationary point under some situations. These motivate us to investigate a strategy that combines the algorithm with and without LSR during the training progress. Let think in this way, one possible scenario is that training one-hot
Algorithm 2 The TSLA algorithm 1: Initialize: w0 ∈ W , T1, θ ∈ (0, 1), η1, η2 > 0
// First stage: SGD with LSR 2: for t = 0, 1, . . . , T1 − 1 do 3: set yLSt = (1− θ)yt + θŷt 4: update wt+1 = wt − η1∇`(yLSt , f(wt;xt)) 5: end for
// Second stage: SGD without LSR 6: for t = T1, 1, . . . , T1 + T2 − 1 do 7: update wt+1 = wt − η2∇`(yt, f(wt;xt)). 8: end for 9: Output: wR, whereR is uniformly sampled from {T1, . . . , T1 + T2 − 1}. Figure 1: Loss on ResNet-18 over CUB-2011.
label could be “easier” than training smoothed label 1. Taking the cross entropy loss in (2) for an example, one need to optimize a single loss function− log ( exp(fk(w;x))/ ∑K j=1 exp(fj(w;x)) ) when one-hot label (e.g, yk = 1 and yi = 0 for all i 6= k) is used, but need to optimize all K loss functions − ∑K i=1 y LS i log ( exp(fi(w;x))/ ∑K j=1 exp(fj(w;x)) ) when smoothed label (e.g., yLS = (1− θ)y + θ 1K so that y LS k = 1− (K − 1)θ/K and yLSi = θ/K for all i 6= k) is used. Nevertheless, training deep neural networks is gradually focusing on hard examples with the increase of training epochs. It seems that training with smoothed label in the late epochs makes the learning progress more difficult. In addition, after LSR, we focus on optimizing the overall distribution that contains the minor classes, which are probably not important at the end of training progress. Please see the blue dashed line (trained by SGD with LSR) in Figure 12, and it shows that there is almost no change in loss after 30 epochs. One question is whether LSR helps at the early training epochs but it has less (even negative) effect during the later training epochs? This question encourages us to propose and analyze a simple strategy with LSR dropping that switches a stochastic algorithm with LSR to the algorithm without LSR.
A natural choice could be just dropping off LSR after certain epochs. For example, one can run SGD with LSR in the first 50 epochs and then run SGD without LSR after that (see red solid line in
Figure 1). From Figure 1, we can see that the drop of LSR after 50 epochs can further decrease the loss function.
With this in mind in this subsection, we propose a generic framework that consists of two stages, wherein the first stage it runs SGD with LSR for T1 iterations and the second stage it runs SGD without LSR up to T2 iterations. This framework is referred to as Two-Stage LAbel smoothing (TSLA) algorithm, whose updating details are presented in Algorithm 2. Although SGD is considered as the pipeline (Line 4 and Line 8 in Algorithm 2) in the convergence analysis, in practice, SGD can be replaced by any stochastic algorithms such as momentum SGD (Polyak, 1964), Stochastic Nesterov’s Accelerated Gradient (Nesterov, 1983), and adaptive algorithms including ADAGRAD (Duchi et al., 2011), RMSProp (Hinton et al., 2012a), AdaDelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), Nadam (Dozat, 2016) and AMSGRAD (Reddi et al., 2018). In this paper, we will not study the theoretical guarantees and empirical evaluations of other optimizers, which can be considered as future work. Please note that the algorithm can use different learning rates η1 and η2 during the two stages. The last solution of the first stage will be used as the initial solution of the second stage. If T1 = 0, then TSLA reduces to the baseline, i.e., SGD without LSR; while if T2 = 0, TSLA becomes to LSR method, i.e., SGD with LSR.
1While this is not a rigorous mathematical conclusion, we want to roughly show that optimizing loss of soft label has to consider minimizing the combination of K functions while one-hot loss only needs to focus on one. For soft label, each fj appears both in numerator (need to maximize) and denominator (need to minimize) of the loss function, while for one-hot label, fj , j 6= k only appears in denominator (need to minimize) of the loss function, indicating that one-hot label has less “constraints” than soft label.
2The blue dashed line is trained by SGD with LSR. The red solid line is trained by SGD with LSR for the first 50 epochs and trained by SGD without LSR (i.e., using one-hot label) for the last 40 epochs.

heading: 4.3 OPTIMIZATION RESULT OF TSLA
text: In this subsection, we will give the convergence result of the proposed TSLA algorithm. For simplicity, we use SGD as the subroutine algorithm A in the analysis. The convergence result in the following theorem shows the power of LSR from the optimization perspective. Its proof is presented in Appendix E. It is easy to see from the proof that by using the last output of the first stage as the initial point of the second stage, TSLA can enjoy the advantage of LSR in the second stage with an improved convergence. Theorem 4. Under Assumptions 1, 2, suppose σ2δ/µ ≤ F (w0), run Algorithm 2, θ = 11+δ , η1 = 1 L , T1 = log ( 2µF (w0)(1+δ) 2δσ2 ) /(η1µ), η2 = 2 2Lσ2 and T2 = 8δσ2 µη2 2 , then ER[‖∇FS(wR)‖2] ≤ 2, where R is uniformly sampled from {T1, . . . , T1 + T2 − 1}.
Remark. It is obvious that the learning rate η2 in the second stage is roughly smaller than the learning rate η1 in the first stage, which matches the widely used stage-wise learning rate decay scheme in training neural networks. To explore the total sample complexity of TSLA, we consider different conditions on δ. For a fixed the target convergence level ∈ (0, 1), let us discuss the total sample complexities of finding -stationary points for SGD with TSLA (TSLA), SGD with LSR (LSR), and SGD without LSR (baseline), where we only consider the orders of the complexities but ignore all constants. When Ω( 2) < δ < 1, LSR does not converge to an -stationary point, while TSLA reduces sample complexity from O ( 1 4 ) to O ( δ 4 ) , compared to the baseline. When δ < O( 2), the total complexity of TSLA is between log(1/ ) and 1/ 2, which is always better than LSR and the baseline. In summary, TSLA achieves the best total sample complexity by enjoying the good property of an appropriate label smoothing (i.e., when 0 < δ < 1). However, when δ ≥ 1, baseline has better convergence than TSLA, meaning that the selection of label ŷ is not appropriate. Since T1 contains unknown parameters, it is difficulty to know its ground-truth value. However, we can tune different values in practice.

heading: 4.4 GENERALIZATION RESULT OF TSLA
text: In this subsection, we study the generalization result of TSLA. First, we give some notations, where most of them are followed by (Hardt et al., 2015; Yuan et al., 2019b). Let wS = A(S) be a solution that generated by a random algorithm A based on dataset S. Recall that problem (1) is called empirical risk minimization in the literature, and the true risk minimization is given by
min w∈W F (w) := E(x,y) [`(y, f(w;x))] . (5)
Then the testing error is dedined as
EA,S [F (wS)]− ES [FS(w∗S)]. (6) We notice that there are several works (Hardt et al., 2015; Yuan et al., 2019b) study the testing error result of SGD for non-convex setting under different conditions such as bounded stochastic gradient ‖∇w`(y, f(w;x))‖ ≤ G and decaying learning rate ηt ≤ ct with a constant c > 0, where t is the optimization iteration. In this paper, we are not interested in establishing fast rate under different conditions, but we want to explore the generalization ability of TSLA with the fewest possible modifications when building a bridge between theory and practice. For example, weight decay is a widely used trick when training deep neural networks. With the use of weight decay, the empirical risk minimization in practice becomes minw∈W{F̂S(w) := FS(w) + λ2 ‖w‖
2}. We present the ERB of TSLA in the following Theorem 5, whose proofs can be found in the Appendix F. Theorem 5. Under Assumption 1, assume that `(y, f(w,x)) is L-smooth andB-Lipschitz, suppose F̂S(w) satisfies Assumption 2 and minw∈W F̂S(w) ≤ FS(w∗S) + λ2 ‖wt‖
2 with λ = 2L, where wt is the intermediate solution in the second stage of Algorithm 2 by running with θ = 11+δ , η1 < 1 3L , T1 = log ( 2µFS(w0)(1+δ) 2δσ2 ) /(η1µ), η2 = O(1/ √ n) and T2 = O(δn) then the testing error ER,A,S [F (wR)]− ES [FS(w∗S)] ≤ O(1/ √ n), where A is TSLA. Remark. Theorem 5 shows that in order to have an bound of testing error in the order of O(1/ √ n), TSLA needs to run T = O(δn) iterations. For the standard analysis of SGD with/without LSR (please see the remarks in Appendix F) under the same conditions, SGD without LSR needs to run T = O(n) iterations to obtain an O(1/ √ n) bound of testing error. Thus, if δ is small enough, the
testing error of TSLA is better than that of SGD without LSR, otherwise, SGD without LSR has better testing error than TSLA. For SGD with LSR, when δ > Ω(1/ √ n), the bound of testing error for LSR can not be bounded by O(1/ √ n); when δ ≤ Ω(1/ √ n), the bound for LSR can be bounded by O(1/ √ n) in T = O( √ n) iterations. This shows that TSLA bas better testing error than LSR.

heading: 5 EXPERIMENTS
text: To further evaluate the performance of the proposed TSLA method, we trained deep neural networks on three benchmark data sets, CIFAR-100 (Krizhevsky & Hinton, 2009), Stanford Dogs (Khosla et al., 2011) and CUB-2011 (Wah et al., 2011), for image classification tasks. CIFAR-100 3 has 50,000 training images and 10,000 testing images of 32×32 resolution with 100 classes. Stanford Dogs data set 4 contains 20,580 images of 120 breeds of dogs, where 100 images from each breed is used for training. CUB-2011 5 is a birds image data set with 11,788 images of 200 birds species. The ResNet-18 model (He et al., 2016) is applied as the backbone in the experiments.
We compare the proposed TSLA incorporating with SGD (TSLA) with two baselines, SGD with LSR (LSR) and SGD without LSR (baseline). The mini-batch size of training instances for all methods is 256 as suggested by (He et al., 2019; 2016). The momentum parameter is fixed as 0.9. We will include more details of experimental settings in Appendix A.

heading: 5.1 STANFORD DOGS AND CUB-2011
text: We separately train ResNet-18 (He et al., 2016) up to 90 epochs over two data sets Stanford Dogs and CUB-2011. We use weight decay with the parameter value of 10−4. For all algorithms, the initial learning rates for FC are set to be 0.1, while that for the pre-trained backbones are 0.001 and 0.01 for Standford Dogs and CUB-2011, respectively. The learning rates are divided by 10 every 30 epochs. For LSR, we fix the value of smoothing strength θ = 0.4 for the best performance, and the label ŷ used for label smoothing is set to be a uniform distribution over all K classes, i.e., ŷ = 1K . The same values of the smoothing strength θ and the same ŷ are used during the first stage of TSLA. For TSLA, we drop off the LSR (i.e., let θ = 0) after s epochs during the training process, where s ∈ {20, 30, 40, 50, 60, 70, 80}. We first report the highest top-1 and top-5 accuracy on the testing data sets for different methods. All top-1 and top-5 accuracy are averaged over 5 independent random trails with their standard deviations. The results of the comparison are summarized in Table 1, where the notation “TSLA(s)” means that the TSLA algorithm drops off LSR after epoch s. It can be seen from Table 1 that under an appropriate hyperparameter setting the models trained using TSLA outperform that trained using LSR and baseline, which supports the convergence result in Section 4.2. We notice that the best top-1 accuracy of TSLA are TSLA(40) and TSLA(50) for Stanford Dogs and CUB-2011,
3https://www.cs.toronto.edu/˜kriz/cifar.html 4http://vision.stanford.edu/aditya86/ImageNetDogs/ 5http://www.vision.caltech.edu/visipedia/CUB-200.html
Stanford Dogs
Stanford Dogs
Stanford Dogs
respectively, meaning that the performance of TSLA(s) is not monotonic over the dropping epoch s. For CUB-2011, the top-1 accuracy of TSLA(20) is smaller than that of LSR. This result matches the convergence analysis of TSLA showing that it can not drop off LSR too early. For top-5 accuracy, we found that TSLA(80) is slightly worse than baseline. This is because of dropping LSR too late so that the update iterations (i.e., T2) in the second stage of TSLA is too small to converge to a good solution. We also observe that LSR is better than baseline regarding top-1 accuracy but the result is opposite as to top-5 accuracy.
We then plot the averaged top-1 accuracy, averaged top-5 accuracy, and averaged loss among 5 trails of different methods in Figure 2. We remove the results for TSLA(20) since it dropped off LSR too early as mentioned before. The figure shows TSLA improves the top-1 and top-5 testing accuracy immediately once it drops off LSR. Although TSLA may not converge if it drops off LSR too late, see TSLA(60), TSLA(70), and TSLA(80) from the third column of Figure 2, it still has the best performance compared to LSR and baseline. TSLA(30), TSLA(40), and TSLA(50) can converge to lower objective levels, comparing to LSR and baseline.

heading: 5.2 CIFAR-100
text: The total epochs of training ResNet-18 (He et al., 2016) on CIFAR-100 is set to be 200.The weight decay with the parameter value of 5 × 10−4 is used. We use 0.1 as the initial learning rates for all algorithms and divide them by 10 every 60 epochs as suggested in (He et al., 2016; Zagoruyko & Komodakis, 2016). For LSR and the first stage of TSLA, the value of smoothing strength θ is fixed as θ = 0.1, which shows the best performance for LSR. We use two different labels ŷ to smooth the
one-hot label, the uniform distribution over all labels and the distribution predicted by an ImageNet pre-trained model which is downloaded directly from PyTorch 6 (Paszke et al., 2019). For TSLA, we try to drop off the LSR after s epochs during the training process, where s ∈ {120, 140, 160, 180}.
CIFAR-100 Algorithm∗ Top-1 accuracy Top-5 accuracy baseline 76.87 ± 0.04 93.47 ± 0.15 LSR 77.77 ± 0.18 93.55 ± 0.11 TSLA(120) 77.92 ± 0.21 94.13 ± 0.23 TSLA(140) 77.93 ± 0.19 94.11 ± 0.22 TSLA(160) 77.96 ± 0.20 94.19 ± 0.21 TSLA(180) 78.04 ± 0.27 94.23 ± 0.15 LSR-pre 78.07 ± 0.31 94.70 ± 0.14 TSLA-pre(120) 78.34 ± 0.31 94.68 ± 0.14 TSLA-pre(140) 78.39 ± 0.25 94.73 ± 0.11 TSLA-pre(160) 78.55 ± 0.28 94.83 ± 0.08 TSLA-pre(180) 78.53 ± 0.23 94.96 ± 0.23
∗TSLA(s)/TSLA-pre(s): TSLA/TSLA-pre drops off LSR/LSR-pre after epoch s.
tion of the prediction function ŷ used for smoothing label is the key to the success of TSLA as well as LSR. Among all methods, the performance of TSLA-pre is the best. For top-1 accuracy, TSLA-pre(160) outperforms all other algorithms, while for top-5 accuracy, TSLA-pre(180) has the best performance.
Finally, we observe from Figure 3 that both TSLA and TSLA-pre converge, while TSLA-pre converges to the lowest objective value. Similarly, the top-1 and top-5 accuracies show the improvements of TSLA and TSLA-pre at the point of dropping off LSR.
We conduct an ablation study for the smoothing parameter θ and the dropping epoch s in TSLA(s). We follow the same settings in Subsection 5.2 but use different values of θ in LSR and TSLA. Specifically, θ ∈ {0.2, 0.4, 0.9}. We use the uniform distribution over all labels to smooth the one-hot label. The results are summarized in Table 3, showing that the different values of θ and s can affect the performances of LSR and TSLA. With the increase of θ, the performances become worse. However, with different values of θ, TSLA can always outperform LSR when an appropriate dropping epoch s is selected. Besides, TSLA(180) has the best performance for each value of θ.

heading: 6 CONCLUSIONS
text: In this paper, we have studied the power of LSR in training deep neural networks by analyzing SGD with LSR in different non-convex optimization settings. The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence. We have proposed a simple yet efficient strategy so-called TSLA whose basic idea is to switch the training from smoothed label to one-hot label. Integrating TSLA with SGD, we observe from its improved convergence result that TSLA benefits from LSR in the first stage and essentially converges faster in the second stage. Our theoretical result also shows that TSLA has better testing error than that of LSR. Throughout extensive experiments, we have shown that TSLA improves the generalization accuracy of deep models on several benchmark data sets.
6https://pytorch.org/docs/stable/torchvision/models.html

heading: A ADDITIONAL EXPERIMENTS
text: All experiments are implemented using PyTorch7 with version 1.7.1 on a machine with 8 NVIDIA P100 GPUs. The code is available at
https://drive.google.com/drive/folders/1QLvSE6iB2YN7UBpRMKB_ kKQGZzK7M4Qr?usp=sharing
A.1 THE ESTIMATION OF δ
As mentioned in previsous sections, the value of δ plays a very important role on the theoretical results. In this section, we provide the estimation of δ for three data sets Stanford Dogs, CUB-2011 and CIFAR-100 that used in the experiments. Our results empirically show that δ < 1 for all three data sets. Specifically, we computed the sample variances (denoted by σ2k and σ̂ 2 k, k = 10, 20, . . . ) of two variance σ2 and σ̂2 every 10 epochs, then based on the definitions in (4), the etimation (denoted by δ̃) of δ was taken as
δ̃ = max k σ̂2k σ2k .
All results of estimated δ are listed in Table 4, empirically showing that δ̃ < 1. We also plot the estimated values of δ across iterations over three different data sets in Figure 4, showing that δ̃ < 1 for all data sets across iterations.
A.2 MORE RESULTS ON DIFFERENT MODELS AND DATASETS
More experiments on different models and datasets are included in this subsection. First, we trained models including ResNet-50, VGG (Simonyan & Zisserman, 2014), MobileNet (Howard et al.,
7A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026–8037, 2019. https://pytorch.org
2017), Inception (Szegedy et al., 2015) and EfficientNet (Tan & Le, 2019) on three datasets CIFAR100, CUB-2011 and Stanford Dogs. We followed the same hyper-parameter setting as in the main paper, except for the smoothing strength θ in LSR. We tuned the parameter θ for LSR and used the one with best performance. For TSLA, the same values of smooth strength θ are used during the first stage of TSLA. The specific values of θ for different models and datasets are listed in Table 5. For the simplicity of presentation, we only report the best performance of TSLA among different drop epochs s, where the values of used hyper-parameter s are listed in Table 5.
All top-1 and top-5 accuracy on the testing datasets are summarized Tables 6, 7 , 8 for CUB-2011, Stanford Dogs, and CIFAR-100, respectively. Overall, the results show that TSLA has the best performance, comparing with baseline and LSR.
Besides, we train ResNet-50 over ImageNet with the total epochs of 180. The initial learning rate is set to be 0.2 for all algorithm and divide them by 10 every 60 epochs as suggested in (He et al., 2016; Zagoruyko & Komodakis, 2016). The mini-batch size of 512 is used. The momentum parameter and weight decay are set to be 0.9 and 10−4 respectively. We fix the smoothing strength θ = 0.1 and the drop epoch s = 120 for TSLA. The top-1 accuracy are presented in Table 9, showing that TSLA and LSR are comparable.

heading: B TECHNICAL LEMMA
text: Recall that the optimization problem is
min w∈W
FS(w) := 1
n n∑ i=1 `(yi, f(w;xi)) (7)
where the cross-entropy loss function ` is given by
`(y, f(w;x)) = K∑ i=1 −yi log ( exp(fi(w;x))∑K j=1 exp(fj(w;x)) ) . (8)
If we set
p(w;x) = (p1(w;x), . . . , pK(w;x)) ∈ RK , pi(w;x) = − log ( exp(fi(w;x))∑K j=1 exp(fj(w;x)) ) , (9)
the problem (7) becomes
min w∈W
FS(w) := 1
n n∑ i=1 〈yi, p(w;xi)〉. (10)
Then the stochastic gradient with respective to w is
∇`(y, f(w;x)) = 〈y,∇p(w;x)〉. (11) Lemma 1. Under Assumption 1 (i), we have
E [∥∥∇`(yLSt , f(wt;xt))−∇FS(wt)∥∥2] ≤ (1− θ)σ2 + θδσ2.
Proof. By the facts of yLSt = (1− θ)yt + θŷt and the equation in (11), we have
∇`(yLSt , f(wt;xt)) = (1− θ)∇`(yt, f(wt;xt)) + θ∇`(ŷt, f(wt;xt)). Therefore,
E [∥∥∇`(yLSt , f(wt;xt))−∇FS(wt)∥∥2]
=E [ ‖(1− θ)[∇`(yt, f(wt;xt))−∇FS(wt)] + θ[∇`(ŷt, f(wt;xt))−∇FS(wt)]‖2 ] (a)
≤ (1− θ)E [ ‖∇`(yt, f(wt;xt))−∇FS(wt)‖2 ] + θE [ ‖∇`(ŷt, f(wt;xt))−∇FS(wt)‖2 ] (b) ≤(1− θ)σ2 + θδσ2,
where (a) uses the convexity of norm, i.e., ‖(1 − θ)a + θb‖2 ≤ (1 − θ)‖a‖2 + θ‖b‖2; (b) uses assumption 1 (i) and the definitions in (4), and Assumption 1 (i).

heading: C PROOF OF THEOREM 3
text: Proof. By the smoothness of objective function F (w) in Assumption 1 (ii) and its remark, we have
FS(wt+1)− FS(wt)
≤〈∇FS(wt),wt+1 −wt〉+ L
2 ‖wt+1 −wt‖2
(a) = − η 〈 ∇FS(wt),∇`(yLSt , f(wt;xt)) 〉 + η2L
2 ∥∥∇`(yLSt , f(wt;xt))∥∥2 (b) = − η
2 ‖∇FS(wt)‖2 +
η
2 ∥∥∇FS(wt)−∇`(yLSt , f(wt;xt))∥∥2 + η(ηL− 1)2 ∥∥∇`(yLSt , f(wt;xt))∥∥2 (c) ≤ − η 2 ‖∇FS(wt)‖2 + η 2
∥∥∇FS(wt)−∇`(yLSt , f(wt;xt))∥∥2 , (12) where (a) is due to the update of wt+1; (b) is due to 〈a,−b〉 = 12 ( ‖a− b‖2 − ‖a‖2 − ‖b‖2 ) ; (c) is due to η ≤ 1L . Taking the expectation over (xt,y LS t ) on the both sides of (12), we have
E [FS(wt+1)− FS(wt)]
≤− η 2
E [ ‖∇FS(wt)‖2 ] + η 2 E [∥∥∇FS(wt)−∇`(yLSt , f(wt;xt))∥∥2]
≤− η 2
E [ ‖∇FS(wt)‖2 ] + η
2
( (1− θ)σ2 + θδσ2 ) . (13)
where the last inequality is due to Lemma 1. Then inequality (13) implies
1
T T−1∑ t=0 E [ ‖∇FS(wt)‖2 ] ≤2FS(w0) ηT + (1− θ)σ2 + θδσ2
(a) =
2FS(w0)
ηT +
2δ
1 + δ σ2
(b) ≤ 2FS(w0) ηT + 2δσ2,
where (a) is due to θ = 11+δ ; (b) is due to 1 1+δ ≤ 1.
D CONVERGENCE ANALYSIS OF SGD WITHOUT LSR (θ = 0)
Theorem 6. Under Assumption 1, the solutions wt from Algorithm 1 with θ = 0 satisfy
1
T T−1∑ t=0 E [ ‖∇FS(wt)‖2 ] ≤ 2FS(w0) ηT + ηLσ2.
In order to have ER[‖∇FS(wR)‖2] ≤ 2, it suffices to set η = min ( 1 L , 2 2Lσ2 ) and T = 4FS(w0)η 2 ,
the total complexity is O ( 1 4 ) .
Proof. By the smoothness of objective function FS(w) in Assumption 1 (ii) and its remark, we have
FS(wt+1)− FS(wt)
≤〈∇FS(wt),wt+1 −wt〉+ L
2 ‖wt+1 −wt‖2
(a) = − η 〈∇FS(wt),∇`(yt, f(wt;xt))〉+
η2L
2 ‖∇`(yt, f(wt;xt))‖2 , (14)
where (a) is due to the update of wt+1. Taking the expectation over (xt;yt) on the both sides of (14), we have
E [FS(wt+1)− FS(wt)] (a) ≤ − ηE [ ‖∇FS(wt)‖2 ] + η2L 2 E [ ‖∇`(yt, f(wt;xt))−∇FS(wt) +∇FS(wt)‖2 ] (b) = − ηE [ ‖∇FS(wt)‖2 ] + η2L 2 E [ ‖∇`(yt, f(wt;xt))−∇FS(wt)‖2 ] + η2L 2 E [ ‖∇FS(wt)‖2
] (c) ≤ − η 2 E [ ‖∇FS(wt)‖2 ] + η2L 2 σ2. (15)
where (a) and (b) use Assumption 1 (i); (c) uses the facts that η ≤ 1L and Assumption 1 (i). The inequality (15) implies
1
T T−1∑ t=0 E [ ‖∇FS(wt)‖2 ] ≤ 2FS(w0) ηT + ηLσ2.
By setting η ≤ 2 2Lσ2 and T = 4FS(w0) η 2 , we have 1 T ∑T−1 t=0 E [ ‖∇FS(wt)‖2 ] ≤ 2. Thus the total
complexity is in the order of O (
1 η 2
) = O ( 1 4 ) .

heading: E PROOF OF THEOREM 4
text: Proof. Following the similar analysis of inequality (13) from the proof of Theorem 3, we have
E [FS(wt+1)− FS(wt)]
≤− η1 2
E [ ‖∇FS(wt)‖2 ] + η1 2 ( (1− θ)σ2 + θδσ2 ) . (16)
Using the condition in Assumption 2 we can simplify the inequality from (16) as
E [FS(wt+1)− FS(w∗S)]
≤(1− η1µ)E [FS(wt)− FS(w∗S)] + η1 2
( (1− θ)σ2 + θδσ2 ) ≤ (1− η1µ)t+1 E [FS(w0)− FS(w∗S)] +
η1 2
( (1− θ)σ2 + θδσ2 ) t∑ i=0 (1− η1µ)i
≤ (1− η1µ)t+1 E [FS(w0)] + η1 2
( (1− θ)σ2 + θδσ2 ) t∑ i=0 (1− η1µ)i ,
where the last inequality is due to the definition of loss function that FS(w∗S) ≥ 0. Since η1 ≤ 1L < 1 µ , then (1− η1µ) t+1 < exp(−η1µ(t + 1)) and ∑t i=0 (1− η1µ)
i ≤ 1η1µ . As a result, for any T1, we have
E [FS(wT1)− FS(w∗S)] ≤ exp(−η1µT1)FS(w0) + 1
2µ
( (1− θ)σ2 + θδσ2 ) . (17)
Let θ = 11+δ and σ̂ 2 := (1 − θ)σ2 + θδσ2 = 2δ1+δσ 2 then 12µ ( (1− θ)σ2 + θδσ2 ) = δσ 2
µ(1+δ) ≤ FS(w0) since δ is small enough. By setting
T1 = log
( 2µFS(w0)
σ̂2
) /(η1µ)
we have
E [F (wT1)− FS(w∗S)] ≤ σ̂2 µ ≤ 2δσ 2 µ . (18)
After T1 iterations, we drop off the label smoothing, i.e. θ = 0, then we know for any t ≥ T1, following the inequality (15) from the proof of Theorem 6, we have
E [FS(wt+1)− FS(wt)] ≤− η2 2
E [ ‖∇FS(wt)‖2 ] + η22Lσ 2
2 .
Therefore, we get
1
T2 T1+T2−1∑ t=T1 E [ ‖∇FS(wt)‖2 ] ≤ 2 η2T2 E [FS(wT1)− FS(wT1+T2−1)] + η2Lσ2
(a) ≤ 2 η2T2 E [FS(wT1)− FS(w∗S)] + η2Lσ2
(18) ≤ 4δσ 2
µη2T2 + η2Lσ
2, (19)
where (a) is due to FS(wT1+T2−1) ≥ FS(w∗S). By setting η2 = 2 2Lσ2 and T2 = 8δσ2 µη2 2 , we have
1 T2 ∑T1+T2−1 t=T1 E [ ‖∇FS(wt)‖2 ] ≤ 2.

heading: F GENERALIZATION ANALYSIS
text: In this section, let
F̂S(w) =FS(w) + λ 2 ‖w‖2 = 1 n n∑ i=1 `(yi, f(w;xi)) + λ
2 ‖w‖2︸ ︷︷ ︸̂̀(yi,f(w;xi))
(20)
F̂ (w) =F (w) + λ
2 ‖w‖2 (21)
Following by Yuan et al. (2019b), we use the following decomposition of testing error.
EA,S [F (wS)]− ES [FS(w∗S)] ≤ ES [EA[FS(wS)− FS(w∗S)]] + EA,S [F (wS)− FS(wS)]. (22)
Let define the gradient update rule Ĝ̀,η as follows Ĝ̀,η(w) = w − η∇w ̂̀(y, f(w,x)). (23)
Then we have the following lemma, which is similar to Lemma 2.5 and Lemma 4.2 in Hardt et al. (2015). Lemma 2. Assume that `(y, f(w,x)) is L-smooth and B-Lipschitz. Let wt+1 = G(wt) and w′t+1 = G′(w′t), then
‖wt+1 −w′t+1‖ = {
(1 + ηL− ηλ)‖wt −w′t‖, G = G′, (1− ηλ)‖wt −w′t‖+ 2ηB, G 6= G′.
Proof. Recall that ̂̀(yi, f(w;xi)) = `(yi, f(w;xi)) + λ2 ‖w‖2. Then (1) When G = G′ and `(·, f(w; ·)) is L-smooth, then
‖wt+1 −w′t+1‖ ≤(1− ηλ)‖wt −w′t‖+ η‖∇w`(yt, f(wt;xt))−∇w`(yt, f(w′t;xt))‖ ≤(1 + ηL− ηλ)‖wt −w′t‖.
(2) When G 6= G′ and `(·, f(w; ·)) is B-Lipschitz, then
‖wt+1 −w′t+1‖ ≤(1− ηλ)‖wt −w′t‖+ η‖∇w`(yt, f(wt;xt))−∇w`(y′t, f(w′t;x′t))‖ ≤(1− ηλ)‖wt −w′t‖+ 2ηB.
F.1 PROOF OF THEOREM 5
Proof. By Theorem 3.2 of Hardt et al. (2015), we have ∆t+1 :=E[‖wt+1 −w′t+1‖]
(a) ≤ (
1− 1 n
) (1 + ηL− ηλ)∆t + 1
n ((1− ηλ)∆t + 2ηtB)
= [ 1 + ( 1− 1
n
) ηL− ηλ ] ∆t + 2ηB
n
(b) ≤(1− ηL)∆t + 2ηB
n
(c) ≤ 2ηB n t0∑ i=0 (1− ηL)i (d) ≤ 2B nL , (24)
where (a) uses Lemma 2; (b) uses λ = 2L; (c) uses ∆t0 = 0; (d) uses ηL < 1. Then by Lemma 3.11 of Hardt et al. (2015) and ̂̀is L+ λ-smooth, we have E[̂̀(wt; z)− ̂̀(w′t; z)] ≤ t0n + 2B(L+ λ)nL ≤ 6B + 1n , (25) where the last inequality holds by selecting t0 = 1 and λ = 2L. Then by Theorem 2.2 of Hardt et al. (2015)
EA,S [F̂ (wt)− F̂S(wt)] ≤ 6B + 1
n , (26)
which is the generalization error. Next, we will bound the optimization error. Since F̂S is (L+ λ)smooth and satisfies µ-PL condition, then follow the similar analysis of Theorem 4, we have
1
T2 T1+T2−1∑ t=T1 ES [EA[F̂S(wt)− F̂S(ŵ∗S)]] ≤ 1 2µT2 T1+T2−1∑ t=T1 E [ ‖∇F̂S(wt)‖2 ] ≤ 2δσ 2
µ2η2T2 + η2(L+ λ)σ
2
2µ , (27)
Then by (22), (26) and (27), we get
1
T2 T1+T2−1∑ t=T1 EA,S [F̂ (wt)]− ES [F̂S(ŵ∗S)] ≤ 2δσ2 µ2η2T2 + η2(L+ λ)σ 2 2µ + 6B + 1 n . (28)

heading: By using the conditions that minw∈W F̂S(w) ≤ FS(w∗S) + λ2 ‖wt‖
text: 2, definitions (20) and (21), we get F (wt)− FS(w∗S) ≤ F̂ (wt)− F̂S(ŵ∗S). (29) Therefore, we have the following inequality by (28) and (29):
1
T2 T1+T2−1∑ t=T1 EA,S [F (wt)]− ES [FS(w∗S)] ≤ 2δσ2 µ2η2T2 + η2(L+ λ)σ 2 2µ + 6B + 1 n . (30)
By setting η2 = O(1/ √ n) and T2 = O(δn) then the testing error ER,A,S [F (wR)]−ES [FS(w∗S)] ≤
O(1/ √ n), where A is TSLA.
Remark 1. (SGD without LSR) Following the similar analysis of Theorem 5 and similar analysis of Theorem 6, we have the testing error for SGD without LSR is
ER,A,S [F (wR)]− ES [FS(w∗S)] ≤ F̂S(w0) µηT + η(L+ λ)σ2 2µ + 6B + 1 n . (31)
By setting η = O(1/ √ n) and T = O(n) then ER,A,S [F (wR)]−ES [FS(w∗S)] ≤ O(1/ √ n), where A is SGD without LSR. Remark 2. (SGD with LSR) Following the similar analysis of Theorem 5 and similar analysis of Theorem 3, we have the testing error for SGD with LSR is
ER,A,S [F (wR)]− ES [FS(w∗S)] ≤ F̂S(w0) µηT + δσ2 µ + 6B + 1 n , (32)
where A is SGD with LSR. If δ > Ω(1/ √ n), the testing error can not be bounded by O(1/ √ n); if δ ≤ Ω(1/ √ n), the testing error can be bounded by O(1/ √ n) in T = O( √ n) iterations.

Review summaries:
This paper makes some analysis on the convergence behaviors of SGD with label smoothing in deep learning. Further, a two-stage label smoothing algorithm is proposed to improve the convergence. Experiments are conducted to verify its effectiveness.

This work theoretically analyzes the convergence behaviors of stochastic gradient descent with label smoothing in deep learning, which implies that LSR may slow down the convergence at the end of optimization. The authors propose the two-stage label smoothing strategy to further improve the convergence. Experiments on several datasets also verify the effectiveness of TSLA.

This paper proved that when performing stochastic gradient descent for a non-convex objective function, an appropriate smooth parameter might speed up the convergence. The authors then proceed to introduce a two-stage TSLA algorithm which firstly learns with LSR. Then switch to learn with hard labels after certain epochs. Empirical evidence demonstrate the effectiveness of the proposed algorithm.

This paper studies the power of Label smoothing regularization (LSR) in training deep neural networks by analyzing SGD with LSR in different non-convex optimization settings. The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence. Besides, this paper proposes a simple yet efficient two-stage label smooth (TSLA) whose basic idea is to switch the training from smoothed label to one-hot label. Integrating TSLA with SGD can produce the improved convergence result that TSLA benefits from LSR in the first stage and essentially converges faster in the second stage. Extensive experiments show that TSLA improves the generalization accuracy of deep models on several benchmark datasets.