Towards Understanding Label Smoothing
Authors: Yi Xu
Yuanhong Xu
Qi Qian
Hao Li
Rong Jin
Venue: ICLR 2022 Submitted (2022)

Abstract: Label smoothing regularization (LSR) is a prevalent component for training deep neural networks and can improve the generalization of models effectively. Although it achieves empirical success, the theoretical understanding about the power of label smoothing, especially about its influence on optimization, is still limited. In this work, we, for the first time, theoretically analyze the convergence behaviors of stochastic gradient descent with label smoothing in deep learning. Our analysis indicates that an appropriate LSR can speed up the convergence by reducing the variance in gradient, which provides a theoretical interpretation on the effectiveness of LSR. Besides, the analysis implies that LSR may slow down the convergence at the end of optimization. Therefore, a novel algorithm, namely Two-Stage LAbel smoothing (TSLA), is proposed to further improve the convergence. With the extensive analysis and experiments on benchmark data sets, the effectiveness of TSLA is verified both theoretically and empirically.

Under review as a conference paper at ICLR 2022
TOWARDS UNDERSTANDING LABEL SMOOTHING
Anonymous authors
Paper under double-blind review
ABSTRACT
Label smoothing regularization (LSR) is a prevalent component for training deep neural networks and can improve the generalization of models effectively. Al- though achieves empirical success, the theoretical understanding about the power of label smoothing, especially about its influence on optimization, is still limited. In this work, we, for the first time, theoretically analyze the convergence behaviors of stochastic gradient descent with label smoothing in deep learning. Our analysis indicates that an appropriate LSR can speed up the convergence by reducing the variance in gradient, which provides a theoretical interpretation on the effectiveness of LSR. Besides, the analysis implies that LSR may slow down the convergence at the end of optimization. Therefore, novel algorithm. namely Two-Stage LAbel smoothing (TSLA), is proposed to further improve the conver- gence. With the extensive analysis and experiments on benchmark data sets, the effectiveness of TSLA is verified both theoretically and empirically.
INTRODUCTION
Due the massive number of parameters, over-fitting becomes of the major challenges in training effective deep neural networks (Müller et al., 2019). Various strategies such as weight decay, dropout (Hinton et al., 2012b), batch normalization (Ioffe & Szegedy, 2015), data augmenta- tion (Simard et al., 1998), etc. have been proposed to alleviate the problem. Unlike regularization on input data or model weights, label smoothing regularization (LSR) (Szegedy al., 2016) perturbs labels of data for better generalization. Concretely, for K-class classification problem, the one-hot label in the prevalent cross entropy loss is replaced by a soft label as y (1 - O)y + 0ÿ, where y is uniform distribution for all labels. By optimizing the soft label
is the one-hot label and instead, LSR helps to improve the generalization of deep neural networks in diverse applications including image classification (Zoph et al., 2018; He et al., 2019), speech recognition (Chorowski & Jaitly, 2017; Zeyer et al., 2018), and language translation (Vaswani et al., 2017: 7; Nguyen & Salazar, 2019). Besides, it is complementary to other regularization methods and can work with others to make learned neural networks applicable (He et al., 2019).
Despite the success of LSR, efforts toward theoretical understanding still limited. Müller et al. (2019) have empirically shown that LSR can help improve model calibration. However, they also found that LSR could impair knowledge distillation when the learned model is applied the teacher network. Yuan et al. (2019a) have proved that LSR provides a virtual teacher model for knowledge distillation. As a widely used trick, Lukasik et al. (2020) have shown that LSR works since it can successfully mitigate label noise. Most of existing works try to study LSR as regularizing label space while little work investigates the direct impact of LSR on training deep neural networks.
In this work, we analyze the benefits of LSR for the learning process from the view of optimiza- tion. Concretely, our theory shows that an appropriate LSR can essentially reduce the variance of stochastic gradient in the assigned class labels and thus it can speed up the convergence of stochas- tic gradient descent, which a standard optimizer for deep learning. Besides, the analysis implies that LSR may incur slow convergence at the end of optimization. Inspired by the result, novel algorithm is developed to accelerate the convergence by a two-stage training paradigm. The effec- tiveness of the proposed method is verified both theoretically and empirically. We summarize the main contributions of this paper as follows.
To the best of our knowledge, is the first work that establishes improved iteration com- plexities of stochastic gradient descent (SGD) (Robbins & Monro, 1951) with LSR for

---

Under review as a conference paper at ICLR 2022
finding c-approximate stationary point (Definition 1) in solving smooth non-convex problem in the presence of an appropriate label smoothing. The results theoretically ex- plain why an appropriate LSR can help speed up the convergence. (Subsection 4.1)
According to the analysis, we propose a simple yet effective method to accelerate the con- vergence of LSR (Subsection 4.2). It adopts two-stage training paradigm that applies LSR in the first stage and then runs on one-hot labels in the second stage.
The theoretical analysis demonstrates that TSLA has an improved iteration complexity (Subsection 4.3) and better generalization (Subsection 4.4) compared to the conventional LSR. Extensive experiments on benchmark data sets confirm the effectiveness of our pro- posed method.
2 RELATED WORK
In this section, we introduce some related work. A closely related idea LSR is confidence penalty proposed by Pereyra et al. (2017), an output regularizer that penalizes confident output distribu- tions by adding its negative entropy to the negative log- likelikheood during the training process. The authors (Pereyra et al., 2017) presented extensive experimental results in training deep neural net- works to demonstrate better generalization comparing to baselines with only focusing on the existing hyper-parameters. They have shown that LSR is equivalent to confidence penalty with reversing direction of KL divergence between uniform distributions and the output distributions.
DisturbLabel introduced by Xie al. (2016) imposes the regularization within the loss layer, where it randomly replaces some of the ground truth labels as incorrect values at each training iteration. Its effect is quite similar LSR that help prevent the neural network training from overfit- ting. The authors have verified the effectiveness of Disturbl Label via several experiments on training image classification tasks.
Recently, many works (Zhang et al., 2018; Bagherinezhad et al., 2018; Goibert & Dohmatob, 2019; Shen et al., 2019; Li al., 2020b) explored the idea of LSR technique. Ding al. (2019) extended an adaptive label regularization method, which enables the neural network to use both correctness and incorrectness during training. Pang et al. (2018) used the reverse cross-entropy loss to smooth the classifier's 's gradients. Wang et al. (2020) proposed graduated label smoothing method that uses the higher smoothing penalty for high-confidence predictions than that for low-confidence predic- tions. They found that the proposed method can improve both inference calibration and translation performance for neural machine translation models. By contrast, we will try to understand the power of LSR from an optimization perspective and try to study how and when to use LSR.
3 PRELIMINARIES AND NOTATIONS
We first present some notations. Let V Fw) denote the gradient of a function Es(w). When the variable to be taken a gradient is obvious, we use (ss(w) for simplicity, We use - to denote the Euclidean norm. Let (---) be the inner product.
In classification problem, labels y E y C. R where y = (y1, y2. .... УK) is a one-hot label, meaning that yt is "I" for the correct class and "0" for the rest. Suppose the example-label pairs are drawn from distribution P, i.e, yx ~ = (IPx, Py). Let S = y1), (x,: yn)} denotes a set of n examples drawn from TP. We denote by Ex the expectation that takes over random variable yx When the randomness is obvious, xw; W x + IR that is as close as possible to y, where w E W is the parameter and W is closed convex set. To this end, we want minimize the following expected loss under IP:
aim to seek classifier to map example x X onto one of K
write Ei.] for simplicity. Our goal is learn a prediction function
min Fs(w) wEW
ely,, x:):
(1)
n
where (: R R, is cross-entropy loss function given by
exp(fi(w;x))
Ely, f(;x)() = Σ Yı log
(2)
Σ 1= exp(f, (w;x))
2

---

(PL) condi- Under review as a conference paper at ICLR 2022
The objective function Fs(w) ) is not convex since fw::)) is non-convex in terms of w. To solve the problem (1), one can simply use some iterative methods such as stochastic gradient descent (SGD). Specifically, at each training iteration t. SGD updates solutions iteratively by W++1 W. ( (W;; xt)), where > is a learning rate.
Next, present some notations and assumptions that will be used in the convergence analysis. Throughout this paper, we also make the following assumptions for solving the problem (1).
Assumption 1. Assume the following conditions hold:
(i) The stochastic gradient of Fs(w) is unbiased, i.e., Exgyi [(;; x))] (Fs(w). and the variance of stochastic gradient is bounded, i.e., there exists constant o² > 0, such that E(x.y) [ fo(:))((()))))) - VFs(w)ll =
(ii) F's ('sww) is smooth with an L-Lipchitz continuous gradient, i.e., it is differentiable and there exists constant L > O such that || [V()) - (V)( I L| u/I, Vw, u E W.
Remark. Assumption 1 (i) and (ii) are commonly used assumptions in the literature of non-convex optimization (Ghadimi & Lan, 2013; Yan et al., 2018; Yuan et al., 2019b; Wang et al., 2019; Li et al., 2020a). Assumption 1 (ii) says the objective function is L-smooth, and it has equivalent expression (Nesterov, 2004) which is
< (VFs w- w- fllwwul ull? Vw,u
W. For classification problem, the smoothed label y is given by
(3) where 0 (0, 1) is the smoothing strength, y is the one-hot label, y is an introduced label. For 1 (Szegedy et al., 2016) for K-class problems. Similar to label
yلs = (1 f)y + 08,
example, one can simply use y = K
y, we suppose the label y is drawn from distribution Iyy. We introduce the variance of stochastic gradient using label ý as follows.
IVV - VFF(()) = =² ====
E(x,))
(4)
where 8 >> 0 a constant and o² is defined in Assumption (i). We make several remarks for (4). Remark. (a) We do not require that the stochastic gradient Ve(y, f(w:x)) is unbiased, i.c., it could be EV,
* FFs(w). (b) The variance 6² is defined based on the label ý rather than the smoothed label yLS. (c) We do not assume the variance σ² is bounded since 5 could be an arbitrary value, however, we will discuss the different cases of 5 in our analysis. If 5 2 1, then 0² 2 0²; while if 0 8 < 1, then Ø² < 0². It is worth mentioning that 5 could be small when an appropriate label is used in the label smoothing. For example, one can smooth labels by using teacher model (Hinton et al., 2014) or the model's own distribution (Reed et al., 2014). In the first paper of label smoothing (Szegedy et al., 2016) and the following related studies (Müller et al., 2019: Yuan al., 2019a), researchers consider uniform distribution over all K classes of labels as the label y. i.e., set y = 1 evaluations of in real-world applications in Appendix A. For example, we show that 8 for CIFAR-100 in practice.
K- Due to the space limitation, we include more discussions and the
We now introduce an important property regarding Fs( i.e. tion (Polyak, 1963). More specifically, the following assumption holds.
Assumption 2. There exists a constant μ > 0 such that 2u(FF(() F's(w))) [[VFs(w)]I (ww Vww E W, where w's E minwew Fs(w) is optimal solution.
Remark. This property has been theoretically and empirically observed in training deep neural networks (Allen-Zhu et al., 2019; Yuan et al., 2019b). This condition is widely used to establish convergence in the literature of non-convex optimization, please see (Yuan et al., 2019b; Wang et al., 2019; Karimi et al., 2016; Li & Li, 2018; Charles & Papailiopoulos, 2018; Li et al., 2020a) and references therein.
To measure the convergence of non-convex and smooth optimization problems as in (Nesterov, 1998; Ghadimi & Lan, 2013; Yan et al., 2018), we need the following definition of the first-order stationary point.
Definition 1 (Fdrst-order stationary point). For the problem of min Fs(w), a point E W is called a first-order stationary point if | ([FFs()) = 0. Moreover. if (VVsF)) [w)] S c, then the point is said to be an e-stationary point, where e E (0, 1) is small positive value.
3

---

Under review as a conference paper at ICLR 2022
Algorithm SGD with Label Smoothing Regularization
1: Initialize: Wo E W, 0 E (0, 1), set as the value Theorem 3.
2: for ===0 0, 1,..., 7 1 do
3: sample Yt). set yt = +)y
4: update W1+1 Wt 17V w f((ex))))
5: end for
6: Output: WR, where R is uniformly sampled from (0, T
TSLA: TWO-STAGE LABEL SMOOTHING ALGORITHM
In this section, we present our main method with its convergence analysis. As a warm-up, we first show the convergence analysis of SGD with LSR understand LS from theoretical side. Then will introduce the proposed TSLA and study its convergence results both in training error (optimiza- tion) and testing error (generalization).
4.1 CONVERGENCE ANALYSIS OF SGD WITH LSR
To understand LSR from the optimization perspective, we consider SGD with LSR in Algorithm for the sake of simplicity. The only difference between Algorithm and standard SGD is the of the output label for constructing a stochastic gradient. The following theorem shows that Al- gorithm converges to an approximate stationary point expectation under some conditions. We include its proof in Appendix C.
Theorem 3. Under Assumption 1, Algorithm 1 with n F < 2Fs(wo) + 280², where R is uniformly sampled from 10, 1).
1 and
It5 then
nil'
Furthermore, given target accuracy level we have the following two results.
(1) when 8 S 1Gz. we set T 4Fs(wo), then Algorishm 1 converges an e-stationary point in expectation, i.e.,
qe²
F.VFFF(Sni < The total sample complexity is T = =0
e.
(2) when 8 > 20z. if we set Fs(wo), then Algorithu does not converge to an e-stationary point, but have F
ndo² 480² < {(6).
Remark. We observe that the variance term is 2502, instead of Lo for standard analysis of SGD without LSR (i.e., 8 = 0, please see the detailed analysis Theorem in Appendix D)- For the convergence analysis, the difference between SGD with LSR and SGD without LSR is that Ve(y, (w: x)) is not an unbiased estimator of VFs(w) when using LSR. The convergence behavior of Algorithm I heavily depends on the parameter 8. When ö is small enough, say 8 s O(e²) with a small positive value E (0, 1). then Algorithm converges to an e-stationary point with the total sample complexity of (2). Recall that the total sample complexity of standard SGD without LSR for finding an e-stationary point is O (1) (GGhadimi & Lan, 2016; Ghadimi Appendix D). The convergence
al., 2016), please also see the detailed analysis of Theorem result shows that if we could find label ý that has reasonably small amount of 6, we will be able to reduce sample complexity for training a machine learning model from O (+) to (+)) (+). Thus, the reduction in variance will happen when an appropriate label smoothing with (0. 1) is introduced. We will find in the empirical evaluations that different label ý lead to different performances and an appropriate selection of label ý has better performance (see the performances of L.R and LSR-pre in Table 2). On the other hand, when the parameter 8 is large such that 5 > 2 that is to say, an inappropriate label smoothing is used, then Algorithm does not converge to an e-stationary point, but it converges to worse level of O()), indicating that it may slow down the convergence at the end of optimization.
4.2 THE TSLA ALGORITIM
Previous subsection have shown that LSR could not find e-stationary point under some situa- tions. These motivate us to investigate a strategy that combines the algorithm with and without LSR during the training progress. Let think in this way, possible scenario is that training one-hot
4

---

that the drop of LSR after 50 epochs can further decrease the Under review as a conference paper at ICLR 2022
Algorithm 2 The TSLA algorithm
Initialize: Wo E W, T1, f E (0, 1), m1. 7/2 > 0
3.0
// First stage: SGD with LSR
2: for (. 1,- , 7]
do
2.5
3: set y
(1 o)y+ 0ÿ,
Loss
4: update Wi+1 = W; 71 f((:))
end for
1.5
// Second stage: SGD without LSR
for Tı, 1........77 T2 do
LSR
1.0
7: update W = w, 1)2 Ve(y,, f(w,;
10
20
70
80
end for
# epoch
9:
Output: WR. where R is uniformly sampled from (T....., T2 1}.
Figure 1: Loss on ResNet-18 over CUB-2011.
label could be "easier" than training smoothed label Taking the cross entropy loss (2) for an example, one need to optimize a single loss funetion log (exp(ff(w::)) Σ exp(f)(w:x)) when one-hot label (e.g, = 1 and y; = 0 for all + k) is used, but need to optimize all K loss functions - -I y log (ex((f((;;)))) x)) ΣØ= e (w; when smoothed label (e.g., y.s.- = f)y + 0 K so that yk ertheless, training deep neural networks is gradually focusing of training epochs. It seems that training with smoothed label in the late epochs makes the learning progress
y/= = (K K and y! = 8 K for all i + k) is used. Nev- hard examples with the increase
difficult. In addition, after LSR, we focus on optimizing the overall distribution that contains the minor classes, which are probably not important at the end of training progress. Please see the blue dashed line (trained by SGD with LSR) Figure 12, and it shows that there is almost no change in loss after 30 epochs. One question is whether LSR helps at the early training epochs but it has less (even negative) effect during the later training epochs? This question encourages us to propose and analyze simple strategy with LSR dropping that switches stochastic algorithm with LSR to the algorithm without LSR.
A natural choice could be just dropping off LSR after certain epochs. For example, one can run SGD with LSR in the first 50 epochs and then run SGD without LSR after that (see red solid line in Figure 1). From Figure 1, loss function.
With this in mind in this subsection, we propose a generic framework that consists of two stages, wherein the first stage it rums SGD with LSR for Ti iterations and the second stage it runs SGD with- out LSR up T3 iterations. This framework is referred to Two-Stage LAbel smoothing (TSLA) algorithm, whose updating details are presented in Algorithm 2. Although SGD is considered as the pipeline (Line 4 and Line 8 in Algorithm 2) in the convergence analysis, in practice, SGD can be placed by any stochastic algorithms such as momentum SGD (Polyak, 1964), Stochastic Nesterov's Accelerated Gradient (Nesterov, 1983), and adaptive algorithms including ADAGRAD (Duchi al., 2011), RMSProp (Hinton et al., 2012a), AdaDelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), Nadam (Dozat, 2016) and AMSGRAD (Reddi et al., 2018). In this paper, we will not study the the- oretical guarantees and empirical evaluations of other optimizers, which can be considered as future work. Please note that the algorithm can use different learning rates ? and 1)2 during the two stages. The last solution of the first stage will be used as the initial solution of the second stage. If Ti = 0. then TSLA reduces to the baseline, i.c., SGD without LSR; while if 2 = 0, TSLA becomes to LSR method, i.c., SGD with LSR.
Whiie this is not a rigorous mathematical conclusion, we want to roughly show that optimizing loss of soft label has to consider minimizing the combination of K functions while one-hot loss only needs to focus on one. For soft label, each fa appears both in numerator (need maximize) and denominator (need to minimize) of the loss function, while for one-hot lahel, fz. * le only appears in denominator (need to minimize) of the loss function, indicating that one-hot label has less "constraints" than soft label.
"The blue dashed line is trained by SGD with LSR. The red solid line is trained by SGD with ISRR for the first 50 epochs and trained by SGD without LR (i.e., using one-hot label) for the last 40 epochs.
5

---

Under review as a conference paper at ICLR 2022
4.3 OPTIMIZATION RESULT OF TSLA
In this subsection, we will give the convergence result of the proposed TSLA algorithm. For sim- plicity, we use SGD as the subroutine algorithm A in the analysis. The convergence result in the following theorem shows the power of LSR from the optimization perspective. Its proof is presented in Appendix E. It is easy to see from the proof that by using the last output of the first stage as the initial point of the second stage, TSLA can enjoy the advantage of LSR in the second stage with an improved convergence.
Theorem 4. Under Assumptions 1. 2, suppose o² 0²8/p S F(wo). run Algorithm 2, 0 I+6' T1 2
86#2 +. Ti = log (2uF(wo) /mpp), 122 = 2Lo and T2 pagez then E S e 200²
where R uniformly sampled {Ti....... Ti + T2 1}.
Remark. It is obvious that the learning rate 7/2 in the second stage is roughly smaller than the learning rate 77 in the first stage, which matches the widely used stage-wise learning rate decay scheme in training neural networks. To explore the total sample complexity of TSLA, we consider different conditions on 8. For a fixed the target convergence level E (0, 1). let us discuss the total sample complexities of finding e-stationary points for SGD with TSLA (TSLA), SGD with LSR (LSR), and SGD without LSR (baseline), where we only consider the orders of the complexities but ignore all constants. When ((+²) < 1, LSR does not converge an e-stationary point, while TSLA reduces sample complexity from O (2:) O (2), compared to the baseline. When 8 < (Ee)) the total complexity of TSLA is between log(1/e) and 1/e², which is always better than LSR and the baseline. In summary, TSLA achieves the best total sample complexity by enjoying the good property of an appropriate label smoothing (i.e., when 0 8 < 1). However, when 6 2 1, baseline has better convergence than TSLA, meaning that the selection of label ý is not appropriate. Since Ti contains unknown parameters, is difficulty to know its ground -ruth value. However, we can tune different values in practice.
4.4 GENERALIZATION RESULT OF TSLA
In this subsection, we study the generalization result of TSLA. First, we give some notations, where most of them are followed by (Hardt et al., 2015; Yuan et al., 2019b). Let ws = A(S) be solution that generated by random algorithm A based on dataset S. Recall that problem (1) is called empirical risk minimization in the literature, and the true risk minimization is given by
min F(w) == Exx) [, f(w;)))] wEW
(5)
Then the testing error is dedined as
EAs((F(ws)] - Ews)ss
(6)
We notice that there are several works (Hardt al., 2015: Yuan et al., 2019b) study the testing error result of SGD for non-conven setting under different conditions such as bounded stochastic gradient [[V f(w; x))l I and decaying learning rate 7): S with a constant e >> O, where f the optimization iteration. In this paper, we are not interested in establishing fast rate under different conditions, but we want to explore the generalization ability of TSLA with the fewest possible modifications when building a bridge between theory and practice. For example, weight decay is widely used trick when training deep neural networks. With the use of weight decay, the empirical
risk minimization in practice becomes min { [w)()) == Fs(w) + 2 [ We present the ERB of TSLA in the following Theorem 5, whose proofs can be found in the Appendix F.
A
Theorem 5. Under Assumption 1, assume that E(y, [(w, x)) is L-smooth and B-Lipschitz, suppose (w(w) satisfies Assumption 2 and minwew Fs(w) S Fs(w3)) + _ Well? with X = 2L, where we is the intermediate solution in the second stage of Algorithm 2 by running with 0 = I+8 n < SE Ti log 2yFss (1 ((HH), 122 ((1//)) and T2 O(ôn) then the testing error ERSAS [( BR.AA.(F(wRR) Es Fs (w3)) O(I/VT), where A is TSLA.
1
26
Remark. Theorem shows that in order to have an bound of testing error in the order of ((1/v)) TSLA needs to run T (6n) iterations. For the standard analysis of SGD with/without LSR (please see the remarks in Appendix F) under the same conditions, SGD without LSR needs to run T O(n) iterations to obtain an (11//n) bound of testing emror. Thus, if d is small enough, the

---

Under review as a conference paper at ICLR 2022
Table Comparison of Testing Accuracy for Different Methods (mean standard deviation, in %).
Stanford Dogs
CUB-2011
Algorithm*
Top-1 accuracy
Top-5 accuracy
Top-1 accuracy
Top-5 accuracy
baseline
82.31 + 0.18
97.76 + 0,06
75.31 = 0.25
LSR
82.80 = 0.07
97.41 + 0.09
76.97 = 0.19
92.73 - 0.12
T2LA(20)
83.15 + 0.02
97.91 + 0.08
76.62 + 0.15
93.60 + 0.18
TSLA(30)
83.89 1 0.16
98.05 + 0.08
93.92 1:922:.00 0.16
(4LA(40)
83.93
98.03 0.05
77:50
TSLA(50)
+.15
98.07 + 0.06
77.57=0.3
TSLA(60)
83.51 + 0.11
97.99 0.06
77.25 + 0.29
TSLA(70)
83.38 + 0.09
97.90
+ 0.09
77.21
+ 0.15
93.31 #
TSLA(80)
83.14 + 0.09
97.73 0.07
77.05 + 0.14
93.05 + 0.08
'TSLA(s): TSLA drops off LSR after epoch
testing error of TSLA is better than that of SGD without LSR, otherwise, SGD without LSR has the bound of testing error
better testing error than TSLA. For SGD with LSR, when > for LSR can not be bounded by O(L/v/): when 6 S 12(1) 22(1/v/r) /r), the bound for LSR can be bounded by 0(1/ in = OV) iterations. This shows that TSLA bas better testing error than LSR.
5 EXPERIMENTS
To further evaluate the performance of the proposed TSLA method, we trained deep neural networks on three benchmark data sets, CIFAR-100 (Krizhevsky & Hinton, 2009), Stanford Dogs (Khosla et al., 2011) and CUB-2011 (Wah et al., 2011), for image classification tasks. CIFAR-100 has 50,000 training images and 10,000 testing images of 32x32 resolution with 100 classes. Stanford Dogs data set contains 20,580 images of 120 breeds of dogs, where 100 images from each breed is used for training. CUB-2011 1 is birds image data set with 1 11,788 images of 200 birds species. The ResNet-18 model (He al., 2016) applied as the backbone in the experiments.
We compare the proposed TSLA incorporating with SGD (TSLA) with two baselines, SGD with LSR (LSR) and SGD without LSR (baseline). The mini-batch size of training instances for all methods is 256 as suggested by (He et al., 2019; 2016). The momentum parameter is fixed as 0.9. We will include more details of experimental settings in Appendix A.
5.1 STANFORD DOGS AND CUB-2011
We separately train ResNet-18 (He et al., 2016) up to 90 epochs over two data sets Stanford Dogs and CUB-2011. We weight decay with the parameter value of 10-4, For all algorithms, the initial learning rates for FC are set to be 0.1, while that for the pre-trained backbones are 0.001 and 0.01 for Standford Dogs and CUB-2011, respectively. The learning rates divided by 10 every 30 epochs. For LSR, we fix the value of smoothing strength = 0.4 for the best performance, and the label used for label smoothing is set to be uniform distribution over all K classes, i.e., ÿ T. The same values of the smoothing strength and the same ý are used during the first stage of TSLA. For TSLA, we drop off the LSR (i.e., let 0) after epochs during the training process, where E {20, 30, 40, 50, 60, 70, 80}.
We first report the highest top-1 and top-5 accuracy on the testing data sets for different methods. All top-1 and top-5 accuracy are averaged over 5 independent random trails with their standard deviations. The results of the comparison are summarized in Table where the notation "TSLA(s)" means that the TSLA algorithm drops off LSR after epoch It can be seen from Table that under an appropriate hyperparameter setting the models trained using TSLA outperform that trained using LSR and baseline, which supports the convergence result in Section 4.2. We notice that the best top-1 accuracy of TSLA are TSLA(40) and TSLA(50) for Stanford Dogs and CUB-2011,
'http://visinn htt.:riiiiinssntooooooeoe htw.:aatttttaoooeeeeedddidaae weweeddd//daaeee tya86/ ImageNet:ogs// htm1
hhttp://ww.rrsiion. cal
7

---

Under review as a conference paper at ICLR 2022
Starlord Dogs
Stanfoodd Dogs
Stanford Dogs
54
83-
TSLA
Teo Accuracy
62-
02
1704
0.0-
TAA (4)
1546 341
0.6-
80
CUB 2011
CUB 2011
CUB 2011
TILA DVS
1516-199 1750
3.0
TSLA
Tepa Accurccy
5 Acuursy
2.5
TSL IVG
beseine
76-
93.
3 2.0
1.5
TRA
T5-A 2011
TNLA
12.A123)
LSA
1.0
50
60
70
20
40
60
80
usech
Figure 2: Testing 1o Top-5 Accuracy and Loss on ResNet-18 over Stanford Dogs and CUB- 2011. TSLA(s) means TSLA drops off LSR after epoch
respectively, meaning that the performance of TSLA(s) is not monotonic over the dropping epoch For CUB-2011, the top-1 accuracy of TSLA(20) is smaller than that LSR. This result matches the convergence analysis of TSLA showing that it can not drop off LSR too early. For top-5 accuracy, we found that TSLA(80) is slightly worse than baseline. This is because of dropping LSR too late so that the update iterations (i.e., T2) in the second stage of TSLA is too small to converge to good solution. We also observe that LSR is better than baseline regarding top-1 accuracy but the result is opposite to top-5 accuracy.
We then plot the averaged top-1 accuracy, averaged top-5 accuracy, and averaged loss among 5 trails of different methods in Figure 2. We remove the results for TSLA(20) since it dropped off LSR too early as mentioned before. The figure shows TSLA improves the top-1 and top-5 testing accuracy immediately once drops off LSR. Although TSLA may not converge if it drops off LSR too late, see TSLA(60), TSLA(70), and TSLA(80) from the third column of Figure 2, it still has the best performance compared to LSR and baseline. TSLA(30), TSLA(40), and TSLA(50) can converge to lower objective levels, comparing to LSR and baseline.
5.2
CIFAR-100
cffr100
difrrDD
cifer 100
2.0
1.8
TELA (385)
TSAA (382)
27-
Tag Accurd
94-
1.6
LSX
3
76
TSLA (160)
T5Uppe (46))
TELA (260)
TSUkpr (44))
TSLA 2001
1504 pn 1340)
75-
beadine
LS
beandine
T5LAvrr 0.00
120
150
100
140
75
100
Figure 3: Testing Top- Top-5 Accuracy and I Loss on ResNet-18 over CIFAR-100. TSLA())TSIA- pre(s) mcansTSLA/TSLA pre drops off LSR/LSR-pre after epoch s.
The total epochs of training ResNet-18 (He et al., 2016) on CIFAR-100 is set to be 200.The weight decay with the parameter value of x 10 - is used. We use 0.1 as the initial learning rates for all algorithms and divide them by 10 every 60 epochs as suggested in (He al., 2016; Zagoruyko & Komodakis, 2016). For LSR and the first stage of TSLA, the value of smoothing strength is fixed as 0 0.1, which shows the best performance for LSR. We use two different labels y to smooth the
8

---

Under review as a conference paper at ICLR 2022
one-hot label, the uniform distribution over all labels and the distribution predicted by ImageNet pre-trained model which is downloaded directly from PyTorch 6 (Paszke et al., 2019). For TSLA, we try to drop off the LSR after epochs during the training process, where s {120, 140, 160, 180).
All top-1 and top-5 accuracy on the test- ing data are averaged over 5 inddpen- dent random trails with their standard deviations. We summarize the results in Table 2, where LSR-pre and TSLA- pre indicate LSR and TSLA use the la- bel y based on the ImageNet pre-trained model. The results show that LSRR- pre/TSLA-pre has a better performance than LSR/TSLA. The reason might be that the pre-trained model-based pre- diction is closer to the ground truth than the uniform prediction and it has lower variance (smaller 6). Then, TSLA (LSR) with such pre-trained model- based prediction converges faster than TSLA (LSR) with uniform prediction, which verifies our theoretical findings in Section 4.2 (Section 4.1). This observa- tion also empirically tells us the selec- tion of the prediction function y used for smoothing label is the key to the success of TSLA as well as LSR. Among all methods, the performance of TSLA-pre is the best. For top-1 accuracy, TSLA-pree(10) outperforms all other algorithms, while for top-5 accuracy, TSLA-pre(188) has the best performance.
Table 2:
Comparison of Testing Accuracy for Different Methods (mean + standard deviation, in %).
CIFAR- 100
Algorithm"
Top-1 accuracy
Top-5 accuracy
baseline
76.87 + 0.04
93,47 + 0.15
LSR
77.77 + 0.18
93.55 + 0.11
TSLA(120)
77.92 + 0.21
94.13 = 0.23
TSLA(140)
77.93
94.11 + 0.22
TSLA(160)
77.96 + 0.20
94.19 + 0.21
TSLA(180)
78.04
94.23 0.15
LSR-pre
78.07 0.31
94.70 0.14
TSLA-pre(1200
78.34:00
94.68 + 0.14
TSLA-pre(140)
78.39 1.3990220 0.25
94.73 + 0.11
PSLA-pre(160)
78.55 + 0.28
94.83 0.08
TSLA-pre(180)
78.53 + 0.23
94.96 + 0.23
TSLA/TSLA-pre
drops off
LSR/LSR- -re after epoch s.
Finally, we observe from Figure 3 that both TSLA and TSLA-pre converge, while TSLA-pra con- verges to the lowest objective value. Similarly, the top-1 and top-5 accuracies show the improve- ments of TSLA and TSLA-pre at the point of dropping off LSR.
We conduct ablation study for the smoothing pa- rameter 9 and the dropping epoch s in TSLA(s). We follow the same settings in Subsection 5.2 but use different values of in LSR and TSLA. Specifically, 0 {0.2, 0.4, 0.9). We use the uniform distribution over all labels to smooth the one-hot label. The results are summarized in Table 3, showing that the different values of 0 and s can affect the performances of LSR and TSLA. With the increase of 6, the performances become worse. However, with different values of 0, TSLA can always outperform LSR when an appropri- ate dropping epoch is selected. Besides, TSLA(180) has the best performance for each value 0.
Table 3: Comparison of Testing Top-1 1 Ac- curacy for Different A of LSR and TSLA (in %).
0
0.2
0.4
0.9
LSR
77.75
77.72
76.40
TSLA(120)
77.92
77.68
76.05
TSLA(140)
78.06
77.61
76.27
TSLA(160)
78.09
77.54
76.37
TSLA(180)
78.10
77.89
76.60
TSLA(ss): TSLA drops off L.SR after epoch
6 CONCLUSIONS
In this paper, we have studied the power of LSR in training deep neural networks by analyzing SGD with LSR in different non-convex optimization settings. The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence. We have pro- posed simple yet efficient strategy so-called TSLA whose basic idea is switch the training from smoothed label to one-hot label. Integrating TSLA with SGD, we observe from its improved con- vergence result that TSLA benefits from LSR the first stage and essentially converges faster in the second stage. Our theoretical result also shows that TSLA has better testing error than that of LSR. Throughout extensive experiments, we have shown that TSLA improves the generalization accuracy of deep models on several benchmark data sets.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, pp. 242-252, 2019.
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi. Label refinery: Improving imagenet classification through label progression. arXiv preprint 1rXiv:185.0006 2018.
Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In International Conference on Machine Learning, pp. 745-754, 2018.
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in sequence to sequence models. Proc. Interspeech 2017, pp. 523-527, 2017.
Qianggang Ding. Sifan Wu, Hao Sun, Jiadong Guo, and Shu- Tao Xia. Adaptive regularization of labels. arXiv preprint arXiv: porXiv:/908. 054744, 2019.
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and zeroth-order methods for nonconvex stochas- tic programming. SIAM Journal Optimization, 23(4):2341-234 2013.
Saeed Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming, Math. Program., 156(1-2):59 2016.
Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. Mini-batch stochastic approximation meth- ods for nonconvex stochastic composite optimization. Mathematical Programming, 155(1-2): 267-305, 2016.
Morgane Goibert and Elvis Dohmatob. Adversarial robustness via adversarial label-smoothing. arXiv preprint arXiv: 1567, 2019.
Moritz Hardt, Benjamin Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint 150.00250,00 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 558-567. 2019.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 2012a.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In NeurIPS Deep Learming Workshop. 2014.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing co-adaptation feature detectors. arXiv preprint arXiv:/207.000 2012b.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: /rXiv:/704.000 1704:04004 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1502.03502.03167 2015.
10

---

Under review as a conference paper at ICLR 2022
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the sojaak-tojssiwwizz condition. Joint European Conference on Ma- chine Learning and Knowledge Discovery in Databases, pp. 795-811 811. Springer, 2016.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In Proc. CVPR Workshop Fime-Grained Visual Categorization (FGVC), volume 2, 2011.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Mas- ter's thesis, Technical report, University of Tronto, 2009.
Xiaoyu Li, Zhenxun Zhuang, and Francesco Orabona. Exponential step sizes for non-convex opti- mization. arXiv preprint arXiv:2002.0527.. 2020a.
Xingjian Li, Haoyi Xiong. Haozhe An, Dejing Dou, and Chengzhong Xu. Colam: Co-
learning of deep neural networks and soft labels via alternating minimization. arXiv preprint arXiv:2004. 12443, 2020b.
Zhize Li and Jian Li. simple proximal stochastic gradient method for nonsmooth nonconvex optimization. In Advances in Neural Information Processing Systems, pp. 5564-5574, 2018.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? arXiv preprint trXiv:2003.02811 2020.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4696-4705, 2019.
Yurii Nesterov. A method of solving a convex programming problem with convergence rate O(1/k²). Soviet Mathematics Doklady, 27:372-376, 1983.
Yurii Nesterov. Introductory lectures on convex programming volume i: Basic course. 1998.
Yurii Nesterov. Introductory lectures on convex optimization: a basic course. Applied optimization.
Kluwer Academic Publ., 2004. ISBN 1-4020-7553-7.
Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv: 2019.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial exam- ples. In Advances in Neural Information Processing Systems, pp. 4579 4589, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, org/ docs/
pp. 8024-8035. 2019. URL models html.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint urXiv: 2017.
Boris Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com- putational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Boris Teodorovich Polyak. Gradient methods for minimizing functionals. Zhurnal Vychislitel noi Matematiki i Matematicheskoi Fiziki, 3(4))643-663 1963.
Sashank Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference Learning Representations, 2018.

---

Under review as a conference paper at ICLR 2022
Scott Reed, Honglak Lee, Dragomir Anguelov. Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks on noisy labels with bootstrapping. arXiv preprint arXiv:1412.6336, 2014.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cul statistics, pp. 400-407, 1951.
Chaomin Shen, Yaxin Peng, Guixu Zhang, and Jinsong Fan. Defending against adversarial attacks by suppressing the largest eigenvalue of fisher information matrix. arXiv preprint arXiv:1909.06137, 2019.
Patrice Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition- tangent distance and tangent propagation. In Neural networks: tricks of the Irade, pp. 239-274. Springer, 1998.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogmition. arXiv preprint aroiv 1556, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826 2016.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference Machine Learning, pp. 4114-6114. PMLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report ENS-TR-2011 2011-001, California Institute of Technology. 2011.
Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. On the inference calibration of neural machine translation. arxiv preprint 2005::200000006333 2020.
Zhe Wang, Kaiyi Ji, Zhou, Yinghin Liang, and Vahid Tarokh. Spiderboost and momentum: Faster variance reduction algorithms. In Advances in Neural Information Processing Systems, pp. 2403-2413, 2019.
Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, and Qi Tian. Disturblabel: Regularizing cnn on the loss layer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4753 4762, 2016.
Yan Yan, Tianbao Yang, 7he Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic momen- tum methods for deep learning. In International Joint Conference on Artificial Intelligence, pp. 2955-2961, 2018
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: a teacher-free framework. arXiv preprint arXiv: arXiv:1909, 10.0/117220 1723, 2019a.
Zhuoning Yuan, Yan Yan, Rong Jin, and Tianbao Yang. Stagewise training accelerates convergence of testing error over sgd. In Advances in Neural Information Processing Systems, pp. 2604-261 2019b.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Confer- ence, 2016.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv: 2012.
12

---

Under review as a conference paper at ICLR 2022
Albert Zeyer, Kazuki Irie, Ralf Schlüter, and Hermann Ney. Improved training of end-to-end atten- tion models for speech recognition. Proc. Interspeech 2018, pp. 7-11,2018
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoe V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697-8710, 2018
13

---

Under review as a conference paper at ICLR 2022
A ADDITIONAL EXPERIMENTS
All experiments are implemented using PyTorch' with version 1.7.1 on machine with 8 NVIDIA P100 GPUs. The code is available
A.1 THE ESTIMATION OF
As mentioned in previsous sections, the value plays a very important role on the theoretical results. In this section, provide the estimation of 5 for three data sets Stanford Dogs, CUB-200 UUB2011 and CIFAR-100 that used in the experiments. Our results empirically show that data sets. Specifically, ) of two variance a² and 6² every 10 epochs, then based on the definitions in (4), the etimation (denoted by ō) of 5 was taken as
for all three computed the sample variances (denoted of and o k 10, 20,
max
All results of estimated 8 are listed in Table 4, empirically showing that ă 1. We also plot the estimated values of 6 across iterations over three different data in Figure showing that ò for all data sets across iterations.
Table 4: The estimated value of å for different data sets
dataset
Stanford Dogs
CUB-2011
CIFAR-100
CIFAR-100 (pretrained)
8
0.16
0.13
0.55
0.38
value
value 5, CIFAR 100
0.30
0.6
0.25
0.5
0.20
0.4
0.15
0.3
0.10
0.05
CUB-2011
0.1
uniform
Stanford Dogs
pretrained
10
20
30
40
50
60
70
80
90
25
50
75
100
125
150
175
200
epoch
# epoch
Figure The estimated values of 5 for different data sets.
A.2 MORE RESULTS ON DIFFERENT MODELS AND DATASETS
More experiments on different models and datasets are included in this subsection. First, we trained models including ResNet-50, VGG (Simonyan & Zisserman, 2014), MobileNet (Howard et al.,
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32: 8026-8033 2019. https: /pttocc..otocoooooooooo
14

---

Under review as a conference paper at ICLR 2022
2017), Inception (Szegedy al., 2015) and EfficientNet (Tan & Le, 2019) three datasets CI- FAR CUB-2011 and Stanford Dogs. We followed the same pprem-err setting as in the main paper, except for the smoothing strength in LSR. We tuned the parameter for LSR and used the one with best performance. For TSLA, the same values of smooth strength are used during the first stage of TSLA. The specific values of 0 for different models and datasets are listed in Table 5. For the simplicity of presentation, we only report the best performance of TSLA among different drop epochs s, where the values of used syper-parametee are listed Table 5.
Table The hyper-paramettrs and for different models and datasets*
Model
CUB
Dogs
CIFAR
CUB
Dogs
CIFAR
ResNet-50
0.4
0.5
0.4
70
50
180
VGG
0.4
0.4
0.4
50
30
180
MobileNet v2
0.4
0.5
0.5
50
30
180
Inception v3
0.1
0.1
0.1
80
40
180
EfficientNet
0.1
0.1
0.3
20
30
180
8: TSLA drops off LSR after 8 epochs.
All top-I and top-5 accuracy on the testing datasets are summarized Tables 6, 7, 8 for CUB-2011, Stanford Dogs, and CIFAR-100, respectively. Overall, the results show that TSLA has the best performance, comparing with baseline and LSR.
Table 6: Comparison of Testing Accuracy (%) for Different Models on CUB-2011
Top-I
accuracy
Top-5 accuracy
Model
baseline
LSR
TSLA
baseline
LSR
TSLA
ResNet-50
80.15
82.76
82.83
94.87
95.08
95.55
VGG
80.58
81.14
81.71
95,46
95.31
95.62
MobileNet v2
78.44
79.91
80.14
94.58
94.39
94.94
Inception v3
79.08
80.12
80.19
94.63
95.24
95.19
EfficientNet 81.12
80.95
81.55
95.53
95.25
95.56
Table 7: Comparison of Testing Accuracy (%) for Different Models on Stanford Dogs
Top-
accuracy
Top-5
accuracy
Model
baseline
LSR
TSLA
baseline
LSR
TSLA
ResNet-50
88. 14
89.50
89.90
99.04
99.10
99.42
VGG
85.96
86.42
86.64
98.61
98.72
98.85
MobileNet
83.05
83.62
84.34
97.98
97.68
98.40
Inception v3
87.10
87.30
87.48
98.93
98.82
98.90
EfficientNet 85.72
85.71
86.26
98.80
98.60
98.78
Table 8: Comparison of Testing Accuracy (%) for Different Models on CIFAR-100
Top-1
accuracy
Top-5 accuracy
Model
baseline
LSR
TSLA
baseline
LSR
TSLA
ResNet-50
76.63
77.52
77.71
93.99
93.08
94.17
VGG
72.99
74.09
74.22
90.59
90.37
91.70
MobileNet v2
74.42
76.64
76.72
93.12
93.13
94.01
Inception v3
79.95
79.71
79.87
95.23
94.76
95.19
EfficientNet
69.73
71.38
71.68
90.50
90.48
91.93
15

---

Under review as a conference paper at ICLR 2022
Besides, we train ResNet-50 over ImageNet with the total epochs of 180. The initial learning rate is set to be 0.2 for all algorithm and divide them by 10 every 60 epochs as suggested in (He et al., 2016; Zagoruyko & Komodakis, 2016). The mini-batch size of 512 is used. The momentum parameter and weight decay are set to be 0.9 and 10-1 respectively. We fix the smoothing strength = 0.1 and the drop epoch 120 for TSLA. The top-1 accuracy are presented in Table showing that TSLA and LSR are comparable.
Table 9: Comparison of Testing Accuracy (%) for ImageNet
Top-I accuracy
Top-5 accuracy
Model
baseline
LSR
TSLA
baseline
LSR
TSLA
ResNet-50
76.35
76.96
77.09
93.14
93.55
93.69
B TECHNICAL LEMMA
Recall that the optimization problem
min Ps(w): == Σ
Cly [(:xx:)
(7)
wEW
i=1
where the cross-entropy loss function l is given by
K
exp(fi(w;x)
f(y,f(w;x)) =
Yo log
(8)
Σ j== exp(f) (w;
1=1
If we set
exp(fi(w:x)
p(w;x)= (p) (w; .:((::)))) E RR,
pi(w;x)= -log
(9)
Σ exp(f (w:x))
the problem (7) becomes
min Fs(w) = 1
n Σ (v.,((:)))))
(10)
WEW
Then the stochastic gradient with respective to w is
(y, vp(wix)).
(11)
Lemma 1. Under Assumption (i), we have
E [1 [ f(wr:x))) - VFF((w))) (1- 0)o² + 08o².
Proof.
By the facts of y!S = (1 0)y: 08, and the equation in (11), we have
Vily!s f(w;:x)) = ( VeV + AV f(wxx))
Therefore,
E
/ f(wxx))) - VFs(W:))1
-E V) V/
| (lVe)))))+OVV///..///..... -VFs)):
(4)
5 11-0) 0} [V vPs(()) + + - |
< (1-0)o² + 2
where (a) uses the convexity of norm, i.c. |1 - 8)a + ²b|l - (1 - 0)]aa|2 + + (b) uses
assumption (i) and the definitions in (4), and Assumption 1 (i).
16

---

Under review as a conference paper at ICLR 2022
C PROOF OF THEOREM 3
Proof. By the smoothness of objective function F(w) in Assumption (ii) and its remark, we have
Fs(W(+1) Fs(W))
1 (VEss (P.) wi) + L [|we+11 - well?
2
77 (VVFs( V f(w(xx)))) + I
2 VV(yy f(wxx)))]
" [W.)s + [VFs(w.) - / ((w
11
|| ((w:x))))
2
2
(e)
7 2 IVFs( 1
|VF((w))
((w,
(12)
where (a) is due the update of W (b) is due to (a, -b) 2/z (I|a bl12
||a/|2 [|bl?); (c)
due to 7 S t. Taking the expectation over (x,, y!s) on the both sides of (12), have
E [FF(WWe+1) - (w:)
s 2 E VFs((w))l (w:))l
7 E [[Fs( - Vely! f(wxx)))]
S "E [IVVFs w.)l ||3}
+ 2
((1 - 0)0² + 080").
(13)
2
where the last inequality is due Lemma
Then inequality (13) implies
T-1
= (Wo(
+ (1 0) + 080
T 1=
1/L"
(a)
(F))
28
+
1/L'
1+
(b) 2F'(( (wo)
S
+ 250
n/
where (a) due to 0 = I 1 (b) is due to
1
< 1.
148
D CONVERGENCE ANALYSIS OF SGD WITHOUT LSR (0 =0)
Theorem 6. Under Assumption 1, the solutions we from Algorithm with 0 satisfy
1 T E FI
(Fs(
+ 11Lo2.
T
n/
In order to have En < è. it suffices to set = min (+. the total complexity is O (+).
2
and T 4Fs(wo)
2L
Proof. By the smoothness of objective function (s(ww) in Assumption 1 (ii) and its remark, we have
Fs(W - -F(ww:)
< <(VPs(We), We-1 - w.) + -1 - -wel²
(VF((w.), Vely,, ((;xx.)(::;xx))))) + T VVlly, ((s;xx)( (14)
17

---

Under review as a conference paper at ICLR 2022
where (a) is due to the update of W++1- Taking the expectation (xAy)) on the both sides of (14), we have
E ([s((We1)) - Fs(ww)]
(0)
-n [Fs((s)))))
E
[ (.s( + VFs(w))l
2
@ 7E VFs (w,)) )1
E
[VE(y: FFs(w,)ll n²
[VFFs
2
(e)
" 2
VIF w: ]|
77
0"
(15)
2
where (a) and (b) use Assumption 1 (i); (c) uses the facts that S 1 and Assumption 1 (i). The inequality (15) implies
I.
T E [
(Wo) nT
+ tnIoo?
T
By setting S 21.22 and T =
(wo)
we have T -
T-1 [ VFs se. Thus the total
complexity is in the order of O 1pe 0 (
PROOF OF THEOREM 4
Proof. Following the similar analysis of inequality (13) from the proof of Theorem we have
E [Fs(we+1) Fs(w:)]
S m E (ws())) + th ((1 - 0)a² + 0800)) 2
(16)
2
Using the condition in Assumption 2 we can simplify the inequality from (16) as
E [FF(wwe11] Fs(w3))
5(1 - (-)) - Fs(w3)) + m1 2
((1- ((1 - 0)0² + 0800²)
< (1 m1) E[Fs(wo) FF(W)) ((1 0)0² 00002)
Σ (1 m11)*
1=0
T1 $ (1 mH) [[Fs(wo)]
(1--)00² 0000 Σ (1 mu))
where the last inequality is due to the definition of loss function that Fs(w") 2 0. Since S <
R² then (1 - m/))+ < exp(- + 1)) and Ei (1 mm)² S "F As result, for any Tı,
we have
[[F((WT))) Ps( (ws)) < pepp( + 2/x ((1 -0)² + +0002).
(17)
Let = I+3 and 62 = (1 - 0) + 0802 = 148
28
0² then ((1 0)o² + 0602) =
802
S
x(1+8)
Fs Fs(w)) since 5 is small enough. By setting
(2uls((W))
Ti = log
((T1))
G2
we have
Ø2
25g
E [F(w))) Fs(w3s)] 5
(18)
18

---

Under review as a conference paper at ICLR 2022
After Ii iterations, drop off the label smoothing, i.c. 0 O, then we know for any t Tı, following the inequality (15) from the proof of Theorem 6, we have
1/ [l:VF((w))ll) + T/ Lo² [[F - Fs(wi) -
2
Therefore, we get
7 7
1
2
Σ
E (Fs(SW,))) - + neLo
T2
1/2
(a)
2
Fs(33)l E [[Fs(WT))) -FF(W) + ?o
1/27
(18)
40o²
1/2T.00
(19)
p12T2
where (a) is due to -)( 2 Fs(w3s) By setting f22 2Lo
2
and T2
850
we have
uuge
DEATT E [Fs(w,))] se..
F
GENERALIZATION ANALYSIS
In this section, let
X [w| Fs(w)= =Fs(w) +
Σ e (w::
111
(20)
[
> マタ1 2
F(w)= =F(w) ||wl
(21)
Following by Yuan et al. (2019b), we use the following decomposition of testing error.
EA.s(F(ws)] - Es|Fs (ss[f((ws)) w'')] s Es (s([ssFs(ss) Fs(w"))] + EE.s(F(ws) - Fs(ws)).
(22)
Let define the gradient update rule Gž., as follows
GE,, (w) = w
f(w,x))
(23)
Then have the following lemma, which is similar to Lemma 2.5 and Lemma 4.2 in Hardt al. (2015).
Lemma Assume that fly, f(w, x)) is L-smooth and B-Lipschitz. Let Wi+1 = G(we) and W (ww)),, then
[w+1 - W'+1ll {{
(1+ nb ,/))lw, wil.
G= g'
(1 (1-n))|[w, -wil 2m
G G
Proof. Recall that ely;, f(w:x:)) Cly,, (w:x + /w/l2 Then (I) When = G' and ((:,, ((:,f(w;:)) is L-smooth, then
[ W'+1ll <(1 (w will m/ Twlly., Vwlly,, /wllye./(w(:xx..)
<(1 + 1/L. -// will-
(2) When 9 G' and {(, (w:))) ))==))))) is B-Liischizz, then
[i41 - W/+1/| <(1 - -)>>i| / - w + n/ www. (wels(.
<(1 - (1-r/A)/[w, w,
www.ww/ w/ll + BnB.
19

---

Under review as a conference paper at ICLR 2022
F.1 PROOF OF THEOREM 5
Proof. By Theorem 3.2 of Hardt et al. (2015), we have
A(t1
F= Wt+1
w/-1ll]
(4)
(1 + 11+nL- 1+ + ((1111)4,, + 24x
11
1
n/. 1/11
D. BnB
1
11
n
(6)
BB (c)
2n B
10
nL)* (d) 2B
S n
(24)
n n
nL''
where (a) uses Lemma 2; (b) uses x = 2L; (c) uses A = 0; (d) uses nL < 1. Then by Lemma 3.11 of Hardt et al. (2015) and is I, + X-smooth, we have
to + 2B(L + A)
6B +
E(((w,; E( (w/: 2)]
(25)
n
nL
n
where the last inequality holds by selecting to = 1 and = 2L. et al. (2015)
Then by Theorem 2.2 of Hardt
EA,((F(w.)) - Fs(w))) 6B +1
(26)
n
which is the generalization error. Next, we will bound the optimization error. Since Fs is (L + A)- smooth and satisfies 4-PL condition, then follow the similar analysis of Theorem 4, we have
T T
Ti+T
1
Es(EalFs(wd) - Fs(W))]l S
E
T3
2p'T2
28
72(T. + A)o²
s
(27)
T2rTTT
24
Then by (22), (26) and (27), get
Ti 7
280²
m/2 12(L + A)o²
6B +
T2 Σ
EAs(F((w))] - Es Ess]ss 5 T2211222
+
(28)
2p
By using the conditions that minwew Fs(w) S Fs(w's) + 2 /wll definitions (20) and (21), we get
- Fs()) < F(w.) - Fs( (WS). Therefore, we have the following inequality by (28) and (29):
(29)
T PT+T--1
25g
1y (L 2)o²
6B +
1
EA.s[F - Es(Fs(w')]
+
(30)
T2
TT
2p
By setting 7/2 = O(1/ Vn) and Tę = ((8) then the testing error ((WR) 0(1/ /n), where A is TSLA.
<
Remark 1. (SGD without LSR) Following the similar analysis of Theorem 5 and similar analysis of Theorem have the testing error for SGD without LSR is
ER.A.S (F(W)) Esll's (w3)) S Fs(wo) T(L + A)o2
6B + 1
(31)
1u7 By setting // = (n)//n) and = On)) then E.R.As(F(WW)) Es(Fs(w3)) S O(1/nn) where A is SGD without LSR.
2/1
n
Remark 2. (SGD with LSR) Following the similar analysis of Theorem 5 and similar analysis of Theorem 3, we have the testing error for SGD with LSR is
FR.A, [P [W Fs(FF(W3)) Fs(wo)
80²
+B+1
(32)
pnyT
/
where A is SGD with LSR. If 8 > ((1/vD), the testing enror can not be bounded by 0(1//n); if 5 < 2(1 (2))/nn,, the testing error can be bounded by O(1//n) in T O(vn) iterations.
20


Reviews:
[{"review_id": "q8bF32mDl4y", "reviewer": "Reviewer_CCDQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\nThis paper provides some theoretical explanation for the function of LSR and design a two-stage training method.\n\nWeaknesses:\nFor the theoretical part, this paper does not make analysis from the overfitting perspective and builds on some assumption that “It seems that training with smoothed label in the late epochs makes the learning progress more difficult”. This is kind of conflicting with the overfitting viewpoint of hard examples issues, where LSR and knowledge distillation are applied to improve the generalization performance.\n \nThe baseline SGD without LSR also should be added into Figure 1. From Figure 2, we can see that for the baseline there is also almost no change in loss after 30 epochs. This effect maybe because of the data characteristics of the relatively small CUB-2011 dataset.\n \nFor the experiments, some strong data augmentation techniques, such as Mixup [1] and RandAugment [2], should be incorporated into the training process to see the effectiveness of the TSLA under data augmentation.\n\nBesides ResNet-18 model, more backbones should be investigated like the ResNet-50 and InceptionNet.\n\nThe experimental datasets mainly focus on fine-grained image classification datasets which has a limited number of classes. To make the results more convincible, much larger datasets like the ImageNet dataset need to be further verified.\n\nSince the weight decay parameters has an important impact on label smoothing regularization, different setups should be investigated.   \n\n[1] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018\n\n[2] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, 2020.", "summary_of_the_paper": "This paper makes some analysis on the convergence behaviors of SGD with label smoothing in deep learning. Further, a two-stage label smoothing algorithm is proposed to improve the convergence. Experiments are conducted to verify its effectiveness.", "summary_of_the_review": "Though this paper provides some theoretical analysis for the LSR, the proposed TSLA builds on some guesses like “training one-hot label could be “easier” than training smoothed label”. Further, much more experiments should be added to prove its effectiveness.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VGjNO-JQmFu", "reviewer": "Reviewer_AcSL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "$\\textbf{Strengths}$\nStrengths can be summarized as below:\n* **Clarity** This paper is well organized and easy to follow.\n* **Rationality** The assumptions adopted in this paper are either commonly used in the literature of non-convex\noptimization or has theoretical/empirical validations. I didn't observe major flaws in the theories as well as proofs.\n\n* **Quality** Overall, this is a novel work that provides insights on understanding the convergence behavior of label smoothing regularization. When optimizing with SGD, it is theoretically depicted that an appropriate LSR which reduces the variance can speed up convergence and require less sample complexity.\n\n$\\textbf{Weaknesses}$\n* **The threshold of $\\delta$ in Theorem 3** In Theorem 3, the threshold of $\\delta$ is shown to be $\\frac{\\epsilon^2}{4\\sigma^2}$ in order to reach an $\\epsilon-$stationary point. Note that this threshold is dependent on $\\epsilon$, it seems that the condition $\\delta\\leq\\frac{\\epsilon^2}{4\\sigma^2}$ might be hard to achieve. It would be better if the authors could provide me with more explanations or empirical validations regarding this threshold.\n* **Estimation of $T_1$**  One potential weakness of this work is the estimation of $T_1$. Clearly, if we adopt a smaller value of $T_1$, the performance of TSLA is very likely to perform worse than learning without LSR. And if estimate $T_1$ is larger than its theoretical form, the convergence (training time) of TSLA will be longer. Besides, I feel like the estimation of $T_1$ is a tough task. Thus, I think this might be a potential weakness. \n* **Ablation study of $\\theta$** In Table 3, as mentioned by authors, large $\\theta$ (i.e., $\\theta>0.4$) frequently results in the worse performance of TSLA than LSR for different value of $T_1$. It seems that a good performance of TSLA relies on the quality of $\\theta$ and $T_1$.   ", "summary_of_the_paper": "This paper proved that when performing stochastic gradient descent for a non-convex objective function, an appropriate smooth parameter might speed up the convergence. The authors then proceed to introduce a two-stage TSLA algorithm which firstly learns with LSR. Then switch to learn with hard labels after certain epochs. Empirical evidence demonstrate the effectiveness of the proposed algorithm.", "summary_of_the_review": "This paper is overall an interesting paper. Although I still have some additional concerns regarding the threshold of $\\delta$ in Theorem 3, and empirical aspects of $T_1$ and $\\theta$ in the proposed TSLA. I am willing to increase the score if the authors well address my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VDAJ4thfgHe", "reviewer": "Reviewer_3bTK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n(1) The presentation and organization are good. This paper is easy to read.\n\n(2) The theoretical analysis for label smoothing understanding is interesting. The authors also provide detailed proof to support the statement.\n\n(3) The proposed TSLA is simple yet effective on several small datasets.\n\nCons:\n\n(1) My main concern lies in the experiments. Such a useful two-stage optimization strategy needs more experiments for verification. The datasets and CNN model adopted by this paper are small and shallow, respectively. I suggest that the authors could conduct experiments on a larger dataset with more deep models. For example, please show the results on ImageNet with ResNet-152. Otherwise, the presented results are not convincing.\n\n(2) According to Table 1, the results of Top-5 accuracy for LSR are lower than the baseline. Is there any theoretical analysis?\n\n(3) How about the generalization of label smoothing? Is it useful for different networks and different datasets? For example, can it improve the performance of ResNet-152, mobilenet, and shufflenet?\n\n(4) For the proposed TSLA, the novelty is somehow limited. It is a combination of two commonly used strategies. But the analysis from the optimization aspect is interesting.", "summary_of_the_paper": "This work theoretically analyzes the convergence behaviors of stochastic gradient descent with label smoothing in deep learning, which implies that LSR may slow down the convergence at the end of optimization. The authors propose the two-stage label smoothing strategy to further improve the convergence. Experiments on several datasets also verify the effectiveness of TSLA.", "summary_of_the_review": "For the label smoothing strategy, this paper theoretically analyzes the convergence behaviors and proposes the TSLA. Experiments on several small datasets demonstrate the effectiveness. More experiments are needed to verify its generalization.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7Sas-ys1nf", "reviewer": "Reviewer_u1nj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper has some positive aspects as follows:\n1. The proposed theoretical analysis of LSR is valuable. This paper gives the theoretical explanation of why an appropriate LSR can help speed up the convergence.\n2.  A improved method is developed. According to the proposed theoretical analysis, this paper further proposes a two-stage label smooth method to enhance the generation ability of LSR.\n3.  The paper is well written, the arguments are clear, and the methodology is well presented.\n\nHowever, the authors need to deal with some issues as follows:\n\n4. In the experiment with different theta, the case with theta=0.1 should be provided in the experimental results, which is a normal setting in the LSR.\n5. On CIFAR-100, the proposed method achieves an extremely weak improvement, thus it is hard to confirm the effectiveness of TSLA.\n6. To conduct a fair comparison, the LSR results with the same epochs as TSLA should be given in the experiments.\n", "summary_of_the_paper": "This paper studies the power of Label smoothing regularization (LSR) in training deep neural networks by analyzing SGD with LSR in different non-convex optimization settings. The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence. Besides, this paper proposes a simple yet efficient two-stage label smooth (TSLA) whose basic idea is to switch the training from smoothed label to one-hot label. Integrating TSLA with SGD can produce the improved convergence result that TSLA benefits from LSR in the first stage and essentially converges faster in the second stage. Extensive experiments show that TSLA improves the generalization accuracy of deep models on several benchmark datasets.", "summary_of_the_review": "The proposed theoretical explanation for LSR is meaningful, however, the experiment in this paper needs to be further enhanced. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors proposed a two-stage algorithm for exploiting label smoothing and provided some analysis based on how label smoothing may have reduced the variance in the stochastic gradient. While the authors provided substantial experiments to justify their work (with additional ones during the response stage), none of the reviewers was very excited in the end, for obvious reasons perhaps: (a) the two-stage algorithm is a straightforward combination of existing practices (first run with label smoothing and then run without label smoothing), without any new, interesting insight from the authors' side; (b) the analysis is a direct consequence of the authors' assumptions. Basically, if label smoothing reduces variance, SGD would converge faster and vice versa, which is nothing surprising or insightful. The key is to understand when and how any particular way to smooth the label would lead to significant reduction of the variance, which the authors did not provide any guidance or insight other than offering some empirical results. Overall, we do not believe this work, in its current form, adds significant value to our understanding of label smoothing.
Decision: Reject