Towards Understanding Label Smoothing
Authors: Yi Xu
Yuanhong Xu
Qi Qian
Li Hao
Rong Jin
Venue:  (2021)
Keywords: Label Smoothing
Non-convex Optimization
Deep Learning Theory

Abstract: Label smoothing regularization (LSR) has a great success in training  deep neural networks by stochastic algorithms such as stochastic gradient descent and its variants. However, the theoretical understanding of its power from the view of optimization is still rare. This study opens the door to a deep understanding of LSR by initiating the analysis. In this paper, we analyze the convergence behaviors of stochastic gradient descent with label smoothing regularization for solving non-convex problems and show that an appropriate LSR can help to speed up the convergence by reducing the variance. More interestingly, we proposed a simple yet effective strategy, namely Two-Stage LAbel smoothing algorithm (TSLA), that uses LSR in the early training epochs and drops it off in the later training epochs. We observe from the improved convergence result of TSLA that it benefits from LSR in the first stage and essentially converges faster in the second stage. To the best of our knowledge, this is the first work for understanding the power of LSR via establishing convergence complexity of stochastic methods with LSR in non-convex optimization.  We empirically demonstrate the effectiveness of the proposed method in comparison with baselines on training ResNet models over benchmark data sets.

Under review as a conference paper at ICLR 2021
TOWARDS UNDERSTANDING LABEL SMOOTHING
Anonymous authors
Paper under double-blind review
ABSTRACT
Label smoothing regularization (LSR) has a great success in training deep neu- ral networks by stochastic algorithms such as stochastic gradient descent and its variants. However, the theoretical understanding of its power from the view optimization still rare. This study opens the door to a deep understanding of LSR by initiating the analysis. In this paper, we analyze the convergence behav- iors of stochastic gradient descent with label smoothing regularization for solving non-convex problems and show that an appropriate LSR can help to speed up the convergence by reducing the variance. More interestingly, we proposed a simple yet effective strategy, namely Two-Stage LAbel smoothing algorithm (TSLA), that uses LSR in the early training epochs and drops off in the later training epochs. We observe from the improved convergence result of TSLA that it bene- fits from LSR in the first stage and essentially converges faster in the second stage. To the best of our knowledge, this is the first work for understanding the power of LSR via establishing convergence complexity of stochastic methods with LSR in non-convex optimization. We empirically demonstrate the effectiveness of the proposed method in comparison with baselines on training ResNet models over benchmark data sets.
INTRODUCTION
In training deep neural networks, one common strategy is to minimize cross-entropy loss with one- hot label vectors, which may lead to overfitting during the training progress that would lower the generalization accuracy (Müller et al., 2019). To overcome the overfitting issue, several regular- ization techniques such as & ---mr or f--norm penalty over the model weights, Dropout which randomly sets the outputs of neurons to zero (Hinton et al., 2012b), batch normalization (Ioffe & Szegedy, 2015), and data augmentation (Simard et al., 1998), are employed to prevent the deep learning models from becoming over-confident. However, these regularization techniques conduct on the hidden activations or weights of a neural network. As an output regularizer, label smoothing regularization (LSR) (Szegedy et al., 2016) is proposed to improve the generalization and learn- ing efficiency of neural network by replacing the one-hot vector labels with the smoothed labels that average the hard targets and the uniform distribution of other labels. Specifically, for K- class classification problem, the one-hot label smoothed by y = (1 f)y + 8y, where y is the one-hot label, E 10 is the smoothing strength and ý K uniform distribution for all labels. Extensive experimental results have shown that LSR has significant successes in many deep learning applications including image classification (Zoph al., 2018; He et al., 2019), speech recognition (Chorowski & Jaitly, 2017; Zeyer et al., 2018), and language translation (Vaswani et al., 2017; Nguyen & Salazar, 2019).
Due to the importance of LSR, researchers try to explore its behavior in training deep neural net- works. Müller al. (2019) have empirically shown that the LSR can help improve model calibration, however, they also have found that LSR could impair knowledge distillation, that is, if one trains a teacher model with LSR, then student model has worse performance. Yuan al. (2019a) have proved that LSR provides a virtual teacher model for knowledge distillation. As a widely used trick, Lukasik et al. (2020) have shown that LSR works since it can successfully mitigate label noise. However, to the best of our knowledge, is unclear, at least from theoretical viewpoint, how the introduction of label smoothing will help improve the training of deep learning models, and to what affirmative answer to this question and try
stage, it can help. In this paper, aim to provide to deeply understand why and how the LSR works from the view of optimization. Our theoretical analysis will show that appropriate LSR can essentially reduce the variance of stochastic gradient

---

Under review as a conference paper at ICLR 2021
in the assigned class labels and thus it can speed up the convergence. Moreover, we will propose novel strategy of employing LSR that tells when to use LSR. We summarize the main contributions of this paper as follows.
It is the first work that establishes improved iteration complexities of stochastic gradient descent (SGD) (Robbins & Monro, 1951) with LSR for finding an e-approximate stationary point (Definition 1) in solving smooth non-convex problem in the presence of appro- priate label smoothing. The results theoretically explain why an appropriate LSR can help speed up the convergence. (Section
We propose a simple yet effective strategy, namely Two-Stage LAbel smoothing (TSLA) algorithm, where in the first stage it trains models for certain epochs using a stochastic method with LSR while in the second stage runs the same stochastic method without LSR. The proposed TSLA is a generic strategy that can incorporate many stochastic algo- rithms. With an appropriate label smoothing, we show that TSLA integrated with SGD has improved iteration complexity, compared to the SGD with LSR and the SGD without LSR. (Section 5)
2 RELATED WORK
In this section, we introduce some related work. A closely related idea to LSR is confidence penalty proposed by Pereyra et al. (2017), an output regularizer that penalizes confident output distribu- tions by adding its negative entropy to the negative log-likelihood during the training process. The authors (Pereyra et al., 2017) presented extensive experimental results in training deep neural net- works to demonstrate better generalization comparing to basclines with only focusing the existing hyper-parameters. They have shown that LSR equivalent to confidence penalty with a reversing direction of KL divergence between uniform distributions and the output distributions.
DisturbLabel introduced by Xie et al. (2016) imposes the regularization within the loss layer, where it randomly replaces some of the ground truth labels incorrect values at each training iteration. Its effect is quite similar to LSR that can help to prevent the neural network training from overfit- ting. The authors have verified the effectiveness of AasturbLabbl via several experiments on training image classification tasks.
Recently, many works (Zhang et al., 2018; Bagherinezhad et al., 2018; Goibert & Dohmatob, 2019; Shen et al., 2019; Li et al., 2020b) explored the idea of LSR technique. Ding et al. (2019) extended an adaptive label regularization method, which enables the neural network to use both correctness and incorrectness during training. Pang et al. (2018) used the reverse cross-entropy loss to smooth the classifier's 's gradients. Wang al. (2020) proposed a graduated label smoothing method that uses the higher smoothing penalty for high- high-confidence predictions than that for low-confidence predic- tions. They found that the proposed method can improve both inference calibration and translation performance for neural machine translation models. By contrast, in this paper, we will try to un- derstand the power of LSR from an optimization perspective and try to study how and when to use LSR.
3 PRELIMINARIES AND NOTATIONS
We first present some notations. Let F(w) denote the gradient of function F(w). When the variable to taken a gradient is obvious, we use F(w) for simplicity. We use to denote the Euclidean norm. Let (---) be the inner product.
In classification problem, we aim to seek a classifier to map an example x E onto one of K labels y E ) C R where y = (y1, ./,..... yk) is a one-hot label, meaning that yi is "1" for the correct class and "0" for the rest. Suppose the example-label pairs are drawn from a distribution P. i.c., (x. y) IP- = (Pxx.Py)) we denote by [ the expectation that takes over random variable y). When the randomness is obvious, we write E[-] for simplicity. Our goal is to learn prediction function (w;;): W x X + K that is as close as possible to y, where w E W the parameter and W is a closed convex set. To this end, we want to minimize the following expected loss under P:
min F(w): == E(x,,) [e(y, f(w:))) wEW
(1)
2

---

Under review as a conference paper at ICLR 2021
where l ( xR* - R4 is cross-entropy loss function given by
explfi(w;x))
e(y,
Yi log
(2)
و== exp(Sy (W;[//(w;))) (wxx))
The objective function F(w) ) is not convex since f(w;x) is non-convex in terms of w. To solve the problem (1), one can simply use some iterative methods such as stochastic gradient descent (SGD).
Specifically, at each training iteration I, SGD updates solutions iteratively by
W++1 = W. - [(::
where 17 > 0 is a learning rate.
Next, we present some notations and assumptions that will used the convergence analysis. Throughout this paper, we also make the following assumptions for solving the problem (1).
Assumption 1. Assume the following conditions hold:
(i) The stochastic gradient of F(w) is unbiased, i.e. (Vely ( F (FF, and the variance of stochustic gradient bounded, ie., there exists a constant o² 0, such that E(x,)) [
FF(W)]] =
(ii) F(w) is smooth with an T--Lipchitz continuous gradient, i.e, it is differentiable and there exists constant L >> 0 such that [|VF(w) FF())l I L||w- Nowwwl. ulI, Vw, W.
Remark, Assumption 1 (i) and (ii) are commonly used assumptions in the literature of non-convex optimization (Ghadimi & Lan, 2013; Yan et al., 2018; Yuan al., 2019b; Wang et al., 2019; Li et al., 2020a). Assumption (ii) says the objective function is L-smooth, and it has an equivalent expression (Nesterov, 2004) which is (w))-FFl S (VF(u), w u) +
4 l|ww -ull Vw, u E W.
For classification problem, the smoothed label y given by
= (1 0)y 08,
where 0- E (0, 1) is the smoothing strength, y is the one-hot label, y is an introduced label. For example, one can simply use y t (Szegedy et al., 2016) for K-class problems. Similar to label y. we suppose the label ý is drawn from distribution Py. We introduce the variance of stochastic gradient using label as follows.
(x))) [[W(() - VF(w)] = = doa.
(4)
where 8 >> O is a constant and o² is defined in Assumption (i). We make several remarks for (4). Remark. (a) We do not require that the stochastic gradient ff is unbiased, i.e., it could be E[V (:: + VF(w). (w). (b) The variance 8² is defined based on the label ÿ rather than the smoothed label y (c) We do not assume the variance 02 is bounded since 6 could be an arbitrary value, however,
then G2 2 o²; while 0 < 5 < 1, then 5² < x². It is worth mentioning that could he small when an appropriate label used in the label smoothing. For example, one can smooth labels by using a teacher model (Hinton et al., 2014) or the model's own distribution (Reed et al., 2014). In the first paper of label smoothing (Szegedy et al., 2016) and the following related studies (Müller al., 2019; Yuan et al., 2019a), researchers consider a uniform distribution over all K classes of labels as the label y, i.e., set y R.
will discuss the different cases of 8 in our analysis. If 8
We now introduce important property regarding F(w), i.e. Polyak-t.ojasiewicz (PL.) condi- tion (Polyak, 1963). More specifically, the following assumption holds.
Assumption 2. There exists constant μ. > O such that 24( F.) < | [V Vw E W, where F. = minwew F(w) is the optimal value.
Remark. This property has been theoretically and empirically observed in training deep neural networks (Allen-Zhu et al., 2019; Yuan et al., 2019b). This condition widely used to establish convergence in the literature of non-convex optimization, please (Yuan et al., 2019b; Wang et al., 2019; Karimi et al., 2016; Li & Li, 2018; Charles & Papailiopoulos, 2018; Li et al., 2020a) and references therein.
3

---

Under review as a conference paper at ICLR 2021
Algorithm SGD with Label Smoothing Regularization
1: Initialize: Wo E W, 0 E (0, 1), set as the value in Theorem 3.
2: for é 0, 1,..., .....,T'- - 1 do
3: sample (t). set LS = (1 +
4: update W1+1 W, V w Ely! f(ww:)))
5: end for
To measure the convergence of non-convex and smooth optimization problems as in (Nesterov, 1998; Ghadimi & Lan, 2013; Yan et al., 2018), need the following definition of the first-order stationary point.
Definition (First-order stationary point). For the problem of minwew F(w), a point w E W is called a first-order stationary point [[]F(w)] = 0. Moreover, if F S then the point w is said to be an e-stationary point, where e E (0,1) 1) a small positive value.
CONVERGENCE ANALYSIS OF SGD WITH LSR
To understand LSR from the optimization perspective, we consider SGD with LSR in Algorithm for the sake of simplicity. The only difference between Algorithm and standard SGD is the use of the output label for constructing stochastic gradient. The following theorem shows that Al- gorithm converges to an approximate stationary point in expectation under some conditions. We include its proof in Appendix B.
Theorem 3. Under Assumption 1, run Algorithm with
I and O 2F(wo) + 25a², where R is uniformly sampled from 10,1....,* -11. Fur- have the following two results.
I46' then
1
F < of
1}.
thermore, given target accuracy level
2 (1) when 8 S 10 we set T 4F(wo), then Algorithm converges an e-stationary point in expectation, i.e., E F [WWR) < 2. The total sample complexity is I ===0=- (2)).
qe²
(2) when 8 > 2.2. if we set T = F(wo), then Algorithm does not converge to an e-stationary point, but have (WR)]] < 28 < 0(6).
gdo²
Remark. We observe that the variance term is 260², instead of TyLO² for standard analysis of SGD without LSR (i.e., 0 = 0, please see the detailed analysis of Theorem 5 in Appendix C). For the convergence analysis, the difference between SGD with LSR and SGD without LSR is that VE(V, f(w; not an unbiased estimator of FF(() when using LSR. The convergence behavior of Algorithm 1 heavily depends on the parameter 6. When is small enough, say 8 < O(²)) with a small positive value e E (0, 1), then Algorithm converges to an e-stationary point with the total sample complexity of (+). Recall that the total sample complexity of standard SGD without LSR for finding an e-stationary point 0 (++ (GGaadimi & Lan, 2016; Ghadimi et al., 2016). please also see the detailed analysis of Theorem in Appendix C). The convergence result shows that we could find a label y that has a reasonably small amount of 8, we will be able to reduce sample complexity for training machine learning model from O (+) to O (+), Thus, the reduction in variance will happen when appropriate label smoothing with E (0, 1) is introduced. We will find in the empirical evaluations that different label ý lead to different performances and an appropriate selection of label y has a better performance (see the performances of LSR and LSR-pre in Table 3). On the other hand, when the parameter 5 is large such that 8 > ((e²). ). that is to say, if an inappropriate label smoothing is used, then Algorithm does not converge to an e-stationary point, but converges to a worse level of ((5).
TSLA: GENERIC TWO-STAGE LABEL SMOOTHING ALGORITHM
Despite superior outcomes in training deep neural networks, some real applications have shown the adverse effect of LSR. Müller et al. (2019) have empirically observed that LSR impairs distilla- tion, that is, after training teacher models with LSR, student models perform worse. The authors believed that LSR reduces mutual information between the input example and output logit. Ko- mblith et al. (2019) have found that LSR impairs the accuracy of transfer learning when training deep neural network models on ImageNet data set. Sco et al. (2020) trained deep neural net- work models for few-shot learning mini-lma and found significant performance drop
4

---

Under review as a conference paper at ICLR 2021
Algorithm 2 The TSLA algorithm
1: Initialize: Wo E W. Ti, 0 E (0,1), .7, >0
2: Input: stochastic algorithm A (c.g., SGD)
// First stage: .A with LSR
3: for t = 0, ......77 T1 do
4: sample Yr). set y: = (1 fly; + By.
5: update Wx+1 A-step( w,; X Vi 71)
one update step of AA
6: end for
// Second stage: A without LSR
7: for tort= = Ti, 1,,,,,.,, 722 do
8: sample Vz)
9: update Wi+1 = A
o one update step of A
end for
with LSR. This motivates us to investigate a strategy that combines the algorithm with and with- out LSR during the training progress. Let think in this way, one possible scenario is that training one-hot label is "easier" than training smoothed label. Taking the cross entropy loss (2) for an example, one need to optimize a single loss funetion loog (oxpff((;;)) Σラ= exp(ss when one-hot label (c.g. Vı: = 1 and y: = 0 for all * k) is used, but need to optimize all K loss functions - - y! log
E exp(f) (w; when smoothed label (e.g., y = (1 O)y + 0 so that v = - (K - 1)0/K and y = 0/K for all i * k) is used. Nev- ertheless, training deep neural networks gradually focusing on hard examples with the increase of training epochs, It seems that training with smoothed label in the late epochs makes the learning progress more difficult. In addition, after LSR, we focus on optimizing the overall distribution that contains the minor classes, which are probably not important at the end of training progress. One question is whether LSR helps at the early training epochs but it has less (even negative) effect dur- ing the later training epochs? This question encourages us to propose and analyze simple strategy with LSR dropping that switches stochastic algorithm with LSR to the algorithm without LSR. In this subsection, stage it runs a stochastic algorithm A (c.g., SGD) with LSR for Ti iterations and the second stage runs the same algorithm without LSR up to Z2 iterations. This framework is referred as Two- Stage LAbel smoothing (TSLA) algorithm, whose updating details are presented in Algorithm 2. The notation A-step(::-, 7) is one update step of stochastic algorithm with learning rate 1). For example, if we select SGD as algorithm A, then
propose generic framework that consists of stages, wherein the first
(EID-step(w;;xx.o yt mh = W. m Vely!s, (wwxx))) SGD-step (W,; Xt, yt, 122) = w. - 1/2 Ve I(we x:)).
LS
(5)
(6)
Although SGD is considered as the subroutine algorithm A in the convergence analysis, in prac- tice, algorithm A can be replaced by any stochastic algorithms such as momentum SGD (Polyak, 1964), Stochastie Nesterov's Accelerated Gradient (Nesterov, 1983), and adaptive algorithms in- cluding ADAGRAD (Duchi et al., 2011), RMSProp (Hinton et al., 2012a), AdaDelta (Zeiler, 2012), Adam (Kingma & Ba, 2015), Nadam (Dozat, 2016) and AMSGRAD (Reddi et al., 2018). In this paper, we will not study the theoretical guarantees and empirical evaluations of other optimizers, which can be considered as future work. Please note that the algorithm can use different learning rates 71 and solution of the second stage. If Ti O, then TSLA reduces to the baseline, i.e., standard stochas- tic algorithm A without LSR; while if T2 0, TSLA becomes to LSR method, i.e., standard stochastic algorithm with LSR.
during the two stages. The last solution of the first stage will be used as the initial
5.1 CONVERGENCE RESULT OF TSLA
In this subsection, we will give the convergence result the proposed TSLA algorithm. For sim- plicity, we use SGD as the subroutine algorithm A in the analysis. The convergence result in the following theorem shows the power of LSR from the optimization perspective. Its proof is presented in Appendix D. It is easy to from the proof that by using the last output of the first stage the initial point of the second stage, TSLA can enjoy the advantage of LSR in the second stage with an improved convergence.
5

---

Under review as a conference paper at ICLR 2021
Table I: Comparisons of Total Sample Complexity
Possibilities on 8
TSLA
LSR
baseline
12(e)) <
x
δ 0(E*)
2
x
(((*) < <8<0
5
( S O(e))
log
(1)
given target accuracy level
- E (0, 1 2 0 is constant
Theorem 4. Under Assumptions 1, 2. suppose 028/2 S F(wo). run Algorithm 2 with A SGD, = 143- "h = T. Ti = log (PPF(w)((+5)) /mmp), 1/2 2L and T2 mage" then Eall F v e, where R is uniformly sampled from (Ti..... Ti + 72 13.
1.
²²
8ã0²
200
Remark. It is obvious that the learning rate 122 in the second stage roughly smaller than the learning rate 771 in the first stage, which matches the widely used stage-wise learning rate decay scheme in training neural networks. To explore the total sample complexity of TSLA, we consider different conditions on 6. We summarize the total sample complexities of finding e-stationary points for SGD with TSLA (TSLA), SGD with LSR (LSR), and SGD without LSR (baseline) in Table 1, where e (0, 1) is the target convergence level, and we only present the orders of the complexities but ignore all constants. When ?(+²) < < 1. LSR does not converge to an e-stationary point (denoted by oo), while TSLA reduces sample complexity from O (+) to O (4), compared to the baseline. When 5 < O(E²), the total complexity of TSLA is between log(1/e) and 1/e², which always better than LSR and the baseline. In summary. TSLA achieves the best total sample complexity by enjoying the good property of an appropriate label smoothing (i.c., when 0 < 8 < 1). However, when 2 1, baseline has better convergence than TSLA, meaning that the selection of label y is not appropriate.
6 EXPERIMENTS
To further evaluate the performance of the proposed TSLA method, we trained deep neural networks on three benchmark data sets, CIFAR- 100 (Krizhevsky & Hinton, 2009), Stanford Dogs (Khosla et al., 2011) and CUB-2011 (Wah et al., 2011), for image classification tasks. CIFAR- has 50,000 training images and 10,000 testing images of 32x32 resolution with 100 classes. Stanford Dogs data set contains 20,580 images of 120 breeds of dogs, where 100 images from each breed is used for training. CUB-2011 is birds image data set with 11,788 images of 200 birds species. The ResNet-18 88 model (He et al., 2016) is applied as the backhone in the experiments. We compare the proposed TSLA incorporating with SGD (TSLA) with two baselines, SGD with LSR (LSR) and SGD without LSR (baseline). The mini-batch size of training instances for all methods is 256 as suggested by He et al. (2019) and He et al. (2016). The momentum parameter fixed as 0.9.
6.1 STANFORD DOGS AND CUB-2011
We separately train R8 (He et al., 2016) up to 90 epochs over two data sets Stanford Dogs and CUB-2011. We use weight decay with the parameter value of 10-1, For all algorithms, the initial learning rates for FC are set to 0.1, while that for the pre-trained backbones are 0.001 and 0.01 for Standford Dogs and CUB-201 epochs. For LSR, we fix the value of smoothing strength
respectively. The learning rates divided by 10 every 30
0.4 for the best performance, and the label ÿ used for label smoothing is set to be a uniform distribution over all K classes, i.c., ÿ R. The same values of the smoothing strength and the same ý are used during the first stage of TSLA. For TSLA, we drop off the LSR (i.e., let 0 = 0) after epochs during the training process, where (20, 30, 40, 50, 60, 70, 80}. We first report the highest top-1 and top-5 accuracy on the testing data sets for different methods. All top-1 and top-5 accuracy are averaged over 5 independent random trails with their standard deviations. The results of the comparison are summarized in Table 2, where the notation "TSLA(s)" means that the TSLA algorithm drops off LSR after epoch s. can be seen from Table 2 that under an appropriate hyperparameter setting the models trained using TSLA
hntts:://ww.cs.torontt hwt.mnttt.edd//kkiii/iiaaa...
"http://visionn wtwp:riiiioostaooodeeeeeeiee tyaet ImageNet "http://w..riiinnt ca
htm1

---

Under review as a conference paper at ICLR 2021
Table 2: Comparison of Testing Accuracy for Different Methods (mean - standard deviation, in %).
Stanford Dogs
CUB-2011
Algorithm*
Top-1 accuracy
Top-5 accuracy
Top-1 accuracy
Top-5 accuracy
bascline
82.31 0.18
97.76 + 0.06
75.31 = 0.25
93.14 + 0.31
LSR
82.80 1 0.07
97.41 + 0.09
76.97 + 0.19
92.73 1 0.12
TSLA(20)
$..15:000
97.91 + 0.08
76.62 = 0.15
93.60 = 0.18
(3LA(30)
13.89:0.11
98.05 0.08
1.119
93.92 0.16
T4LA(40)
0.13
98.03 + 0.05
7:500::.00
93.99
TSLA(50)
83.91
98.07 0.06
77.57 = 0.21
93.86 =
TSLA(60)
83.51
97.99 0.06
77.25 = 0.29
94.43 0.18
TSLA(70)
83.38 + 0.09
97.90 + 0.09
77.21
+ 0.15
93.31 + 0.12
TSLA(80)
83.14 0.09
97.73 + 0.07
77.05 + 0.14
93.05 + 0.08
'TSLA(s): TSLA drops off LSR after epoch 8.
Stanford
Shanford
Scanford
34
PaLA (d4)
83-
L5h
3
TSAN 1931
TALA 240
DMA 1621
TSAA
TELA (39)
0.6
79
30
40
50
80
90
20
30
70
80
20
40
85
apoch
spoch
CUB-2011
CUB 2011
CUB-2001
TSAA H4:
TSUA (00
TS4A (23)
Top:k Accuraa
Top-5 Acturery
2.5-
78
2.0-
3
1.5
TIL
TEL 103)
TSLA (90
TOL (00)
3.0-
20
50
600
70
50 80
80
Twooch
# eepoch
Figure 1: Testing Top-1, Top-5 Accuracy and Loss on ResNet-18 over Stanford Dogs and CUB- 2011. TSLA(s) means TSLA drops off LSR after epoch s.
outperform that trained using LSR and baseline, which supports the convergence result Section 5. We notice that the best top-1 accuracy of TSLA are TSLA(40) and TSLA(50) for Stanford Dogs and CUB-2011, respectively, meaning that the performance of TSLA(s) is not monotonic over the dropping epoch s. For CUB-2011, the top-1 I accuracy of TSLA(20) is smaller than that of LSR. This result matches the convergence analysis of TSLA showing that it can not drop off LSR too early. For top-5 accuracy,
found that TSLA((8)) is slightly worse than baseline. This is because of dropping LSR too late so that the update iterations (i.e., T2) in the second stage of TSLA is too small converge to good solution. We also observe that LSR is better than baseline regarding top- accuracy but the result is opposite as to top-5 accuracy. We then plot the averaged top-1 accuracy, averaged top-5 accuracy, and averaged loss among 5 trails of different methods in Figure 1. We remove the results for TSLA(20) since it dropped off LSR too early as mentioned before. The figure shows TSLA improves the top-1 and top-5 testing accuracy immediately once drops off LSR. Although TSLA may not converge if it drops off LSR too late, see TSLA(60), TSLA(70), and TSLA(80) from the third column of Figure still has the best performance compared to LSSR and baseline. TSLA(30), TSLA(40), and TSLA(50) can converge lower objective levels, comparing to LSR and baseline.
6.2 CIFAR-100
The total epochs of training ResNet-18 (He et al., 2016) on CIFAR-100 is set to be 200.The weight decay with the parameter value of x 10 4 is used. We use 0.1 as the initial learning rates for all algorithms and divide them by 10 every 60 epochs as suggested in (He al., 2016; Zagoruyko & Komodakis, 2016). For LSR and the first stage of TSLA, the value of smoothing strength is fixed as 0 0.1, which shows the best performance for LSR. We use two different labels y to smooth the
7

---

Under review as a conference paper at ICLR 2021
Table 3: Comparison of Testing Accuracy for Different Methods (mean + standard deviation, in %).
CIFAR-I00
Algorithm"
Top-1 accuracy
Top-5 accuracy
baseline
76.87 = 0.04
93.47 + 0.15
LSR
77.77 = 0.18
93.55 + 0.11
TSLA(120)
17.920022
TSLA(140)
77.93 = 0.19
94.11 0.22
TSLA(160)
77.96 = 0.20
94.19 + 0.21
TSLA(180)
78.04 = 0.27
94.23 + 0.15
LSR-ph pre
78.07 = 0.31
94.70 + 0.14
ISLA-pret 120)
78.34 = 0.31
94.68 + 0.14
TSLA-pree 140)
78.39 = 0.25
94.73 0.11
-re 160)
78.55 = 0.28
94.83 + 0.08
TSLA-pre (180)
78.53 = 0.23
94.96 + 0.23
'TSLA())/TSLA -rr(s)) TSLA/TSLA -re drops off LSR/LSR pre after epoch 8.
cfar100
cfor100
cilar000
2.0
3.8
TSGA
TEAA (2:00
T5
Top- Acuuaey
LSI
bereine
Top0 1000--Acuuue
76
1180
pa (162)
15
Taappr 2001
1.2
T2k 043
(120)
1.0
LSR
92
0.8
74
80
102
120
140
180
300
120
140
180
200
100
125
150
175
spoce
#ooch
Figure 2: Testing Top-1, Top-5 Accuracy and Loss on R8sNet-18 over CIFAR-100. TSLA($)TSLA- pre(s) mcansTSLA/TSLA-prr drops off LSR/LSR-pre after epoch s.
one-hot label, the uniform distribution over all labels and the distribution predicted by ImageNet pre-trained model which downloaded directly from PyTorch 4 (Paszke et al., 2019). For TSLA, we try to drop off the LSR after epochs during the training process, where All top-1 and top-5 accuracy on the testing data set are averaged over 5 independent random trails with their standard deviations. We summarize the results in Table 3, where LSRRpre and TSLA-pre indicate LSR and TSLA use the label y based on the ImageNet pre-trained model. The results show that LSR-pre/TSLA-pr has a better performance than LSR/TSLA The reason might be that the pre-trained model-based prediction is closer to the ground truth than the uniform prediction and it has lower variance (smaller 8). Then, TSLA (LSR) with such pre-trained model-based prediction converges faster than TSLA (LSR) with uniform prediction, which verifies our theoretical findings in Section (Section 4). This observation also empirically tells us the selection of the prediction function y used for smoothing label is the key to the success of TSLA as well as LSR. Among all methods, the performance of ASLA-pre is the best. For top-1 accuracy, TSLA A-pre(166 outperforms all other algorithms, while for top-5 accuracy, TSLA-pre( (po((10))))))))) has the best performance. Finally, observe from Figure 2 that both TSLA and TSLA-pa converge, while TSLA-pre converges to the lowest objective value. Similarly, the top-1 and top-5 accuracies show the improvements of TSLA and TSLA-pre at the point of dropping off LSR.
{120, 140, 160, 180).
CONCLUSIONS
In this paper, we have studied the power of LSR in training deep neural networks by analyzing SGD with LSR in different non-convex optimization settings. The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence. We have proposed a simple and efficient strategy so-called TSLA that can incorporate many stochastic algorithms. The basic idea of TSLA is to switch the training from smoothed label to one-hot label. Integrating TSLA with SGD, we observe from its improved convergence result that TSLA benefits from LSR in the first stage and essentially converges faster in the second stage. Throughout extensive experiments, we have shown that TSLA improves the generalization accuracy of deep models on benchmark data sets.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over- parameterization. International Conference on Machine Learning, pp. 242-252, 2019.
Hessam Bagherinezhad, Maxwell Horton, Mohammad Rastegari, and Ali Farhadi. Label refinery: Improving imagenet classification through label progression. arXiv preprint arXiv:1805.00 2018.
Zachary Charles and Dimitris Papailiopoulos. Stability and generalization of learning algorithms that converge to global optima. In International Conference Machine Learning, pp. 745-754, 2018.
Jan Chorowski and Navdeep Jaitly. Towards better decoding and language model integration in sequence sequence models. Proc. Interspeech 2017, pp. 523-527, 2017.
Qianggang Ding, Sifan Wu, Hao Sun, Jiadong Guo, and Shu-Tao Xia. Adaptive regularization of labels. arXiv preprint arXiv: 1908.05444 2019.
Timothy Dozat. Incorporating nesterov momentum into adam. 2016.
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121-2159, 2011.
Saeed Ghadimi and Guanghui Lan. Stochastic first-and rede methods for nonconvex stochas- tic programming. SIAM Journal on Optimization, 23(4):2341-2368, 2013.
Saced Ghadimi and Guanghui Lan. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. Math. Program.. (56(1-2):59-99 2016.
Saeed Ghadimi, Guanghui Lan, and Hongchao Zhang. ods for nonconvex stochastic composite optimization. Mathematical Programming, 155(1-2): 267-305, 2016.
stochastic approximation meth-
Morgane Goibert and Elvis Dohmatob. Adversarial robustness via adversarial label-smoothing. arXiv preprint arXiv: 1906v:1906..1355 11567, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Tong He. Zhi Zhang, Hang Zhang. Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 558-567 067,577 2019.
Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. 2012a
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurlPS Deep Learning Workshop. 2014.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv:1207.0588. 2012b.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167. 2015.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the -ajaak--ojasiwwizz condition. Joint European Conference on Ma- chine Learning and Knowledge Discovery in Databases, pp. 795 811. Springer, 2016.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine- grained image categorization: Stanford dogs. In Proc. CVPR Workshop on Fine- Grained Visual Categorization (FGVC), volume 2, 2011.

---

Under review as a conference paper at ICLR 2021
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Simon Kornblith, Jonathon Shlens, and Quoe V Le. Do better imagenet models transfer better? In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2661-2671, 2019.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Mas- ter's thesis, Technical report, University of Tronto, 2009.
Xiaoyu Li, Zhenxun Zhuang, and Francesco Orabona. Exponential step sizes for non-convex opti- mization. arXiv preprint arXiv:2002.05273, 2020a.
Xingjian Li, Haoyi Xiong, Haozhe An, Dejing Dou, and Chengzhong Xu. Colam: Co-
learning of deep neural networks and soft labels via alternating minimization. arXiv preprint arXiv:2004. 12443, 2020b.
Zhize Li and Jian Li. A simple proximal stochastic gradient method for nonsmooth nonconvex optimization. In Advances Neural Information Processing Systems, pp.
2018.
Michal Lukasik, Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? arXiv preprint arXiv:2003.028/1 2020.
Rafael Müller, Simon Korbblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4696-4705, 2019.
Yurii Nesterov. A method of solving a convex programming problem with convergence rate 0(1/k²). Soviet Mathematics Doklady, 27:372-376. 1983.
Yurii Nesterov. Introductory lectures convex programming volume i: Basic course. 1998.
Yurii Nesterov. Introductory lectures
optimization a basic course. Applied optimization. Kluwer Academic Publ., 2004. ISBN 1-4020-7553-7.
Toan Q Nguyen and Julian Salazar. Transformers without tears: Improving the normalization of self-attention. arXiv preprint arXiv: 2019.
Tianyu Pang, Chao Du, Yinpeng Dong, and Jun Zhu. Towards robust detection of adversarial exam- ples. In Advances in Neural Information Processing Systems, pp. 4579 4589, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035, 2019. URL /ttps://pytoon models.html.
org/ docs/ stw.ltttabtteeeeeee staale//orchiissir...
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint arXiv:1701.06545 2017.
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Com- putational Mathematics and Mathematical Physics, 4(5):1-17, 1964.
Boris Teodorovich Polyak. Gradient methods for minimizing functionals. Zhurnal Vychislitel noi Matematiki Matematicheskoi Fiziki, 3(4):643-653. 1963.
Sashank J Reddi, Satyen Kale, and Sanjiv Kumar. On the convergence of adam and beyond. In International Conference on Learning Representations, 2018
Scott Reed, Honglak Lee, Dragomir Anguclov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. Training deep neural networks noisy labels with bootstrapping, arXiv preprint arXiv: 2014.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
10

---

Under review as a conference paper at ICLR 2021
Jin- Woo Seo, Hong-Gyu Jung, and Seong- Whan Lee. Self-auggmentation: Generalizing deep net- works to unseen classes for fho-shot learning. arXiv preprint arXiv:2004.00251, 2020.
Chaomin Shen, Yaxin Peng, Guixu Zhang, and Jinsong Fan. Defending against adversarial attacks by suppressing the largest eigenvalue of fisher information matrix. arXiv preprint arXiv:1909.066 17, 2019.
Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition- tangent distance and tangent propagation. In Neural networks: tricks of the Irade, pp. 239-274, Springer, 1998.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818 2826, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 1998-6008 2017.
C. Wah, S. Branson, P. Welinder, Perona, and Belongie. The Caltech-UCSD Birds-200-200 Dataset. Technical report, 2011.
Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu. On the inference calibration of neural machine translation. arXiv preprini arXiv:2005. 00963, 2020.
Zhe Wang, Kaiyi Ji, Yi Zhou, Yingbin Liang, and Vahid Tarokh. Spiderboost and momentum: Faster variance reduction algorithms. Advances in Neural Information Processing Systems, pp- 2403-2413, 2019.
Lingxi Xie, Jingdong Wang, Zhen Wei, Meng Wang, and Qi Tian. Disturblabel: Regularizing cnn on the loss layer. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4753 4762, 2016.
Yan Yan, Tianbao Yang, Zhe Li, Qihang Lin, and Yi Yang. A unified analysis of stochastic momen- tum methods for deep learning. In International Joint Conference on Artificial Intelligence, pp. 2955-2961, 2018
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: teacher-free framework. arXiv preprint arXiv:/909, 1199999117233 1723, 2019a.
Zhuoning Yuan, Yan Yan, Rong Jin, and Tianbao Yang. Stagewise training accelerates convergence of testing error over sgd. Advances in Neural Information Processing Systems, pp. 2604-2614, 2019b.
Sergey Zagoruyko and Nikos Komodakis, arXiv:1605.07146. 2016.
Wide residual networks.
arXiv preprint
Matthew D Zeiler. 2012.
Adadelta: an adaptive learning rate method. arXiv preprint 1rXiv:1212.5701,
Albert Zeyer, Kazuki Irie, Ralf Schlüter, and Hermann Ney. Improved training of end-to-end atten- tion models for speech recognition. Proc. Interspeech 2018, pp. 7-11, 2018.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8697-87 10, 2018
11

---

Under review as a conference paper at ICLR 2021
A
TECHNICAL LEMMA
Recall that the optimization problem is
min F(w) == Exx,y) [e(y, J(w;))); WEW
where the cross-entropy loss function e is given by
exp(fi(w;x))
[(;,](;;x)) =
yı log
(8)
プラ exp(S (w;()))))))
k
we set
exp(Ji(w:x))
p(w;x)= = (pr (w;x), x)........ PK (w: E IK p:(w;x) = log
(9)
Σ exp(s (w;)))
the problem (7) becomes
min F(w) = (xx)) [(,,)(;;>))].
(10)
wEW
Then the stochastic gradient with respective to
Vlly, f(;;)) (y, Vp(w:x)).
(11)
Lemma 1. Under Assumption (i), have
VF(W:)) [ ; - (1- 0)0² + 0802.
Proof.
By the facts of y!S = (1 0)y: 08, and the equation in (11), we have
Vily!s. f(w:x))) (1 V11-)Velyy..lll f(wxx.) AV ((wx))))
Therefore,
E Vely!" f(wxx)) VF(W))]1
E [1 -
OlV -VF(:)
(n)
s - (1- f(mxx))))(wxx))) VF(W:)) + oE - VF(w))
(6) (1 - 0)= + A
where (a) uses the convexity norm, i.c., -1(1-))) + 0bl s (1 0)||a|² + 0l[b/12 (b) uses
assumption (i) and the definitions in (4), and Assumption 1 (i).
B PROOF OF THEOREM 3
Proof. By the smoothness of objective function F(w) in Assumption (ii) and its remark, we have
F(W--1) - F(w:)
(VF(Ww,), Wet1 - w.) + /| -wel2
(4)
(VF(W.), Ve(y!S f(w::x))) T |[VELy!S. | ((wx))))]
( 1
/VF(w))l1 |/VF(..) - (ICY!s,((Wax))) + 7(LL-1)
Vely!s, ((w)x
2
2
/VF(:))1 -
(12)
12

---

Under review as a conference paper at ICLR 2021
where (a) is due the update of w1+1i (b) is due to (a. -b) [||l- -||a||2 ||b|l?) (c)
due to 7) + Taking the expectation over y!.) on the both sides of (12), have
E [[F(w(+1) - F(w))]
s E (VF(w:))) |1
- 7 VFF(w; Ve(y!' f(we:)))]
2
2
S " w: ||3)
(
+
((1 0)0² + +500))
(13)
where the last inequality is due Lemma
Then inequality (13) implies
2F(wo) 11
+ (1 - -0)6² + 080²
T/T
(a) (F)ww)
28
0²
T/T
1+
(b) 2F(w)) 280
T/T
where (a) due to 0 1 (b) due 1 S 1.
C CONVERGENCE ANALYSIS OF SGD WITHOUT LSR (0 0)
Theorem 5. Under Assumption 1, the solutions we from Algorithm I with A 0 satisfy
2F(wu)
Σ E [[VFF()))]]]
+ "Loo2.
In order to have E < 2, it suffices set 1j min
t
. and T
4F(wo) ne-
the
2Lo
total complexity is O )
Proof. By the smoothness of objective function F(w) in Assumption (ii) and its remark, we have F(W(+1) - F(w.)
L
<(VF(we), Wi+1 we) 1 W well?
n (VF(w)) Ve(ye. f(w;x)))) + n
[
(14)
where (a) is due to the update of w.41- Taking the expectation (xAy)) on the both sides of (14), we have
E [[F(w(+1) F(wi)]
(4)
T
+ < E [IVF(())))
E
VF(w:) + VF(w:))1
2
( E (1)
7/24 E
172 L²=LE E
V (VF((:))'
I 0².
" [ 19
(15)
2
where (a) and (b) use Assumption 1 (i); (c) uses the facts that S 1 inequality (15) implies
I. and Assumption 1 (i). The
T-1 Σ
2F(wu)
E
VF
< nf
+ +LLo?
T
L=
By setting 7) S 2f.22 and T
2
1F(wo) nr²
we have 1 EI-LE E (VF((:))) $ S. Thus the total
complexity is in the order of 0
0 (1)
13

---

Under review as a conference paper at ICLR 2021
D PROOF OF THEOREM 4
Proof. Following the similar analysis of inequality (13) from the proof of Theorem 3, we have
E [[ - F(we)]
" [I + ?h ((1 0)= + 08o²). 2
(16)
Using the condition in Assumption 2 we can simplify the inequality from (16) as
E [[F(w++1) F.]
<(1 MM)E F.1+ ((1 0)0² +8880²)
< (1-- - mp) (1-mp)) r [[F((w)) F.] 71 (1-0 +800)
Σ (1-m11/2) m2/22²
S - (1-mw))+
+ 2 ((1 - 0)0² 2 + 08o²)
11
Σ (1 m1/22)²
where the last inequality is due to the definition of loss function that F. 2 0. Since Th S I < 10
then (1 - 1-) < exp(-m + 1)) and Ei=o (1 - -mm)²= S mp As a result, for any Ti, we have
B[F(wT,) - F.] < exp( (1sp--111TT;/PP F(wo) + ((1 0)o² + +0002)
(17)
2p
Let 0 I+8 and 6h = (1 (1-0)o 0)0² + 080 = I+5
24
0²
then žn ((1 0)o² + 0600) < F(wo) since d
small enough and 7 s 1. By setting
(2F((wo)
Ti log
02
we have
0²
250²
B[F(WT.) F.]
(18)
After Ti iterations, we drop off the label smoothing, i.e. 0 0, then we know for any t 2 T₁ following the inequality (15) from the proof of Theorem 5, we have
72 E [w. + Lo² 2
E I( F(w.)] s
2
Therefore, we get
T++--
[FF(w))) <
E [F((wT) - F(WT,+73-1)) + To?
T
T2
(a) 2 [F(WTT)) - F.] + 1/2 L02
T2
(18) 4802
+ ?oLo?
(19)
umpT2
where (a) is due F(WT.+T,-1) 2 F.. By setting 1i2 FLoz and Ta
850
we have
Ts Ei. +T [VF(()) Se
14

---

Under review as a conference paper at ICLR 2021
E ADDITIONAL EXPERIMENTS
In this section, we study an ablation study for the smoothing parameter 8. We follow the same settings in Subsection 6.2 but use different values of 0 in LSR and TSLA. We summarize the results in Table 4, The results show that the different values of can affect the performances of LSR and TSLA. Besides, TSLA(180) has the best performance for each value of B.
Table 4: Comparison of Testing Top-1 I Accuracy for Different of LSR and TSLA (in %).
0
LSR
TSLA(120)
TSLA(140)
TSLA(160)
TSLA(180)
0.2
77.75
77.92
78.06
78.09
78.10
0,4
77.72
77.68
77.61
77.54
77.89
0.9
76.40
76.05
76.27
76.37
76.60
*TSLA(s): TSLA drops off LSR after epoch
15


Reviews:
[{"review_id": "wPPet3n6Q4U", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[strong points]\n1. This paper is smooth and well-motivated. Label smoothing is a well-known trick to improve the multi-class neural network. This is the first work to theoretically understand the effect of label smoothing by analyzing the convergence speed.\n\n2. Based on some reasonable assumption, this paper proves that SGD with LSR can have faster convergence speed than SGD without LSR when delta is small, and it will converge to a worse level of O(delta) when \\delta is large.\n\n3. By observing that LSR may have adverse effects during the later training epochs, a simple two-stage algorithm is proposed to take advantage of (i) fast convergence speed of SGD with LSR; (ii) easy training and better convergence guarantee of SGD without LSR.\n\n[negative points]\n1. Experiment is not enough. \n  (i). The authors mention several times that LSR may not work well for training teacher models. However, there is no evidence or analysis of whether the TSLA method can help solve this problem. \n  (ii) In theorem  3, \\theta is chosen to be 1/(1+delta). However, it is not clear whether different \\theta will influence performance. \n  (iii). The experiments are tested based on SGD. It is better to show some results on the other optimizers frequently used in deep learning, such as Adam or Adagrad. Meanwhile, it will be better to show that the bound in SGD can be generalized to the other stochastic optimizers.\n\n2. Even though the convergence speed is theoretically analyzed, it still does not explain why LSR can improve over SGD, namely the better convergence point. In other words, these bounds do not explain the gap among baseline, LSR, and TSLA in Figure 1. \n\n3. Lack of experiment setup. Little information is provided about the training techniques such as dropout, batch normalization. These techniques may have an influence on LSR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "W_w8QfwOjDD", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summarizing the paper:\nLabel smoothing is a popular regularization method that is used for deep learning models with classification problems, but has not been studied from a theoretical point of view extensively. This paper studies label smoothing from a optimization view and shows the convergence behavior when SGD is used, showing the improvement over the one-hot label learning under certain conditions. The paper then proposes a new method called Two-Stage Label smoothing algorithm (TSLA), which simply turns off the smoother after a certain number of epochs. According to the analysis, TSLA has improved convergence under certain conditions over the original label smoothing baseline. The experiments show how the proposed TSLA performs better than the label smoothing baseline for 3 image datasets.\n\nStrengths of the paper:\nThe paper's motivation is clear: to study a well known regularizer, label smoothing, from an optimization point of view. It compares the convergence properties of SGD without label smoothing and SGD with label smoothing. It further proposes an original method that first uses label smoothing but later turns it off, which is shown to be empirically better with some convergence properties.\n\nWeaknesses and issues of the paper:\nIn the beginning of Section 5, it discusses several papers that show the harm of label smoothing, under problem settings such as few-shot learning, knowledge distillation, and transfer learning, as the motivation for thinking of a two step strategy. However, this paper focuses on the original ordinary learning (no transfer learning and not few-shot learning) setup and the connection between these were not so clear.\n\nThe intuition behind why label smoothing may be harmful is given in page 5, but it would be nice to have some experiments to support these claims. For example, if optimizing all K loss functions with label smoothing is harder, having more classes should lead to more harm when label smoothing is used.\n\nFor Stanford Dogs and CUB-2011, 90 epochs looks like it is quite small. The learning curves for accuracy/loss still seems to be going up/down respectively for (60), (70), and (80).\n\nOther comments:\nIn the experiments, it would be better to have a validation dataset to choose the best hyper-parameter (the epoch number when LBR is dropped) and report the test performance based on the chosen one. (I'm guessing it will still be better than the baselines since in most hyper-parameter candidates, the proposed method seems to be working better.)\n\nIn page 8: op-5 --> top-5\n\nIn page 4: different --> difference\n\n================= after response \n\nThank you for providing the response. I would like to add some comments to the response.\n\nAbout Q1-- I agree that it is important to work on the original ordinary learning as the first paper in this direction, but it makes the story of the paper inconsistent. The adverse affect of LSR might only show up in other problem settings (distillation/few-shot/transfer learning) and not in ordinary learning, and if that is the case, it is unclear why the paper focuses on ordinary learning. (This is just a comment on the story/motivation of the paper.)\n\nAbout Q2-- Although Fig.1 shows the benefits of the one-hot label over label smoothed version in latter stages of training, I believe it does not show how the number of K plays a role in the performance and the difficulty.\n\nAbout Q3-- I agree that 90 epochs is sufficient for the LSR baseline, and the current figures demonstrate the benefits of the proposed method, but I was more interested in the difference of (s) in TSLA(s). From Fig.1, it still seems like TSLA(60,70,80) are improving at the end of 90 epochs, while TSLA (30,40,50) seem to have converged.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hkf5MNMqNg", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analyzes the convergence of SGD with a biased gradient, where the bias comes from label smoothing. The paper positions itself as a theorectical work towards understanding the success of such smoothing trick, but I feel the analysis does not add too much to the literature and its main results are somewhat trivial/misleading.\n\nIn particular, given that the analysis hinges on a specific loss function, i.e. Eq. (2), which is linear in terms of $y_i$, label smoothing quickly translates to additive terms that can be well-controlled separately from the unbiased gradient. It is thus not surprising that standard analysis of non-convex SGD goes through here with a new additive term appearing in the final gradient upper bound, as shown in the paper. For this reason, I do not see quite technical novelty in this paper.\n\nMy another concern is that the role of $\\delta$ is unclear. By Eq. (4), this quantity is data-independent and should be treated as only depending on the distribution of $(x, \\hat{y})$. Now if we take a close look at the main result, i.e. Theorem 3, it is fairly a weak result saying that SGD converges to a point with constant gradient. This is because the first case in Theorem 3 is essentially ensuring convergence to $\\epsilon$-stationary point only when $\\epsilon \\geq \\Omega(\\sqrt{\\delta}) \\geq \\Omega(1)$, i.e. a point with constant gradient. Likewise, the second case also boils down to the same guarantee.\n\nAs such, the comparison in Table 1 seems problematic since $\\epsilon$ should be treated as a variable arbitrarily close to 0. In words, only the first row of Table 1 will make sense. Yet, there are still two issues here.\n- I believe that the infinite iteration complexity of LSR is due to the drawback of Theorem 3 as I just pointed out. Namely, running LSR only gives you a point with constant gradient. Although TSLA does converge to $\\epsilon$-stationary point, such performance guarantee is *not* due to your design of TSLA, but follows from standard SGD. In fact, simply running SGD from the very beginning already gives such guarantee. Thus, the first row to compare TSLA and LSR is somewhat misleading.\n- Now the question boils down to why not simply running SGD. In the first row of the table (and main text), it is argued that TSLA (i.e. LSR + SGD) has improved iteration complexity of $O(\\delta / \\epsilon^4)$ which is better than the $O(1/\\epsilon^4)$ of vanilla SGD. I do not really agree with the conclusion because $\\delta$ is independent of $\\epsilon$. In fact, it can be a dimension-dependent quantity, and e.g. even blow up to $O(d)$ or so, making the iteration complexity of TSLA worse than SGD by a dimension-dependent factor.\n\nTo make the work qualify for a top-tier venue, authors need to either present a new algorithm/analysis with vanishing gradient, i.e. an upper bound of the form $O(\\delta \\cdot \\epsilon)$, or show hardness result, say any algorithm that takes the label smoothing must incur a stationary point with gradient $\\geq \\Omega(\\epsilon + \\delta)$.\n\n\n---------------------Updates after author response---------------------\n\n\nThe authors basically posted their response at the last minute of the window which eliminates the possibility for further discussion. While it is lengthy and point-to-point, I found it failed to clear up any of my concerns.\n\nI am very disappointed that even after they recognized the misleading arguments in \"convergence\" analysis, in Theorem 3 of the revised version, they are still claiming $\\| \\nabla F(w_R) \\| \\leq \\epsilon$ when $\\delta < O(\\epsilon^2)$ as \"convergence\". Such conclusion will be EXTREMELY MISLEADING if readers missed or did not carefully think about the condition on $\\epsilon$. Authors may want to refer to a calculus textbook for the rigorous definition of convergence.\n\nRegarding technical contribution,\n\n- Authors acknowledged that their analysis does *not* guarantee convergence to stationary point, since there is an additive $O(\\delta)$ term in the gradient upper bound. This immediately diminishes their theoretical contribution.\n\n- Authors did not justify their technical novelty. In fact, if we carefully look at their analysis in the appendix, it follows from standard SGD with a slight adjustment to the biased gradient induced by label smoothing (which is acknowledged by the authors). This is an additive gradient and thus is easily controlled; it is also the main reason why in their main theorem, the gradient upper bound suffers a non-vanishing $O(\\delta)$ term.\n\n- Authors argued that compared to running SGD from scratch, the benefit of LSR+SGD is the introduction of $\\delta$. However, this quantity itself is out of control. Note that even they were able to show $\\delta=O(1)$, it is not strong enough here since this can easily be obtained by running SGD. The only way that I see will save the paper is to show under distributional assumption of the data, that $\\delta = O(1/d)$ for example. However, I did not see how to make it happen, and authors completely ignored such analysis in their response.\n\nOverall, this is a paper playing tricks on its technical parts. It is decorated with bunch of mathmatical analysis most of which is known and standard, and the introduction of new insights is minimal. I will be shocked if it gets accepted in ICLR or equivalent conferences.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CAymXgoBYRy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Overview**\n\nThis paper investigates the effects of Label Smoothing Regularization and proposes two contributions: improved theoretical convergence guarantees for SGD with label-smoothing and a practical two-stage training strategy (TSLA) that uses label smoothing during the first several epochs and disables it thereafter.\n\nThe performance gains from TSLA are demonstrated both theoretically and in three image classification experiments (all three use ResNet18 + momentum) against reasonably tuned baselines. The overall readability of the paper is decent. There are numerous minor issues (see below), they are all trivial to fix.\n\n\n**Score & justification**\n\nWhile this paper definitely has scientific merit, i cannot recommend acceptance in its current form.\nMy main issue with the paper is with the claim that TSLA can work with any stochastic gradient optimizer (momentum, adam, etc, see p5 second to last paragraph). While I have no doubts that one can plug in any optimizer, the theoretical guarantees were provided for SGD, and, more importantly, the experimental results only cover SGD+Momentum (not to mention, a single model architecture). More complex optimizers (e.g. adam) and architectures (e.g. Transformer) may introduce problems when switching between training phases. Therefore, I suggest further supporting authors' claim by conducting additional experiments in other popular LSR use cases. \n\nFor example, one such use case is Machine Translation. Most recent models for this task use label smoothing to improve BLEU (e.g. [1, 2]). One can train a transformer-base model with Adam optimizer for IWSLT14 de-en[3] task in around the same time that it takes to train ResNet18 on CIFAR (see fairseq[4]). Of course, any other task would also support the claim, provided it doesn’t use the same model and/or optimizer.\n\n**[resolved]** Another issue I have is a subjective one (so it didn’t affect my score): it feels that the paper contains two independent lines of research with limited synergy. For instance, the theoretical study estimates label smoothing at convergence while TSLA seemingly disregards this by disabling label smoothing during the final stages of training.\n\n\n**Questions:**\n\n> Page 8: We use two different labels to smooth the one-hot label, the uniform distribution over all labels and the distribution predicted by an ImageNet pre-trained model which downloaded directly from PyTorch\n\n**[resolved]** Please clarify: how exactly do you adapt the predictions from 1000 ImageNet classes to CIFAR-100 classes?\nAs far as I understand, there are no pre-trained models in PyTorch itself. Did you mean TorchVision or torch Hub? If so, it would be great to state the exact model and library version to facilitate reproducibility. Alternatively, publishing the source code of the proposed solution should also do the trick as long as it clearly describes all the requirements.\n\n\n> Page 6 (Experiments): The momentum parameter is fixed as 0.9\n\n**[resolved]** Momentum SGD and other popular algorithms maintain a set of statistics accumulated over training batches. Please clarify: do these statistics carry over from stage 1 to stage 2 of TSLA? If not, what happens to them at T1?\n\n\n> Page 7,8: the value of theta\n\n**[resolved]** As far as i understand, you select theta by maximizing the performance of LSR baseline and then use the same theta for TSLA (if not, please clarify). Please elaborate on how exactly you tuned theta. Is there some intuition why optimal theta for LSR would generalize to TSLA? If not, how does TSLA perform under exaggerated theta?\n\n\n\n**Math**\n\n*Definition 1:*\n**[resolved]** The formulae use lowercase f, suggesting that the gradient/jacobian of the __model output__  should be zero. Did you mean capital F?\n\n*Assumption 2:*\n**[resolved]** Q1: Neural networks typically have many global optima due to inherent invariances in the model. Does this assumption need to hold for every F*?\n\n\n*Equation 4: {\\hat y} vs {y^{LS}}:*\n**[i was wrong]** As I understood, \\hat y is the common smoothing vector that is used to construct y^LS. Did you mean to use y^LS in the objective function? Note: the same expression with \\hat y is also used later on pages 3-4.\n\n\n**Typos & presentation:**\n\n**Page 2 (last paragraph)** *the example-label pairs are draw*\n\n**Page 3 (after remark 1)** *label y is draw from*\n\nI’d recommend changing “draw” to “drawn”.\n\n\n**Page 3 (after remark 1)** *Let y be a label introduced for smoothing label*\n\nWhile the sentence is technically correct, I would suggest paraphrasing.\n\n\n**Page 3, remark 2** *we do not require the stochastic gradient … is unbiased*\n\nSuggestion: “require **that** ... is unbiased” or \"require ... **to be** unbiased\"\n\n\n**Page 3 (assumption 2)** *i.e. there is no very bad local optimum on the surface of objective function*\n\nAgain, technically correct, but i’d recommend paraphrasing.\n\n\n**Page 4 (remark)**  *sample complexity for training a learning model from*\n\nDid you mean “training a model” (w/o learning) or “training a machine learning model”?\n\n\n**Page 4** *the different between SGD with LSR and SGD without LSR*\n\nDid you mean “the difference”?\n\n\n**Page 4 (bottom)** *miniImageNet*\n\nThis issue is of negligible importance, but i’d recommend using one of mini-ImageNet, Mini-ImageNet or MinImageNet.\n\n\n**Page 4 (bottom)** *between input example and output logit*\n\nMaybe there’s a missing “the”\n\n\n**Algorithm 2**\n\nShould T1 be listed as an input (along with theta, eta1, eta2)\n\n\n**Page 5** *It seems that training smoothed label in the late epochs*\n\nTraining **with** smoothed labels?\n\n\n**Page 5** *it runs a stochastic algorithm A(e.g., SGD) with LSR in T1 iterations*\n\nRuns … **for** T1 iterations?\n\n\n**Page 6** *LSR dose not converge*\n\nLSR **does** not converge\n\n\n**Page 7 (bottom)** *CIFRA-100*\n\nDid you mean **CIFAR**-100?\n\n\n**Page 7** *TSLA may not converges if it drops*\n\nmay not **converge** (without s)\n\n\n**Page 7** *… and divide them by 10 every 60 epochs suggested in ...*\n\n… **as** suggested in …?\n\n\n**Page 8** *pre-trained model which downloaded*\n\nwhich **is/was** downloaded?\n\n\n**Page 8 (conclusion)** *that TSLA benefits by LSR in the first stage*\n\nbenefits **from** LSR?\n\n\n**Page 8** *which verifies our theoretical findings in Sections 5 (Section 4)* \n\nSection/sections & formatting\n\n\n**References**\n\n[1] Attention is all you need, Vaswani et al. https://arxiv.org/pdf/1706.03762.pdf\n[2] Joint Source-Target Self Attention with Locality Constraints, Iyer et al, https://arxiv.org/pdf/1905.06596v1.pdf \n[3] http://workshop2014.iwslt.org/downloads/proceeding.pdf\n[4] see Fairseq library, examples/translation is a tutorial for IWSLT14 github.com/pytorch/fairseq/tree/master/examples/translation \n\n\n**EDIT** my original score was 5; i have read the authors response and updated my score to 6.\n\nMy score represents the experimental side of the paper, i do not feel confident judging the impact of theoretical contributions.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers ways to understand label smoothing methods, which are widely used in many applications.  There is some theory on the performance of SGD with and without the methods of the paper, but there is s significant gap in terms of how the theory offers insight into label smoothing.  There are some empirical results, but they are insufficient and there is not much description of the experimental setup.  There was a diversity of reviews.  But, after a discussion among reviewers, it was felt that, overall, another iteration on improving the coherence and presentation of the paper will make it much better for the community. 
Decision: Reject