Towards Understanding Label Smoothing

Abstract: Label smoothing regularization (LSR) is a prevalent component for training deep neural networks and can improve the generalization of models effectively. Although it achieves empirical success, the theoretical understanding about the power of label smoothing, especially about its influence on optimization, is still limited. In this work, we, for the first time, theoretically analyze the convergence behaviors of stochastic gradient descent with label smoothing in deep learning. Our analysis indicates that an appropriate LSR can speed up the convergence by reducing the variance in gradient, which provides a theoretical interpretation on the effectiveness of LSR. Besides, the analysis implies that LSR may slow down the convergence at the end of optimization. Therefore, a novel algorithm, namely Two-Stage LAbel smoothing (TSLA), is proposed to further improve the convergence. With the extensive analysis and experiments on benchmark data sets, the effectiveness of TSLA is verified both theoretically and empirically.

Reviews and author responses:

 Dear reviewer,

We would like to thank all reviewers for their precious time, their feedback, and their constructive suggestions. We tried our best to clarify the main concerns and we have updated the draft (all changes in the revision are marked in red for convenience). 

Specifically, we added the results of hyper-parameters $\theta$ and $s$ for different models and datasets in Table 5. We also included the results of additional models (ResNet-50, VGG, MobileNet, Inception, and EfficientNet) for three datasets CUB-2011, Stanford Dogs and CIFAR-100, which can be found in Tables 6, 7, 8. For the larger dataset, we add a new result for ImageNet with ResNet-50 (Table 9) due to the limited time and compute resources. We will add more results of ImageNet in the final version. 

We hope that our added new results and our responses will address your concerns. Please let us know if you have better suggestions. Thanks a lot!

Best,
Authors



 Dear Reviewer,

Thank you so much for reviewing our paper and our responses to the quetions are list below. 

(1) In the experiment with different theta, the case with theta=0.1 should be provided in the experimental results, which is a normal setting in the LSR.

A. First, we tuned the $\theta$ in a wide range for LSR and used the $\theta$ with the best performance of LSR. For a fair comparison, we used the some $\theta$ for TSLA without any additional tuning. Second, some tasks use $\theta=0.1$ due to they are trained from scratch while the tasks on CUB and Standford Dogs in this paper are fine-tuned on pre-trained models. Third, we conducted the experiments with $\theta=0.1$ and the results are presented as follows.

**Results for $\theta=0.1$**

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| dataset  | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
  | CUB-2011 | 75.75 | 75.25 | 76.80 ||| 93.46 | 91.80 | 93.73 |
 |   Stanford Dogs | 82.54 | 81.61 | 82.86 ||| 97.76  | 97.04 | 97.79 |

(2) On CIFAR-100, the proposed method achieves an extremely weak improvement, thus it is hard to confirm the effectiveness of TSLA.

A: TSLA obtained 78.55% top-1 accuracy on CIFAR-100, improving the LSR by 0.48%, which is not weak. It is worth mentioning that the improvements are obtained without any additional computation cost.



(3) To conduct a fair comparison, the LSR results with the same epochs as TSLA should be given in the experiments.

A: We did use the same epochs both for LSR and TSLA. Specifically, on Stanford Dogs and CUB, we trained 90 epochs for LSR as well as TSLA; while on CIFAR-100, we trained 200 epochs for LSR as well as TSLA. 



 Dear reviewer,

Thank you very much for spending time reviewing our paper. We would like to response the questions as follows.

(1) About the threshold $\delta$

A: We agree that, in theory, the condition of $\delta \le \epsilon^2/\sigma^2$ might be hard to achieve especially when the accuracy level is very high ($\epsilon$ is very small) and the variance $\sigma^2$ is large. The goal of using this threshold is to better understand LS from the view of optimization. The most important message is that it tells us that one needs to select an appropriate $\widehat{y}$ with a small variance $\widehat\sigma^2:=\delta\sigma^2$. For example, Table 2 shows that LSR-pre is better than LSR, where LSR-pre used a label $\widehat{y}$ based on the ImageNet pre-trained model and LSR used a label $\widehat{y} = 1/K$, and Table 4 (in Appendix) shows that the estimated $\delta$ for LSR-pre is 0.38, which is smaller than that for LSR (which is 0.55). On the other hand, even if this threshold is not achieved, TSLA can also has a better performance only requiring a weaker condition of $\delta < 1$ (please refer to Theorems 4 and 5). Please note that, in Appendix, we have estimated the value of $\delta$ in practice, which matches our theoretical findings. 


(2) Estimation of $T_1$
A: In fact, this is a very general but difficult problem in the field. Many existing works introduce hyperparameters and rely them to achieve good performances. Since the theoretical value of $T_1$ depends on many unknown parameters, its estimation is tough (also mentioned by the reviewer). As many previous works, one simple way to deal with this hyperparameter issue is to tune it in a wide range in practice (like learning rate). We agree that $T_1$ cannot be too small or too large to achieve its optimal value. However, we find in our experiments that TSLA somehow is robust to $T_1$: under different values of $T_1$, TSLA can better than LSR (please refer to Tables 1,2,3 for more details). 


(3) Ablation study of $\theta$.
A: First, we have tuned $\theta$ in a wide range for LSR and reported the one with best performance. For a fair comparison, we used the some $\theta$ for TSLA, while fine-tuning it may further improve our performance. Second, it is not surprise that the performance of TSLA depends on $\theta$ since it builds on LSR, whose performance relies on the quality of $\theta$. About $T_1$, please refer to Answer to the second question “estimation of $T_1$” for more details. We can see from Tables 1-3 that TSLA is robust to $T_1$. 


 Dear reviewer,

Thank you for reviewing our paper and providing many valuable suggestions. We have included the additional experiments in Appendix A2 of the revised version. We summarize the questions and our responses as follows.

(1) About overfitting perspective and builds on some assumption that “It seems that training with smoothed label in the late epochs makes the learning progress more difficult”. 

A: First, we did give analysis from the overfitting perspective by conducting generalization analysis. Please refer to Section 4.4 and Appendix F for details. Second, due to the true label is changed to a smoothed label, LSR is optimizing a shifted problem rather the original problem (similar to an out-of-distribution problem). That is to say, the model learned by LSR may be far away to the model of original problem since LSR could fit the artificial hard example (due to the artificial label such as $1/K$ used in label smoothing). However, generalization analysis is interested in the original problem, thus at the end of training, one needs to drop label smoothing to approach the original problem for better generalization.

(2) TSLA under data augmentation.

A: This is out of scope of this paper. As we mentioned in the paper, our main goal is to theoretical understanding label smoothing, and we did not consider any augmentations in our theoretical analysis since we wanted to better study the theoretical property of label smoothing without any additional effects. However, this is a very good suggestion and we can consider it as a future work.

(3) More backbones and datasets in experiments.

A: Thanks for the valuable suggestion!  Due to the time limitation of rebuttal, we try our best to do experiments as many as possible. We present 5 different backbones for datasets CUB-2011, Stanford Dogs, CIFAR-100, as well as the results for ImageNet in the following tabels. More experimental details can be found from Appendix A2 in the revision.

**CUB-2011**:

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| model     | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
| ResNet-50 |  80.15 |  82.76 |  82.83   | |   | 94.87 | 95.08 | 95.55|
   | VGG | 80.58 | 81.14  | 81.71   ||| 95.46 | 95.31 | 95.62  |
  |  MobileNet  | 78.44 |  79.91 | 80.14  ||| 94.58  | 94.39  |  94.94 |
  |  Inception | 79.08 | 80.12  | 80.19 |||  94.63 | 95.24  |  95.19 |
  |  EfficientNet | 81.12 | 80.95  | 81.55   ||| 95.53   | 95.25 | 95.56  |


**Stanford Dogs**:

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| model     | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
| ResNet-50 | 88.14 | 89.50  | 89.90  ||| 99.04 | 99.10  | 99.42  |
 |   VGG | 85.96 | 86.42  | 86.64   ||| 98.61  | 98.72 | 98.85 |
|    MobileNet | 83.05 | 83.62  | 84.34   ||| 97.98   | 97.68  | 98.40   |
|    Inception | 87.10 | 87.30   | 87.48  ||| 98.93  | 98.82  | 98.90   |
| EfficientNet | 85.72 | 85.71   | 86.26   ||| 98.80   | 98.60  | 98.78  |


**CIFAR-100**:

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| model     | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
  | ResNet-50 | 76.63 | 77.52 | 77.71 ||| 93.99 | 93.08 | 94.17 |
 |   VGG | 72.99 | 74.09 | 74.22 ||| 90.59 | 90.37 | 91.70 |
 |   MobileNet | 74.42 | 76.64 | 76.72||| 93.12 | 93.13 | 94.01|
  |  Inception | 79.95 | 79.71 | 79.87 ||| 95.23| 94.76 | 95.19|
 |   EfficientNet | 69.73 | 71.38 | 71.68 ||| 90.50 | 90.48 | 91.93 |

**ImageNet**:

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| model     | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
  | ResNet-50 | 76.35 | 76.96 | 77.09 | || 93.14 | 93.55 | 93.69|

(4) about weight decay.

A: For a fair comparison, we have adopted the best weight decay parameter for the two baselines (with and without label smoothing) that is shared in many existing works, while fine-tuning it may further improve the performance of TSLA.



 Dear reviewer,

Thank you very much for your hard work and valuable comments. We have conducted additional experiments and presented the details in Appendix A2 of the revised version. Here are our response to the questions. 

(1) With limited time and compute resources, we only obtained the results (see the table below) of ImageNet with ResNet-50 at this point. The result shows that TSLA and LSR are comparable.

**ImageNet**:

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| model     | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
  | ResNet-50 | 76.35 | 76.96 | 77.09 | || 93.14 | 93.55 | 93.69|

(2) We do not have any theoretical analysis for Top-5 accuracy. To the best of our knowledge, there are no studies on the theoretical analysis (both on optimization and generalization sides) of Top-5 accuracy. We list the Top-5 accuracy since it is an important measurement for evaluating models, which is also commonly used in some existing deep learning works. 

(3) We implemented 5 different models for datasets CUB-2011, Stanford Dogs, CIFAR-100. The results of top-1 accuracy in the following tables show that LSR outperforms baseline on most models except for EfficientNet (on CUB-2011 and Stanford Dogs) and Inception (on CIFAR-100). Besides, we also included the results of TSLA, showing that overall TSLA can outperform both baseline and LSR.

**CUB-2011**:

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| model     | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
| ResNet-50 |  80.15 |  82.76 |  82.83   | |   | 94.87 | 95.08 | 95.55|
   | VGG | 80.58 | 81.14  | 81.71   ||| 95.46 | 95.31 | 95.62  |
  |  MobileNet  | 78.44 |  79.91 | 80.14  ||| 94.58  | 94.39  |  94.94 |
  |  Inception | 79.08 | 80.12  | 80.19 |||  94.63 | 95.24  |  95.19 |
  |  EfficientNet | 81.12 | 80.95  | 81.55   ||| 95.53   | 95.25 | 95.56  |


**Stanford Dogs**:

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| model     | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
| ResNet-50 | 88.14 | 89.50  | 89.90  ||| 99.04 | 99.10  | 99.42  |
 |   VGG | 85.96 | 86.42  | 86.64   ||| 98.61  | 98.72 | 98.85 |
|    MobileNet | 83.05 | 83.62  | 84.34   ||| 97.98   | 97.68  | 98.40   |
|    Inception | 87.10 | 87.30   | 87.48  ||| 98.93  | 98.82  | 98.90   |
| EfficientNet | 85.72 | 85.71   | 86.26   ||| 98.80   | 98.60  | 98.78  |


**CIFAR-100**:

|           |      | top-1 |     |  | |      | top-5 |     |
| :----:       | :----:       |    :----:   |        :----: |    :----: |     :----: |     :----: |     :----: |     :----: |
| model     | baseline      | LSR  | TSLA     |  |  | baseline      | LSR  | TSLA     |
  | ResNet-50 | 76.63 | 77.52 | 77.71 ||| 93.99 | 93.08 | 94.17 |
 |   VGG | 72.99 | 74.09 | 74.22 ||| 90.59 | 90.37 | 91.70 |
 |   MobileNet | 74.42 | 76.64 | 76.72||| 93.12 | 93.13 | 94.01|
  |  Inception | 79.95 | 79.71 | 79.87 ||| 95.23| 94.76 | 95.19|
 |   EfficientNet | 69.73 | 71.38 | 71.68 ||| 90.50 | 90.48 | 91.93 |

(4) First, TSLA is simple yet efficient and new. Second, we provided a theoretical understanding of the convergence behavior of LSR, which has not been done yet. Based on the insight of theoretical result, the new algorithm TSLA was proposed. Third, we gave the analysis of TLSA both from optimization and generalization aspects, which is not trivial. We would like to mention that this paper gives a new way to use label smoothing, and its novelty lies at not only the proposed new algorithm but also the technical analysis. 


This paper makes some analysis on the convergence behaviors of SGD with label smoothing in deep learning. Further, a two-stage label smoothing algorithm is proposed to improve the convergence. Experiments are conducted to verify its effectiveness. Strengths:
This paper provides some theoretical explanation for the function of LSR and design a two-stage training method.

Weaknesses:
For the theoretical part, this paper does not make analysis from the overfitting perspective and builds on some assumption that “It seems that training with smoothed label in the late epochs makes the learning progress more difficult”. This is kind of conflicting with the overfitting viewpoint of hard examples issues, where LSR and knowledge distillation are applied to improve the generalization performance.
 
The baseline SGD without LSR also should be added into Figure 1. From Figure 2, we can see that for the baseline there is also almost no change in loss after 30 epochs. This effect maybe because of the data characteristics of the relatively small CUB-2011 dataset.
 
For the experiments, some strong data augmentation techniques, such as Mixup [1] and RandAugment [2], should be incorporated into the training process to see the effectiveness of the TSLA under data augmentation.

Besides ResNet-18 model, more backbones should be investigated like the ResNet-50 and InceptionNet.

The experimental datasets mainly focus on fine-grained image classification datasets which has a limited number of classes. To make the results more convincible, much larger datasets like the ImageNet dataset need to be further verified.

Since the weight decay parameters has an important impact on label smoothing regularization, different setups should be investigated.   

[1] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018

[2] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, 2020. Though this paper provides some theoretical analysis for the LSR, the proposed TSLA builds on some guesses like “training one-hot label could be “easier” than training smoothed label”. Further, much more experiments should be added to prove its effectiveness.

This work theoretically analyzes the convergence behaviors of stochastic gradient descent with label smoothing in deep learning, which implies that LSR may slow down the convergence at the end of optimization. The authors propose the two-stage label smoothing strategy to further improve the convergence. Experiments on several datasets also verify the effectiveness of TSLA. Pros:

(1) The presentation and organization are good. This paper is easy to read.

(2) The theoretical analysis for label smoothing understanding is interesting. The authors also provide detailed proof to support the statement.

(3) The proposed TSLA is simple yet effective on several small datasets.

Cons:

(1) My main concern lies in the experiments. Such a useful two-stage optimization strategy needs more experiments for verification. The datasets and CNN model adopted by this paper are small and shallow, respectively. I suggest that the authors could conduct experiments on a larger dataset with more deep models. For example, please show the results on ImageNet with ResNet-152. Otherwise, the presented results are not convincing.

(2) According to Table 1, the results of Top-5 accuracy for LSR are lower than the baseline. Is there any theoretical analysis?

(3) How about the generalization of label smoothing? Is it useful for different networks and different datasets? For example, can it improve the performance of ResNet-152, mobilenet, and shufflenet?

(4) For the proposed TSLA, the novelty is somehow limited. It is a combination of two commonly used strategies. But the analysis from the optimization aspect is interesting. For the label smoothing strategy, this paper theoretically analyzes the convergence behaviors and proposes the TSLA. Experiments on several small datasets demonstrate the effectiveness. More experiments are needed to verify its generalization.

This paper proved that when performing stochastic gradient descent for a non-convex objective function, an appropriate smooth parameter might speed up the convergence. The authors then proceed to introduce a two-stage TSLA algorithm which firstly learns with LSR. Then switch to learn with hard labels after certain epochs. Empirical evidence demonstrate the effectiveness of the proposed algorithm. $\textbf{Strengths}$
Strengths can be summarized as below:
* **Clarity** This paper is well organized and easy to follow.
* **Rationality** The assumptions adopted in this paper are either commonly used in the literature of non-convex
optimization or has theoretical/empirical validations. I didn't observe major flaws in the theories as well as proofs.

* **Quality** Overall, this is a novel work that provides insights on understanding the convergence behavior of label smoothing regularization. When optimizing with SGD, it is theoretically depicted that an appropriate LSR which reduces the variance can speed up convergence and require less sample complexity.

$\textbf{Weaknesses}$
* **The threshold of $\delta$ in Theorem 3** In Theorem 3, the threshold of $\delta$ is shown to be $\frac{\epsilon^2}{4\sigma^2}$ in order to reach an $\epsilon-$stationary point. Note that this threshold is dependent on $\epsilon$, it seems that the condition $\delta\leq\frac{\epsilon^2}{4\sigma^2}$ might be hard to achieve. It would be better if the authors could provide me with more explanations or empirical validations regarding this threshold.
* **Estimation of $T_1$**  One potential weakness of this work is the estimation of $T_1$. Clearly, if we adopt a smaller value of $T_1$, the performance of TSLA is very likely to perform worse than learning without LSR. And if estimate $T_1$ is larger than its theoretical form, the convergence (training time) of TSLA will be longer. Besides, I feel like the estimation of $T_1$ is a tough task. Thus, I think this might be a potential weakness. 
* **Ablation study of $\theta$** In Table 3, as mentioned by authors, large $\theta$ (i.e., $\theta>0.4$) frequently results in the worse performance of TSLA than LSR for different value of $T_1$. It seems that a good performance of TSLA relies on the quality of $\theta$ and $T_1$.    This paper is overall an interesting paper. Although I still have some additional concerns regarding the threshold of $\delta$ in Theorem 3, and empirical aspects of $T_1$ and $\theta$ in the proposed TSLA. I am willing to increase the score if the authors well address my concerns.

This paper studies the power of Label smoothing regularization (LSR) in training deep neural networks by analyzing SGD with LSR in different non-convex optimization settings. The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence. Besides, this paper proposes a simple yet efficient two-stage label smooth (TSLA) whose basic idea is to switch the training from smoothed label to one-hot label. Integrating TSLA with SGD can produce the improved convergence result that TSLA benefits from LSR in the first stage and essentially converges faster in the second stage. Extensive experiments show that TSLA improves the generalization accuracy of deep models on several benchmark datasets. This paper has some positive aspects as follows:
1. The proposed theoretical analysis of LSR is valuable. This paper gives the theoretical explanation of why an appropriate LSR can help speed up the convergence.
2.  A improved method is developed. According to the proposed theoretical analysis, this paper further proposes a two-stage label smooth method to enhance the generation ability of LSR.
3.  The paper is well written, the arguments are clear, and the methodology is well presented.

However, the authors need to deal with some issues as follows:

4. In the experiment with different theta, the case with theta=0.1 should be provided in the experimental results, which is a normal setting in the LSR.
5. On CIFAR-100, the proposed method achieves an extremely weak improvement, thus it is hard to confirm the effectiveness of TSLA.
6. To conduct a fair comparison, the LSR results with the same epochs as TSLA should be given in the experiments.
 The proposed theoretical explanation for LSR is meaningful, however, the experiment in this paper needs to be further enhanced. 

Meta-review: The authors proposed a two-stage algorithm for exploiting label smoothing and provided some analysis based on how label smoothing may have reduced the variance in the stochastic gradient. While the authors provided substantial experiments to justify their work (with additional ones during the response stage), none of the reviewers was very excited in the end, for obvious reasons perhaps: (a) the two-stage algorithm is a straightforward combination of existing practices (first run with label smoothing and then run without label smoothing), without any new, interesting insight from the authors' side; (b) the analysis is a direct consequence of the authors' assumptions. Basically, if label smoothing reduces variance, SGD would converge faster and vice versa, which is nothing surprising or insightful. The key is to understand when and how any particular way to smooth the label would lead to significant reduction of the variance, which the authors did not provide any guidance or insight other than offering some empirical results. Overall, we do not believe this work, in its current form, adds significant value to our understanding of label smoothing.

Decision: Reject