Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper makes some analysis on the convergence behaviors of SGD with label smoothing in deep learning. Further, a two-stage label smoothing algorithm is proposed to improve the convergence. Experiments are conducted to verify its effectiveness.
main_review: Strengths:
This paper provides some theoretical explanation for the function of LSR and design a two-stage training method.

Weaknesses:
For the theoretical part, this paper does not make analysis from the overfitting perspective and builds on some assumption that “It seems that training with smoothed label in the late epochs makes the learning progress more difficult”. This is kind of conflicting with the overfitting viewpoint of hard examples issues, where LSR and knowledge distillation are applied to improve the generalization performance.
 
The baseline SGD without LSR also should be added into Figure 1. From Figure 2, we can see that for the baseline there is also almost no change in loss after 30 epochs. This effect maybe because of the data characteristics of the relatively small CUB-2011 dataset.
 
For the experiments, some strong data augmentation techniques, such as Mixup [1] and RandAugment [2], should be incorporated into the training process to see the effectiveness of the TSLA under data augmentation.

Besides ResNet-18 model, more backbones should be investigated like the ResNet-50 and InceptionNet.

The experimental datasets mainly focus on fine-grained image classification datasets which has a limited number of classes. To make the results more convincible, much larger datasets like the ImageNet dataset need to be further verified.

Since the weight decay parameters has an important impact on label smoothing regularization, different setups should be investigated.   

[1] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018

[2] Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical automated data augmentation with a reduced search space. In CVPR Workshops, 2020.
summary_of_the_review: Though this paper provides some theoretical analysis for the LSR, the proposed TSLA builds on some guesses like “training one-hot label could be “easier” than training smoothed label”. Further, much more experiments should be added to prove its effectiveness.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work theoretically analyzes the convergence behaviors of stochastic gradient descent with label smoothing in deep learning, which implies that LSR may slow down the convergence at the end of optimization. The authors propose the two-stage label smoothing strategy to further improve the convergence. Experiments on several datasets also verify the effectiveness of TSLA.
main_review: Pros:

(1) The presentation and organization are good. This paper is easy to read.

(2) The theoretical analysis for label smoothing understanding is interesting. The authors also provide detailed proof to support the statement.

(3) The proposed TSLA is simple yet effective on several small datasets.

Cons:

(1) My main concern lies in the experiments. Such a useful two-stage optimization strategy needs more experiments for verification. The datasets and CNN model adopted by this paper are small and shallow, respectively. I suggest that the authors could conduct experiments on a larger dataset with more deep models. For example, please show the results on ImageNet with ResNet-152. Otherwise, the presented results are not convincing.

(2) According to Table 1, the results of Top-5 accuracy for LSR are lower than the baseline. Is there any theoretical analysis?

(3) How about the generalization of label smoothing? Is it useful for different networks and different datasets? For example, can it improve the performance of ResNet-152, mobilenet, and shufflenet?

(4) For the proposed TSLA, the novelty is somehow limited. It is a combination of two commonly used strategies. But the analysis from the optimization aspect is interesting.
summary_of_the_review: For the label smoothing strategy, this paper theoretically analyzes the convergence behaviors and proposes the TSLA. Experiments on several small datasets demonstrate the effectiveness. More experiments are needed to verify its generalization.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proved that when performing stochastic gradient descent for a non-convex objective function, an appropriate smooth parameter might speed up the convergence. The authors then proceed to introduce a two-stage TSLA algorithm which firstly learns with LSR. Then switch to learn with hard labels after certain epochs. Empirical evidence demonstrate the effectiveness of the proposed algorithm.
main_review: $\textbf{Strengths}$
Strengths can be summarized as below:
* **Clarity** This paper is well organized and easy to follow.
* **Rationality** The assumptions adopted in this paper are either commonly used in the literature of non-convex
optimization or has theoretical/empirical validations. I didn't observe major flaws in the theories as well as proofs.

* **Quality** Overall, this is a novel work that provides insights on understanding the convergence behavior of label smoothing regularization. When optimizing with SGD, it is theoretically depicted that an appropriate LSR which reduces the variance can speed up convergence and require less sample complexity.

$\textbf{Weaknesses}$
* **The threshold of $\delta$ in Theorem 3** In Theorem 3, the threshold of $\delta$ is shown to be $\frac{\epsilon^2}{4\sigma^2}$ in order to reach an $\epsilon-$stationary point. Note that this threshold is dependent on $\epsilon$, it seems that the condition $\delta\leq\frac{\epsilon^2}{4\sigma^2}$ might be hard to achieve. It would be better if the authors could provide me with more explanations or empirical validations regarding this threshold.
* **Estimation of $T_1$**  One potential weakness of this work is the estimation of $T_1$. Clearly, if we adopt a smaller value of $T_1$, the performance of TSLA is very likely to perform worse than learning without LSR. And if estimate $T_1$ is larger than its theoretical form, the convergence (training time) of TSLA will be longer. Besides, I feel like the estimation of $T_1$ is a tough task. Thus, I think this might be a potential weakness. 
* **Ablation study of $\theta$** In Table 3, as mentioned by authors, large $\theta$ (i.e., $\theta>0.4$) frequently results in the worse performance of TSLA than LSR for different value of $T_1$. It seems that a good performance of TSLA relies on the quality of $\theta$ and $T_1$.   
summary_of_the_review: This paper is overall an interesting paper. Although I still have some additional concerns regarding the threshold of $\delta$ in Theorem 3, and empirical aspects of $T_1$ and $\theta$ in the proposed TSLA. I am willing to increase the score if the authors well address my concerns.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the power of Label smoothing regularization (LSR) in training deep neural networks by analyzing SGD with LSR in different non-convex optimization settings. The convergence results show that an appropriate LSR with reduced label variance can help speed up the convergence. Besides, this paper proposes a simple yet efficient two-stage label smooth (TSLA) whose basic idea is to switch the training from smoothed label to one-hot label. Integrating TSLA with SGD can produce the improved convergence result that TSLA benefits from LSR in the first stage and essentially converges faster in the second stage. Extensive experiments show that TSLA improves the generalization accuracy of deep models on several benchmark datasets.
main_review: This paper has some positive aspects as follows:
1. The proposed theoretical analysis of LSR is valuable. This paper gives the theoretical explanation of why an appropriate LSR can help speed up the convergence.
2.  A improved method is developed. According to the proposed theoretical analysis, this paper further proposes a two-stage label smooth method to enhance the generation ability of LSR.
3.  The paper is well written, the arguments are clear, and the methodology is well presented.

However, the authors need to deal with some issues as follows:

4. In the experiment with different theta, the case with theta=0.1 should be provided in the experimental results, which is a normal setting in the LSR.
5. On CIFAR-100, the proposed method achieves an extremely weak improvement, thus it is hard to confirm the effectiveness of TSLA.
6. To conduct a fair comparison, the LSR results with the same epochs as TSLA should be given in the experiments.

summary_of_the_review: The proposed theoretical explanation for LSR is meaningful, however, the experiment in this paper needs to be further enhanced.
review_title: null

Meta-review: The authors proposed a two-stage algorithm for exploiting label smoothing and provided some analysis based on how label smoothing may have reduced the variance in the stochastic gradient. While the authors provided substantial experiments to justify their work (with additional ones during the response stage), none of the reviewers was very excited in the end, for obvious reasons perhaps: (a) the two-stage algorithm is a straightforward combination of existing practices (first run with label smoothing and then run without label smoothing), without any new, interesting insight from the authors' side; (b) the analysis is a direct consequence of the authors' assumptions. Basically, if label smoothing reduces variance, SGD would converge faster and vice versa, which is nothing surprising or insightful. The key is to understand when and how any particular way to smooth the label would lead to significant reduction of the variance, which the authors did not provide any guidance or insight other than offering some empirical results. Overall, we do not believe this work, in its current form, adds significant value to our understanding of label smoothing.
Decision: Reject