Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: [strong points]
1. This paper is smooth and well-motivated. Label smoothing is a well-known trick to improve the multi-class neural network. This is the first work to theoretically understand the effect of label smoothing by analyzing the convergence speed.

2. Based on some reasonable assumption, this paper proves that SGD with LSR can have faster convergence speed than SGD without LSR when delta is small, and it will converge to a worse level of O(delta) when \delta is large.

3. By observing that LSR may have adverse effects during the later training epochs, a simple two-stage algorithm is proposed to take advantage of (i) fast convergence speed of SGD with LSR; (ii) easy training and better convergence guarantee of SGD without LSR.

[negative points]
1. Experiment is not enough. 
  (i). The authors mention several times that LSR may not work well for training teacher models. However, there is no evidence or analysis of whether the TSLA method can help solve this problem. 
  (ii) In theorem  3, \theta is chosen to be 1/(1+delta). However, it is not clear whether different \theta will influence performance. 
  (iii). The experiments are tested based on SGD. It is better to show some results on the other optimizers frequently used in deep learning, such as Adam or Adagrad. Meanwhile, it will be better to show that the bound in SGD can be generalized to the other stochastic optimizers.

2. Even though the convergence speed is theoretically analyzed, it still does not explain why LSR can improve over SGD, namely the better convergence point. In other words, these bounds do not explain the gap among baseline, LSR, and TSLA in Figure 1. 

3. Lack of experiment setup. Little information is provided about the training techniques such as dropout, batch normalization. These techniques may have an influence on LSR.
review_title: An Interesting Paper Understanding Label Smoothing Techniques

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summarizing the paper:
Label smoothing is a popular regularization method that is used for deep learning models with classification problems, but has not been studied from a theoretical point of view extensively. This paper studies label smoothing from a optimization view and shows the convergence behavior when SGD is used, showing the improvement over the one-hot label learning under certain conditions. The paper then proposes a new method called Two-Stage Label smoothing algorithm (TSLA), which simply turns off the smoother after a certain number of epochs. According to the analysis, TSLA has improved convergence under certain conditions over the original label smoothing baseline. The experiments show how the proposed TSLA performs better than the label smoothing baseline for 3 image datasets.

Strengths of the paper:
The paper's motivation is clear: to study a well known regularizer, label smoothing, from an optimization point of view. It compares the convergence properties of SGD without label smoothing and SGD with label smoothing. It further proposes an original method that first uses label smoothing but later turns it off, which is shown to be empirically better with some convergence properties.

Weaknesses and issues of the paper:
In the beginning of Section 5, it discusses several papers that show the harm of label smoothing, under problem settings such as few-shot learning, knowledge distillation, and transfer learning, as the motivation for thinking of a two step strategy. However, this paper focuses on the original ordinary learning (no transfer learning and not few-shot learning) setup and the connection between these were not so clear.

The intuition behind why label smoothing may be harmful is given in page 5, but it would be nice to have some experiments to support these claims. For example, if optimizing all K loss functions with label smoothing is harder, having more classes should lead to more harm when label smoothing is used.

For Stanford Dogs and CUB-2011, 90 epochs looks like it is quite small. The learning curves for accuracy/loss still seems to be going up/down respectively for (60), (70), and (80).

Other comments:
In the experiments, it would be better to have a validation dataset to choose the best hyper-parameter (the epoch number when LBR is dropped) and report the test performance based on the chosen one. (I'm guessing it will still be better than the baselines since in most hyper-parameter candidates, the proposed method seems to be working better.)

In page 8: op-5 --> top-5

In page 4: different --> difference

================= after response 

Thank you for providing the response. I would like to add some comments to the response.

About Q1-- I agree that it is important to work on the original ordinary learning as the first paper in this direction, but it makes the story of the paper inconsistent. The adverse affect of LSR might only show up in other problem settings (distillation/few-shot/transfer learning) and not in ordinary learning, and if that is the case, it is unclear why the paper focuses on ordinary learning. (This is just a comment on the story/motivation of the paper.)

About Q2-- Although Fig.1 shows the benefits of the one-hot label over label smoothed version in latter stages of training, I believe it does not show how the number of K plays a role in the performance and the difficulty.

About Q3-- I agree that 90 epochs is sufficient for the LSR baseline, and the current figures demonstrate the benefits of the proposed method, but I was more interested in the difference of (s) in TSLA(s). From Fig.1, it still seems like TSLA(60,70,80) are improving at the end of 90 epochs, while TSLA (30,40,50) seem to have converged.

review_title: Gives theoretical understanding of label smoothing from the optimization point of view and gives a practical extension of label smoothing

final_score: rating: 1: Trivial or wrong
review_content: This paper analyzes the convergence of SGD with a biased gradient, where the bias comes from label smoothing. The paper positions itself as a theorectical work towards understanding the success of such smoothing trick, but I feel the analysis does not add too much to the literature and its main results are somewhat trivial/misleading.

In particular, given that the analysis hinges on a specific loss function, i.e. Eq. (2), which is linear in terms of $y_i$, label smoothing quickly translates to additive terms that can be well-controlled separately from the unbiased gradient. It is thus not surprising that standard analysis of non-convex SGD goes through here with a new additive term appearing in the final gradient upper bound, as shown in the paper. For this reason, I do not see quite technical novelty in this paper.

My another concern is that the role of $\delta$ is unclear. By Eq. (4), this quantity is data-independent and should be treated as only depending on the distribution of $(x, \hat{y})$. Now if we take a close look at the main result, i.e. Theorem 3, it is fairly a weak result saying that SGD converges to a point with constant gradient. This is because the first case in Theorem 3 is essentially ensuring convergence to $\epsilon$-stationary point only when $\epsilon \geq \Omega(\sqrt{\delta}) \geq \Omega(1)$, i.e. a point with constant gradient. Likewise, the second case also boils down to the same guarantee.

As such, the comparison in Table 1 seems problematic since $\epsilon$ should be treated as a variable arbitrarily close to 0. In words, only the first row of Table 1 will make sense. Yet, there are still two issues here.
- I believe that the infinite iteration complexity of LSR is due to the drawback of Theorem 3 as I just pointed out. Namely, running LSR only gives you a point with constant gradient. Although TSLA does converge to $\epsilon$-stationary point, such performance guarantee is *not* due to your design of TSLA, but follows from standard SGD. In fact, simply running SGD from the very beginning already gives such guarantee. Thus, the first row to compare TSLA and LSR is somewhat misleading.
- Now the question boils down to why not simply running SGD. In the first row of the table (and main text), it is argued that TSLA (i.e. LSR + SGD) has improved iteration complexity of $O(\delta / \epsilon^4)$ which is better than the $O(1/\epsilon^4)$ of vanilla SGD. I do not really agree with the conclusion because $\delta$ is independent of $\epsilon$. In fact, it can be a dimension-dependent quantity, and e.g. even blow up to $O(d)$ or so, making the iteration complexity of TSLA worse than SGD by a dimension-dependent factor.

To make the work qualify for a top-tier venue, authors need to either present a new algorithm/analysis with vanishing gradient, i.e. an upper bound of the form $O(\delta \cdot \epsilon)$, or show hardness result, say any algorithm that takes the label smoothing must incur a stationary point with gradient $\geq \Omega(\epsilon + \delta)$.


---------------------Updates after author response---------------------


The authors basically posted their response at the last minute of the window which eliminates the possibility for further discussion. While it is lengthy and point-to-point, I found it failed to clear up any of my concerns.

I am very disappointed that even after they recognized the misleading arguments in "convergence" analysis, in Theorem 3 of the revised version, they are still claiming $\| \nabla F(w_R) \| \leq \epsilon$ when $\delta < O(\epsilon^2)$ as "convergence". Such conclusion will be EXTREMELY MISLEADING if readers missed or did not carefully think about the condition on $\epsilon$. Authors may want to refer to a calculus textbook for the rigorous definition of convergence.

Regarding technical contribution,

- Authors acknowledged that their analysis does *not* guarantee convergence to stationary point, since there is an additive $O(\delta)$ term in the gradient upper bound. This immediately diminishes their theoretical contribution.

- Authors did not justify their technical novelty. In fact, if we carefully look at their analysis in the appendix, it follows from standard SGD with a slight adjustment to the biased gradient induced by label smoothing (which is acknowledged by the authors). This is an additive gradient and thus is easily controlled; it is also the main reason why in their main theorem, the gradient upper bound suffers a non-vanishing $O(\delta)$ term.

- Authors argued that compared to running SGD from scratch, the benefit of LSR+SGD is the introduction of $\delta$. However, this quantity itself is out of control. Note that even they were able to show $\delta=O(1)$, it is not strong enough here since this can easily be obtained by running SGD. The only way that I see will save the paper is to show under distributional assumption of the data, that $\delta = O(1/d)$ for example. However, I did not see how to make it happen, and authors completely ignored such analysis in their response.

Overall, this is a paper playing tricks on its technical parts. It is decorated with bunch of mathmatical analysis most of which is known and standard, and the introduction of new insights is minimal. I will be shocked if it gets accepted in ICLR or equivalent conferences.
review_title: trivial results, strong rejection after reading author response 

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Overview**

This paper investigates the effects of Label Smoothing Regularization and proposes two contributions: improved theoretical convergence guarantees for SGD with label-smoothing and a practical two-stage training strategy (TSLA) that uses label smoothing during the first several epochs and disables it thereafter.

The performance gains from TSLA are demonstrated both theoretically and in three image classification experiments (all three use ResNet18 + momentum) against reasonably tuned baselines. The overall readability of the paper is decent. There are numerous minor issues (see below), they are all trivial to fix.


**Score & justification**

While this paper definitely has scientific merit, i cannot recommend acceptance in its current form.
My main issue with the paper is with the claim that TSLA can work with any stochastic gradient optimizer (momentum, adam, etc, see p5 second to last paragraph). While I have no doubts that one can plug in any optimizer, the theoretical guarantees were provided for SGD, and, more importantly, the experimental results only cover SGD+Momentum (not to mention, a single model architecture). More complex optimizers (e.g. adam) and architectures (e.g. Transformer) may introduce problems when switching between training phases. Therefore, I suggest further supporting authors' claim by conducting additional experiments in other popular LSR use cases. 

For example, one such use case is Machine Translation. Most recent models for this task use label smoothing to improve BLEU (e.g. [1, 2]). One can train a transformer-base model with Adam optimizer for IWSLT14 de-en[3] task in around the same time that it takes to train ResNet18 on CIFAR (see fairseq[4]). Of course, any other task would also support the claim, provided it doesn’t use the same model and/or optimizer.

**[resolved]** Another issue I have is a subjective one (so it didn’t affect my score): it feels that the paper contains two independent lines of research with limited synergy. For instance, the theoretical study estimates label smoothing at convergence while TSLA seemingly disregards this by disabling label smoothing during the final stages of training.


**Questions:**

> Page 8: We use two different labels to smooth the one-hot label, the uniform distribution over all labels and the distribution predicted by an ImageNet pre-trained model which downloaded directly from PyTorch

**[resolved]** Please clarify: how exactly do you adapt the predictions from 1000 ImageNet classes to CIFAR-100 classes?
As far as I understand, there are no pre-trained models in PyTorch itself. Did you mean TorchVision or torch Hub? If so, it would be great to state the exact model and library version to facilitate reproducibility. Alternatively, publishing the source code of the proposed solution should also do the trick as long as it clearly describes all the requirements.


> Page 6 (Experiments): The momentum parameter is fixed as 0.9

**[resolved]** Momentum SGD and other popular algorithms maintain a set of statistics accumulated over training batches. Please clarify: do these statistics carry over from stage 1 to stage 2 of TSLA? If not, what happens to them at T1?


> Page 7,8: the value of theta

**[resolved]** As far as i understand, you select theta by maximizing the performance of LSR baseline and then use the same theta for TSLA (if not, please clarify). Please elaborate on how exactly you tuned theta. Is there some intuition why optimal theta for LSR would generalize to TSLA? If not, how does TSLA perform under exaggerated theta?



**Math**

*Definition 1:*
**[resolved]** The formulae use lowercase f, suggesting that the gradient/jacobian of the __model output__  should be zero. Did you mean capital F?

*Assumption 2:*
**[resolved]** Q1: Neural networks typically have many global optima due to inherent invariances in the model. Does this assumption need to hold for every F*?


*Equation 4: {\hat y} vs {y^{LS}}:*
**[i was wrong]** As I understood, \hat y is the common smoothing vector that is used to construct y^LS. Did you mean to use y^LS in the objective function? Note: the same expression with \hat y is also used later on pages 3-4.


**Typos & presentation:**

**Page 2 (last paragraph)** *the example-label pairs are draw*

**Page 3 (after remark 1)** *label y is draw from*

I’d recommend changing “draw” to “drawn”.


**Page 3 (after remark 1)** *Let y be a label introduced for smoothing label*

While the sentence is technically correct, I would suggest paraphrasing.


**Page 3, remark 2** *we do not require the stochastic gradient … is unbiased*

Suggestion: “require **that** ... is unbiased” or "require ... **to be** unbiased"


**Page 3 (assumption 2)** *i.e. there is no very bad local optimum on the surface of objective function*

Again, technically correct, but i’d recommend paraphrasing.


**Page 4 (remark)**  *sample complexity for training a learning model from*

Did you mean “training a model” (w/o learning) or “training a machine learning model”?


**Page 4** *the different between SGD with LSR and SGD without LSR*

Did you mean “the difference”?


**Page 4 (bottom)** *miniImageNet*

This issue is of negligible importance, but i’d recommend using one of mini-ImageNet, Mini-ImageNet or MinImageNet.


**Page 4 (bottom)** *between input example and output logit*

Maybe there’s a missing “the”


**Algorithm 2**

Should T1 be listed as an input (along with theta, eta1, eta2)


**Page 5** *It seems that training smoothed label in the late epochs*

Training **with** smoothed labels?


**Page 5** *it runs a stochastic algorithm A(e.g., SGD) with LSR in T1 iterations*

Runs … **for** T1 iterations?


**Page 6** *LSR dose not converge*

LSR **does** not converge


**Page 7 (bottom)** *CIFRA-100*

Did you mean **CIFAR**-100?


**Page 7** *TSLA may not converges if it drops*

may not **converge** (without s)


**Page 7** *… and divide them by 10 every 60 epochs suggested in ...*

… **as** suggested in …?


**Page 8** *pre-trained model which downloaded*

which **is/was** downloaded?


**Page 8 (conclusion)** *that TSLA benefits by LSR in the first stage*

benefits **from** LSR?


**Page 8** *which verifies our theoretical findings in Sections 5 (Section 4)* 

Section/sections & formatting


**References**

[1] Attention is all you need, Vaswani et al. https://arxiv.org/pdf/1706.03762.pdf
[2] Joint Source-Target Self Attention with Locality Constraints, Iyer et al, https://arxiv.org/pdf/1905.06596v1.pdf 
[3] http://workshop2014.iwslt.org/downloads/proceeding.pdf
[4] see Fairseq library, examples/translation is a tutorial for IWSLT14 github.com/pytorch/fairseq/tree/master/examples/translation 


**EDIT** my original score was 5; i have read the authors response and updated my score to 6.

My score represents the experimental side of the paper, i do not feel confident judging the impact of theoretical contributions.

review_title: Initial review

Meta-review: The paper considers ways to understand label smoothing methods, which are widely used in many applications.  There is some theory on the performance of SGD with and without the methods of the paper, but there is s significant gap in terms of how the theory offers insight into label smoothing.  There are some empirical results, but they are insufficient and there is not much description of the experimental setup.  There was a diversity of reviews.  But, after a discussion among reviewers, it was felt that, overall, another iteration on improving the coherence and presentation of the paper will make it much better for the community. 
Decision: Reject