Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Label smoothing is a popular approach to regularize modern neural networks. However, the amount of smoothing is the same across all samples in the dataset, which can be sub-optimal. In this paper, the authors of the paper proposed a novel adaptive label smoothing method so that each sample gets a different amount of smoothing. The key insight of the paper involves using the entropy of predictions from past timestamps as a way to quantify the amount of label smoothing applied to samples. The authors of the paper demonstrate empirically the effectiveness of the proposed method. 

main_review: Strengths 
- Overall, the paper is well written, and the proposed method is well motivated. 
- Experiments seem to demonstrate the effectiveness of the proposed method. 
- Thorough empirical analysis is done on several benchmark datasets. 
- A careful ablation study is also conducted to further demonstrate the effectiveness of the proposed method. 

Weakness 
- The main weakness of the paper in my opinion is the lack of novelty. The proposed method in my opinion is very similar to the previous works like [1], which proposed to use predictions from previous time stamps for self-distillation, and [2], which also proposed a method for adaptive label smoothing based on predictions from previous time stamps. While the authors of the paper addressed the key difference between the proposed method and [1], I do still feel that the proposed method lack novelty, despite good improvements in performance. On top of [1] and [2], [3] and [4] are also potentially relevant prio works to potentially discuss in literature review and benchmark against. 

[1] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets, 2021.

[2] Zhang, Zhilu, and Mert R. Sabuncu. "Self-distillation as instance-specific label smoothing." arXiv preprint arXiv:2006.05065 (2020).

[3] Li, Xingjian, et al. "One Generation Knowledge Distillation by Utilizing Peer Samples." (2019).

[4] Yun, Sukmin, et al. "Regularizing Predictions via Class-wise Self-knowledge Distillation." (2019).

summary_of_the_review: Overall, despite the good empirical performance, I think the lack of novelty is a significant weakness of the paper. As such, I recommend weakly rejecting the paper for now. 

After Rebuttal:
I would like to thank the authors of the paper for all clarifications and additional experiments. After reading the response and other reviewers' comments, I am raising my score to 6.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method that is based on label smoothing.
The original label smoothing uses the same level of smoothness among samples and among time steps. 
This paper tries to extend this to a dynamic nature with different smoothness between samples and also throughout training.
The main idea is to use a normalized version of the entropic level of model prediction for a certain input data point for the smoothness parameter.
This is combined with self-knowledge which is used as the distribution of label smoothing.
main_review: 
1. Strengths

The paper proposes a simple extension for label smoothing that enables adaptive and instance-level smoothing. Experiments show good results w.r.t. model performance measures and calibration error metrics, even when compared with recent self-distillation methods such as TF-KD and SKD-PRT. Ablation study and gradient analysis provide further insights into the proposed method. It is also nice that the smoothing hyper-parameter tuning is not required anymore.

2. Weaknesses

There are a few instance-level label smoothing methods that are not discussed (nor compared) in the paper: Li et al. "Regularization via structural label smoothing" (AISTATS2020). Zhang and Sabuncu "Self-distillation as instance-specific label smoothing" (NeurIPS2020).

Although already discussed in the final section, it would be interesting to see results for datasets from other domains, such as images. Since most label smoothing methods were proposed to work well with image datasets, it would make the paper much better to show how it will perform under under other tasks/datasets, such as image classification.
summary_of_the_review: This paper provides a simple way to extend label smoothing to instance-specific label smoothing with an adaptive smoothing parameter. The benefits of the method is shown empirically, with an ablation study, and through gradient analysis.

After rebuttal: Thank you for the additional experiments. It is good to see that the proposed method tend to perform better than the additional baselines. I do not have further comments/questions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on adaptive label smoothing that could reflect the change in probability distribution mapped by a model over the course of training. To deal with this issue, this paper proposes a label soothing scheme that brings dynamic nature into the smoothing parameter and the prior label distribution from the distilled knowledge. Specifically, the smoothing parameter is computed with the entropic level of model probability distribution per sample on the fly during the forward propagation in training.  Besides, the prior label distribution is selected from the self-knowledge distillation. Experiments on various datasets demonstrate that the proposed adaptive label smoothing achieves state-of-the-art performance.
main_review: Strengths:
1.	This paper proposes the adaptive \alpha computed by the entropic level of model probability distribution per sample, which leads to updating the model parameters to lower the predictive score on the ground-truth target, as opposed to the effect of the cross-entropy with hard targets.  The hyperparameter search on \alpha in label smoothing is removed.   
2.	This paper set an assumption that the teacher network makes a less confident prediction than that of the student and extends gradient analysis in the perspective of regularization effect in the proposed adaptive label smoothing. 

Weakness:
1.	This paper demonstrates why self-knowledge distillation as a prior distribution is a form of regularization with theoretical analysis on the gradients. However, there is no theoretical result about the effectiveness of assigning the self-knowledge distillation to label smoothing. 
2.	The authors claim that “There are a number of benefits of adopting the adaptive smoothing parameter”. However, they only show that the hyperparameter search on \alpha is removed and the adaptive smoothing parameter can be connected to the gradient rescaling effect on self-distillation. More details should be reported to show the benefits of adopting the adaptive smoothing parameter. 
3.	In the ablation study, the authors only consider the fixed \alpha as the base. However, \alpha could also be changed in the training process. Therefore, the dynamic \alpha by hyperparameter searching should also be added as a base.  


[1]  Learning Better Structured Representations Using Low-rank Adaptive Label Smoothing, ICLR 2020.
[2]  Adaptive label smoothing,  arXiv 2020.

summary_of_the_review: This paper proposes a label soothing scheme that brings dynamic nature into the smoothing parameter and the prior label distribution from the distilled knowledge. More theoretical results and empirical results should be reported to validate the effectiveness of the proposed adaptive label smoothing. The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a simple yet effective way of smoothing the labels for each data point. The smoothing parameter alpha is dynamically adapted for each data point based on the (normalized) entropy of the predicted label-distribution for this data point. The distribution that is used for label-smoothing originates from the same but at earlier epoch of the training, i.e., a teacher-student learning framework like in knowledge distillation is employed, where the teacher is a model learned in an earlier epoch. From among these earlier model-candidates, the one is chosen that obtains the best evaluation metric g (Eq. 5). Function g does not have to be the training loss, but can be the (possibly different) evaluation metric. The validity of this approach is theoretically supported by a gradient analysis, and experimentally corroborated by improved evaluation metrics, improved calibration, and an ablation study.

main_review: Positive points:

+ the paper proposes a simple way of adapting the smoothing parameter alpha. And it works well in the experiments.
+ Instead of a simple distribution that is often used for smoothing, the paper proposes to use the distribution of the same model but from an earlier training epoch.
+ experimental results show improved accuracy and improved calibration compared to baselines

Negative points:

- it is not clear to me how important it is in the proposed approach that function g in  Eq 5 is the metric that is eventually used in evaluation (e.g., BLEU), rather than the training-objective itself, e.g., likelihood/cross entropy during training. Given that alpha is tuned dynamically based on function g, it seems that this could be a unique/unfair advantage of the proposed approach over the other baselines, which are only trained based on likelihood/cross-entropy, but have no access to the evolution metric (e.g., BLEU), which is different. For this reason, it could also be interesting to have experiments where the evaluation metric and training loss are the same function, e.g., log likelihood. Also, in Table 1, where different evaluation metrics are shown (BLEU, METEOR,...), is the 'Ours"-model always trained with function g = BLEU, or is function g chosen to be identical to the evaluation metric, e.g., when evaluated w.r.t. METEOR, then g-METEOR is used during training?

- also, it might be good to add the original paper that proposed knowledge distillation:  Bucilua, Caruana and Niculescu-Mizil: Model compression. KDD 2006.

summary_of_the_review: A theoretically simple yet empirically effective approach to label smoothing.

+++++++++++++++++

Update based on authors' response:
Thanks for the clarifications. I will maintain my current score.
review_title: null

Meta-review: The paper proposes an approach to performing label smoothing, with the amount of smoothing being sample-dependent and guided by the model's prediction (similar to self-distillation). While the reviewers find the studied problem relevant and important, they find the contributions (in their current state) to be borderline, mainly on the basis of lack of novelty and missing discussion with some related papers. While authors' response was able to partially resolve these concerns, at the end none of the reviewers was a strong advocate for accepting the paper and all scores remained at the borderline (although on the positive side). In concordance with the reviewers, I believe this submission can be made much stronger by digging a bit deeper into the problem, and also making broader connections with the existing literature.

As a concrete example/suggestion (among many other possibilities for strengthening this work), the authors may want to go a bit deeper into the theoretical analysis. Currently, their analysis shows the approach is able to reduce model's confidence, which is what happens in label smoothing and self-distillation. However, self-distillation is more than confidence reduction, and the information contained in the "dark knowledge" can provide a much stronger regularization than a sole confidence reduction argument. There are already some papers in the literature on the regularization/generalization effects of self-distillation, which the authors might want to use as a stepping-stone.
Decision: Reject