Adaptive Label Smoothing with Self-Knowledge
Authors: Dongkyu Lee
Ka Chun Cheung
Nevin Zhang
Venue: ICLR 2022 Submitted (2022)
Keywords: Regularization
Model Calibration
Adaptive Label Smoothing
Self-Knowledge Distillation
Overconfidence
Natural Language Generation

Abstract: Overconfidence has been shown to impair generalization and calibration of a neural network. Previous studies remedy this issue by adding a regularization term to a loss function, preventing a model from making a peaked distribution. Label smoothing smoothes target labels with a predefined prior label distribution; as a result, a model is learned to maximize the likelihood of predicting the soft label. Nonetheless, the amount of smoothing is the same in all samples and remains fixed in training. In other words, label smoothing does not reflect the change in probability distribution mapped by a model over the course of training. To address this issue, we propose a regularization scheme that brings dynamic nature into the smoothing parameter by taking model probability distribution into account, thereby varying the parameter per instance. A model in training self-regulates the extent of smoothing on the fly during forward propagation. Furthermore, inspired by recent work in bridging label smoothing and knowledge distillation, our work utilizes self-knowledge as a prior label distribution in softening target labels, and presents theoretical support for the regularization effect by knowledge distillation. Our regularizer is validated comprehensively on various datasets in machine translation and outperforms strong baselines not only in model performance but also in model calibration by a large margin.

Under review as a conference paper at ICLR 2022
ADAPTIVE LABEL SMOOTHING WITH SNOW-KNOWE
Anonymous authors Paper under double-blind review
ABSTRACT
Overconfidence has been shown impair generalization and calibration of neu- ral network. Previous studies remedy this issue by adding a regularization term loss function, preventing model from making a peaked distribution. Label smoothing smoothes target labels with pre-defined prior label distribution; as a result, model is learned to maximize the likelihood of predicting the soft label. Nonetheless, the amount of smoothing is the same in all samples and remains fixed in training. In other words, label smoothing does not reflect the change prob- ability distribution mapped by model over the course of training. To address this issue, we propose a regularization scheme that brings dynamic nature into the smoothing parameter by taking model probability distribution into account, thereby varying the parameter per instance. A model in training self-regulates the extent of smoothing on the fly during forward propagation. Furthermore, inspired by recent work in bridging label smoothing and knowledge distillation, our work utilizes self knowledww as a prior label distribution in softening target labels, and presents theoretical support for the regularization effect by knowledge distillation and the dynamic smoothing parameter. Our regularizer validated comprehen- sively on various datasets in machine translation and illustrates greater results in model performance and model calibration than when training with hard targets.
INTRODUCTION
In common practice, a neural network trained to maximize the expected likelihood of observed targets, and the gradient with respect to the objective updates the learnable model parameters. With hard targets (one-hot encoded), the maximum objective can be approached when a model assigns high probability mass to the corresponding target label over the output space. That is, due to the normalizing activation functions (i.e. so tmax), a model is trained in order for logits to have a marked difference between the target logit and the other classes logits (Müller et al. 2019).
Despite its wide application and use, the maximum likelihood estimation with hard targets has been found to incur an overconfident problem: the predictive score of a model does not reflect the actual accuracy of the prediction. Consequently, this leads to degradation in model calibration (Pereyra al. 2017). as well in model performance (Müller et al. 2019) Additionally, this problem stands out more clearly with a limited number of samples, as model is more prone to overfitting. To remedy such phenomenon, [Szegedy et al.] (2016) proposed label smoothing, in which one-hot encoded targets are replaced with smoothed targets. Label smoothing has boosted performance in computer vision (Szegedy et al., 2016). and has been highly preferred other domains, such as Natural Language Processing (Vaswani et al. 2017 Lewis et al. 2020).
However, there are several aspects to be discussed in label smoothing. First, it comes with certain downsides, namely the static smoothing parameter. The smoothing regularizer fails to account for the change in probability mass over the course of training. Despite the fact that a model can benefit from adaptive control of the smoothing extent depending on the signs of overfitting and overconfi- dence, the smoothing parameter remains fixed throughout training in all instances.
Another aspect of label smoothing for consideration is its connection to knowledge distillation (Hin- ton et al. (213).. There have been attempts to bridge label smoothing with knowledge distillation, and the findings suggest the latter is adaptive form of the former (Tang al. 2021 Yuan et al.

---

Under review as a conference paper at ICLR 2022
However, the regularization effect overconfidence by self-knowledge distillation is still 2020).
poorly understood and explored.
To tackle the issues mentioned above, this work presents adaptive label smoothing with self- knowledge as a prior label distribution. Our regularizer allows a model to self-regulate the ex- tent of smoothing based on the entropic level of model probability distribution, varying the amount per sample and per time step. Furthermore, our theoretical analysis suggests that self-knowledge distillation and the adaptive smoothing parameter have strong regularization effect by rescaling gradients on logit space. Our work validates the efficacy of the proposed regularization method on machine translation tasks and achieves superior results model performance and model calibration compared to other baselines.
2
PRELIMINARIES & RELATED WORK
2.1 LABEL SMOOTHING
Label smoothing (Szegedy et al. 2016) was first introduced to prevent a model from making peaked probability distribution. Since its introduction, it has been in wide application as means of regularization (Vaswani et al. 2017 Lewis et al. 2020). label smoothing, one-hot encoded ground-truth label (y) and pre-defined prior label distribution (q) mixed with the weight, the smoothing parameter (a), forming a smoothed ground- label. model with label smoothing is learned to maximize the likelihood of predicting the smoothed label distribution. Specifically,
( y (1) = Σ (1-a) a)y;"" (n) log Po(yi( + aqq log (y((((o)
(1)
|C| denotes the number classes, (n) the index of a sample a batch, and Po the probability distribution mapped by model. is commonly set to 0.1, and remains fixed throughout train- ing (Vaswani et al. 2017; Lewis et al., 2020). A popular choice of is an uniform distribution (q ((CC) while unigram distribution is another option for dealing with imbalanced label
distribution (Vaswani et al. 2017, Szzgedy et al., 2016; Müller et al. 2019 Pereyra et al.) 2017). The pre-defined prior label distribution remains unchanged, hence the latter cross-entropy term in Equation iss equivalent to minimizing the KL divergence between the model prediction and the pre- defined label distribution. (ConlPenalty) that adds negative entropy term to the loss function. thereby minimizing the KL di- vergence between the uniform distribution and model probability distribution. Ghoshal et al. (202T) proposed low-rank adaptive label smoothing (LORAS) that jointly learns a noise distribution for softening targets and model parameters. [i et al. 2020); Krothapalli & Abbott (2020) introduced smoothing schemes that are data-dependent.
line with the idea, Perryra al. (2117) proposed confidence penalty
2.2 KNOWLEDGE DISTILLATION
Knowledge distillation (Hiton et al. 2015) aims to transfer the dark knowledge of (commonly) a larger and better performing teacher model student model (Buciluundefiine al.) (2006). The idea is to mix the ground-truth label with the model probability distribution of a teacher model, the result acting an adaptive version of label smoothing (Tang et al. 2021).
C (n) (m)) - - a)v," (m) log (o())( + a aPoo (m)) log (o (x))
(2)
and denote the parameters teacher model and a student model respectively. Similar to label smoothing, è remains unchanged in training; thus student model is learned to minimize the KL divergence between its probability distribution and that of the teacher model. When Po follows a uniform distribution with temperature set to 1, the loss function of knowledge distillation is identical to that of uniform label smoothing.
Training a large teacher model can be computationally expensive; for this reason, there have been attempts to replace the teacher model with the student model itself, called infwllneweedee distillation
(Zhag et al. 2019 Yuan et al. 2020) Kim al. 2021 Zhang & Sabuneu 2020). TF-KD Yuuan
2

---

Under review as a conference paper at ICLR 2022
Parameter Update
Propagation
Pasi Checkpoints
Backward Propseation
Closer View
12 NNxiC]
in
fX; (
Mixc
usk
f(X: Φ))
Figure I: Overview of the proposed regularization. and |N] are input dimension size and a batch size respectively. Time step is not described in the figure, yet one can easily extend the above to sequential classification tasks.
et al. 2020) trains student with a pre-trained teacher that is identical to the student in terms of structure. SKD-PRT (Kim et al. 2021) utilizes the previous epoch checkpoint as teacher with linear increase in a. Zhang & Sabuncu (020) incorporate beta distribution sampling (BETA) and self-knowledge distillation (SD), and introduce instance-specine prior label distribution. Yun et al. (2020) utilize seff-knowledge distillation to minimize the predictive distribution of samples with the same class, encouraging consistent probability distribution within the same class.
METHODOLOGY
The core components of label smoothing two-fold: smoothing parameter (a) and prior label dis- tribution. The components determine how much to smooth the target label using which distribution. process that requires careful choice of selection. In this section, we illustrate how to make the smoothing parameter adaptive. We also demonstrate how our adaptive smoothing parameter and self-knowledge distillation a prior distribution act form of regularization with theoretical analysis on the gradients.
3.1 ADAPTIVE
intuitive and ideal way of softening the hard target to bring dynamic nature into choosing a; sample with low entropic level in model prediction receives high smoothing parameter to further smooth the target label. In another scenario, in which high entropy of model prediction (flat distribution) is seen, the smoothing factor is decreased.
With the intuition, our method computes the smoothing parameter on the fly during the forward propagation in training, relying the entropic level of model probability distribution per sample, and per time step in case sequential classification
IC
H ((Po(y) (m)) = Σ (o( (")) log (x(()((())
The entropy quantifies the level of probability mass distributed across the label space; therefore, low entropy is indication of overfitting and overconfidence (Pereyra et al. 2017 Meister al. 2020
Since entropy does not have fixed range between 0 and 1, one simple scheme is to normalize the entropy with maximum entropy (log CC). Hence, the normalization is capable of handling variable
For notational simplicity, time step is included in the equation hereafter.

---

Under review as a conference paper at ICLR 2022
size of class set among different datasets.
a (n) = H(Po(y)ao())
(4)
log |C|
With this mechanism, a sample with high entropy is trained with low and sample with low entropy receives high a. The computation for is excluded from the computation graph for the gradient calculation, hence, the gradient does not flow through adaptive a!
There are two essential benefits of adopting the adaptive smoothing parameter. As the smoothing extent is determined by its own probability mass over the output space, the hyperparameter search on a is removed. Furthermore, it is strongly connected to the gradient rescaling effect on self- knowledge distillation, which will be dealt Section 3.3 in detail.
3.2 SELF-KNOWLEDGE AS A PRIOR
Similar to Kim et al. point teacher network parameters in the course of training. The intuition is to utilize past self- knowledge which is a generalized supervision, thereby hindering the model from overfitting to ob- servations in the training set.
202I Liu et al. 2021), our regularizer loads a past student model check-
gegmaxg(f(X'ss 0.), Y'), where e θ
(5)
is a set of past model checkpoints in training, and function ʃ is a specific task, which our work is machine translation. X' and Y' are sets of imput and ground-truth samples from validation d and the function g could be any proper evaluation metric for model generalization (i.c. accuracy)
fl Our work utilizes the n-gram matching score, BLEU (Papineni et al. 2002) being the function for finding the prior label distribution (q ~ Po).
Combining the adaptive smoothing parameter and self- saff-knowedgge prior distribution, our loss function is as follows:
10
((x(")) ((()) Σ (1 a ("))""" log (o(viar(())
(6)
1=1
+ a (n) P (y(laa log (o
3.3 GRADIENT ANALYSIS
Tang et al. 2021): Kim et al. (021) theoretically find that the success of knowledge distillation is related to the gradient rescaling in the logit space; the difficulty of a sample determines the rescaling factor, and difficulttto-learn samples receive higher rescaling factors than those of the easy-to-learn samples f We further extend the gradient analysis in the perspective of regularization effect, and discuss the importance of the adaptive smoothing parameter.
Before dissecting the gradients, we first a hypothesis: teacher network makes a less confident prediction than that of the student. In self-knowledge distillation with a past checkpoint as the teacher, the assumption holds true. The expected predictive score on target label by the teacher model is lower than that of the current checkpoint in training (Kim et al. 2021).
The gradient with respect the logit (z) by the cross entropy loss (Lee) is as follows
dL.e
Po(y)) y:
"Nte that validation datasct is used calculate generalization error, not for training. There parameter change by the function J.
Depending the objective of the function g, argmin, can also be used.
For further understanding of gradient rescaling with knowledge distillation, please refer to Proposition
in Tang et al. 2021).
For notational siplicity, is omitted hereafter.
4

---

Under review as a conference paper at ICLR 2022
With knowledge distillation (Cka), the gradient the logit is
0Lkd - a))((()) - y:) a(Po()) Pa(u.)
(8)
0z;
The following compares the ratio of the gradient from knowledge distillation and with that of the cross entropy.
Po(us) ( 0Lkd/Oz (1 a) Po(y:) y/s
(9)
0c /0z;
When i = J. with j being the index of the ground truth, is worth noting that the denominator of the second term in Equation 9 has range Po(y:)
0], and the range of the numerator is confined to Po (yt) - (1( E [0, 1J. Therefore, the equation can be written as
aCka/dzi = (1- - al (v( - Pa(u) OL
(10)
(o(
The norm of the gradient drastically diminishes when there large difference between the pre- dictions by the models, and when the predictive score of student model is high, which is a sign of overconfidence. In terms of the direction of the gradient, when the following is seen,
a1-a) a1 Po(y:) Po(y))
(11)
(o(y)) -
the direction of the gradients with respect to knowledge distillation becomes the opposite to that of the cross entropy, pushing parameters to lower the likelihood of the target index.
The same applies when i is not equal to the target index (i + j). From Equation the following can be derived.
OLud/Oz;
(a(
(12)
aCee/Oe
Po()))
With the generalized teacher, the expected predictive score the incorrect labels by the teacher model higher than that of the student model. Therefore, in addition to the shrinking norm effect, Pa(u.)
the direction of the gradient can be reversed when 1 < a Poty.) similar to Equation 11 as result, it leads updating model parameters increase the likelihood the incorrect classes, an opposite behavior to that of the cross entropy with hard targets.
Overall, in either case, the theoretical support depicts strong regularization effects with the general- ized supervision by the teacher. As label smoothing is also closely linked to the above, the theoretical support can be easily extended to label smoothing if (1.) is replaced with in case of uniform label smoothing, and ((c;) in unigram label smoothing.
The adaptive is another factor to be discussed regarding the gradient analysis. As clearly demon- strated in Equation IT and 12 a high a, an indication peak probability distribution, not only leads drastic decrease the gradient norm, but is likely make the gradient go the opposite direc- tion to that of the cross entropy. It leads to updating the model parameters to lower the predictive score on the ground-trun -trut target, as opposed to the effect of the cross entropy with hard targets. Fur- thermore, as the parameters updates are performed by aggregating the losses of samples, adaptive smoothing acts as a gradient rescaling mechanism. The gradient rescaling by adaptive a reweights the gradients in aggregating the losses, hence, the regularization effect is stronger on the samples with signs of overconfidence than on other samples. The use adaptive a not only intuitive in terms of tackling overconfidence, but it also serves an important aspect in the theoretical support.
4 EXPERIMENT
4.1 DATASET& & EXPERIMENT SETUP
We validate the proposed regularizer on three popular translation corpora: IWSLT14 German- English (DE-EN) (Cettolo al. 2014). IWSLT15 Vitiiis---ettteeeeeee.. (EN-VI) (Cettolo et al. 2013). and Multi30K Serman-Englinn pair (Elliott et al. 2016). IWSLT14 DE-EN contains 160K sentence pairs in training, 7K in validation, and 7K in testing. IWSLT15 EN-VT has 133K, 1.5K
5

---

Under review as a conference paper at ICLR 2022
The scores are reported in percentage and are averaged over three runs with different random seeds. Except for the results from the cross entropy with hard targets, denoted as Base hereinafter, other scores are absolute difference from those of Base. Bold numbers indicate the best performance among the methods.
Table 1:
Corpus
Method
BLEU(1)
METEOR(1)
WER(()
ROUGE-L(1)
NIST())
Base
40.64
73.31
38.76
69.21
7.96
Uniform LS
+1.90
+1.47
-0.76
+0.96
+0.12
Unigram LS
+1.87
+1.30
-1.22
+1.09
+0.17
ConfPenalty
+2.50
+1.72
-0.81
+1.33
+0.15
Multi30K
LORAS
+1.14
+1.00
-0.73
+0.63
+0.16
DE-
TF-KD
+1.13
+1.02
-1.21
+0.73
+0.15
SKD-PRT
+1.31
+0.95
-0.31
+0.54
+0.09
BETA
+1.26
+0.94
-0.20
+0.46
+0.07
SD
+2.76
+2.09
-1.26
+1.55
+0.18
Ours
+3.75
+2.91
-2.19
+2.17
+0.32
Base
30.17
59.09
54.02
63.91
7.18
Uniform LS
+0.57
+0.62
-0.40
+0.42
+0.05
Unigram LS
+0.62
+0.48
-0.79
+0.43
+0.08
ConfPenalty
+0.93
+0.56
-1.02
+0.59
+0.12
IWSLTI5 EN-VI
LORAS
-0.04
-0.10
-0.23
-0.19
+0.02
TF-KD
-0.01
-0.15
-0.01
-0.08
-0.02
SKD-PRT
+1.03
+0.95
-1.31
+0.80
+0.16
BETA
+0.30
+0.20
-0.63
+0.17
+0.07
SD
+0.68
+0.46
-0.63
+0.45
+0.08
Ours
+1.37
+1.14
-1.80
+1.05
+0.21
Base
35.96
64.70
48.17
61.82
8.47
Uniform LS
+0.86
+0.61
-0.67
+0.59
+0.14
Unigram LS
+1.01
+0.68
-0.87
+0.76
+0.16
ConiPenalty
+1.15
+0.86
-1.08
+0.85
+0.19
IWSLT14 DEーEN
LORAS
+0.36
+0.23
+0.61
+0.16
-0.03
TF-KD
+0.39
+0.19
-0.33
+0.29
+0.06
SKD-PRT
+1.53
+1.11
-1.69
+1.25
+0.24
BETA
+0.95
+0.69
-0.30
+0.58
+0.08
SD
+1.39
+1.06
-0.84
+0.88
+0.16
Ours
+1.86
+1.55
-2.08
+1.59
+0.32
and 1.3K in training, validation, and testing dataset respectively. Lastly, 28K training, 1K validation, and 1K testing sentences used in Multi30K dataset. Byte pair encoding (Sennrich al. 2016) is used to process words into sub-word units.
Telsa V100. For generation, beam size is set to 4 in the inference stage. The training configuration follows the instruction of fa rseq (Ott et al. 2019 For the quality of the generated outputs, we report the popular metrics for machine translation: BL.UU (Papineni et al. 2002), METEOR (Banerjee &
All of the experiments conducted with transformer architecture (Vaswani al. 2017)
Lavie 2005), Word Error Rate (WER), ROUGE-L (Lin (004). and NIST (Doddington) 2002).
4.2 EXPERIMENTAL RESULT & ANALYSIS
Automatic evaluation results on the three test datasets are shown in Table I Though most of the methods achieve meaningful gain, the most noticeable difference is with method. Our regularization scheme shows solid improvements on all of the metrics on the datasets without any additional learnable parameter. For example, the absolute gain BLEU compared to the base method in Multi30K dataset is around 3.75. which is 9.2% relative improvement. Not only does our method excel in n-gram precision score, but it shows superior performance in having the longest

---

Under review as a conference paper at ICLR 2022
0.9
200
Base
Uniform
Unigrem
0.7
ConTPssaly
LORLAE
0/9
1.88-
TP KD
SKD
Adeplive
0.5
100
Bana
0.79
04
0.3
IWSLTIC
08
IWSLT14
0.8
0
85X
BUK
7HR
100K
125K
150K
0o0
10K
80X
50K
40K
BOK
of Updiattn
of Updates
(a) Adaptive
(b) Gradient Norm
Figure 2: (a) displays the change in the smoothing parameter a throughout the training on the tested corpora. (b) plots the gradient norm of the methods on IWSLT15 EN-VI.
1.C
10
1.0
Confiteenee
Confidencee Accurscy
Confidaaiin
Acsuracy
Accuragy
Roclo
0.6
088
044
044
044
0.0
0.1
0.0.0.0.00 Bins
0.7 1008810 000.000888811
0.0
0.1 00000000000000
1.7.0...000 3.009.
0.0
0.1 L.00000000 0.
000007 07088810 Bints
BLи
(a) Basc
(b) Label Smoothing
(c) Ours
Figure 3: Reliability diagram Base method, label smoothing, and ours. Predictions on IWSLT14
DE-EN test set are binned 10 groups based on the predictive scores. Each bar indicates the average confidence score and accuracy of each bin.
common subsequence with the reference text, as well as the informativeness of the n-grams. The empirical result demonstrates that regularizer improves the base method across all the metrics by a large margin.
In Figure 2a the changes in during training visualized. As expected, the smoothing parameters start with very small number, as the entropic level must be high due to the under-fitted models. As training continues, the predictive scores of the models increase, and accordingly, adaptive increases to prevent overconfidence. One notable aspect is the convergence at certain level. Each training the corpora ends up with different a, and the model training self-regulates the amount of smoothing and the value converges.
Furthermore, our adaptive affects the norm of the gradients as depicted in Figure 2b The gradient norm of our regularizer is considerably smaller than that of the other methods. This empirical finding mainly conforms with the gradient analysis Section 3.3] where the importance of adaptive a and generalized teacher model was discussed.
4.2. MODEL CALIBRATION
In addition to the automatic evaluation, in which the improved generalization is seen through the performance gains, we look into the calibrations of the models trained with the methods. Figure 3 depicts how the cross entropy with hard targets tends to make model overconfident in prediction. In the reliability diagram, the confidence score each bin is larger than that of accuracy, the gap of which is noticeable. Label smoothing mitigates the problem to accuracy and the confidence score still remains clear. On the other hand, the proposed regularizer significantly reduces the gap, showing the enhanced model calibration.
extent, yet the gap between the

---

Under review as a conference paper at ICLR 2022
Table 2: We report Expected Calibration Error (ECE) and Maximum Calibration Error (MCE), in percentage, on the test sets of the corpora for evaluating the calibration ability.
Multi3OK DE-+EN
IWSLTI5 EN-VI
IWSLT14 DE-+N
Method
ECE (4)
MCE (4)
ECE (1)
MCE (1)
ECE (4)
MCE (1)
Base
14.95
26.01
14.05
20.38
12.98
19.29
Uniform LS
9.17
17.22
8.33
12.13
6,43
9.98
Unigram LS
9.12
17.78
7.89
11.71
6.12
9.46
ConfPenalty
48.21
73.46
43.94
59.28
48.19
57.58
LORAS
20.27
40.86
12.41
19.15
10.54
15.29
TF-KD
21.18
42.87
13.30
19.29
12.20
17.60
SKD-PRT
14.75
26.69
9.34
14.18
5.63
8.88
BETA
11.71
21.90
9.57
14.97
8.63
13.21
SD
6.87
12.38
5.01
9.64
7.82
13.71
Ours
4.76
12.41
2,15
4.40
1.76
3.64
The improvement in calibration is more clear with expected calibration enror (ECE) and maximum calibration error (MCE) reported in Table 2 For instance, on the IWSLT14 dataset, errors with label smoothing drop significantly in both metrics, which is around 6% absolute decrease in ECE and 10% in MCE. Nonetheless, ECE of the proposed method results in 1.76% which is around 11% absolute decrease and 86% relative improvement. In addition, which is 81 % relative improvement over the base method. The improved calibration with our method seen across the datasets, demonstrating the effectiveness of our method in model calibration.
method achieves 3.64% in MCE,
One important finding is the gap between the performance in automatic evaluation and the cali- bration error. Confidence penalty (Pereyra al. 2017) is highly competitive in 7-gram matching scores (BLEU) on all of the dataset tested. Nevertheless, the calibration error is the highest among the methods. Similar the finding in (Guo al., 2017). the discrepancy between the performance and model calibration exists, and it calls for caution in training neural network when considering model calibration.
4.3 ABLATION STUDY
Table 3 shows the change in performance when adding our core components the base method on the IWSLT14 dataset. When using the fixed smoothing parameter with self-knowledge as prior, the BLBU score increases by small mar- gin, and the ECE does not drop significantly. In another case where the smoothing parameter is adaptive, and the prior label distribution is set to uniform distribution, there is meaningful in- crease in BLEU score. However, impairs the ECE score noticeably. We empirically find that the result mainly comes from underconfidennce of model. The confidence score is largely lower than that of the accuracy. In an experiment with linearly increasing a with self-knowledge prior, the BLEU score improves by around 1.6 score, yet the ECE score still has room for improvement. Since value is shared among samples in this ab- lation experiment, there is no gradient rescaling by adaptive which may explain ECE score be- ing high compared to that our adaptive a. We also look into observe that both JALEU and INLI greatly improve the scores. As has the purpose of selecting a generalized teacher from the set of past checkpoints, proper metric would serve the purpose. In
Table 3: (+) denotes adding the following components to the base method. (n))) de- notes our adaptive a, and a indicates a lin- ear increase in in the course of training. SK and Uniform denote Knolwedee and Uniform distribution prior label distribu- tion respectively. 9NL.I. and JALEV indicates function set negative log likelihood and BLEU respectively.
Method
BLEU
ECE
Base
35.96
12.98
(+) Fixed & SK
36.27
T3.56
(+) (n) (#)
Uniform
37.30
18.76
(+) & SK
37.52
5.58
Ours (gNLL)
37.74
1.30
Ours (GBLEV)
37.82
1.76
with different function: BLEU and Negative Log Likelihood (NLL). We
8

---

Under review as a conference paper at ICLR 2022
conclusion, while the adaptive smoothing parameter plays important role in regularizing model, both adaptive smoothing parameter and the choice of prior label distribution greatly affect model calibration.
5 CONCLUSION & FUTURE WORK
In this work, we propose regularization scheme that dynamically smooths the target label with self-knowledge. Our regularizer self-regulates the amount of smoothing with respect to the entropic level of the model probability distribution, making the smoothing parameter dynamic per sample, and per time step. The given idea is theoretically supported by gradient rescaling and direction, and the finding is backed up by the empirical results, both model performance and calibration.
The future work is expected to validate the proposed regularization on other domains outside of Nat- ural Language Processing. The scope of application is not confined to natural language generation tasks, and it can be easily extended other classification tasks.
REPRODUCIBILITY STATEMENT
For reproducibility, we report the three random seeds tested: {0000, 3333, 5555). For all of the experiments, this work utilizes the transformer architecture (Vaswani al.) 2017). Both the encoder and the decoder are composed of transformer layers with 4 attention heads. The hidden dimension size of the both 512. For training configuration, the maximum tokens to 4,096. For optimization, Adam (Kingma & Ba 2015) is used with beta I and beta 2 set to 0.9 and 0.98 respectively. We slowly increase the learning rate up to 0.005 throughout the first 4,000 steps, and the learning rate decreases from then on. The source code and training script are included in the supplementary materials.
batch is set
REFERENCES
Satanjeev Banerjee and Alon Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on In- trinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https::/acclantholooy...o www.orrr//000009990
Cristian Bucilaundefined, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDI) '06, pp. 535-541, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1,595933995.
doi: 10. 15402.1 150464. URL httph::/s/ org/
10. T0.1145/1150402.1 1150444
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign. In /WSLT, 2014.
Mauro Cettolo, Jan Nichues, Sebastian Stüker, Luisa Bentivogli, Roldano Cattoni, and Marcello Federico. The iwslt 2015 evaluation campaign. In IWSLT, 2015.
George Doddington. Automatic evaluation of machine translation quality using n-gram co- occurrence statistics. In Proceedings of the Second International Conference on Human Language Technology Research, HLT '02, pp. 138-145, San Francisco, CA, USA, 2002. Morgan Kaufmann Publishers Inc.
Desmond Elliott, Stella Frank, Khalil Sima'an an, and Lucia Specia. Multi30k: Multilingual english- german image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pp- 70-74. Association for Computational Linguistics, 2016. doi: 10.186533 URL
http:://ww... wwwww....w......... www.g/ggggtggggooooooooooooooooe W16 611662622 3210
Asish Ghoshal, Xilun Chen, Sonal Gupta, Luke Zettlemoyer, and Yashar Mehdad. Learning better structured representations using low-rank adaptive label smoothing. In International Confer- ence on Learning Representations, 2021. URL https:://///pt https:://openeeview.. net htt.oouummmeeetemee/mmmmmmm forum?id
5N
1IbsS pbSv.

---

Under review as a conference paper at ICLR 2022
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1321-1330. PMLR, 06-11 Aug 2017. URL https: httt:s://proceedingga wtt../g......./////..... guol7a. lt.ml
Geoffrey Hinton, Oriol Vinyals, and Jeffree Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. http://:///ax
org/abs/ 150.. 02531
Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distilla- tion with progressive refinement targets, 2021.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representutions (ICIR), 2015.
Ujwal Krothapalli and A. Lynn Abbott. Adaptive label smoothing. CoRR, abs/2009.064333 2020.
URL httts:t/aaxxiiiai org/ abs/2009 06432
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-t--sqqueece pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, On- line. July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2220..o acl- man.am/......mmii............. URL
org/ 2020 main. 703
Weizhi Li, Gautam Dasarathy, and Visar Berisha. Regularization via structural label smoothing. In Silvia Chiappa and Roberto Calandra (eds.). The 23rd International Conference on Artificial Intelligence and Statistics, AISTATS 2020, 26-28 August 2020, Online [Palermo, Sicily, Italy], volume 108 of Proceedings of Machine Learning Research, pp. 1453-1463, PMLR, 2020. URL mml. wtress/v10//112000 html.
http://proceedlin
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL https: http//claatthooooooo wttp://oyy...////--1--
Yang Liu, Sheng Shen, and Mirella Lapata. Noisy self-knowledge distillation for text summariza- tion. In Proceedings of the 2021 Conference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies, pp. 692-703, Online, June 2021. Association for Computational Linguisties. doi: 10.18653/v1/021..nem 1aw.00221n.aa-mmmi..5555 URL attps://aclanthology hwtphhhwwwo......///.o/////
owg.orgg//222 mttp:/222n....c1....m nwacl maaeec--miinn main......
Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized entropy regularization or: There's nothing special about label smoothing. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, Seattle, USA, July 2020. Association for Computational Linguisties.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Bue, E. Fox, and R. Garnett (eds.), Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Myle Ott, Sergey Edunov, Alexci Bacvski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic
evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002.
Association for Computational Linguistics. doi: 10.31151103303.1077333. URL https: /
10

---

Under review as a conference paper at ICLR 2022
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. Regulariz- ing neural networks by penalizing confident output distributions. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Workshop Track Proceedings, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Anmual Meeting of the Association for Com- putational Linguistics (Volume Long Pupers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: N0.18653/v1/PP/ 1/P16-1162. URL https:
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016. doi: 10. 10.1009/VP..22006.3338
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. Un- derstanding and improving knowledge distillation, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3903-3911, 2020.
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via self-knowledge distillation. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation, 2019.
Zhilu Zhang and Mert R. Sabuneu. Self-distillation as instance-specific label smoothing. In Hugo Larochelle, Marc' Aurelio Ranzato. Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL ht hss / neurips wwass.ccpppppee cc/paperr2220/haatt 9c65c 0b4b Abstract.
1731592aca5fb461
A APPENDIX
A.1 BASELINES
A.1.1 CONFIDENCE PENALTY
Confidence penalty (Pereyra al. 2017) adds a negative entropy term to the loss function, hence the model is encouraged to maintain entropy at certain level.
|C|
(x) (())
y (n) log (o(u)(u(")) - (Po
(13)
i=1
For the regularization-specine hyperparameter, following (Meister et al., 22220) was set to 0.78.
A.1.2 TF-KD
TF-KD Yuan et al. 2020), similar to conventional knowledge distillation, trains teacher model prior to training student; but is different in that the model architecture same with that of the student. For the hyperparameters used in this paper, empirically find that high smoothing parameter leads to better performance. Thus, we set the smoothing parameter to 0.9 and temperature scaling 20.
11

---

Under review as a conference paper at ICLR 2022
A.1.3 SKD-PRT
SKD-PRT (Kim al. 2021) is self-knowledge distillation method, where a student model (epoch () is trained with its own last epoch checkpoint (epoch t - 1) in the course of training. Though the idea is similar to ours. yet there are two core differences. The first that we find the teacher model that generalizes well with a function g. Another difference is that SKD-PRT linearly increases throughout the training, and this practice inevitably adds two hyperparameters (max a and max epoch). Following the original work (Kim et al. (02T), we set that maximum to 0.7 and maximum epoch to 150 in our experiments.
A.1.4 LORAS
LORAS Ghhshal et 202I jointly learns soft target and model parameters in training in the aim of increasing model performance and model calibration, with low rank assumption. For hyper- parameters, 7), a, rank and dropout probability are 0.1, 0.2, 25 and 0.5 respectively.
A.1.5 BETА & SD
Zhang & Sabuncu (2020) propose amortized MAP interpretation of teacher-student training, and introduce Beta smoothing which is an instance-specific smoothing technique that is based on the prediction by a teacher network. For SD-specific hyperparameters, this work sets to 0.3 and temperature to 4.0. For BETA-specific hyperparameters, a and a are set to 0.4, 4.0 respectively.
12


Reviews:
[{"review_id": "xbeGi5PZ6M", "reviewer": "Reviewer_528k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n- Overall, the paper is well written, and the proposed method is well motivated. \n- Experiments seem to demonstrate the effectiveness of the proposed method. \n- Thorough empirical analysis is done on several benchmark datasets. \n- A careful ablation study is also conducted to further demonstrate the effectiveness of the proposed method. \n\nWeakness \n- The main weakness of the paper in my opinion is the lack of novelty. The proposed method in my opinion is very similar to the previous works like [1], which proposed to use predictions from previous time stamps for self-distillation, and [2], which also proposed a method for adaptive label smoothing based on predictions from previous time stamps. While the authors of the paper addressed the key difference between the proposed method and [1], I do still feel that the proposed method lack novelty, despite good improvements in performance. On top of [1] and [2], [3] and [4] are also potentially relevant prio works to potentially discuss in literature review and benchmark against. \n\n[1] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distillation with progressive refinement of targets, 2021.\n\n[2] Zhang, Zhilu, and Mert R. Sabuncu. \"Self-distillation as instance-specific label smoothing.\" arXiv preprint arXiv:2006.05065 (2020).\n\n[3] Li, Xingjian, et al. \"One Generation Knowledge Distillation by Utilizing Peer Samples.\" (2019).\n\n[4] Yun, Sukmin, et al. \"Regularizing Predictions via Class-wise Self-knowledge Distillation.\" (2019).\n", "summary_of_the_paper": "Label smoothing is a popular approach to regularize modern neural networks. However, the amount of smoothing is the same across all samples in the dataset, which can be sub-optimal. In this paper, the authors of the paper proposed a novel adaptive label smoothing method so that each sample gets a different amount of smoothing. The key insight of the paper involves using the entropy of predictions from past timestamps as a way to quantify the amount of label smoothing applied to samples. The authors of the paper demonstrate empirically the effectiveness of the proposed method. \n", "summary_of_the_review": "Overall, despite the good empirical performance, I think the lack of novelty is a significant weakness of the paper. As such, I recommend weakly rejecting the paper for now. \n\nAfter Rebuttal:\nI would like to thank the authors of the paper for all clarifications and additional experiments. After reading the response and other reviewers' comments, I am raising my score to 6. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NbZ2fprb0dW", "reviewer": "Reviewer_WMdY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1. Strengths\n\nThe paper proposes a simple extension for label smoothing that enables adaptive and instance-level smoothing. Experiments show good results w.r.t. model performance measures and calibration error metrics, even when compared with recent self-distillation methods such as TF-KD and SKD-PRT. Ablation study and gradient analysis provide further insights into the proposed method. It is also nice that the smoothing hyper-parameter tuning is not required anymore.\n\n2. Weaknesses\n\nThere are a few instance-level label smoothing methods that are not discussed (nor compared) in the paper: Li et al. \"Regularization via structural label smoothing\" (AISTATS2020). Zhang and Sabuncu \"Self-distillation as instance-specific label smoothing\" (NeurIPS2020).\n\nAlthough already discussed in the final section, it would be interesting to see results for datasets from other domains, such as images. Since most label smoothing methods were proposed to work well with image datasets, it would make the paper much better to show how it will perform under under other tasks/datasets, such as image classification.", "summary_of_the_paper": "This paper proposes a new method that is based on label smoothing.\nThe original label smoothing uses the same level of smoothness among samples and among time steps. \nThis paper tries to extend this to a dynamic nature with different smoothness between samples and also throughout training.\nThe main idea is to use a normalized version of the entropic level of model prediction for a certain input data point for the smoothness parameter.\nThis is combined with self-knowledge which is used as the distribution of label smoothing.", "summary_of_the_review": "This paper provides a simple way to extend label smoothing to instance-specific label smoothing with an adaptive smoothing parameter. The benefits of the method is shown empirically, with an ablation study, and through gradient analysis.\n\nAfter rebuttal: Thank you for the additional experiments. It is good to see that the proposed method tend to perform better than the additional baselines. I do not have further comments/questions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "BImYFN7X3wH", "reviewer": "Reviewer_PF2S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positive points:\n\n+ the paper proposes a simple way of adapting the smoothing parameter alpha. And it works well in the experiments.\n+ Instead of a simple distribution that is often used for smoothing, the paper proposes to use the distribution of the same model but from an earlier training epoch.\n+ experimental results show improved accuracy and improved calibration compared to baselines\n\nNegative points:\n\n- it is not clear to me how important it is in the proposed approach that function g in  Eq 5 is the metric that is eventually used in evaluation (e.g., BLEU), rather than the training-objective itself, e.g., likelihood/cross entropy during training. Given that alpha is tuned dynamically based on function g, it seems that this could be a unique/unfair advantage of the proposed approach over the other baselines, which are only trained based on likelihood/cross-entropy, but have no access to the evolution metric (e.g., BLEU), which is different. For this reason, it could also be interesting to have experiments where the evaluation metric and training loss are the same function, e.g., log likelihood. Also, in Table 1, where different evaluation metrics are shown (BLEU, METEOR,...), is the 'Ours\"-model always trained with function g = BLEU, or is function g chosen to be identical to the evaluation metric, e.g., when evaluated w.r.t. METEOR, then g-METEOR is used during training?\n\n- also, it might be good to add the original paper that proposed knowledge distillation:  Bucilua, Caruana and Niculescu-Mizil: Model compression. KDD 2006.\n", "summary_of_the_paper": "The paper proposes a simple yet effective way of smoothing the labels for each data point. The smoothing parameter alpha is dynamically adapted for each data point based on the (normalized) entropy of the predicted label-distribution for this data point. The distribution that is used for label-smoothing originates from the same but at earlier epoch of the training, i.e., a teacher-student learning framework like in knowledge distillation is employed, where the teacher is a model learned in an earlier epoch. From among these earlier model-candidates, the one is chosen that obtains the best evaluation metric g (Eq. 5). Function g does not have to be the training loss, but can be the (possibly different) evaluation metric. The validity of this approach is theoretically supported by a gradient analysis, and experimentally corroborated by improved evaluation metrics, improved calibration, and an ablation study.\n", "summary_of_the_review": "A theoretically simple yet empirically effective approach to label smoothing.\n\n+++++++++++++++++\n\nUpdate based on authors' response:\nThanks for the clarifications. I will maintain my current score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8XCwfWOb1OY", "reviewer": "Reviewer_xcg8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1.\tThis paper proposes the adaptive \\alpha computed by the entropic level of model probability distribution per sample, which leads to updating the model parameters to lower the predictive score on the ground-truth target, as opposed to the effect of the cross-entropy with hard targets.  The hyperparameter search on \\alpha in label smoothing is removed.   \n2.\tThis paper set an assumption that the teacher network makes a less confident prediction than that of the student and extends gradient analysis in the perspective of regularization effect in the proposed adaptive label smoothing. \n\nWeakness:\n1.\tThis paper demonstrates why self-knowledge distillation as a prior distribution is a form of regularization with theoretical analysis on the gradients. However, there is no theoretical result about the effectiveness of assigning the self-knowledge distillation to label smoothing. \n2.\tThe authors claim that “There are a number of benefits of adopting the adaptive smoothing parameter”. However, they only show that the hyperparameter search on \\alpha is removed and the adaptive smoothing parameter can be connected to the gradient rescaling effect on self-distillation. More details should be reported to show the benefits of adopting the adaptive smoothing parameter. \n3.\tIn the ablation study, the authors only consider the fixed \\alpha as the base. However, \\alpha could also be changed in the training process. Therefore, the dynamic \\alpha by hyperparameter searching should also be added as a base.  \n\n\n[1]  Learning Better Structured Representations Using Low-rank Adaptive Label Smoothing, ICLR 2020.\n[2]  Adaptive label smoothing,  arXiv 2020.\n", "summary_of_the_paper": "This paper focuses on adaptive label smoothing that could reflect the change in probability distribution mapped by a model over the course of training. To deal with this issue, this paper proposes a label soothing scheme that brings dynamic nature into the smoothing parameter and the prior label distribution from the distilled knowledge. Specifically, the smoothing parameter is computed with the entropic level of model probability distribution per sample on the fly during the forward propagation in training.  Besides, the prior label distribution is selected from the self-knowledge distillation. Experiments on various datasets demonstrate that the proposed adaptive label smoothing achieves state-of-the-art performance.", "summary_of_the_review": "This paper proposes a label soothing scheme that brings dynamic nature into the smoothing parameter and the prior label distribution from the distilled knowledge. More theoretical results and empirical results should be reported to validate the effectiveness of the proposed adaptive label smoothing. The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to performing label smoothing, with the amount of smoothing being sample-dependent and guided by the model's prediction (similar to self-distillation). While the reviewers find the studied problem relevant and important, they find the contributions (in their current state) to be borderline, mainly on the basis of lack of novelty and missing discussion with some related papers. While authors' response was able to partially resolve these concerns, at the end none of the reviewers was a strong advocate for accepting the paper and all scores remained at the borderline (although on the positive side). In concordance with the reviewers, I believe this submission can be made much stronger by digging a bit deeper into the problem, and also making broader connections with the existing literature.

As a concrete example/suggestion (among many other possibilities for strengthening this work), the authors may want to go a bit deeper into the theoretical analysis. Currently, their analysis shows the approach is able to reduce model's confidence, which is what happens in label smoothing and self-distillation. However, self-distillation is more than confidence reduction, and the information contained in the "dark knowledge" can provide a much stronger regularization than a sole confidence reduction argument. There are already some papers in the literature on the regularization/generalization effects of self-distillation, which the authors might want to use as a stepping-stone.
Decision: Reject