Pseudo Knowledge Distillation: Towards Learning Optimal Instance-specific Label Smoothing Regularization
Authors: Peng Lu
Ahmad Rashid
Ivan Kobyzev
Mehdi Rezagholizadeh
Philippe Langlais
Venue: ICLR 2022 Submitted (2022)
Keywords: Knowledge Distillation
Label Smoothing
Supervised Learning
Image Classification
Natural Language Understanding

Abstract: Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training. Although a complete understanding of KD is elusive, a growing body of work has shown that the success of both KD and label smoothing comes from a similar regularization effect of soft targets. In this work, we propose an instance-specific label smoothing technique, Pseudo-KD, which is efficiently learnt from the data. We devise a two-stage optimization problem that leads to a deterministic and interpretable solution for the optimal label smoothing. We show that Pseudo-KD can be equivalent to an efficient variant of self-distillation techniques, without the need to store the parameters or the output of a trained model. Finally, we conduct experiments on multiple image classification (CIFAR-10 and CIFAR-100) and natural language understanding datasets (the GLUE benchmark) across various neural network architectures and demonstrate that our method is competitive against strong baselines.

Under review as a conference paper at ICLR 2022
PSEUDO KNOWLEDGE DISTILLATION: LEARNING OPTIMAL INSTANCE-SPECIFIG LABEL SMOOTHING REGULARIZATION
TOWARDS
Anonymous authors Paper under double-blind review
ABSTRACT
Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training. Al- though a complete understanding of KD is elusive, growing body of work has shown that the success of both KD and label smoothing comes from a similar regularization effect of soft targets. In this work, we propose an instance-specific label smoothing technique, Pscudo-KD KD, which is efficiently learnt from the data. We devise a two-stage optimization problem that leads to a deterministic and in- terpretable solution for the optimal label smoothing. We show that Pseudo-KD can be equivalent to an efficient variant of self-distillation techniques, without the need store the parameters the output of a trained model. Finally, we con- duct experiments on multiple image classification (CIFAR-I0 and CIFAR-100) and natural language understanding datasets (the GLUE benchmark) across vari- ous neural network architectures and demonstrate that our method is competitive against strong baselines.
INTRODUCTION
Neural networks, predominantly, pre- praitrdined language models have grown massively in scale over the past few years (Fedus et al.] 2021 Brown et al.) Zeng et al.) 2021). Some researchers have gone so far as to term these over-parameterized models as Foundation Models (Bommasani et al.) 2021. Although larger models trained on more data tend to dominate various benchmarks (Wang et al. networks is an option but empirical evidence suggests that training larger models and then com- pressing them can help achieve a higher accuracy per unit computation (Li et al.).
2019b a Deng al.) 2009), deploying them on edge devices is challenge. Training smaller
Knowledge distillation (KD) (Hinton et al. 2013 Buciluă et al., 2006) which is a knowledge trans- fer technique, from single
ensemble of neural networks another, is widely applied neural model compression technique across domains ranging from computer vision (He et al.] [019 Xu
et al. 2020 Jafari et al. 2021) natural language processing (Jiao et al. 2020 Sanb Sanh et al.] 2019).
This empirical success is attributed to a few factors such as a) deeper teacher networks learn bet- ter representations, b) the teacher is able transfer dark knowledge (Hinton
al. (004]) in its
predictions and c) the regularization effect of the soft labels in the KD loss (Yuan et al.) 2020). Nonetheless, it unclear how exactly student networks benefit from these soft labels. Recent work suggests that with KD, the student is unable to match the teacher on the training distribution but improves generalization on the test set (Stanton al., 202T).
In trying to understand KD, series of investigations have looked at regularization and have demon- strated that the success of both KD and label smoothing due to similar regularization effect of soft targets (Yuan et al.] 2020 Zhang & Sabuncu 2020). Label smoothing (LS) (Szegedy et al.) 2016) is a popular technique for enhancing the generalization of a wide range of models without incurring additional computational costs. It encourages the model to treat each non-target class as equally likely for classification by using a uniform distribution to smooth the distribution of the hard labels. Although combining the uniform distribution with the original hard label is beneficial for regularisation, LS does not take into account the true relationships between different label cate- gories like the way KD does. ZZhan & Sabuneu 2020) demonstrate the importance of instance

---

Under review as a conference paper at ICLR 2022
specific LS regularization. They demonstrate better performance compared to LS, but trained model to infer prior knowledge on the label space and thereby sacrifice some the efficiency of LS.
In this work, investigate whether maintaining the efficiency of LS. Motivated by this, we first revisit the conventional LS technique and generalize the uniform LS method instance-specifie smoothing regularization framework. Within this framework, we demonstrate that both LS and KD can be interpreted as instances of a smoothing distribution. Next, propose learn the optimal smoothing function along with the model training, by devising a two-stage optimization problem. We solve the first-stage problem by giving deterministic and interpretable solution and apply gradient based optimization to solve the second stage.
bridge the performance gap benween LS and KD, while
Our contributions can be summarized as follows:
Our method, Pseudo-KD, provides general framework, that improves upon conventional LS without an additional training cost.
We justify our framework by demonstrating the new formulation unifies the conventional LS and KD objectives, which confirm their relationship from a different perspective.
Based on our framework, we further propose to learn the smoothing distribution along with training by formulating it as bi-level optimization problem and we derive a deterministic and interpretable solution.
We conducted extensive experiments on image classification (CIFARIO and CIFAR and natural language understanding (the GLUE benchmark) and show that our method outperforms label smoothing, and performs on par with KD while being significantly faster in training (approximately up to 3x for some settings).
RELATED WORK
We will provide brief overview of KD) and LS and teacher-free methods which reconcile these two.
Knowledge Distillation. KD came into prominence after the work of Hinton al. (2015). Given access to a trained teacher model, assume that we want to train a student. Denote by Pr and Po(x) the teacher's and student's predictions respectively. For classification problem, the total KD loss is defined as:
-- po(x)
where ((x) is the one-hot label, (1(()), p())) is the cross-entropy loss between the student's output and the labels, DEL is the KI. divergence and is the scaling parameter. Note that assume temperature of 1 and omit it without loss of generality. KD training is equivalent to training with smoothed label P(z) such that:
Pr(z) (1 ())(()(() + aPr(z)
We observe that the loss in equation 2 is similar in form to the LS loss Yuuan et al. 2020 In LS the KI. divergence is between the student output and a uniform distribution define label smoothened label as:
Label Smoothing
K where K is the number of classes. Therefore
Po(x) = (1 a))(()) aU
(3)
Both LS and KD incorporate training a model with smoothened label. KD tends to perform better as it leads to a more accurate student (Müller et al. 2019: Shen et al. 2021). However, LS is more efficient since it does not need pre-trained teacher network. We note from equation 3 that higher a can lead to a smoother label distribution and a lower a to a peakier label. Yuan et al. (2020) exploited this to propose better LS loss to bridge the performance gap between KD and L.S Chen itt al. (202T) note that training on high entropy label distributions can also lead to better robustness under adyersarial attacks.
2

---

Under review as a conference paper at ICLR 2022
CB|PIs. So)
(s.
CB St)
Sp
T
S9
Pt.
S
y
Ps. Py (r(ja))
23 y
F y
I
Pu
U(-))
(0.3)
Pg( pyllls,)
2 2,)
(6) Label Smoothing
(b) Knowledge Distillation
(c) Pseudo-KD (Ours)
Figure I: Our Rseudo-KD method is compared with the label smoothing and knowledge distillation techniques.
Self Distillation We take advantage of distillation without necessarily using cumbersome teacher. A student can improve generalization by learning for the output of a trained, identically parameterized model Furlancllo et al. (2018); Yuan al. (2020). Zhang et (2018) propose to train an ensemble of students without a static teacher, which benefits from learning and teaching among different students collaboratively. Another self- self-boosting direction is to apply regularization on the intermediate layers instead of predictive distributions. During the training stage, [Zhang et al.) 2019) introduce auxiliary classifiers on top of intermediate layers and penalize the difference be- tween the predictions of these classifiers and the penultimate layer. Along this direction, Liu et al. 2020)) further propose a top-down feature fusion process to handle the capacity gap between the intermediate layers with different depths. However, all these methods require extra parameterized modules for training which are abandoned at inference. In contrast, some recent works focus on boosting the performance with explicitly regularized soft targets. [Zhng & Sabuncu (2020) interpret student-teacher training as an amortized maximum a posteriori estimation and derive an equivalence between siff-dstilllaionn and instance-specifie label smoothing. This analysis helped them to devise a regularization scheme, Beta Smoothing. However, they still use an extra model to infer a prior dis- tribution on their smoothing technique during the training. Yun et al. (2020) introduce an additional regularization to penalize the predictive output between different samples of the same class. Ding al. (022) propose capture the relation of classes by introducing decoupled labels table which increases O(K K x K)) space complexity. The concurrent work (Kim et al.) 2021) utilizes the model trained at i-th epoch as the teacher to regularize the training at (I+1) epoch along with annealing techniques. The difference emerges as it still requires separated model teacher, saving all soft labels generated at i-th epoch for the next epoch training which introduces O(N x K) space complexity. Our work has different basis: develop Pseudo-KD motivated by principled start- ing point: to generalize the smoothing distribution to general form with neither time nor space complexity increases during the training and inference.
stage.[Zhang
METHODOLOGY
In this section, we derive our Pseudo-KI framework from the two-stage optimization perspective. First, we revisit the conventional uniform LS method and introduce a closed-form solution to our optimal instance-specific label smoothing. Then, we describe a teacher-free KD implementation under this formulation.
3.1 REVISITING LABEL SMOOTHING
Suppose that we have a K KCclass classification task to learned by a neural network, (.). Given training of (Ix, y J. samples where x; is data sample and y; is the ground-truth label, the
3

---

Under review as a conference paper at ICLR 2022
model So(-) outputs the probability po (2,) of each class fl,...,Ky KY:
exp(zi.)
po(j)z;) = Σラ= (2.g))
(4)
where zi, = [Sx(x is the logit for j-th class of input Zy..
general label smoothing can be formally written as:
P.o(il.)) = { aa- Pialilas), j= k, where is the ground-truth class; (a. Pi(loa))
(5)
j
where P/s is smoothed label, Pts is a smoothing distribution, and a is a hyperparameter that determines the amount of smoothing. If 0, we obtain the original one-hot encoded label. For the original label smoothing method, the probability Pta(-1z) is independent on the sample æ and taken to be the uniform distribution Pr.(j) = U(), However, the training can benefit from instance- specific regularization. (uaan al. 2020: Zhang & Sabuncu 2020p. In this work, we consider the general form of LS Pi,(-1z)) which is instance-specific and not necessarily uniform.
Let us consider the Cross Entropy (CE) loss with the smoothed labels:
N
K Σ
N
Ca(P)))
Σ
(o) = Σ E A. log (o)la());; N
1
(6)
Note that computing the weighted sum of negative log-likelihood of each probability of label can be viewed as taking expectation of the negative log-likelihood over label space under cer- tain distribution Pis. We modify this loss by adding DRL (-a((((a))]((U(()))) into 6 which encourages the sample-wise smoothing distribution Pis( (2:) to be close to the uniform distribution.
(KL) divergence term
Ro(P.s)
Σ
where B is hyper-parameter. The instance-specific smoothing (tx(-) can be viewed as the prior knowledge over the label space for sample 24. This KL divergence term can be understood as a measure of the 'smoothness' or "overconfidence" of each training label. We choose Uniform distribution to constrain Pts in the KL term because we would like Pte contain more information on non- non-target classes. plays role similar to temperature distribution Pt.. On the one hand side, in case of KD, it regulates the amount of prior knowledge we inject in the soft label. On the other hand side, it reduce the overconfidence of the trained network by increasing the entropy of soft labels (a phenomenon studied by Zhang & Sabuncu (2020)).
KD and controls the sharpness of the
Next we show the post-hoc interpretability of our new formulation. The following two Remarks discuss the relationship of our objective with LS and KD.
Remark. When Pia(-)z;;) is taken to be the uniform distribution for any I;, the objective in Eq. 7 reduces back to the one in Eq. 6 since (U.,(U())IUU()))) =0.
Remark. When (tu(-)a;) is laken to be Pr(+(x;) which is the output probability of teacher model T(-) for any F4, our objective can be converted to the conventional KD objective with temperature 1 and plus constant a log K
N
1
Ro(1T) =
E Ep, |- -oxp)))t))))
(8)
N
1=1
1
(11 (L))a)Cc((u...p)
(po aDk
+ a log K,
N
where Pr(j(z,) (1 (1-a Pr for k the ground truth, and Pr(j)z)) = a Pr (x;))
for j ht. Our objective function Eq. demonstrates that the both objective KD and IS can
4

---

Under review as a conference paper at ICLR 2022
be rewritten an expectation of negative log-likelihood w.r.t transformed distribution Pr or U.) plus a KL term between the smoothing distribution and uniform distribution. This is not only to confirm previous findings from different perspective but also justify why our framework is reasonable.
Our objective gives unified formulation for instance-spp -estanc--ppecffc label smoothing in a general form. is easy to bridge the objective of label smoothing and knowledge distillation by converting our objective KD or LS via replacing the (i;(- in Eq. 7 with Pr(-|x;) or U(), respectively, There is an inherent relation among these methods. KD and LS, in particular, can be interpreted as instances of our method with a fixed smoothing distribution.
3.2 PSEUDO-KS KD VIA TWO-STAGE OPTIMIZATION
The choice of the distribution Pt, determines the label regularization method. As described above, in KD the smoothing distribution is an output of the pre-trained teacher model, and in LS it is uniform distribution. Generally speaking, the optimal smoothing distribution is unknown, and ideally, we would like to learn the optimal Pis. In this regard, we set up the following two-stage optimization problem:
min (Pt))
arg min Ro (Pts)- subject PT arg
Pt.
This optimization setting, also called bilevel optimization (Colson al. 2007). strongly NP hard (Jeroslow] 1985) so getting an exact solution is difficult, Luckily, in our case the inner optimization loop has closed-form solution. The following Theorem gives the explicit formulation of Ro Pi.). In particular, we will show that Ro (Pts) is convex function w.r.t Pts. and we will obtain the optimal solution Pt = arg minp, Ro (Pts) by using a Lagrangian multiplier. For the details of this derivation please refer to Appendix A.
Theorem 1. The solution to the inner loop of optimization in Eq. 9 is given by:
pa(ji
PL(]).:) Σÿ Po Po (o(i)(z))3
(10)
where po po (.]x:))))))))) ) is the output probability of j-th class of model So(-), a is the smoothing coefficient defined in Eq. and B is defined in Eq.
As result, we reduced the two-stage optimization problem in Eq. 9 to a regular single-stage mini- mization:
min Ro (P))),
(1I)
where
K
1 N
(P(PP)
((o)) (- (jlx:)) BPT Pf.Gilx))
(12)
Pi (jlæz) is given in Theorem and P (zlz;) ) is defined in Eq. Note that the solution Pf is deterministic and interpretable. Moreover, we show the post-hoc interpretability of the solution in the two remarks below by demonstrating the relation of the solution with LS and self-distillation methods.
Remark. When 3 is extremely large, Pf will be close the Uniform distribution, and our objective function will be equivalent to optimizing the CE loss with Uniform LS regularizer. When B is extremely small, the smoothing distribution collapses to the one-hot vector.
Remark. There is an intrinsic connection between the P distribution and generating softmax outputs with a temperature factor. Specifically, when 3 = T, we could have
expl ZLL
(13)
- exp ELL
5

---

Under review as a conference paper at ICLR 2022
Since the smoothing distribution in this case becomes the temperature smoothed output of the model, our Pseudo-KD technique can be considered as a generalized version of the self reff-ditiillation method
To summarize, our method can be expressed as an alternating two-stage process. We generate opti- mal smoothed labels Pf, using Theorem [ in the first stage. Then, in the second phase, we fix the Pis to compute loss Ro (P.)) and update the model parameters. We execute our method in an online manner, which eliminates the need for additional memory or storage for the parameters or outputs of the pre-trained model. The online training process is shown in Algorithm
Algorithm Online Training with Pscudo
Input: Training set {(Ts M:) M:) batch size n, number of training epochs T, learning rate yy
1: for + to T do
2:
for t < 1--10 to N/n do
3:
Randomly sample mini-batch S = {(i 3)) from training set;
4: Compute the Pt. according to Pr: solution for the mini-batch data;
D. 1st Stage
5: Update the model 0 Os 11 Ro (Pis S):
2nd Stage
6: end for
7:
end for
4 EXPERIMENTS
In this section, we evaluate the performance of our proposed Pseudo-KD method. Experiments are conducted on both the image classification tasks CIFAR-10 and CIFAR- 100 (Krizhevsky al.) 2009) for the ResNet-based Model with of various sizes (parameters) and the General Language Understanding Evaluation (GLUU) tasks (Wang et 2019b) with the RoBERTa-based model (Liu et al. 2019).
4.1 EXPERIMENTS ON IMAGE CLASSIFICATION
Data. CIFAR is a benchmark dataset for image classification (Krizhevsky et al. 2009). We use both CIFAR-IO and CIFAR-100, which have images 50,000 training samples and 10,000 test sam- ples divided into 10 and 100 different object classes, respectively. The validation set is made up of 10% of the training data.
Experimental settings. We evaluated our method on different model architectures including Mo- bileNetV2 (Sandler et al., 2018), ShuffleNetV2 (Ma et al.] (018). (esNeXt29(8x64d) (Xie et al.) 2017). ResNet18, ResNet50 and ResNet101 (He et al. 2016). We follow standard data augmenta- tion schemes: random crop and horizontal flip to augment the original training images. The models trained for 200 epochs, with batch size of 128 for MobileNetV2, ShuffleNetV2 and ResNet18, and 64 for ResNeXt129 (8x64d), ResNet50 and ResNet101. For optimization we used stochastic gradient descent with momentum of 0.9, and weight decay set 5e-4. The learning rate starts 0.1 and is then divided by 5 at epochs 60, 120 and 160. All experiments are repeated 5 times with random initialization. For the KD experiments (KD-Shuffle and KD-29), used ShuffleNetv2 and Res2eX129 as the teachers. All teacher models are trained from scratch and picked based on their best accuracy. To explore the best hhper-parameter
we conduct grid search over parameter pools. We explore {0.1, 0.2, 0.5, 0.9} for a and {5, 10, 20, 40} for KD temperature. For Pesudo-KD, we also perform explore {1.5, 2.5, 4) for B. The best hyper-parameterd can be found in Appendix C.
We first compare our proposed Pseudo KD method with uniform label smoothing and conventional KD with different teachers. All experiments are repeated 5 times with different random initialization.
Results Tab. 4.1 shows the test accuracy for each method using various network architectures. Our method significantly improves classification performance than conventional LS on all six models with over 0.85 or 0.35 points higher accuracy average for CIFARIOR and CIFAR 10, respectively.
Thhe derivation be found in Appendix B.

---

Under review as a conference paper at ICLR 2022
Table 1: Accuracy CIFAR 100 and CIFAR 10 test set. "mean (+ std)" reported on 5 runs.
Base
L.S
KD-Shuffle
KD-29
Pseudo- KD (Ours)
ShufileNetv2
70,45 (10.22)
70.78 (10035)
72.09 (100211)
71.97 (10.266
71.95 (10.266
MobileNetv2
67.98 (1:0.13)
68.69 (+0.33)
70.93 (10030)
70,99 (2:0.16)
71.05 (10.35)
R8
76.92 (::0.11)
77.67 (4:0.27)
77.73 (1:0.10)
77.78 (+:0.22)
78.10 (+0.15)
CIFARIO0
ResNeX129
81.07 (=00330
82.02 (=0.17)
81.86 (#0016)
82.24 (#:0.18)
82.19 (=0.16)
ResNer50
79.28 (+:0.12)
(+:0.2)
79.51 (=00188)
79.65 (+0.36)
79.76 (+0.31)
ResNet101
78.81 (=:0.35)
79.11 (#0.33)
79.30 (+0.20)
79.58 (1:0211)
79.45 (#0.25)
ShuffleNetv2
91.32 (#:0.77)
91.67 (=0024)
92.30 (=00.09
92.26 (+:0.15)
91.99 (+0.22)
MobileNetv2
90.55 (.10.20)
90.82 (100.12)
91.46 (100.25)
91.52 0.15)
91.53 (100113
ResNer18
94.82 (=:0.09)
95.02 (=0.15)
95.05 (=0.05)
95.28 (2:0008)
95.21 (=0.09)
CIFARIO
ResNeX129
95.72 (1:0.11)
95.71 (1:0.09)
95.75 (1:013)
96.06 (1:0.07)
95.82 (1.0.14
ResNet50
95.15 (110.11)
05.10
95.21 (1.0.08)
95.52 (1.0.08)
95.43 (100.16
ResNet101
95.25 (+00113)
94.98 (+0.11)
95.29 (+0.07)
95.62 (+0.07)
95.44 (+0.15)
This indicates its adaptability to different networks. Compared to KD methods, our method can give comparable or better performance, while being much more efficient training. This is because our method could be implemented in an online manner, which only need one forward pass as Base or LS methods. Furthermore, our method does not necessitate the use of a pre-trained teacher, which saves computational overhead.
We also observe that lightweight models like ShuffleNetv2 and MobileNetv2 benefit more from LS, KD and method than complex models (ResNeXt29 and ResNet101). Furthermore, as teacher, using a model with a low capacity can provide comparable performance for improvements for small models to using large model. Specifically, by distilling three low-capacity models (ShuffleNetv2, MobileNetv2, and ResNet18) 8) and using ShuffleNet as the teacher, all three models can achieve comparable performance, with ShuffleNet outperforming the others.
Table 2: Comparison between different Teacher-free methods. Average test accuracy and training time are reported. The training time is measured on a single NVIDIA V100 GPU.
CIFAR (Ace. Time)
CIFAR
Time)
MobileNetv2
ResNet18
ResNet50
MobileNetv2
ResNet18
Res5o
Base
67.98 is
1.1h
76.9220
1.1h
79. 79.28
3.6h
90,552m02
94,8220.0
1.1h
95, 15e0.
3.6h
KT>29
70.9910 16
4.1h
77.7820-2
4.2h
79.652036
14.4h
91.52g0015
4.1h
95.28200e
4.2h
95.522005
14.4h
68.69
77.622007
1.1h
79.1610-48
90.8220.11
95.0220:1
10..000000
CS CS-KD
70.3610 24
1.2h
77.95
1.4h
79.33
3.9h
91.17
1.2h
94 g4.90go.0s
1.4h
95.25
3.9h
70.08110.3
1.1h
27
1.3h
79.192034
3.7h
90.9720.19
95,052000
1.3h
95.1220
3.7h
B.ta-LS
20.45.10.55
1.6h
77.8310.10
1.5h
79.55
4.8h
90.89 jacce
1.6h
94.87
1.8h
95.221..08
4.5h
KR-IS
70.1220 29
1.1h
77. 77.8220077
1.1h
19.4820018
3.6h
M0.6720.0
1.1h
94, 94.7620.11
1.1h
95.30120
3.6h
1.2h
79.83=020
91.56
1.2h
95.14
95.39=0.18
Pseudo-KD
71.0510.55
1.1h
78. 18
1.1h
79.7618.1
3.6h
91.53_n-33
1.1h
95.21
1.1h
95.433.033
3.6h
Next, we conduct a series of experiments on three models to compare our approach with other methods without requiring any pre-trained models (except for KD-29 reported as reference). All experiments are repeated 5 times with different random initialization. The baseline methods include, CS-KD which distill the soft output between different samples of the same class to student (Yun) et al. 2020). TF-reg which regularize the training with manually designed teacher (Yuan al.] 2020 and Beta-LS which leveraging a model with same capacity to learn mnstance-speciin prior over the smoothing distribution (Zhang & Sabuncu 2020). We follow their best parameerr settings. The time for all baselines is measured based on its original implementations.
Tab. shows the test accuracy and training time for 200 epoch of different methods three mod- els. It can be seen that our method consistently improves the classification performance on both
htapss/ggitub.com/aaalaaaass
hhatss//gtthbb.cm///hhihhnag 23/neurips 2020 distillation
7

---

Under review as a conference paper at ICLR 2022
lightweight and complex models, which indicates its robustness to different networks. Besides, shows the training time of our method is equivalent to Base or LS methods, which show its stable efficiency over other baselines such as Beta-LS, which still requires separated model output learned prior over the smoothing distribution. We further compare two recent works KR-LS (Ding et al. 202T and PS-KD (Kim et al. (021). Our method achieves comparable better performance than PSKD and consistently better than KR-LS. We have to mention the implementation of KR-LS and PS-KD increase space complexity O(K K) and O(N x K) for class-wise and sample-wist smoothing labels, respectively. However, our method neither increases time nor space complexity during the training.
4.2 RESULTS ON NATURAL LANGUAGE UNDERSTANDING
Data. We run experiments on 7 classification tasks from the GLUE benchmark (Wang et al. 2019b). These datasets can broadly divided into families of problems. Single set tasks which include linguistic acceptability (CoLA) and sentiment analysis (SST-2), similarity and paraphrasing tasks (MRPC and QQP), and inference tasks which include Natural Language Inference (MNLI and RTE) and Question Answering (QNLI). Following prior works (Jafari et al.) 2021 Rashid et al., 2021], we report Spearman correlation for STS-B, Matthews correlation for CoLA, and accuracy for the other tasks for the development set. We report "F1" for the QQP and MRPC and the corre- sponding test sets.
Experimental Setup. We choose the pre-trained DistilRoBERTa (6-layer) (Sanh et al. 2019) as the base model for all our experiments. For KD experiments, we use RoBERTa-Large model (24-layer) (Liu et al., 2019) fine-tuned on each of the tasks as the corresponding teacher model. All models are trained with the Adam optimizer with the default settings [oshchilov & Hutter 2019). To explore the best
conduct grid search over the learning rate [le 5, 2e - 3e 5}. batch size E {8, 16, 32, 64), a E {0.1, 0.2, 0.5) and KD temperature [1. 5, 10}. Moreover, for Pesudo-KD, perform search over three / values in {1.5, 2, 4). The best
can be found Appendix. We repeat our experiments times with different random initialization.
Table 3: GLUE results on Dev set.
CoLA
MRPC
RTE
SST-2
QNLI
QQP
MNLI
Avg.
Teacher Student
68.1
91.9
86.3
96.4
94.6
91.5
89.9
88.4
61 .02022
86.52034
69.1
92.72005
91.21002
91.31001
83.42a19
82.2
w/ KD
61.4=054
86.1
70.2+0.17
92.6=a11
91.5+1014
91.8+0.0
84.4=024
82.6
LS
60.3+a.54
86.42042
7..3
92.8=aos
91.3+018
91.5+0.00
83.8=a
82.3
w/ Pseudo-KE KD (Ours)
61.4
86.9:029
7..4
93.0=0.15
91.5++0.4
91.5
84.1=021
82.7
Table 4: GLUE results on Test set for best model on dev sets.
CoLA
MRPC
RTE
SST-2
QNLI
QQP
MNLI
Avg.
Student
50.5
88.9
63.5
92.8
90.8
88.9
83.7
79.8
w/ KD
51.5
88.6
63.3
93.0
91.5
89.2
84.5
80.2
w/ LS
49.6
88.6
64.6
93.0
90.8
89.0
83.8
79.9
w/ Pseudo-KD KD (Ours)
51.2
88.9
63.2
93.5
91.0
89.2
84.0
80.1
Results Table 3 presents the dev set results on GLUE, We present the average result for each ex- periment, over three runs, as well as the standard deviation. As is standard on the GLUE benchmark, present average result across the datasets well. We compare Pseudo-KD simple fine- tuning, LS and KD. We observe that our method is better on all datasets compared to fine-tuning and at par (QQP only) better all datasets compared to LSS. We are comparable average to
8

---

Under review as a conference paper at ICLR 2022
ResNet18
ResNet50
betalothha-1.5 bethalshaad
betaöpha-1. beialepha-25
betsapphsse
77.0
79.0
77.6
76.8
Figure 2: Effect of different combinations of and 3 on the performance of ResNet18 and ResNet50. For illustration, all values of / are divided by corresponding a.
KD, with KD doing better on MNLI and QQP and Pseudo-KD doing better CoLA, MRPC, RTE and SST-2.
Furthermore, Table 4 shows the GLUE leaderboard results. We chose the model with the best dev results for each method and submitted it to the official GLUE benchmark to get the test results. We see similar trend and observe that Pseudo-KD better than LS and fine-tuning on all datasets except RTE. On average the results are comparable to KD as before. KD is better on CoLA, RTE, QNLI and MNLI and Pseudo KD better on MRPC, SST-2 and at par on QQP. We observe that on average all the models are close in performance, with an average difference of 0.4 between the best and the worst model.
4.3 EFFECT OF HYPER-PARAMENT PARAMETER AND 3
We conducted ablation tests to probe how the and 3 hyper-parameters affects the performance of our method. We also compared our results to LS while choosing the same a for both. Specif- ically, chose
{0.1, 0.2, ... 0.6). For Pseudo-KD, also verified the influence of 3/a E (1.5, 2.5, 43. Following the discussion on equation 13 we observe that B/ox can be in- terpreted as the temperature in self-distillation.
We plot the accuracy on CIFAR100 for two architectures (ResNet18 and ResNet50) as shown in Fig. 2 The plot shows that our method can consistently outperform the LS method across different a values even when varying the 3/a parameter. The results are consistent for both architectures. As expected, the accuracy both LS and our method gradually decreases when a increases from 0.1 to 0.6. It is interesting that even at an of 0.6 the drop in accuracy is less than 1% of the peak value.
We observe that temperature value between 1.5 and 2.5 is the best obtain optimal performance.
5 CONCLUSION
Our aim in this work to fill the accuracy gap between KD and LS techniques, while maintaining the training efficiency of LS. We proposed learning an instance-specific label smoothing regulariza- tion simultaneously with training our model on the target. We began by revisiting the traditional LS method and introduced our objective function by substituting the uniform distribution with general, instance-specific, discrete distribution. Within the new objective, we explained the relation- ship between the LS and KD. Particularly, our method can be interpreted self-distillation. Then, using a two-stage optimization approach, we obtained an approximation for the optimal smoothing function. We conducted extensive experiments to compare model with LS, KD and various teacher-free free methods on popular CV and NLU benchmarks and showed the effectiveness and efficiency of our technique.
online version of
9

---

Under review as a conference paper at ICLR 2022
We want to extend this research in two directions in the future. First want to improve algorithm further and explore replacing the alternating approach to two-stage optimization with a more complex algorithm to reach closer to the optimal smoothing.
Next we want to explore practical applications where our method can replace KD. One interesting application is the pre-training of compressed language models. Typically, the best compressed mod- els are pre-trained using distillation from a much larger teacher which makes the training process cumbersome and computationally intensive. It would be interesting to explore if Pseudo-KD can achieve comparable performance to KD in that setting.
REFERENCES
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv:2 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 2rXiv:2005.14166 2020.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535-541, 2006.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may mitigated by properly learned smoothening. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL wtt.eeewweeeeeeeeemeeeeeeeeeeeee net/ stttt//foommmidddzyyuuuu
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of Operations Research, 153(())235-256, 2007.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Qianggang Ding, Sifan Wu, Tao Dai, Hao Sun, Jiadong Guo, Zhang-Hus Hua Fu, and Shutao Xia. Knowledge refinery: Learning from decoupled label. In Thirty-Fifth AAAI Conference Ar- tificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelli- gence, KAA/ 2021, Virtual Event, February 2-9, 2021, pp. 7228-7235. AAAI Press, 2021. URL https: //tjs//jjjs/.../ aaai. org/ index. php/ AAAI /tt.//aaticceeeeee///aaa.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, arXiv preprint arXiv:2101.03961, 2021.
Tommaso Furlanello, Zachary Chase Lipton. Michael Tschannen, Laurent Itti,, and Anima Anandku- mar. Born-again neural networks. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning. ICML 2018, Stockholmsmüssaan Stockholm, Sweden, July 15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1602 mlr www. v80 Eurlanellol
1611. PMLR, 2018. TtRLhttp:// http:tttttttppppeeeeeeeeeee http:://prrccceeingss html
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pp. 770-778. IEEE Computer Society, 2016. doi: 10.1
Utthhttps:// ozt.:////oooo/////////////// 10. 2119///V..22111. 90
Tong He, Zhi Zhang, Hang Zhang, Zhongyue Zhang, Junyuan Xie, and Mu Li. Bag of tricks for image classification with convolutional neural networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 558-567, 2019.
10

---

Under review as a conference paper at ICLR 2022
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Dark knowledge. Presented as the keynote in BayLearn, 2:2, 2014.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:/503.0255: 2015.
Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge distilla- tion. arXiv preprini arXiv:2 2021.
R. G. Jeroslow. The polynomial hierarchy and simple model for competitive analysis. Mathemat- ical Programming, 32(2):146-164, 1985.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 3163-41744 2020.
Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-knowledge distil- lation with progressive refinement of targets. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6567-6576, October 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joseph E Gon- zalez. Train large, then compress: Rethinking model size for efficient training and inference of transformers.
Benlin Liu, Yongming Rao, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Metadistiller: Network self- boosting via meta-learned top-down distillation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020 16th European Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV, volume 12359 of Lecture Notes in Computer Science, pp. 199-709. Springer, 2020. doi: 10.1007/978----00-5550 008--000006664444 URL https://doi. org/oo..oooooooooooooooooooo. 1007 0000009778 3 000003366----000 6_
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 0tbs/190..11660 16922, 2019. URL htllitttp//aax...rrrttt abs 1907 11692.1699
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. wtttfooum//////// i dw
OpenReview.net, 2019. URL /
Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet V2: practical guidelines for efficient CNN architecture design. In Vittorio Ferrari, Martial Hebert, Cristian Sminchis- escu, and Yair Weiss (eds.), Computer Vision ECCV 2018 15th European Conference, Mu- nich, Germany, September 8-14, 2018, Proceedings, Part XIV, volume 11218 of Lecture Notes in Computer Science, pp. 122 138. Springer, 2018 doi: 1111110977-300000011111 100000777700 03000 01264-9 9_8
URL
htttps://doi.org
Rafael Müller, Simon Korbblith, and Geoffrey E. Hinton. When does label smooth- ing help? In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Flo- rence d'Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural In- formation Processing Systems 32: Annual Conference on Neural Information Process- ing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 4696 2019. URL https: http:s:///ppccccegssssss.mm new.eeeeddnn.....riii....e. ips. ppappr/201//aaaa/e #1748d6b0 (11133999911111111 7etal17eb222222 Abst.ratttttttttttt
Ahmad Rashid, Vasileios Lioutas, and Mehdi Rezagholizadeh. MATE-KD: Masked adversarial TExt, a companion to knowledge distillation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1062-1071, Online. August 2021. Association for Computational Linguistics. doi: 10.18653/v1/200 long..8... URL https:
7/acl mocclantholscc www.atooooll..ogo//o...o .al.a.clllo long. aln...

---

Under review as a conference paper at ICLR 2022
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In 2018 IEEE Conference on Computer Vision and Pattem Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4510-4520. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10.1 10./CRR. 2018.00474.
thecvf. com/ coom/content _eppntt____r__________ 2018//tttt 8/html/ Sandler Www.ileeeetyl.. wtwpaleeeettt__eeeeeeeeee Receeteeeeeeiiiiiee Residuals CVPR 2018
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1rXiv:1910.01108, 2019.
Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides. Is label smoothing truly incompatible with knowledge distillation: An empirical study. In 9th International Conference on Learning Representations, ICLR 2021, Viruall Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:://penteviee. net for forum?? id= PCbuuGVrGaZ
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Gordon Wil- son. Does knowledge distillation really work? arXiv preprint arXiv:2 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Superglue: stickier benchmark for general-purpose language un- derstanding systems. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 3266-3280, 2019a.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benehmark and analysis platform for natural language understanding, In 7th International Conference on Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019b. URL
net/ fwrummrrmmmmmmm iid=
FJ4km2R5t7
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated resid- ual transformations for deep neural networks. In 2017 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 5987-5995. IEEE Computer Society, 2017, doi: 00.1009/VVR.2217..6644 AtLhttps: : /ooi...../ erg/10. LLC9/CVPR.2010 634
Kunran Xu, Lai Rui. Yishi Li, and Lin Gu. Feature normalized knowledge distillation for image classification. In Computer Vision - ECCV 2020, pp. 664-680. Springer International Publishing, 2020.
Li Yuan, Francis E. H. Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEF/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pp. 3902- 3910. Computer Vision Foundation IEEE, 2020. doi: 10.11099CVP422200.2200 URL
/ttps://openaccest the.tet..eeece.. com content wwwpeett______eeeeeeeeeeeeee 2020/htm1/ via Label Smoothing
Yuan Rewisiiiiiiiieeeeeee Kwww Baaige_DDittee www.tiistiilio wtwpg//wwwwweeeeeeeeaaaaaaaaaaee
Regularization _at_aaatttttaaaaaaaaaeeeaeea CVPR 01FF__0200__0 1020_paper.html
Sukmin Yun, Jongjin Park, Kimin Lee, and Jinwoo Shin. Regularizing class-wise predictions via
self-knowledge distillation. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle WA, USA, June 13-19, 2020, pp. 13873-13882, IEEE, 2020. doi: 10111///CVVR222000000000 2020. 01389
URLhttss:
https: httt::/:/ooo..ccce//..oo///////
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang. ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. Pangu-o: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint 104 /2369, 2021.
12

---

Under review as a conference paper at ICLR 2022
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In 2019 IEEE/CVF International Conference Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp- 3712-3721. IEEE, 2019. doi: 10. 1..1009CC..2019000000 URL https://doii .org 10. 1111/109CCCCC 20000..0...1111111111
Ying Zhang, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mu- tual learning. In 2018 IEEE Conference on Computer Vision and Pattern Recogni- tion, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pp. 4320-4328. Com- doi: 10.1109/CVP.22018 thecve. com content evpr 2018/htm1/ Zhang_ Bae___Deee________ Mutual Leeetuuulllaaaaaeee weeeeernnn_nveppe 201 1 _appaaae.... wwppp..ttm............ppp....mmp
puter Vision Foundation / IEEE Computer Society, 2018. 00454. URL
Zhilu Zhang and Mert R. Sabuneu. Self-distillation as instance-specific label smoothing. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Con- ference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL http://t////roooeoooooooooooe
1721572caff347733411665532 neurips CC/ html
htt.ea//wpp..capppp////////////0 oc/ppper//222/hhat/
A DERIVATION OF THE PROOF FOR THEOREM 1
Proof. First, observe that the function Rę defined in Eq. 7 is convex combination of N nonnegative N.. We will show that
functions R2 = E E [- log Po (j [z,)] )] 3D... (Pis [u))) (-)), for = I,...,N.. each of R. is a convex function of the components of simplex Pt (-1x;) by computing the Hessian matrix of R. with respect Pt. (-Ix:):
8 R DP. 17 8 R
RRR. BP. (1))))() 8 (2)
8 R. 8 R.
((R((P)(1))"" P()(K)) (((())))))))))))))
(P)o(2)a
DP. (2)
(14)
8
dRR. (2)
8²R. (K)
JPP. (3)PP.
8P.
3
diag(
(1) (2)
(i.
When 3 is greater than zero, the Hessian is positive definite. Therefore, each RG is convex function of the componets of Pta(-lar.). As result, Rs is convex function of the collection of components of every simplex Pia(-(r.).
For simplicity, we derive the global optimum solution for each R, with Lagrangian multiplier:
K
Pioli))
K
(a(Pi,AAL) =
ΣL F'1
-
P((()) log
+ A
-
1/K
f'=1
where we omit the dependency on æ; to simplify the notation. We set the corresponding gradients equal to 0 to obtain the global optimum for =,..., K.
OL, OP,(() a log + 3 log Pioli) + 3 + 3log K + Ar = 0
(15)
P((() = exp( a
logp(j)) exp( -3 3 log K X1
B
=xxp( logp(j)) Cts
B
13

---

Under review as a conference paper at ICLR 2022
Since Σ/= Pio(j')
have
Σ ='=1
K
K Pio(')) = Σ
exp
a
log p())) exp( -B - 8 log K - -xy =
(16)
='=1
3 log K Xu
AL)
1
C₁ = exp(
2y'=1 exp( log
So the optimal Pi (j) is given by the formula:
PG(()) exp( logr())
p(j)a/8
(17)
- exp() log
B DERIVATION FROM OPTIMAL SMOOTHING TO SOFTMAX OUTPUT WITH TEMPERATURE
When 3 = = T, we could have
effic ttem
po(clar)
PT.
Eopo(i)oz)+
(c
6
(18)
Ee(ets)
Σ
e 2id
This is the commonly used soft label in KD approach.
C HYPER-PARAMETRRS FOR PSEUDO-KD
Model
Q
8
batch size
Ir
CIFARIO0
0.2
1.5
CIFARIO
0.2
2.5
CoLA
0.1
2
32
le-5
RTE
0.1
2
2e-5
SST-2
0.1
5
64
2c-5
MRPC
0.1
2
2e-5
QNLI
0.1
4
64
2e-5
QQP
0.1
4
32
2c-5
MNLI
0.1
2
32
2c-5
14


Reviews:
[{"review_id": "tSovZ-c8X2w", "reviewer": "Reviewer_bwuZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is well-written and well-organized. Codes are provided.\n2. The paper proposes an adaptive label smoothing method, which is novel to some extent.\n3. The proposed bi-level framework and its implementation look reasonable.\n\nCons:\n\n1. The motivation of the paper is my main concern. The method is called Pseudo Knowledge Distillation and the authors claim the method bridges the gap between label smoothing and knowledge distillation, however it seems to have no connection to Knowledge Distillation. The generated smoothed labels don't consider the relevance of classes which is defined as the dark knowledge in [1]. The motivation of adding a KL divergence term into Eq.7 is not elaborated clearly. \n2. The effectiveness of the proposed method is not convincing. The improvement of Pseudo-KD in practice is not significant. \n3. Some baselines are missing, such as Deep Mutual Learning [2] and Knowledge Refinery [3].\n\n[1] Geoffrey E. Hinton, Oriol Vinyals, Jeffrey Dean. Distilling the Knowledge in a Neural Network\n[2] Zhang, Ying, Tao Xiang, Timothy M. Hospedales, and Huchuan Lu. Deep mutual learning. CVPR2018\n[3] Qianggang Ding, Sifan Wu, Tao Dai, Hao Sun, Jiadong Guo, Zhang-Hua Fu, Shutao Xia. Knowledge Refinery: Learning from Decoupled Label. AAAI 2021", "summary_of_the_paper": "The paper proposes a novel teacher-free label regularization method named Pseudo-KD which bridges the gap between label smoothing and knowledge distillation. The proposed bilevel programming framework is implemented by an alternating two-stage process. The optimal smoothed labels are generated in the upper optimization, and the backbone network is optimized by the inner loop. The overall framework is equivalent to self-distillation without the need for extra computational cost. The authors claim that the extensive experiments on both image classification and natural language understanding tasks validate the effectiveness of the proposed method. ", "summary_of_the_review": "The idea is interesting and novel enough. However, considering the motivation and the experimental results, I vote for \"5: marginally below the acceptance threshold\".\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XAjD0MEHWMJ", "reviewer": "Reviewer_JBRd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a self-distillation approach that is derived from a series of observation made regarding how LS and KD work, linking in the two together in the process.\n\nPro:\n+ The approach and the mathematical proofs provided are sound\n+ The two step training process is fast and matches or outperform the vanilla KD baseline.\n+ The performance is tested on both vision and language data showing that the approach generalizes well.\n\nCons:\n- There is a relatively large body of work on self-distillation however there is no comparison against them, neither on explaining how they differ nor empirically. A non-exhaustive list:\n[A] Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation, Zhang et al, ICCV 2019\n[B] Self-Knowledge Distillation with Progressive Refinement of Targets, Kim et al, ICCV 2021\n[C] MetaDistiller: Network Self-Boosting via Meta-Learned Top-Down Distillation, Liu et al, ECCV 2020\n[D] Regularizing Class-Wise Predictions via Self-Knowledge Distillation, Yun et al, CVPR 2020\n- How does this method work when combined with other distillation losses (such as feature matching)? KD generally can boost their performance, is the same true here?\n- Little comparison with state-of-the-art on knowledge distillation and while surpassing the KD baseline is important in this context its unclear if applying such method in practice will suffice. As such the authors should offer a comparison with the latest work on KD too.\n- How well does this scale to larger datasets, such as ImageNet?\n\nMinor:\nTable 2, some values are wrongly in bold (ex: KD outperforms P-KD on CIFAR-10)\n\n", "summary_of_the_paper": "The authors propose a self-distillation approach that is derived from a series of observation made regarding how LS and KD work, linking in the two together in the process. The paper then experimentally shows the superiority of the proposed approach against KD on CIFAR-10/100 and languge datasets. Important comparisons against closely-related methods are however missing. ", "summary_of_the_review": "While the paper is interesting and the idea of performing \"distillation\" (pseudo) is appealing the experimental section needs to be improved.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "PrmIWB_aa6W", "reviewer": "Reviewer_CUiW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an online self-distillation method. The paper is well-written and I enjoy reading it. However, the proposed link between label smoothing and knowledge distillation, in my opinion, is still based on previous research findings (e.g., Yuan et al., 2020). When compared to previous research, it would be interesting to see what new insights can be gained from the newly proposed loss function (7) (other than adding the smoothing kl term to the original ce loss). Or, to put it another way, I believe it is nearly equivalent if we use the formulation proposed in previous work (Yuan et al., 2020). However, in section 3.2, this paper presents a closed-form solution to the two-stage optimization problem.\n\nIn the evaluation part, it would be great if authors could compare Pseudo-KD to the recent efforts on self-distillation (instance-specific label smoothing), for example:\n   Zhang, Zhilu, and Mert R. Sabuncu. \"Self-distillation as instance-specific label smoothing.\"\n   Yuan, Li, et al. \"Revisiting knowledge distillation via label smoothing regularization.\"\n\n\n\n\n\n\n\n", "summary_of_the_paper": "This paper presents an instance-specific label smoothing (self-distillation) technique, Pseudo-KD, formulated as a two-stage optimization problem. On popular CV and NLU benchmarks, authors conducted experiments to compare Pseudo-KD against LS, KD, and various teacher-free methods, demonstrating the usefulness and efficiency of the proposed technique.", "summary_of_the_review": "1. novelty is limited compared to the recent work on self distillation and instance-specific label smoothing\n2. missing relevant baselines in the evaluation section", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9eCnjLOnm2", "reviewer": "Reviewer_piYX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The paper proposes a new formulation of an objective function for instance-specific label smoothing regularization to unify label smoothing (LS) regularization and knowledge distillation (KD), which is quite interesting. \n\n2. It is good to find a closed-form solution for the inner level of the bi-level optimization problem of the proposed objective function.\n\n3. Experiments have been done with data from different domains (images and texts) and have confirmed the effectiveness of the proposed objective function. \n\nWeakness:\nPresentation of the experimental results can be improved:\nIn Tables 4.2 and 4.4, it would be interesting to also show the standard deviation of the test accuracy/GLUE scores. The results of the propose Pseudo-KD model does not seem particularly attractive in these two tables. \n\nMinor presentation issues:\nPage 5: \"in the second phase, we fix the\" => \"in the second stage, we fix\"\nPage 6: It would be good to add references for the KD-Shuffle and KD-29 models. \n", "summary_of_the_paper": "This paper proposes a new formulation of an objective function for label smoothing regularization to unify label smoothing (LS) regularization and knowledge distillation (KD). The objective function learns an instance-specific label smoothing regularization while training a model towards its prediction target. When the instance-specific label smoothing regularization uses a simple uniform distribution for all training samples, the objective function falls back to traditional LS. When the instance-specific label smoothing regularization uses the output distribution of a teacher model, the objective function learns a KD model. The paper formulates the learning of the optimal instance-specific label smoothing regularization as a bi-level optimization problem and derived a closed-form solution for the inner level of the problem. Experimental results on both image classification and natural language understanding tasks show that the proposed objective and learning algorithm are effective and yield results comparable to the baseline LS and KD models. ", "summary_of_the_review": "The paper proposes a new formulation of an objective function for instance-specific label smoothing regularization to unify label smoothing (LS) regularization and knowledge distillation (KD), which is quite interesting but is also largely based on the CVPR 2020 work by Yuan et al. The paper further formulates the proposed objective function as a bi-level optimization problem and finds a closed-form solution for the inner level of the problem. Experiments confirmed the effectiveness of the proposed objective function on both image classification and natural language understanding tasks, while the improvements over the baselines are somewhat limited. Overall this is a solid study with reasonable contributions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes an instance-specific label smoothing method, which is formulated as a two-stage optimization problem for finding the optimal label smoothing. The authors show that the proposed approach can be equivalent to an efficient variant of self-distillation techniques (i.e. no need to store the parameters or the output of a trained model). Experiments on image classification (CIFAR-10 and CIFAR-100) and natural language understanding datasets (the GLUE benchmark) demonstrate that our method is competitive against strong baselines.

The reviewers find the proposed approach reasonable, and the presentation clear. However, they all rated the paper as borderline, due to some concerns that the submission has in its current form. These include limited novelty (by CUiW) [the link between label smoothing and knowledge distillation is largely based on previous research findings (e.g., Yuan et al., 2020)], nonconvincing results on the effectiveness of the proposed method (by bwuZ) [Improvement of Pseudo-KD in practice is not significant in terms of test accuracy gains], and lack of comparison with some recent related methods (by JBRd as well as other reviewers). The authors responded to these (and other concerns), but this did not convince the reviewers about the concerns listed above. I recommend the authors to resubmit after addressing these issues.
Decision: Reject