PSEUDO KNOWLEDGE DISTILLATION: TOWARDS LEARNING OPTIMAL INSTANCE-SPECIFIC LABEL SMOOTHING REGULARIZATION
Year: 2021

Abstract:
Knowledge Distillation (KD) is an algorithm that transfers the knowledge of a trained, typically larger, neural network into another model under training. Although a complete understanding of KD is elusive, a growing body of work has shown that the success of both KD and label smoothing comes from a similar regularization effect of soft targets. In this work, we propose an instance-specific label smoothing technique, Pseudo-KD, which is efficiently learnt from the data. We devise a two-stage optimization problem that leads to a deterministic and interpretable solution for the optimal label smoothing. We show that Pseudo-KD can be equivalent to an efficient variant of self-distillation techniques, without the need to store the parameters or the output of a trained model. Finally, we conduct experiments on multiple image classification (CIFAR-10 and CIFAR-100) and natural language understanding datasets (the GLUE benchmark) across various neural network architectures and demonstrate that our method is competitive against strong baselines.

heading: 1 INTRODUCTION
text: Neural networks, predominantly, pre-trained language models have grown massively in scale over the past few years (Fedus et al., 2021; Brown et al., 2020; Zeng et al., 2021). Some researchers have gone so far as to term these over-parameterized models as Foundation Models (Bommasani et al., 2021). Although larger models trained on more data tend to dominate various benchmarks (Wang et al., 2019b;a; Deng et al., 2009), deploying them on edge devices is a challenge. Training smaller networks is an option but empirical evidence suggests that training larger models and then compressing them can help achieve a higher accuracy per unit computation (Li et al.).
Knowledge distillation (KD) (Hinton et al., 2015; Buciluǎ et al., 2006) which is a knowledge transfer technique, from a single or an ensemble of neural networks to another, is a widely applied neural model compression technique across domains ranging from computer vision (He et al., 2019; Xu et al., 2020; Jafari et al., 2021) to natural language processing (Jiao et al., 2020; Sanh et al., 2019). This empirical success is attributed to a few factors such as a) deeper teacher networks learn better representations, b) the teacher is able to transfer dark knowledge (Hinton et al., 2014) in its predictions and c) the regularization effect of the soft labels in the KD loss (Yuan et al., 2020). Nonetheless, it is unclear how exactly student networks benefit from these soft labels. Recent work suggests that with KD, the student is unable to match the teacher on the training distribution but improves generalization on the test set (Stanton et al., 2021).
In trying to understand KD, a series of investigations have looked at regularization and have demonstrated that the success of both KD and label smoothing is due to a similar regularization effect of soft targets (Yuan et al., 2020; Zhang & Sabuncu, 2020). Label smoothing (LS) (Szegedy et al., 2016) is a popular technique for enhancing the generalization of a wide range of models without incurring additional computational costs. It encourages the model to treat each non-target class as equally likely for classification by using a uniform distribution to smooth the distribution of the hard labels. Although combining the uniform distribution with the original hard label is beneficial for regularisation, LS does not take into account the true relationships between different label categories like the way KD does. Zhang & Sabuncu (2020) demonstrate the importance of an instance
specific LS regularization. They demonstrate better performance compared to LS, but use a trained model to infer prior knowledge on the label space and thereby sacrifice some of the efficiency of LS.
In this work, we investigate whether we can bridge the performance gap between LS and KD, while maintaining the efficiency of LS. Motivated by this, we first revisit the conventional LS technique and generalize the uniform LS method to an instance-specific smoothing regularization framework. Within this framework, we demonstrate that both LS and KD can be interpreted as instances of a smoothing distribution. Next, we propose to learn the optimal smoothing function along with the model training, by devising a two-stage optimization problem. We solve the first-stage problem by giving a deterministic and interpretable solution and apply gradient based optimization to solve the second stage.
Our contributions can be summarized as follows:
• Our method, Pseudo-KD, provides a general framework, that improves upon conventional LS without an additional training cost.
• We justify our framework by demonstrating the new formulation unifies the conventional LS and KD objectives, which confirm their relationship from a different perspective.
• Based on our framework, we further propose to learn the smoothing distribution along with training by formulating it as a bi-level optimization problem and we derive a deterministic and interpretable solution.
• We conducted extensive experiments on image classification (CIFAR10 and CIFAR100) and natural language understanding (the GLUE benchmark) and show that our method outperforms label smoothing, and performs on par with KD while being significantly faster in training (approximately up to 3x for some settings).

heading: 2 RELATED WORK
text: We will provide a brief overview of KD and LS and teacher-free methods which reconcile these two.
Knowledge Distillation. KD came into prominence after the work of Hinton et al. (2015). Given access to a trained teacher model, assume that we want to train a student. Denote by PT (x) and pθ(x) the teacher’s and student’s predictions respectively. For a classification problem, the total KD loss is defined as:
L = (1− α)H(q(x), pθ(x)) + αDKL(PT (x), pθ(x)) (1)
where q(x) is the one-hot label,H(q(x), p(x)) is the cross-entropy loss between the student’s output and the labels, DKL is the KL divergence and α is the scaling parameter. Note that we assume a temperature of 1 and omit it without loss of generality. KD training is equivalent to training with a smoothed label P̂ (x) such that:
P̂T (x) = (1− α)q(x) + αPT (x) (2)
Label Smoothing We observe that the loss in equation 2 is similar in form to the LS loss (Yuan et al., 2020). In LS the KL divergence is between the student output and a uniform distribution U(k) = 1K where K is the number of classes. Therefore we can define label smoothened label as:
P̂U (x) = (1− α)q(x) + αU (3)
Both LS and KD incorporate training a model with a smoothened label. KD tends to perform better as it leads to a more accurate student (Müller et al., 2019; Shen et al., 2021). However, LS is more efficient since it does not need a pre-trained teacher network. We note from equation 3 that a higher α can lead to a smoother label distribution and a lower α to a peakier label. Yuan et al. (2020) exploited this to propose a better LS loss to bridge the performance gap between KD and LS. Chen et al. (2021) note that training on high entropy label distributions can also lead to better robustness under adversarial attacks.
Self Distillation We can take advantage of distillation without necessarily using a cumbersome teacher. A student can improve generalization by learning for the output of a trained, identically parameterized model Furlanello et al. (2018); Yuan et al. (2020). Zhang et al. (2018) propose to train an ensemble of students without a static teacher, which benefits from learning and teaching among different students collaboratively. Another self-boosting direction is to apply regularization on the intermediate layers instead of predictive distributions. During the training stage, Zhang et al. (2019) introduce auxiliary classifiers on top of intermediate layers and penalize the difference between the predictions of these classifiers and the penultimate layer. Along this direction, Liu et al. (2020) further propose a top-down feature fusion process to handle the capacity gap between the intermediate layers with different depths. However, all these methods require extra parameterized modules for training which are abandoned at inference. In contrast, some recent works focus on boosting the performance with explicitly regularized soft targets. Zhang & Sabuncu (2020) interpret student-teacher training as an amortized maximum a posteriori estimation and derive an equivalence between self-distillation and instance-specific label smoothing. This analysis helped them to devise a regularization scheme, Beta Smoothing. However, they still use an extra model to infer a prior distribution on their smoothing technique during the training. Yun et al. (2020) introduce an additional regularization to penalize the predictive output between different samples of the same class. Ding et al. (2021) propose to capture the relation of classes by introducing decoupled labels table which increases O(K ×K) space complexity. The concurrent work (Kim et al., 2021) utilizes the model trained at i-th epoch as the teacher to regularize the training at (i+1)-th epoch along with annealing techniques. The difference emerges as it still requires a separated model as teacher, or saving all soft labels generated at i-th epoch for the next epoch training which introduces O(N × K) space complexity. Our work has a different basis: we develop Pseudo-KD motivated by a principled starting point: to generalize the smoothing distribution to a general form with neither time nor space complexity increases during the training and inference.

heading: 3 METHODOLOGY
text: In this section, we derive our Pseudo-KD framework from the two-stage optimization perspective. First, we revisit the conventional uniform LS method and introduce a closed-form solution to our optimal instance-specific label smoothing. Then, we describe a teacher-free KD implementation under this formulation.

heading: 3.1 REVISITING LABEL SMOOTHING
text: Suppose that we have a K-class classification task to be learned by a neural network, Sθ(·). Given a training set of {xi, yi}Ni=1 samples where xi is a data sample and yi is the ground-truth label, the
model Sθ(·) outputs the probability pθ(j|xi) of each class j ∈ {1, . . . ,K}:
pθ(j|xi) = exp(zi,j)∑K
j′=1 exp(zi,j′) , (4)
where zi,j = [Sθ(xi)]j is the logit for j-th class of input xi.
A general label smoothing can be formally written as: P̂ls(j|xi) = { 1− α+ α · Pls(j|xi), j = k,where k is the ground-truth class; α · Pls(j|xi), j 6= k , (5)
where P̂ls is a smoothed label, Pls is a smoothing distribution, and α is a hyperparameter that determines the amount of smoothing. If α = 0, we obtain the original one-hot encoded label. For the original label smoothing method, the probability Pls(·|x) is independent on the sample x and is taken to be the uniform distribution Pls(j) = U(j), However, the training can benefit from instancespecific regularization. (Yuan et al., 2020; Zhang & Sabuncu, 2020). In this work, we consider the general form of LS P̂ls(·|xi) which is instance-specific and not necessarily uniform. Let us consider the Cross Entropy (CE) loss with the smoothed labels:
Lθ(Pls) = 1
N N∑ i=1 − K∑ j=1 P̂ls(j|xi) log pθ(j|xi)  = 1 N N∑ i=1 EP̂ls [− log pθ(j|xi)] . (6)
Note that computing the weighted sum of negative log-likelihood of each probability of label can be viewed as taking expectation of the negative log-likelihood over label space under a certain distribution P̂ls. We modify this loss by adding a Kullback–Leibler (KL) divergence term DKL(Pls(·|xi)‖U(·)) into Eq. 6 which encourages the sample-wise smoothing distribution Pls(·|xi) to be close to the uniform distribution.
Rθ(Pls) = 1
N N∑ i=1 [ EP̂ls [− log pθ(j|xi)] + βDKL(Pls(·|xi)‖U(·)) ] , (7)
where β is a hyper-parameter. The instance-specific smoothing Pls(·|xi) can be viewed as the prior knowledge over the label space for a sample xi. This KL divergence term can be understood as a measure of the ‘smoothness’ or ‘overconfidence’ of each training label. We choose a Uniform distribution to constrain Pls in the KL term because we would like Pls contain more information on non-target classes. It plays a role similar to temperature in KD and controls the sharpness of the distribution Pls. On the one hand side, in case of KD, it regulates the amount of prior knowledge we inject in the soft label. On the other hand side, it reduce the overconfidence of the trained network by increasing the entropy of soft labels (a phenomenon studied by Zhang & Sabuncu (2020)).
Next we show the post-hoc interpretability of our new formulation.The following two Remarks discuss the relationship of our objective with LS and KD. Remark. When Pls(·|xi) is taken to be the uniform distribution U(·) for any xi, the objective in Eq. 7 reduces back to the one in Eq. 6 since DKL ( U(·)‖U(·) ) = 0.
Remark. When Pls(·|xi) is taken to be PT (·|xi) which is the output probability of a teacher model T (·) for any xi, our objective can be converted to the conventional KD objective with temperature τ = 1 and plus a constant α logK
Rθ(PT ) = 1
N N∑ i=1 [ EP̂T [− log pθ(j|xi)] + αDKL(PT (·|xi)‖U(·)) ] , (8)
= 1
N N∑ i=1 [(1− α)LCE(yi, pθ(·|xi)) + αDKL(PT (·|xi)‖pθ(·|xi))] + α logK,
where P̂T (j|xi) = (1−α+α ·PT (k|xi)) for j = k the ground truth, and P̂T (j|xi) = α ·PT (j|xi) for j 6= k. Our objective function Eq. 7 demonstrates that the both objective of KD and LS can
be rewritten as an expectation of negative log-likelihood w.r.t. a transformed distribution P̂T or U(·) plus a KL term between the smoothing distribution and uniform distribution. This is not only to confirm previous findings from a different perspective but also justify why our framework is reasonable.
Our objective gives a unified formulation for instance-specific label smoothing in a general form. It is easy to bridge the objective of label smoothing and knowledge distillation by converting our objective to KD or LS via replacing the Pls(·|xi) in Eq. 7 with PT (·|xi) or U(·), respectively. There is an inherent relation among these methods. KD and LS, in particular, can be interpreted as instances of our method with a fixed smoothing distribution.

heading: 3.2 PSEUDO-KD VIA TWO-STAGE OPTIMIZATION
text: The choice of the distribution Pls determines the label regularization method. As described above, in KD the smoothing distribution is an output of the pre-trained teacher model, and in LS it is a uniform distribution. Generally speaking, the optimal smoothing distribution is unknown, and ideally, we would like to learn the optimal Pls. In this regard, we set up the following two-stage optimization problem:
min θ Rθ(P ∗ls), subject to P ∗ls = argmin Pls Rθ(Pls). (9)
This optimization setting, also called bilevel optimization (Colson et al., 2007), is strongly NP hard (Jeroslow, 1985) so getting an exact solution is difficult. Luckily, in our case the inner optimization loop has a closed-form solution. The following Theorem 1 gives the explicit formulation ofRθ(P ∗ls). In particular, we will show that Rθ(Pls) is a convex function w.r.t Pls, and we will obtain the optimal solution P ∗ls = argminPls Rθ(Pls) by using a Lagrangian multiplier. For the details of this derivation please refer to Appendix A. Theorem 1. The solution to the inner loop of optimization in Eq. 9 is given by:
P ∗ls(j|xi) = pθ(j|xi) α β∑K
j′=1 pθ(j ′|xi)
α β
, (10)
where pθ(j|xi) is the output probability of j-th class of model Sθ(·), α is the smoothing coefficient defined in Eq. 5, and β is defined in Eq. 7.
As a result, we reduced the two-stage optimization problem in Eq. 9 to a regular single-stage minimization:
min θ Rθ(P ∗ls), (11)
where
Rθ(P ∗ls) = 1
N N∑ i=1  K∑ j=1 P̂ ∗ls(j|xi) · (− log pθ(j|xi)) + βP ∗ls(j|xi) log(K · P ∗ls(j|xi))  , (12) P ∗ls(j|xi) is given in Theorem 1, and P̂ ∗ls(j|xi) is defined in Eq. 5. Note that the solution P ∗ls is deterministic and interpretable. Moreover, we show the post-hoc interpretability of the solution in the two remarks below by demonstrating the relation of the solution with LS and self-distillation methods. Remark. When β is extremely large, P ∗ls will be close to the Uniform distribution, and our objective function will be equivalent to optimizing the CE loss with a Uniform LS regularizer. When β is extremely small, the smoothing distribution collapses to the one-hot vector.
Remark. There is an intrinsic connection between the P ∗ls distribution and generating softmax outputs with a temperature factor. Specifically, when β = α · τ , we could have
P ∗ls(j|xi) = exp( zi,j τ )∑K
j′=1 exp( zi,j′ τ ) (13)
Since the smoothing distribution in this case becomes the temperature smoothed output of the model, our Pseudo-KD technique can be considered as a generalized version of the self-distillation method 1.
To summarize, our method can be expressed as an alternating two-stage process. We generate optimal smoothed labels P ∗ls using Theorem 1 in the first stage. Then, in the second phase, we fix the P ∗ls to compute lossRθ(P ∗ls) and update the model parameters. We execute our method in an online manner, which eliminates the need for additional memory or storage for the parameters or outputs of the pre-trained model. The online training process is shown in Algorithm 1.
Algorithm 1 Online Training with Pseudo-KD Input: Training set {(xi, yi)}|Ni=1, batch size n, number of training epochs T , learning rate η;
1: for i← 1 to T do 2: for t← 1 to N/n do 3: Randomly sample a mini-batch S = {(xi, yi)}|ni=1from training set; 4: Compute the P̂ls according to P̂ ∗ls solution for the mini-batch data; . 1st Stage 5: Update the model θt+1 = θt − η∇θRθ(P̂ ∗ls, S); . 2nd Stage 6: end for 7: end for

heading: 4 EXPERIMENTS
text: In this section, we evaluate the performance of our proposed Pseudo-KD method. Experiments are conducted on both the image classification tasks CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) for the ResNet-based Model with of various sizes (parameters) and the General Language Understanding Evaluation (GLUE) tasks (Wang et al., 2019b) with the RoBERTa-based model (Liu et al., 2019).

heading: 4.1 EXPERIMENTS ON IMAGE CLASSIFICATION
text: Data. CIFAR is a benchmark dataset for image classification (Krizhevsky et al., 2009). We use both CIFAR-10 and CIFAR-100, which have images 50,000 training samples and 10,000 test samples divided into 10 and 100 different object classes, respectively. The validation set is made up of 10% of the training data.
Experimental settings. We evaluated our method on different model architectures including MobileNetV2 (Sandler et al., 2018), ShuffleNetV2 (Ma et al., 2018), ResNeXt29(8×64d) (Xie et al., 2017), ResNet18, ResNet50 and ResNet101 (He et al., 2016). We follow standard data augmentation schemes: random crop and horizontal flip to augment the original training images. The models are trained for 200 epochs, with a batch size of 128 for MobileNetV2, ShuffleNetV2 and ResNet18, and 64 for ResNeXt29 (8×64d), ResNet50 and ResNet101. For optimization we used stochastic gradient descent with a momentum of 0.9, and weight decay set to 5e-4. The learning rate starts at 0.1 and is then divided by 5 at epochs 60, 120 and 160. All experiments are repeated 5 times with random initialization. For the KD experiments (KD-Shuffle and KD-29), we used ShuffleNetv2 and ResNeXt29 as the teachers. All teacher models are trained from scratch and picked based on their best accuracy. To explore the best hyper-parameters, we conduct a grid search over parameter pools. We explore {0.1, 0.2, 0.5, 0.9} for α and {5, 10, 20, 40} for KD temperature. For Pesudo-KD, we also perform explore {1.5, 2.5, 4} for β. The best hyper-parameters can be found in Appendix C. We first compare our proposed Pseudo-KD method with uniform label smoothing and conventional KD with different teachers. All experiments are repeated 5 times with different random initialization.
Results Tab. 4.1 shows the test accuracy for each method using various network architectures. Our method significantly improves classification performance than conventional LS on all six models with over 0.85 or 0.35 points higher accuracy on average for CIFAR100 and CIFAR10, respectively.
1The derivation can be found in Appendix B.
This indicates its adaptability to different networks. Compared to KD methods, our method can give comparable or better performance, while being much more efficient in training. This is because our method could be implemented in an online manner, which only need one forward pass as Base or LS methods. Furthermore, our method does not necessitate the use of a pre-trained teacher, which saves computational overhead.
We also observe that lightweight models like ShuffleNetv2 and MobileNetv2 benefit more from LS, KD and our method than complex models (ResNeXt29 and ResNet101). Furthermore, as a teacher, using a model with a low capacity can provide comparable performance for improvements for small models to using a large model. Specifically, by distilling three low-capacity models (ShuffleNetv2, MobileNetv2, and ResNet18) and using ShuffleNet as the teacher, all three models can achieve comparable performance, with ShuffleNet outperforming the others.
Next, we conduct a series of experiments on three models to compare our approach with other methods without requiring any pre-trained models (except for KD-29 reported as a reference). All experiments are repeated 5 times with different random initialization. The baseline methods include, CS-KD which distill the soft output between different samples of the same class to student (Yun et al., 2020). TF-reg which regularize the training with a manually designed teacher (Yuan et al., 2020) and Beta-LS which leveraging a model with same capacity to learn instance-specific prior over the smoothing distribution (Zhang & Sabuncu, 2020). We follow their best hyper-parameter settings. The time for all baselines is measured based on its original implementations. 2345
Tab. 2 shows the test accuracy and training time for 200 epoch of different methods on three models. It can be seen that our method consistently improves the classification performance on both
2https://github.com/alinlab/cs-kd 3https://github.com/yuanli2333/Teacher-free-Knowledge-Distillation 4https://github.com/ZhiluZhang123/neurips 2020 distillation 5https://github.com/lgcnsai/PS-KD-Pytorch
lightweight and complex models, which indicates its robustness to different networks. Besides, it shows the training time of our method is equivalent to Base or LS methods, which show its stable efficiency over other baselines such as Beta-LS, which still requires a separated model to output a learned prior over the smoothing distribution. We further compare two recent works KR-LS (Ding et al., 2021) and PS-KD (Kim et al., 2021). Our method achieves comparable or better performance than PSKD and consistently better than KR-LS. We have to mention the implementation of KR-LS and PS-KD increase space complexity O(K ×K) and O(N ×K) for class-wise and sample-wise smoothing labels, respectively. However, our method neither increases time nor space complexity during the training.

heading: 4.2 RESULTS ON NATURAL LANGUAGE UNDERSTANDING
text: Data. We run experiments on 7 classification tasks from the GLUE benchmark (Wang et al., 2019b). These datasets can be broadly divided into 3 families of problems. Single set tasks which include linguistic acceptability (CoLA) and sentiment analysis (SST-2), similarity and paraphrasing tasks (MRPC and QQP), and inference tasks which include Natural Language Inference (MNLI and RTE) and Question Answering (QNLI). Following prior works (Jafari et al., 2021; Rashid et al., 2021), we report Spearman correlation for STS-B, Matthews correlation for CoLA, and accuracy for the other tasks for the development set. We report “F1” for the QQP and MRPC and the corresponding test sets.
Experimental Setup. We choose the pre-trained DistilRoBERTa (6-layer) (Sanh et al., 2019) as the base model for all our experiments. For KD experiments, we use a RoBERTa-Large model (24-layer) (Liu et al., 2019) fine-tuned on each of the tasks as the corresponding teacher model. All models are trained with the AdamW optimizer with the default settings Loshchilov & Hutter (2019). To explore the best hyper-parameters, we conduct a grid search over the learning rate ∈ {1e − 5, 2e − 5, 3e − 5}, batch size ∈ {8, 16, 32, 64}, α ∈ {0.1, 0.2, 0.5} and KD temperature ∈ {1, 5, 10}. Moreover, for Pesudo-KD, we perform a search over three β values in {1.5, 2, 4}. The best hyper-parameters can be found in Appendix. We repeat our experiments 3 times with different random initialization.
Results Table 3 presents the dev set results on GLUE. We present the average result for each experiment, over three runs, as well as the standard deviation. As is standard on the GLUE benchmark, we present an average result across the datasets as well. We compare Pseudo-KD to simple finetuning, LS and KD. We observe that our method is better on all datasets compared to fine-tuning and at par (QQP only) or better on all datasets compared to LS. We are comparable on average to
KD, with KD doing better on MNLI and QQP and Pseudo-KD doing better o CoLA, MRPC, RTE and SST-2.
Furthermore, Table 4 shows the GLUE leaderboard test results. We chose the model with the best dev results for each method and submitted it to the official GLUE benchmark to get the test results. We see a similar trend and observe that Pseudo-KD is better than LS and fine-tuning on all datasets except RTE. On average the results are comparable to KD as before. KD is better on CoLA, RTE, QNLI and MNLI and Pseudo KD is better on MRPC, SST-2 and at par on QQP. We observe that on average all the models are close in performance, with an average difference of 0.4 between the best and the worst model.
4.3 EFFECT OF HYPER-PARAMETER α AND β
We conducted ablation tests to probe how the α and β hyper-parameters affects the performance of our method. We also compared our results to LS while choosing the same α for both. Specifically, we chose α ∈ {0.1, 0.2, . . . , 0.6}. For Pseudo-KD, we also verified the influence of β/α ∈ {1.5, 2.5, 4}. Following the discussion on equation 13 we observe that β/α can be interpreted as the temperature in self-distillation.
We plot the accuracy on CIFAR100 for two architectures (ResNet18 and ResNet50) as shown in Fig. 2. The plot shows that our method can consistently outperform the LS method across different α values even when varying the β/α parameter. The results are consistent for both architectures. As is expected, the accuracy of both LS and our method gradually decreases when α increases from 0.1 to 0.6. It is interesting that even at an α of 0.6 the drop in accuracy is less than 1% of the peak value.
We observe that a temperature value between 1.5 and 2.5 is the best to obtain optimal performance.

heading: 5 CONCLUSION
text: Our aim in this work is to fill the accuracy gap between KD and LS techniques, while maintaining the training efficiency of LS. We proposed learning an instance-specific label smoothing regularization simultaneously with training our model on the target. We began by revisiting the traditional LS method and introduced our objective function by substituting the uniform distribution with a general, instance-specific, discrete distribution. Within the new objective, we explained the relationship between the LS and KD. Particularly, our method can be interpreted as an online version of self-distillation. Then, using a two-stage optimization approach, we obtained an approximation for the optimal smoothing function. We conducted extensive experiments to compare our model with LS, KD and various teacher-free methods on popular CV and NLU benchmarks and showed the effectiveness and efficiency of our technique.
We want to extend this research in two directions in the future. First we want to improve our algorithm further and explore replacing the alternating approach to two-stage optimization with a more complex algorithm to reach closer to the optimal smoothing.
Next we want to explore practical applications where our method can replace KD. One interesting application is the pre-training of compressed language models. Typically, the best compressed models are pre-trained using distillation from a much larger teacher which makes the training process cumbersome and computationally intensive. It would be interesting to explore if Pseudo-KD can achieve comparable performance to KD in that setting.

heading: A DERIVATION OF THE PROOF FOR THEOREM 1
text: Proof. First, observe that the functionRθ defined in Eq. 7 is a convex combination ofN nonnegative functionsRi = EP̂ls [− log pθ(j|xi)]+βDKL(Pls(·|xi)‖U(·)), for i = 1, . . . , N . We will show that each of Ri is a convex function of the components of simplex Pls(·|xi) by computing the Hessian matrix ofRi with respect to Pls(·|xi):
H(Ri(Pls(1), · · · , Pls(K)) =  ∂2Ri ∂Pls(1)2 , ∂ 2Ri ∂Pls(1)∂Pls(2) , · · · , ∂ 2Ri ∂Pls(1)∂Pls(K) ∂2Ri ∂Pls(2)∂Pls(1) , ∂ 2Ri ∂Pls(2)2 , · · · , ∂ 2Ri ∂Pls(2)∂Pls(K) ..., ..., . . . , ...
∂2Ri ∂Pls(3)∂Pls(1) , ∂ 2Ri ∂Pls(3)∂Pls(2) , · · · , ∂ 2Ri ∂Pls(K)2
 (14)
= diag( β
Pls(1) ,
β Pls(2) , · · · , β Pls(K) )
When β is greater than zero, the Hessian is positive definite. Therefore, eachRi is a convex function of the componets of Pls(·|xi). As a result, Rθ is a convex function of the collection of components of every simplex Pls(·|xi). For simplicity, we derive the global optimum solution for eachRi with a Lagrangian multiplier:
Li(Pls, λL) = K∑ j′=1 [ −P̂ls(j′) log p(j′) + β · Pls(j′) log Pls(j ′) 1/K ] + λL( K∑ j′=1 Pls(j ′)− 1),
where we omit the dependency on xi to simplify the notation. We set the corresponding gradients equal to 0 to obtain the global optimum for j = 1, . . . ,K.
∂Li ∂Pls(j) =− α log p(j) + β · logPls(j) + β + β logK + λL = 0 (15)
P ∗ls(j) = exp( α β log p(j)) · exp(−β − β logK − λL β )
= exp( α
β log p(j)) · Cls
Since ∑K j′=1 Pls(j ′) = 1, we have
K∑ j′=1 Pls(j ′) = K∑ j′=1 exp( α β log p(j′)) · exp(−β − β logK − λL β ) = 1 (16)
Cls =exp( −β − β logK − λL
β ) = 1∑K j′=1 exp( α β log p(j ′))
So the optimal P ∗ls(j) is given by the formula:
P ∗ls(j) = exp(αβ log p(j))∑K
j′=1 exp( α β log p(j
′)) = p(j)α/β∑K j′=1 p(j ′)α/β . (17)

heading: B DERIVATION FROM OPTIMAL SMOOTHING TO SOFTMAX OUTPUT WITH TEMPERATURE
text: When β = α · τ , we could have
P ∗ls(c|xi) = pθ(c|xi) 1 τ∑
j pθ(j|xi) 1 τ
= ( e
zi,c∑ m e
zi,m ) 1 τ∑
j( ezi,j∑ m e zi,m ) 1 τ
= (ezi,c) 1 τ∑
j(e zi,j )
1 τ
= e zi,c τ∑
j e zi,j τ
(18)
This is the commonly used soft label in KD approach.

heading: C HYPER-PARAMETERS FOR PSEUDO-KD
text: Model α β batch size lr
CIFAR100 0.2 1.5 - - CIFAR10 0.2 2.5 - - CoLA 0.1 2 32 1e-5 RTE 0.1 2 16 2e-5 SST-2 0.1 5 64 2e-5 MRPC 0.1 2 8 2e-5 QNLI 0.1 4 64 2e-5 QQP 0.1 4 32 2e-5 MNLI 0.1 2 32 2e-5

Review summaries:
The paper proposes a novel teacher-free label regularization method named Pseudo-KD which bridges the gap between label smoothing and knowledge distillation. The proposed bilevel programming framework is implemented by an alternating two-stage process. The optimal smoothed labels are generated in the upper optimization, and the backbone network is optimized by the inner loop. The overall framework is equivalent to self-distillation without the need for extra computational cost. The authors claim that the extensive experiments on both image classification and natural language understanding tasks validate the effectiveness of the proposed method. 

This paper proposes a new formulation of an objective function for label smoothing regularization to unify label smoothing (LS) regularization and knowledge distillation (KD). The objective function learns an instance-specific label smoothing regularization while training a model towards its prediction target. When the instance-specific label smoothing regularization uses a simple uniform distribution for all training samples, the objective function falls back to traditional LS. When the instance-specific label smoothing regularization uses the output distribution of a teacher model, the objective function learns a KD model. The paper formulates the learning of the optimal instance-specific label smoothing regularization as a bi-level optimization problem and derived a closed-form solution for the inner level of the problem. Experimental results on both image classification and natural language understanding tasks show that the proposed objective and learning algorithm are effective and yield results comparable to the baseline LS and KD models. 

The authors propose a self-distillation approach that is derived from a series of observation made regarding how LS and KD work, linking in the two together in the process. The paper then experimentally shows the superiority of the proposed approach against KD on CIFAR-10/100 and languge datasets. Important comparisons against closely-related methods are however missing.