Do Not Blindly Imitate the Teacher: Loss Perturbation for Knowledge Distillation
Authors: Rongzhi Zhang
Jiaming Shen
Tianqi Liu
Jialu Liu
Michael Bendersky
Marc Najork
Chao Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: distillation
loss function
natural language processing

Abstract: Knowledge distillation (KD) is a popular model compression technique to transfer knowledge from large teacher models to a small student model. Typically, the student learns to imitate the teacher by minimizing the KL divergence of its output distribution with the teacher's output distribution. We argue that such a learning objective is sub-optimal because there exists a discrepancy between the teacher's output distribution and the ground truth label distribution, and forcing the student to blindly imitate the unreliable teacher output distribution leads to inferior performance. To this end, we propose a novel knowledge distillation objective PTLoss by first representing the vanilla KL-based distillation loss function via a Maclaurin series and then perturbing the leading-order terms in this series. This perturbed loss improves the student generalizability by effectively distilling knowledge from a shifted distribution closer to the ground truth data. We also propose a method to compute this shifted teacher distribution, named Proxy Teacher, which enables us to select the perturbation coefficients in PTLoss. We theoretically show the perturbed loss reduces the deviation from the true population risk compared to the vanilla KL-based distillation loss functions. Experiments on three tasks with teachers of different scales show that our method significantly outperforms vanilla distillation loss functions and other perturbation methods.

Under review as a conference paper at ICLR 2023
Do NOT BLINDLY IMITATE THE TEACHER: Loss PERTURBATION FOR KNOWLEDGE DISTILLATION
Anonymous authors Paper under double-bling review
ABSTRACT
Knowledge distillation (KD) is a popular model compression technique to trans- fer knowledge from large teacher models small student model. Typically. the student learns to imitate the teacher by minimizing the KL divergence of its output distribution with the teacher's output distribution. We argue that such a learning objective is sub-optimal because there exists a discrepancy between the teacher's 's output distribution and the ground truth label distribution, and forcing the student to blindly imitate the unreliable teacher output distribution leads to inferior performance. To this end, we propose novel knowledge distillation ob- jective PTLoss by first representing the vanilla KL-based distillation loss function via a Maclaurin series and then perturbing the leading-order terms in this series. This perturbed loss improves the student generalizability by effectively distilling knowledge from shifted distribution closer to the ground truth data. We also pro- pose method compute this shifted teacher distribution, named Proxy Teacher, which enables us to select the perturbation coefficients in PTLoss. We theoreti- cally show the perturbed loss reduces the deviation from the true population risk compared to the vanilla KL-based distillation loss functions. Experiments on three tasks with teachers of different scales show that our method significantly outper- forms vanilla distillation loss functions and other perturbation methods.
INTRODUCTION
Deep neural networks (DNNs) have achieved enormous success due to their massive sizes, expres- sive power, and the availability of large-scale data for model training. Accompanied by their success is the increasing need of deploying such large-scale DNNs on resource-limited devices. Knowledge distillation (KD) a widely-used model compression technique, which distills knowledge from large teacher models into much smaller student model to preserve the predictive power of teacher models (Buciluă et al. 2006 Hinton et al., 2015). In the deacher-studens learning paradigm of KD, the student model is encouraged to imitate the teacher models' outputs distillation dataset.
The typical training objective in KD such as KL loss (Hinton et al., 2015 Menon et al., 2021 Stanton et al. 2021) encourages the student's 's outputs to be close to the teacher's outputs much as possible. This implicitly assumes the teacher's outputs on the distillation data are perfect. However, the teacher's s predictive distributions can be biased from the ground truth due to various factors, such as the inductive bias encoded in the teacher's 's model architecture, miscalibration in the training pro- cedure (Menon al. (02T), I) the bias in the source dataset used for learning the teacher model (Liu et al. 2021 Lukasik et al. [2021). D). Enforcing the student to blindly imitate the teacher's 's outputs can make the student inherit such biases and produce suboptimal predictions.
To overcome this challenge, one commonly used approach is to scale the teacher's 's logits via a tem- perature parameter (Hinton et al., 2015). Menon al.] (202T) show that proper temperature can improve the quality of the teacher model's predictive distribution, but the shifting space offered by the temperature scaling limited, and the optimal temperature value relies on expensive grid search. Along separate line, label smoothing (Szegedy et al. 2016) is proposed as a general tech- nique to regularize the neural networks, and modulated loss functions (Lin et al. [0177] Leng et al.) 2022) are designed to address several statistical issues (e.g..overfitting issues and data imbalance) in model training. However, there lack works that explore tailoring such techniques for more robust knowledge distillation.

---

Under review as a conference paper at ICLR 2023
We propose PTLoss for knowledge distillation, which revises the vanilla loss function in knowledge distillation to implicitly create a debiased teacher distribution closer to the ground truth. Instead of forcing out-and-out imitation of the original teacher model, relax the KL loss and add per- turbations to the distillation objective. Specifically, we approximate the standard KI. loss using the Maclaurin series, which allows us construct a more flexible objective and to perturb the leading- order terms, To determine the perturbation extent, we design a method to compute the equivalent distribution of the implicitly shifted teacher by perturbation, namely Proxy Teacher. With the com- puted proxy teacher distribution, we measure the empirical deviation between the perturbed teacher and the ground truth data. It leads a systematic searching strategy for the perturbation coefficients, Lee, the near-optimal perturbation coefficients should minimize the deviation between distilled risk and population risk on the validation set.
Theoretically, we justify the effectiveness of the PTLLss by proving that it can reduce the devi- ation from the distilled empirical risk compared KI. loss. We draw connection between the PTLoss and other perturbation method (i.e., label smoothing Szegedy et al., 2016)). We illustrate that the PTLoss can debias the teacher to produce higher-fidelife outputs via a finer-grained pertur- bation, while subsuming existing perturbation techniques special cases. Experiments datasets with different-sized teacher models demonstrate the empirical advantage of the PTLoss. Moreover, the Proxy Teacher method for perturbation coefficient search significantly outperforms the PTsos with random searched coefficients, which shows the superiority of this systematic pa- rameter search method.
three
In summary, our key contributions are:
A perturbed loss function PTLoss, which formulates the vanilla knowledge distillation loss in the form of Maclaurin series and perturbs to improve the fidelity of teacher models;
A Proxy Teacher method to solve the implicitly shifted teacher and to determine the perturbation coefficients in PTL.ss;;
Theoretical analysis proves that we can lower the distilled empirical risk bound with PTLoss and establishes the connection with other perturbation methods;
Comprehensive experiments three public datasets with different-sized teacher models demon- strating the advantage the PTLoss and the Proxy Teacher method.
RELATED WORK
2.1 KNOWLEDGE DISTILLATION
Knowledge distillation first proposed in (Buciluà et al. (006) to compress the large model ensem- bles to smaller, faster models without significant performance drop. This technique is generalized by (Hinton et al. 2013). where the temperature parameter is introduced to smooth the prediction and the student loss and distillation loss integrated. With the prevalence of pre-trained language models (Devlin et al. 2018). it becomes more urgently needed to distill such large models to de- ploy on edge devices with limited resources. For example, DistillBERT (Sanh et al., 2019) uses the teacher's 's soft prediction probability to train the student model; TinyBERT (Jiao et al. 2019) aligns the student's layer outputs (including attention outputs and hidden states) with the teacher's. s; MobileBERT {Sun et al., 2020) also adopts layer-wise training objective and equips bottleneck structure to distill from BERT-Large.
2.2 DISTILLATION THEORY
In parallel with the empirical success of the application of knowledge distillation, many works are devoted to answering mechanism. Hinton et al. (2015) propose the teacher's soft labels can provide "dark knowledge" via weights on the wrong labels. Menon et al. (2021) present a statistical perspective on distillation, they observed that a good teacher model should be Bayesian to lower the variance of the student objective via the teacher's prediction distribution. Stanton et al. (2021) show the discrepancy between the teacher and the student regarding their output distribution and identify the optimization difficulty in knowledge distillation. Ji & Zhu (2020); Zhou et al. (022))
Hsu et al. (202T) study distillation from several different aspects, but there remains a gap between the theoretical analysis and the better distillation techniques.
2

---

Under review as a conference paper at ICLR 2023
2.3 LOss DESIGN
Our work is also related to loss function design and learning. Lin et al. (2017) propose to reshape the cross-entropy loss focus on the hard examples, but it cannot be directly applied in our setting because the data imbalance issue targeted by their approach do not directly correlated with the biased teacher model knowledge distillation Leng al. (2022) propose to expand cross-entropy loss and focal loss to a linear combination of polynomial functions and study the Poly-1 formulation on computer vision tasks. However, the motivation is not clear under the fully supervised settings and they skirt around the problem of papre-paraeeter search in high-dimension space when the order of polynomials is high. Notably, TaylorGLO (Gonzalez & Miikkulainen) 2021) utilizes Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to optimize multivariate Taylor parameterization of a loss function and learning rate schedule during training. But they fail to provide a principled analysis regarding the performance gain after perturbation. Instead, we theoretically and empirically demonstrate the necessity of adding perturbation to the learning objective for knowledge distillation, where high-fidelity teacher is required to provide quality supervision for the student training.
3 PRELIMINARIES
We study knowledge distillation for classification tasks. For brevity, we discuss binary classification in this paper, but the formulation can be extended to multi-class settings.
In binary classification, we are given training sample set 8 y >n) IPN, for distribu- tion IP over instances and labels {0, 13 Our goal is to learn logits predictor f : X R² with minimal risk:
R(f)= E(a,))> [(=)))
Here, fyy, f(r)) is the loss of predicting f(x) E IR? when the true label is y E IR3 The predicted
probabilities are p(x) = (11())) 4of(()))))) ) which is invariant constant shift both
fa))
f2(())
(I) and fz()). Thus we can simplify the f(x) as (0, fx))..
In this paper, we consider the KL divergence:
fly, fxx) y" log(y) - 5f f(x) log ((f(z)),
(2)
where Z is the partition function Z(x) 1_exx for R For finetuning from true labels, approximate the risk R(f) via the empirical risk
R(()
1
Σ ((((sa))
N
RE[N]
for one-hot encoding yn E (0, 132, and vector of losses for each possible label [(f(x:) E R2, which is defined as
((((z)) = [-f(x) + log Z(f(x))
(4)
The term y log(y) is omitted in the above equation because it is always zero for one-hot hard labels.
For distillation from teacher's predictions, we first compute teacher class-probability estimates p'(x) (p' (0)z), (1[=)), where p' (y((z) estimates how likely to be classified as y. Then we train student model to minimize the distilled risk
Rp f) = N
Σ (p((=..)"((((x))))) p (x.)"" log(p' (x.)) nE[N]
(5)
Note that each loss from the above risk the widely used distillation loss, proposed in (Hinton et al., 2015). It is the KI. divergence of the student output distribution from the teacher output distribution
(1() p (T.)
((p' ((xw) = ExL (p'(x.), ((w.)."((..))) = KL. (p' (x.) (xx))lp"(Zz))) = Σ (Ta) log
Je(o.13
(6)
Forr simplicity, assume the teacher model here does not contain temperature scaling, we will discuss the impact of different temperatures in the experiment section.
3

---

Under review as a conference paper at ICLR 2023
[0
10604
1.0
1-order P Tooss Value
P [0
p'- (05....
p² (073..3)
p'- 1aA0
p' (p-1,0.)
Optimal student probabilitt
0.8
8.6
KLue
Toake erlees
8.4
0.2
Teddu PTuss
0.0
0.4
1/0
00
02
04 0.6
1.0
Student Probability P3
Teacher probability pb
(a) The I-order PTLoss (M 1, 71 1) with dif-
(b) Comparison between the standard KI.-loss
ferent teacher probabilities p'. For each curve, p' is fixed and we search the optimal student probability po on class 0. Black crosses denote pó achieving the lowest loss value given p'.
with PTLLss with perturbation order from to
10. Each perturbation coefficient and ft are
set as 2 for this plot.
Figure I Intuitive understanding of Poossss via basic forms.
where p) (2,,) and (Z,) denote the probability of the i-th example In belonging the j-th class according to the teacher (student) model.
Instead of directly learning a student model p from 8, knowledge distillation learns the student p" from trained teacher model p" by replacing the ground truth one-hot label yv. with the teacher output probabilistic label estimates p' (Ir)). Then, the student model is learned by minimizing the empirical distillation risk P((p:; p", 8)) N En lKL (p' (xx): (*,,).
4 PERTURBED DISTILLATION LOSS
Using KL divergence as the distillation loss (in short "KL loss"" essentially assumes the teacher model is perfect and thus forces the student model to mimic the teacher's output label distribu- tion. In reality, the teacher model can produce a biased estimate of label distribution and lead to sub-optimal student model, as demonstrated by both empirical results (Müller et al., (019)) and theoretical analysis (Menon et al. 2021).
In this work, we present novel distillation loss that relaxes the standard KL loss form and accom- modates the distribution gap between (possibly biased) teacher output distribution and underlying ground truth distribution. Inspired by the PolyLoss (Leng et al. 2022), propose to first replace the logarithmic terms in the standard KL loss with their corresponding Maclaurin series:
log(z) = log(1 - (1 =)) = (1=x)"
log(1 - -r) =
m
T7
Then perturb the polynomial terms with two perturbation coefficients Yy and cm
log(1
Σ "m)"", log(x: ~ x - -
cm x)""
(8)
m==1
Here, essentially replace the original coefficient 1 of the m-th order polynomial term in the standard KL loss to ( m 7mm) or ( + G.)) When "/m 6m [- m oo), can further separate these perturbation coefficients with the original coefficient and obtain:
log(1 - x) ~ log(1 - x) (mm)""
log(z) z log(z)
> (em)(1-2))
(9)
Now, by replacing the term log (*,) and log pi (a:.. log (1 (*.)) in the standard KL loss
Eq. 6 with Eqs. and 9 we have:
(KL (p' (2'(x)) (xx), ( (xx)) -H (p (zx)) + px(2)) | log (x + lo
- (2))) p((wn)
log (wx)
cm (1 pM
(10)
+ pb()))
log(1 (o(())))
Tm (p6(
4

---

Under review as a conference paper at ICLR 2023
where H.I (x,)) = (z,( p' (zn, is the entropy of the teacher output distribution and thus student-inden mudeent-ndependdet. By further separating out perturbation coefficients on the right hand side of the above Eq. 10 have:
Ext. (p' (an), p' (xn)) H. (p' (zn)) + p)((n)) log (o(xx)) (1 pō(xnn) |- log(1 po(xn)))
+ po(
Σ (m(11p6(cx.)) + (1 - -p-((e))
Σ 7m (pō((())))
(11)
where the first line the right hand side is essentially the original KL loss. Therefore, finally define our perturbed distillation loss as:
PP = EKT + (T( cm (1 -i.() + (1 ((()))
Σ m=
Ym (p)(xx)))
(12)
we only perturb the leading M-th order terms, then we have:
M
M Σ
PPT-M EEL ()(an)
Σ m=1
Fm
(i)(p))()) + po((n))
7m (P)()))))
(13)
m=1
In vector form, it is
M
bPT-M Cxz (xx)) Σ nm (1 a01-pp(ce) p²(x)))"
(14)
where @ is the clement-wise product, and 7m (Cm, )mm)
Figure shows some basic forms of the PTLoss to give an intuitive understanding of the loss func- tion family and the perturbation extent. In Figure Tal present the simplest 1-order PTLoss to show how the perturbation influences the student training. For example, when p' = [0.1, 0.9), the optimal student probability on class between 0 to In general, This specific PTLoss makes polarized adjustment to the student prediction. In Figure Ib, we show PTLoss with different per- turbation orders, i.e., M 21, 10). For KL loss, the identical curve implies it encourages a total imitation of the teacher's behavior. On the other hand, the PTLoss relaxes the objective by perturbations and thus prevents the student model from overfitting the biased teacher. The pertur- bation granularity can be adjusted by the perturbation order and coefficients. Note that we set all the perturbation coefficients as for simplicity Figure [bb but in practice, we use a systematic approach to determine the perturbation coefficients, as described in the below Section 4.1
4.1 PROXY TEACHER
In this section, we theoretically prove that the introduced PTLoss allows for reducing the deviation from the true population risk. We first show that PTLoss implicitly transforms the teacher's predic- tions, then select the minimizer in terms of the perturbation coefficients in PTLoss to bound the distilled empirical risk with introduced perturbation. Finally, we prove that PTLoss helps us get a closer estimation of the predictor to the oracle signal function.
4.1.1
THEORY OF DISTILLATION
We start from the population risk R(f) defined Eq. I and write it in a Bayes classifier form
R(f) = F. Fy f()))]] = F. (p" (((=()
(15)
where (1))) Plyla) with y E (0. 1) is the Bayes class probability distribution in the label space, and fly, f(z)) and ((f(=)) were defined in Eqs. 2 and respectively. Again, the term y" log(y) is omitted in the above equation because it is always zero for one-hot hard labels.
When labeled examples are limited, people learn the student (*(x) by minimizing the distilled risk R(p', S) to achieve low population risk R(f*). For any teacher model's predictions p', we have the following proposition:
5

---

Under review as a conference paper at ICLR 2023
Proposition 1. Given bounded loss teacher model p' with its distilled empirical risk defined in Eq. 5 and any predictor f + R. we have:
(R(p',f:8) - S v [[((a))(((()))]
(16)
log (og'()))))))))))))
Es
where VI. denotes the variance of random variable.
We defer the proof appendix. The above proposition states that the deviation between distilled risk and population risk depends on three terms: (1) variance of KL. loss for a random example, (2) MSE between teacher's predictions p' and the true distribution p", and (3) scales of entropy of the teacher's predictions. The latter two terms dominate in the large N regime because the first term can be bounded by the Cauchy-Schwartz inequality and is of order O1)/N). The second term quantifies how close the teacher is to the true distribution. The third term quantifies the teacher's s uncertainty. A well-calibrated and certain teacher yields improved bounds on the generalization emror of the student.
4.1.2 PROXY TEACHER
In order to lower the population risk and the generalization error of the student model, we introduce PTLoss that implicitly transforms the teacher's predictions Then we leverage the calculated proxy teacher to guide the search of the perturbation coefficients that can minimize the right hand side of Eq. [6 Since the KI. loss on the original teacher's predictions is a special case of PTLoss with all perturbed coefficients to be zero, we can improve the student by searching for the optimal coefficients for perturbation.
Proxy Teacher under the KL loss.
Concretely, we replace the loss function with a perturbed form as
ne 2 (1
f(#)
Epr(y, f(x)) ) = Ely, f(x)( +
(17)
7(f(z))
[M]
where @ denotes element- wise multiplication between two vectors and 7mm = (Emi E RR are hyperparameters for the perturbation order m..
Similarly, we define pfr(f(z)) = L(f(x)) EmEjM Tm @ e
2(((z))
Replace the distillation loss by the perturbed loss, we have
A f:S)
1 N
Σ (p'(=n)"lrr p' (' log(p' (zx))) nE(N)
(18)
We first solve Proxy Teacher PTPT such that R(p'Pr, f: 8) is close to Rpr(p', f)S). Then we find the optimal perturbation coefficients that yield the best risk bound.
Concretely, we define a transformation function
[0, 1]2 + [0, 1]2 such that
pd PT (TT()) (x)))(((()) + +pp SP (ar) Io)""oo(pp PT (=)) = p' (z))) + + p'(x)
(19)
where p PPP (x)==gte (p)()))) is the Proxy Teacher. We transform teacher's predictions as 2 (x) = g (x)) so that the distilled risk Rpr(f; 8) with the original teacher's predictions f'(x) is close to R(p', 8) with the proxy teacher's predictions p(er(z)) In practice, f(x) is unknown, but we know the minimizer of the left hand side in Eq. 19 ftrr (x) with softmax(f*rr (a:) = p (a)). Assume the student functional space is complicated enough, then we can replace the f by flr (x) and solve the above equation by numerical method.
Note that the boundedness assumption on the loss is standard BBocchronn et al.
2005
Theorem 4.1;
Menon 202T) Proposition 2)

---

Under review as a conference paper at ICLR 2023
4.1.3 BEST PROXY TEACHER
For each MMM1] mm I [M]]]. we can compute the risk deviation upper bound according
to Proposition I
Jomitting the O(1/N) variance term:
D( = (Ex [|pp" PT (x) p (z))la))) E. (D""T)())) (x)) log (a)) ( )
(20)
To compute the above deviation, we replace expectation by the sample mean. We also replace p" (Ix) with yn because it is an unbiased estimate of (xx))
Then we minimize the empirical deviation:
D(nA))=
Σ PE[M]
[Ip'"T(x.)) (xx) -xa)-yalaa]
+
1 N
Σ REIN]
(Du) log p (mx) (xT) (
(21)
N
The first term is the deviation between the equivalent teacher and the ground truth, which encourages the teacher to be well-calibrated. The second term encourages the equivalent teacher to be as certain as possible. We randomly generate 100 sets of Vaaij and pick the optimal M[a that minimizes D.
4.2 CONNECTION TO OTHER PERTURBATION METHODS
We compare PoLoss with the label smoothing method and claim that label smoothing proposed in (Szegedy al. 2016) a special case of PTLoss. Per the implementation in Szegedy et al. (2016),
we can smooth the teacher labels in KD by
P tu (= = (1 - (1-8))%((6 (z.) + 8/2,
(22)
with a smoothing parameter Starting from Eq. 11 we can replace the term (xr,) by smooth (1.))) which holds for binary classification. Then the original Eq. 11 with label smoothing is:
version po (xn)) and use (2x)
tk (p' (zn). se IH (xn) + P (In) log pū(x)) + P (xx) [- po
+ +
1 (xn)
Σ cm -((xx)))
+ (1- Po (xr.))
Yom Tm (Pū())))
(23)
For the entropy of the teacher output, the smooth version Iil (I,) is different from the original H (p'(x)) with only a constant C. We introduce Ap(( = 5/2 (ph(.,) and replace all the
Po in Eq. 23 by ti (In) (z() + A2b( then we get:
de (p' (n.). (zx))) =xx (a.), ,, (xx,)
+ +Apb(#.)) [[ log pū(())) + + (p((())) [- log(1 po(x))))
+Ap)(zn)
(1 pM( + (-Ap((Zzn)
I' (zm)))
(24)
Again, by replacing the term log (1r.) and log (1 p8(#x)) with Eqs.
and have:
de (p (2x), p=())) =xxi(p' (sn), p²(x))
+ + Σ (1 po(xx))) (-Ap) (p. [I
+ A
Σ 6m - pm( + (-Api((zm)
I'm (I)
(25)
Thus, by setting
App(xx)
Aph ( + 7);
cm
(T()
m
fm); 7/5
(26)
get the same form Eq.
12
Cxr (p'(sn)) (sn)) p" (Hx)) = tx (In)
Σ
G
(1 (T.))) +(1 -1)(2.))) Σ
(p((==))) (27)
cm
m=1
m=1

---

Under review as a conference paper at ICLR 2023
Dataset
Task
Train
Distillation
Dev
Test
MNLI
Natural Language Inference
58,905
314,161
19,636
9.832
SST-2
Sentiment Analysis
6,734
53,870
6,736
872
BoolQ
Boolean Question Answering
2,500
5,927
1,000
3,270
Table I: Dataset Statistics
5 EXPERIMENTS
Tasks and Datasets. We conduct experiments on three public benchmark datasets, including MNLI Williams et al. (2017) for multi-genre natural language inference, SST-2 (Wang et al.] 2018)
sentiment analysis, and BoolQ (Clark et al. 2019) for boolean question answering. The dataset statistics are shown in Table
Model Architecture. For the teacher model, we choose the T3 architecture (Raffel al. 2020) and select three teacher models of different scales. Specifically,
use T5-xxl with II billion parameters, T5-x1 with billion parameters, and T5-large with 770 million parameters. For the student model,
BERT-base model (Devlin al.. 2018 with 1 110 million parameters.
Compared Methods. We compare PTLoss with the following baselines: 1) Standard KL loss (Kullback) 1959): use standard KL loss in knowledge distillation; 2) Temperature scaling (Hin- ton et al. 2015): scale the model logits via temperature hyper-parameter; 3) Label smoothing (Szegedy al.] 2016): smooth the teacher's output by small scalar; 4) Focal loss (Lin al. 201 modulated cross-entropy loss to focus learning on hard examples.
5.1
MAIN RESULTS
Dataset
Teacher Size
Teacher Acc.
KI.
Temp.
Smoothing
Focal
Los
T5-xxl
94.68
90.21
90.79
90.15
90.30
90.94
MNLI
T5-xl
92.42
90.41
90.32
88.94
88.83
90.84
T5-large
93.56
89.95
90.36
90.46
90.42
90.62
T5-xxl
96.44
88.88
89.56
89.56
90.14
90.25
SST-2
T5-xl
95.18
89.67
89.68
90,02
89.22
90.25
T5-large
95.53
88.89
89.56
89.56
89.45
90.02
T5-xxl 89.14
69.57
72.23
68.38
68.78
72.69
BoolQ
T5-xI
87.52
70.40
72.66
68.10
69.51
72.87
T5-large 77.91
69.39
70.03
69.39
69.27
70.83
Table 2: Main Results three datasets. The student model is distilled from teacher models with different size. We show student model's test accuracy (%) and list the teacher model's validation accuracy (%) in the colored column for reference. The details of the hyper-parameter search for each method are introduced in Appendix A.l
Table 2 shows the performance of the PTLoss and the baselines. In this set of experiments, we set the perturbation order as 5 in the PTLoss, the corresponding perturbation coefficients obtained through our Proxy Teacher method. We found that PTLoss outperform all the baselines under 9 settings, the average performance improvement the standard KL 1.31%. We found that for the most challenging task BoolQ, PTLLss presents the most prominent improvement to the underlying standard KI. loss.
Among all the baselines, temperature scaling performs strongly on MNLI and BoolQ, while label smoothing and focal loss yield competitive results on SST-2, but the PTLoss holds lead to the strongest baseline method on all the tasks. Besides, present different-sized teacher models on each dataset. The results show that the performance gain is consistent regardless of the capacity of teacher models. Notably, we report the baseline results with exhaustive syper-parameter search as shown Appendix A..I Instead, we only give the perturbation order for PTLoss and obtain the perturbation coefficients via the Proxy Teacher method. Although the hyper-parameter search space
8

---

Under review as a conference paper at ICLR 2023
y= 0.228'' 0.737 R2-0.84
weccer coefficient
random coofiicien
0.75
Student Test Accuracy
912
$1.01
MNUI Accuracy
91.0
0.70
80.2
0.85
loss
0.80
2 Perturbation Order
0.15
0.20
Teacher Model Validation TVD
(a) The Proxy Teacher method v.s. random search for the perturbation coefficient selection. We conduct experiments MNLJ with T5-xl teacher.
(b) Correlation between the validation TVD the teacher model and the test accuracy of the student model. Experiments are conducted BoolQ.
Figure 2: PTLLss analysis.
of PTLLss is much larger than the baselines, we still save much human labor owing to the automatic process of Proxy Teacher method compared to the baselines.
5.2 PERTURBATION COEFFICIENTS SEARCH VIA THE PROXY TEACHER
We validate the effectiveness of the Proxy Teacher method for perturbation coefficients selection. We use MNLI as the representative dataset to show the difference between the Proxy Teacher and the random search method. For each perturbation order, Proxy Teacher follows the procedure in Section 2.ction[41..3 4.1.3 select the coefficients on the development set by minimizing the empirical deviation shown in Eq. 21 while random search just samples M-dimension vector with each perturbation coefficient between [-1,
In Fig. 2a we range the perturbation order ranged from 5 and report the student model accuracy with different perturbation coefficients obtained by either the Proxy Teacher or random search. The consistent improvement over the random coefficient demonstrates the importance of adding appro- priate perturbation via systematic coefficients search. If we just randomly set the perturbation coefficients, the student performance can drop by up to 1.2%. Comparing different perturbation or- ders, we also found that the higher the perturbation order, the higher performance difference between the Proxy Teacher and the random coefficients. This is because the higher-dimension space, is harder for random search to get set of appropriate perturbation coefficients, which makes the random PTLLsseven worse than the standard KI. loss. Instead, equipped with the perturbation coef- ficients obtained via Proxy Teacher PTLoss significantly outperforms the underlying KL loss.
We limit the perturbation order in practice because when the order is high, the corresponding per- turbation term will be a low value after the power operation, compared to the leading terms. Also, too high perturbation order will make difficult to search for the optimal perturbation coefficients.
5.3 CORRELATION BETWEEN THE TEACHER MODEL'S TVD AND THE STUDENT MODEL PERFORMANCE
presents the student model performance positively correlated with the total variance distance between the teacher model's output and the ground truth. By calculating the validation TVD of the teacher model, we measure the discrepancy between the teacher's output and the ground truth data. The results demonstrate that the teacher model with higher-fidelity predictive distribution yields a better distilled student.
Fig. 2b
6 CONCLUSION
We proposed PTLoss to perturb the teacher model's output distribution to a high-fidelity one for student model training in knowledge distillation, followed by the Proxy Teachermethod to system- atically search perturbation coefficients by calculating the implicitly shifted teacher. Moreover, theoretically established a bounded distillation risk the proposed PTLoss and illustrated advan- tage the standard KL loss. We also demonstrated the other perturbation methods such as label smoothing fall into the special cases of PTLoss. The empirical study further supported our theory and validated the effectiveness of our method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Stéphane Boucheron, Olivier Bousquet, and Gábor Lugosi. Theory of classification: A survey of some recent advances. ESAIM: probability and statistics, 9:323-375, 2005.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 535-541, 2006.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 19Xi::9905.10444 2019.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1et::8880.04005 2018.
Santiago Gonzalez and Risto Miikkulainen. Optimizing loss functions through multi-variate tay- lor polynomial parameterization. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 305-313, 2021.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint wxXi:1553.002511 2(7), 2015.
Daniel Hsu, Ziwei Ji, Matus Telgarsky, and Lan Wang. Generalization bounds via distillation. arXiv preprint 1tti::2004.0566. 2021.
Guangda Ji and Zhanxing Zhu. Knowledge distillation in wide neural networks: Risk bound, data efficiency and imperfect teacher. Advances Neural Information Processing Systems, 33:20823- 20833, 2020.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv: /909. 10351, 2019.
Solomon Kullback. Statistics and information theory, 1959.
Zhaoqi Leng, Mingxing Tan, Chenxi Liu, Ekin Dogus Cubuk, Jay Shi, Shuyang Cheng, and Dragomir Anguelov. Polyloss: polynomial expansion perspective of classification loss functions. In International Conference Learning Representations, 2022. URL https:
//openreview. wwweeerwwieww.eeeeffeeeeeeee htw.:eeet....oo.....oooaaeaaaaaa
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 2980-2988, 2017.
Boxiao Liu, Shenghan Zhang, Guanglu Song, Haihang You, and Yu Liu, Rectifying the data bias in knowledge distillation. In 2021 IEEE/CVF International Conference Computer Vision Work- shops (ICCVW), pp. 1477-1486, 2021. doi: 10.11800CC2W554122220.000
Michal Lukasik. Srinadh Bhojanapalli, Aditya Krishna Menon, and Sanjiv Kumar. Teacher's pet: understanding and mitigating biases in distillation. arXiv preprint arXiv:2106. 10494, 2021.
Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statisti- cal perspective on distillation. In International Conference on Machine Learning, pp. 7632-7642. PMLR, 2021.
Rafael Müller, Simon Kornblith, and Geoffrey E. Hinton. When does label smoothing help? In NewrlPS, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1-67 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 19Xiv: 10.0/108. 2019.
10

---

Under review as a conference paper at ICLR 2023
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances in Neural Information Processing Systems, 34:6906-699 6919, 2021.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- bert: a compact tank-agn bert for resource- limutee devices. arXiv preprint arXiv:2004.0298. 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2818-2826, 2016.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804 100.:/804.74.11, 2018.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint (0MXvv//74405422 2017.
Helong Zhou, Liangchen Song, Jiajie Chen, Zhou, Guoli Wang, Junsong Yuan, and Qian Zhang. Rethinking soft labels for knowledge distillation: A bias-variance tradeoff perspective. arXiv preprint arXiv:2 102. 02.00000 2021.
A APPENDIX
A.
We list the search range of hyperparamters in Table The search of batch size and learning rate is applied to all the methods. And for each baseline, we search for the best baseline-specific hyper- parameters.
Elype-ppaaaette
Search Range
Learning Rate Batch Size
{2, 5} 10-5
{8. 16, 32, 64, 128, 256] (0.1, 0.2, 0.5, 1.0, 2.0, 10) (0.02, 0.05, 0.1, 0.1
Temperature T
Label Smoothing 8
Focal Loss
0.5, {0.1, 0.,, 1,200,
Random PTLoss Ej
11, 10
Random PTLosS 7
1,
Table 3: The search range of parrameaarasstrs..
A.2
Running time For each perturbation order, randomly sample 100 coefficients from [-1, Then we compute and choose the best scores according to Eq. 21 on the dev dataset with 1000 examples. The whole process takes less than two minutes CPU with 64G memory.
A.3 PERTURBATION COFFFICIENT
Figure shows parameter study of the perturbation coefficients and 7. We adopt 1-order PTLoss and search e and 7 in [-1, 5] to show how the perturbation extent determined by the perturbation coefficients.
11

---

Under review as a conference paper at ICLR 2023
3-
BL
113
0.30gm10
06
04
Cptnee
0.2
025
033
035
020 375
150 015
Teacher
00
prunn mi
Eay
o-2 pmid
g-
0-4.30 g-2
Expo
E7.p1..0 67
13 57
o-3 --32226--22
p-4-22g-222
00
0:2
200
015
1000
3.25
075
100
300
055
2:00
035
2000
0.00
025
0.00
375
2.0
8.00
322
350
035
Teacher postice
Sacher
Teacher toe 35b
Teacher painuee prob
Figure 3: Parameter study of PTLoss We fix the perturbation order and varies the coefficients.
A.4 PROOF OF PROPOSITION I
Proof. By Eqs. and 15
R(p',, R(( = Σ ((((E))
1 N
Σ p'(xn) log(p"(xx)) - E. (1) ((f()))
+
RE[N]
(28)
Let
A
Σ (((())) E (=)
DE(N]
and
H
Σ log("))) N
RE[N]
then
E (R(p', f:8) - R(F)))]
= [A H)
2E A 2E [H]]]
2V | + 2 | + 2E [
where the second line is by the inequality (a + 6)2 < 2a² 26 and linearity of expectation, and the third line is by E [
+ E | Observe that V[A]
V
E [A] = E, [[' (z) p" (()))((((()
< E, [|l'' - (=(=))22
< E, [ll'(x) - -x)(=))2- (I
s E, [>) (((e)))p((x))l) (x)))))
12

---

Under review as a conference paper at ICLR 2023
where the second line is by the Cauchy-Schwartz Schwartz inequality, the third line by the equivalence of norms with constant C1, and the last line is by the boundedness of loss term.
Since R(f) is constant,
V [ - v Rtp'' f: N V[D())"(((<>)
By plugging in everything above, we finish the proof.
13


Reviews:
[{"review_id": "rwrCT4arhUD", "reviewer": "Reviewer_yv5C", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a KD algorithm that can improve the generalizability of students through perturbation of the teacher's output distribution. The PTLoss proposed in this paper is obtained by expressing a KL-based loss function through a Maclaurin series and then perturbing the terms of the preceding order in this series. The paper shows theoretically and experimentally that this new loss function can show higher KD performance than other competitors.", "summary_of_the_review": "Overall, I read this paper interesting and lean to positive. Additional experiments will help to confirm the effectiveness of the proposed algorithm.", "strength_and_weaknesses": "The proposed algorithm is systematic, well-organized, and has a solid theoretical background. This paper goes beyond simply pertuㄷrbating the distribution of teachers and proposes the best perturbation strategy, and several results presented in the experimental results demonstrate that this strategy is effective.\n\nDespite the advantages mentioned above, I believe the experiment was conducted only in a relatively narrow space. Empirical validation for more diverse scenarios will be more helpful in demonstrating the effectiveness of the algorithm.\nFirst of all, how does this algorithm work with students of larger capacity? Some KD algorithms test not only if a student has a lower capacity than the teacher but also if they have the same or greater capacity.\nNext, how does the proposed algorithm work if the teacher is not a single network but an ensemble of multiple networks?\nFinally, I wonder how the proposed algorithm can work for images, which is another big area where KD is applied.\n\n\n---\nAfter reading the reviews of other PCs and the authors' replies, \n\nI lower my primary score to 6.\nI still think this paper is well written and has enough contributions to be accepted, but as reviewer BGUG points out, this paper does not sufficiently cover state-of-the-art KD algorithms. For example, it would have been nice to have compared more recent methods in Table 5.1, as well.\nAlso, the application of the proposed algorithm is limited at this point. As the authors also mentioned, it would be good to explore computer vision further, the effectiveness of the proposed PTLoss according to the relationship between student and teacher capacity, etc., in future work.\n\n---", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "4N9Htp8L86", "reviewer": "Reviewer_Djov", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "KL divergence between the teacher outputs and the student outputs is commonly used for knowledge distillation. However, this may not be optimal. The authors propose a method to perturb the KL divergence by modifying terms in its Maclaurin expansion. This implicitly modifies the teacher outputs to form a \"Proxy Teacher\". The authors claim and derive a proof that the proxy teacher has the ability to reduce the gap between the true risk and the distillation risk. Connection to Label Smoothing (LS) is also derived. Experiments on multiple NLP datasets show improved performance from the proposed PTLoss over the usual KL loss as well as related methods like LS.", "summary_of_the_review": "I think the authors are tackling an important problem, and they have an interesting idea to improve the usual KL loss for knowledge distillation. However, I don't think the theoretical analysis is sound and I believe the experiments needs to be done better as well. Given these concerns, I am recommending rejection. \n\nI hope the authors can respond to my review in case I have misunderstood some parts of the paper. \n\nUPDATE AFTER AUTHORS' RESPONSE:\n\nThe authors use a KD setup which is different from the usual setup of having access to both ground-truth labels and teacher outputs -- only the teacher outputs are used for KD in this paper. I think the theoretical analysis makes more sense to me now. However, having experiments which also use the cross entropy loss assuming known ground-truth can help make the paper more broadly useful. And also including an experiment on CIFAR-100 and/or Tiny ImageNet to make it easier to put this work in context with the dozens of existing KD works. I do appreciate the results on the toy dataset which seems to agree with their theoretical analysis. Given these improvements to the paper, I am increasing my rating to a 6. ", "strength_and_weaknesses": "Strengths:\n\n1. The authors consider an interesting modification to the usual KL objective by perturbing the leading terms in its Maclaurin expansion for knowledge distillation.\n\n2. They show how this perturbation subsumes label smoothing as a special case.\n \n3. Experimental results on multiple NLP datasets seem to show that the proposed objective outperforms the usual KL loss, LS loss and others. \n\nWeaknesses and suggestions for improvement:\n\nI think there are quite a few weaknesses and things that are unclear in this version of the paper, I list them below:\n\n1. Based on my reading of the paper, it appears that only the KL loss, LS loss or the proposed PTLoss is used for knowledge distillation. It is my understanding that a combination of cross-entropy loss and a version of the KL loss is used usually. I am wondering if the authors missed mentioning this in the paper or if they have a different setup.\n\n2. Definition of true risk $R(\\mathbf{f})$ is defined differently in Eq. (1) and in Eq. (28) in the supplement. I think Eq. (28) makes more sense in that there is a true distribution $\\mathbf{p}^*(x)$ that is inaccessible and we are trying to approximate with the PTLoss and the proxy teacher. If $\\mathbf{y}$ (the one-hot labels) were the true distribution, then we would use it directly.\n\n3. It is not at all clear from the paper as to how the best Proxy Teacher is found in Eq. (19) and how that relates to Eq. (21) to find the best $\\eta$. For Eq. (19), the authors just say that it is solved numerically, perhaps they should elaborate. My guess is that random $\\eta$'s are sampled, and for each $\\eta$, $\\mathbf{p}^{t_{PT}}$ is found numerically and the corresponding ${\\hat{D}}$ in Eq. (21) is computed. Finally, the $\\eta$ that corresponds to the $\\mathbf{p}^{t_{PT}}$ that results in the smallest ${\\hat{D}}$ is selected. The authors should clarify whether this is the procedure, and should make this clear in the paper. \n\n4. There are parts of in Section 4.1.2 and 4.1.3 that are seem to be wrong or assumptions that are not justified. The authors write towards the end of 4.1.2 that \"Assume the student functional space is complicated enough, then we can replace the $\\mathbf{f}$ by $\\mathbf{f}^{t_{PT}}(x)$. I don't understand what this means or how we can do this.\n\nIn Section 4.1.3, the authors write: \"We also replace $\\mathbf{p}^*(x_n)$ with $\\mathbf{y}_n$.\" This does not make a lot of sense to me. If the whole point is to get closer to $\\mathbf{p}^*(x)$, why replace it with one-hot labels. If one-hot labels were good, using just the cross-entropy loss should be fine right? l think this is the main problem with the theoretical analysis.\n\n5. All the experiments are done with real-world datasets with unknown $\\mathbf{p}^*(x)$. What the paper needs is an illustrative experiment with a toy dataset where the $\\mathbf{p}^*(x)$ is known exactly, so that the authors can show that the proposed PTLoss gets closer to the desired output compared to one-hot labels, KL loss and LS loss. I would suggest the authors look at Section 2.2 in this recent paper which does exactly this for essentially the same problem:\n\nRin et al. BETTER SUPERVISORY SIGNALS BY OBSERVING LEARNING PATHS, ICLR 2022\n\nI think this is an important and necessary experiment\n\n6. All the theoretical analysis presented is for binary classification. Can the authors provide details how to extend these ideas for multi-class problems?\n\n7. Experimental results should be reported over multiple runs, with means, standard deviations and statistical significance results. Otherwise, it is not possible to determine whether the performance improvements exist.\n\n8. I would suggest the authors discuss the FilterKD that I mentioned above. I think the ideas are very closely related. Ideally, experimental comparisons should be provided.\n\n9. Similarly, I would like the authors to present some comparisons against SOTA KD methods such as Contrastive Representation Distillation which I believe would help readers a lot, but this is not required.\n\n10. I am not very familiar with the datasets used in the paper. It would help me a lot and many readers I think if the authors could consider an image recognition dataset such as CIFAR-100 or ImageNet-100. Again, this is just a suggestion, not too important.\n\n11. This is minor, but using $[0,1]^2$ as the support for a probability distribution is not perfect, as not all pairs of numbers in the $[0,1]$ interval would form a probability vector.\n\n12. I think it should be $\\mathbf{y}^T log(Z(f(x)))$ in Eq. (2) and there should be a $\\mathbf{y}^T$ multiplying the RHS is Eq. (4) right? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2lY3TR0z3X", "reviewer": "Reviewer_BGUG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Summary.\n\nThis paper is dedicated to investigating knowledge distillation. The authors argue that the vanilla learning objective of knowledge distillation is sub-optimal since there exists a discrepancy between the teacher's output distribution and the ground truth label distribution. Therefore, they introduce the PTLoss with a Maclaurin series and perturbed leading-order terms. Both theoretical and empirical results are conducted.", "summary_of_the_review": "A borderline paper needs more studies.", "strength_and_weaknesses": "Pros.\n\n1. Both empirical and theoretical analyses are provided.\n2. The paper is well-written and easy to follow.\n\n\nCons.\n\n1. Related works are outdated and not sufficient. Take knowledge distillation as an example. There are tens of KD papers from 2021 and 2022, but none of them are cited.\n2. Only limited datasets are considered. The paper's proposals seem to be general methods, which should be validated in diverse representative tasks (e.g., ImageNet classification, detection, segmentation, and GLUE NLP benchmarks) to support their effectiveness.\n3. The improvements are marginal. Need multiple runs and an error bar to show the significance of benefits.\n4. Missing comparisons with important literature like \"Revisiting Knowledge Distillation via Label Smoothing Regularization\" and \"Do We Need Zero Training Loss After Achieving Zero Training Error?\".\n5. The authors argue the limitation of a discrepancy between the teacher's output distribution and the ground truth label distribution. It is hard to say the discrepancy is a bad thing. Because lots of input samples like ImageNet images in vision can be multi-label but the ground truth is usually a one-hot label. Thus, a soft label is actually more appropriate to represent the true data distribution, ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject