Knowledge Distillation as Semiparametric Inference
Authors: Tri Dao
Govinda M Kamath
Vasilis Syrgkanis
Lester Mackey
Venue: ICLR 2021 Poster (2021)
Keywords: knowledge distillation
semiparametric inference
generalization bounds
model compression
cross-fitting
orthogonal machine learning
loss correction

Abstract: A popular approach to model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, this two-step knowledge distillation process often leads to higher accuracy than training the student directly on labeled data. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabilities as nuisance, and the teacher probabilities as a plug-in nuisance estimate.  By adapting modern semiparametric tools, we derive new guarantees for the prediction error of standard distillation and develop two enhancements—cross-fitting and loss correction—to mitigate the impact of teacher overfitting and underfitting on student performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.

Published as a conference paper at ICLR 2021
KNOWLEDGE DISTILLATION AS SEMIPARAMETRIC INFERENCE
Tri Dao', Govinda M. Kamath", Vasilis Syrgkanis Lester Mackey² 1 Department of Computer Science, Stanford University 2 Microsoft Research, New England t bridastanford. edu, (govinda aeooinda.kamathll h, vasy, /macker/@ndcrrooot..ccm
ABSTRACT
A popular approach model compression is to train an inexpensive student model to mimic the class probabilities of a highly accurate but cumbersome teacher model. Surprisingly, this two-step knowledge distillation process often leads higher ac- curacy than training the student directly on labeled data. To explain and enhance this phenomenon, we cast knowledge distillation as a semiparametric inference problem with the optimal student model as the target, the unknown Bayes class probabili- ties as nuisance, and the teacher probabilities as a plug-in nuisance estimate. By guarantees for the prediction adapting modern semiparametrie tools, we derive cenerceeeetttcoosss-tte cross-fitting and loss error of standard distillation and develop two enhancements correction mitigate the impact of teacher overfitting and underfitting performance. We validate our findings empirically on both tabular and image data and observe consistent improvements from our knowledge distillation enhancements.
student
INTRODUCTION
Knowledge distillation (KD) (Craven & Shavlik, 1996; Breiman & Shang, 1996; Bucila et al., 2006; Li al., 2014; Ba & Caruana, 2014; Hinton al., 2015) is a widely used model compression technique that enables the deployment of highly accurate predictive models on devices such as phones, watches, and virtual assistants (Stock et al., 2020). KD operates by training a compressed student model to mimic the predicted class probabilities of an expensive, high-quality teacher model, Remarkably and across a wide variety of domains (Hinton et al., 2015; Sanh al., 2019; Jiao al., 2019; Liu et al., 2018; Tan et al., 2018 18; Fakoor et al., 2020), this two-step process often leads to higher accuracy than training the student directly on the raw labeled dataset.
While the practice of KD is now well developed. general theoretical understanding of its successes and failures is still lacking. As we detail below, a number of authors have argued that the success of KD lies in the more precise "soft labels" provided by the teacher's 's predicted class probabilities. Recently, Menon et al. (2020) observed that these teacher probabilities can serve as a proxy for the Bayes probabilities (i.e., the true class probabilities) and that the closer the teacher and Bayes probabilities, the better the student's 's performance should be.
Building on this observation, cast KD as plug-in approach semiparametric inference (Kosorok, 2007): that is, we view KD as fitting student model î in the presence of nuisance (the Bayes probabilities Po) with the teacher's probabilities ý as a plug-in estimate of Po- This insight allows us to adapt modern tools from semiparametric inference analyze the error of a distilled student in Sec. 3. Our analysis also reveals two distinct failure modes of KD: one due to teacher overfitting and data reuse and the other due to teacher underfitting from model misspecification or insufficient training. In Sec. 4, we introduce and analyze two complementary KD enhancements that correct for these failures: cress-fitting- popular technique from semiparametric inference (see, e.g., Chernozhukov et al., teacher underfitting by reducing the bias of the plug-in estimate p. The latter enhancement was inspired by the orthogonal machine learning (Chermozhukov et al., 2018; Foster & Syrgkanis, 2019) approach to semiparametric inference w which suggests a particular adjustment for the teacher's 's log probabilities. We argue in Sec. 4 that this orthogonal correction minimizes the teacher bias but often at the cost of
teacher overfitting through data partitioning while loss correction mitigates

---

Published as a conference paper at ICLR 2021
unacceptably large variance. Our proposed correction avoids this variance explosion by balancing the bias and variance terms in our generalization bounds.
In Sec. 5, we complement our theoretical analysis with pair of experiments demonstrating the value of our enhancements on six real classification problems. On five real tabular datascts, gross-fittingg and loss correction improve student performance by up to 4% AUC over vanilla KD. Furthermore, on CIFAR-10 (Krizhevsky & Hinton, 2009), a benchmark image classification dataset, our enhancements improve vanilla KD accuracy by up 1.5% when the teacher model overfits.
Related work. Since we cannot review the vast literature on KD in its entirety, we point the interested reader to Gou et al. (2020) for recent overview of the field. We devote this section to reviewing theoretical advances in the understanding of KD and summarize complementary empirical studies and applications of in the extended literature review in App. A.
A number of papers have argued that the availability of soft class probabilities from the teacher rather than hard labels enables us to improve training of the student model. This was hypothesized in Hinton et al. (2015) with empirical justification. Phuong & Lampert (2019) consider the case in which the teacher is a fixed linear classifier and the student is either a linear model or deep linear network. They show that the student can learn the teacher perfectly if the number of training examples exceeds the ambient dimension. Vapnik & Izmailov (2015) discuss the setting of learning with privileged information where one has additional information training time which is not available at test time. Lopez-Paz et al. (2015) draw a connection between this and KD, arguing that KD is effective because the teacher learns better representation allowing the student to learn a faster rate. They hypothesize that a teacher's 's class probabilities enable student improvement by indicating how difficult each point classify. Tang et al. (2020) argue using empirical evidence that label smoothing and reweighting of training examples using the teacher's s predictions are key to the success of KD. Mobahi et al. (2020) analyzed the case of self-distillation in which the student and teacher function classes are identical. Focusing on kernel ridge regression models, they proved that self-distillation can act as increased regularization strength. Bu et al. (2020) considers more generic model compression in a rate-distortion framework, where the rate is the size of the student model and distortion is the difference in excess risk between the teacher and the student. Menon et al. (2020) consider the case of losses such that the population risk is linear in the Bayes class probabilities. They consider distilled empirical risk and Bayes distilled empirical risk which are the risk computed using the teacher class probabilities and Bayes class probabilities respectively rather than the observed label. They show that the variance of the Bayes distilled empirical risk is lower than the empirical risk. Then using analysis from Maurer & Pontil (2009); Bennett (1962), they derive the excess risk of the distilled empirical risk function of the f2 distance between the teacher's is class probabilities and the Bayes class probabilities. We significantly depart from Menon et al. (2020) in multiple ways: i) our Thm. allows for the common practice of data re-use, ii) our results cover the standard KD losses SEL and ACE which are non-linear in po, iii) we use localized Rademacher analysis to achieve tight fast rates for standard KD losses, and iv) we use techniques from semiparametric inference to improve upon vanilla KD.
2 KNOWLEDGE DISTILLATION BACKGROUND
We consider a multiclass classification problem with is classes and training datapoints 2; = (zx;,;) sampled independently from some distribution P. Each feature vector x belongs to a set X', each label vector y E {ei,,..,E*} C (0.1) } is one-hot encoding of the class label, and the conditional probability of observing each label is the Bayes class probability function po(z) = E[Y |X = =r). Our aim is to identify a scoring rule f: X + IR that minimizes a prediction loss on average under the distribution P.
Knowledge distillation. Knowledge distillation (KD) is a two-step training process where one first uses a labeled dataset to train a teacher model and then trains a student model to predict the teacher's predicted class probabilities Typically the teacher model is larger and more cumbersome, while the student is smaller and more efficient. Knowledge distillation was first motivated by model compression (Bucila et al., 2006). to find compact yet serhoheerfrminggggg models to be deployed (such as mobile devices).
In training the student to match the teacher's prediction probability, there are several types of loss functions that are commonly used. Let p(x)) E IR* be the teacher's vector of predicted class probabilities, f() R be the student model's output, and [k] A 1,, The most popular distillation loss
2

---

Published as a conference paper at ICLR 2021
functions' ((:f(()) include the squared error logit (SEL) loss (Ba & Caruana, 2014)
(x)x:(x)), -
(x)))
(SEL)
and the annealed cross-entropy (ACE) loss (Hinton et al., 2015)
fz(z:f(z) 5 Eiel Pe(z)s log
(x)
exp(Sf, (x))
(ACE)
exp(f
for an inverse temperature 3 > 0. These loss functions measure the divergence between the probabilities predicted by the teacher and the student.
A student model trained with knowledge distillation often performs better than the same model trained from scratch (Bucila et al., 2006; Hinton et al., 2015). . Secs. 3 and 4, we will adapt modern tools from semiparametric inference to understand and enhance this phenomenon.
DISTILLATION AS SEMIPARAMETRE INFERENCE
In semiparametric inference (Kosorok, 2007), one aims to estimate a target parameter or function fo, but that estimation depends on an auxiliary nuisance muisanceffuctionn function po that is unknown and not of primary interest.
We cast the knowledge distillation process semiparametric inference problem, by treating the unknown Bayes class probabilities Po as nuisance and the teacher's predicted probabilities as a plug-in estimate of that nuisance. This perspective allows bound the generalization of the student in terms of the mean squared error (MSE) between the teacher and the Bayes probabilities. In the next section (Sec. 4) we use techniques from semiparametric inference to enhance the performance of the student. The interested reader could consult Tsiatis (2007) for more details on semiparametric inference.
Our analysis starts from taking the following perspective on distillation. For given pointwise loss function f (P))P(()z) (z)), we view the goal of the student as minimizing an oracle population loss over function class F, Lp
(o()) with fo a Je
The main hurdle is that this is objective depends on the unknown Bayes probabilities pv- We view the teacher's ss model p as approximate version of Po and bound the distillation error of the student as function of the teacher's estimation error.
Typical semiparametric inference considers cases where fo is finite dimensional parameter, however recent work of Foster & Syrgkanis (2019) extends this framework to infinite dimensional models fo and to develop statistical learning theory with a nuisance component framework. The distillation problem fits exactly into this setup.
Bounds on vanilla KD As first step we derive a vanilla bound on the error of the distilled student model without any further modifications of the distillation process, i.e., assume that the student is trained on the same data as the teacher and is trained by running empirical risk mininization (ERM) on the plug-in loss, plugging in the teacher's 's model instead of po. i.e.,
f argmin J (.. for [,((fp) & E, (x)
(Vanilla KD)
where E., [X] = n E X, denotes the empirical expectation of random variable.
Technical definitions Before presenting our main theorem we introduce some technical notation. For a vector valued function f that takes input random variable X, use the shorthand notation Il/l] A [II)
=I [I] [[I](()) (x) [[g] 1/q Let V and V, denote the partial derivatives of ((236,=). with respect to its second and third input correspondingly and Vex the Jacobian of cross partial
derivatives, i.e.. ( Do,dm (236,5)) Finally, let ((x)((())X))))))))))))))))))))))
a²
and (1)p())= (app(r))EEruamme -Er (4... (FFF--omn(o..)1///.0000000000000 (x))
Critical radius Finally, need define the notion of the critical radius (see, e.g., Wainwright (2019, 14.1.1)) of a function class, which typically provides tight learning rates for statistical learning theory tasks. For any function class FF we define the localized Rademacher complexity as:
R(8;F)= - Ex...... (sup)e ferr: fllasá (a:r::/llaaa3
(Xx,) {
where e, are i.i.d. random variables taking values equiprobably in [-1,1]. The critical radius of class F. taking values in H. HJ, the smallest positive solution 5,, to the inequality R(5;F) -
'These loss functions do not depend on the ground-truth label y. but we use the augmented notation ((=:(()) (x)) to accomooate the enhanced distillation losses presented in Sec. 4.
3

---

Published as a conference paper at ICLR 2021
Theorem 1 (Vanilla KD analysis). Suppose fu belongs a convex set F satisfying the l ratio condition sup fo fff C and that the teacher estimates p E P from the same dataset used to train the student. Let 8n, /oster/() for universal constants Co,01 and on n
an upper bound on the critical radius of the function class
g {2 - (r((((; (((((=)())))())))))))))))))))))) -ltl::()))(((fffff) f(=:f(())))))))))))))))))))))( (=)))(()))))))-))))))))))))))))
Let p(z) = /lp ((2:0)((x))) and assume that the loss ((2:0.T) is o-strongly convex in & for
each z and that each g G is uniformly bounded in [- H, H. Then the Vanilla KD satisfies
|| -6 C C² H= + $10 (fl-o))(pplllll11lll (2,2) with probability at least 1-Ç.
Thm. 1, proved in App. C, shows that vanilla distillation yields an accurate student whenever the teacher generalizes well (i.e.. Ilo-poll2.2 is small) and the student and teacher model classes F and P are not too complex. The l2/ L4 ratio requirement can be removed at the expense of replacing || 4 by || [u(2) in the final bound. Moreover, we highlight that the strong convexity requirement for & is satisfied by all standard distillation objectives including SEL and ACE, as it is strong convexity with respect to the output of I and not the parameters of f. Even this requirement could removed, but this would yield slow rate bounds of the form: || - fo 12,2 = 0 + In (f- po) 102
Failure modes of vanilla KD Thm. I also hints at two distinct ways in which vanilla distillation could fail. First, since the student only learns from the teacher and does not have access to the original labels, we would expect the student to be erroneous when the teacher probabilities are inaccurate due to model misspecification, an overly restrictive teacher function class, or insufficient training. Prop. 2, proved in App. D, confirms that, in the worst case, student error suffers from inaccuracy due to this teacher underfitting even when both the student and teacher belong to low complexity model classes.
Proposition 2 (Impact of teacher underfitting on vanilla KD). There exists a classification problem in which the following properties all hold simultaneously with high probability for fo = log(Po):
The teacher learns f(z) = m([+x] Bi=1b all x E X via ridge regression with X 0=
Vanillu KD with SEL loss and constant f satisfies f So l 2 11 po ( po) 52( (mn
matching the dependence of the Thm. upper bound up to constant factor.
Enhanced KD with SEL loss, g(0) =diag( 5 ), and constant f satisfies I| So 0
Second, the critical radius in Thm. I depends on the complexity of the teacher model class P. If has a large critical radius, then the student error bound suffers due to potential teacher overfitting even if the teacher generalizes well. Prop. 3, proved in App. E, shows that, in the worst case, this teacher overfitting penalty is unavoidable and does fact lead increased student error. This occurs as the student only has access to the teacher's 's training set probabilities which, due to overfitting, need not reflect its test set probabilities.
Proposition 3 (Impact of teacher overfitting on vanilla KD). There exists a classification problem in which the following properties all hold simultaneously with high probability for = (ooo=llloppp))
The critical radius ön of the teacher-student function class in Thm. 1 is a non-vanishing constant, due to the complexity of the teacher's 's function class.
The Vanilla KD error [ Jo 11.222 for constant f with SEL loss is lower bounded by a non-vaiishing
constant, mutching the 5n dependence of the Thm. I upper bound up to a constant factor.
Enhanced KD with SEL loss, (0)) = O, and constant satisfies ||f fo 12,2 =O 44/(44d))
These examples serve to lower bound student performance in the worst case by the teacher's ss critical radius and class probability MSE, matching the upper bounds given in Thm. 1. However, we note that in other better-case scenarios vanilla distillation can perform better than the upper-bounding Thm. I would imply. In the next section, we adapt and generalize techniques from semiparametric inference to mitigate the effects of teacher overfitting and underfitting in all cases.
4 ENHANCING KNOWLEDGE DISTILLATION
To address the two distinct inefficiencies of vanilla distillation revealed in Sec. 3, we will adapt and gen- eralize two distinct techniques from semiparametric inference: orthogonal correction and cross-fitting.
4

---

Published as a conference paper at ICLR 2021
4.1 COMBATING TEACHER UNDERHITTING WITH LOSS CORRECTION
We can view the plug-in in distillation loss f() (()))(())))))))))))))))))))))()) as a zeroth order Taylor approximation to the ideal loss f (z) pp((z)) around p. An ideal first-order approximation would take the form
{(z:f(x) ()xz)))+( - (=((::((=))(((a)
However, its computation also requires knowledge of po- Nevertheless, since po(x) X == we can always construct an unbiased estimate of the ideal first order term by replacing po with y: fortho (z) (()))))((()(((()( (x); (()))) (()) EVV, <( ([(=;((() (x). [=])
(1)
For standard distillation base losses like SEL and ACE, the orthogonal loss (1) has an especially simple form, as <= f of Bregman divergence losses.
.(ar)) linear in f. Indeed, this true more generally for the following class
Definition (Bregman divergence losses). Any Bregman divergence loss function of the form has V (()) const (2)
with the second lerm bilinear (x) and For the SEL loss, v((s) - 11 g(p) = log(p).
and the correction matrix V v)g(p())) "( diag( 1 Similarly, the ACE loss falls into the class of Bregman divergence losses.
We will show that orthogonal correction (1) can significantly improve student bias due to teacher underfitting; however, for standard distillation losses (SEL and ACE), the same orthogonal correction term often introduces unreasonably large variance due to division by small probabilities appearing in the correction matrix (see Definition 1). To grant ourselves more flexibility in balancing bias and variance, we propose and analyze a family of r-corrected losses, parameterized by matrix valued function y -I R I
(zr,)))) = ((=:f(x).
(v-p(=))T (())))
to mimic the bilinear structure of Bregman orthogomal losses (2). Note that we can always recover the vanilla distillation loss by taking y ==0. O. We denote the associated population and empirical risks by Lp(f A (2 (X),)(()))
and Ln A E., [e, (X: Observe that at Pu the correction term is mean-zert and hence (f po: is independent of
(x))
for all ?
The 7-ooreccedd loss has strong connections to the literature on Neyman orthogonality (Chernozhukov et al., 2018; Chernozhukov et al., 2016; Nekipelov et al., 2018; Chernozhukov et al., 2018; Foster & Syrgkanis, 2019). In particular, if the function 7 is set appropriately, then one can show that the y-corrected loss function satisfies the condition of a Neyman orthogonal loss defined by Foster & Syrgkanis (2019). We begin our analysis by showing general lemma for any estimator f. which adapts the main theorem of Foster & Syrgkanis (2019) to account for approximate orthogonality; the proof can be found in App. F.
analysis). Consider any estimation algorithm that produces Lemma 4 (Algorithm-agnnose estimate f with small plug-in in excess risk, i.e.,
- (Df(.))) ( sc(f.p))
If the loss Ln is a-strongly convex with respect to J and F is convex set, then
fl follă, <(f.p.))+ 21
If, in addition, sup,. V Vo (. SM then
I1( ( 22 (2 (7) (p-pp)))l 11,
Connection to Neyman orthogonality Remarkably, if we set 7 fo then the 7-corrected loss Neyman orthogonal (Foster& & Syrgkanis, 2019), and the student MSE bound depends only on the squared MSE of the teacher. Moreover, g an observable quantity for any Bregman divergence loss (Definition 1) as foo is independent of fo- However, we note that this setting of the y can lead to larger variance, i.e., the achievable excess risk can be much larger than the excess risk without the correction. For instance, in the case of the SEL loss (fo (x) = p which can be excessively large when ş is close to 0, leading to a large increase in the variance of our loss. Thus, in a departure from the standard approach in semiparametric inference, we will be choosing 7 in practice to balance bias and variance.
5

---

Published as a conference paper at ICLR 2021
Example instantiation of student's estimation algorithm If we use plug-in empirical risk minimization, i.e., argmin fer T. (f :D:7), estimate fo with ý estimated sample, then the results Maurer Pontil (2009) directly imply that as long as the loss function ((2;0,T)) is uniformly bounded in [- H,, then, with probability at least 1-6,
independent
<(f =(f sup sup Var(t, (2:(() /aap Mlg((ra)/0)
(lgg(7())/8)
where (1 and No (c.F. is the so empirical covering number of function class Fin the worst-case over all realizations of m data points and at approximation level e. resule has two drawbacks: is slow rate result that scales as 1 V for parametric or bounded Vapnik-Chervonee (VC)-dimension classes, and it requires the student to be fit on completely separate dataset from the teacher's. In the next theorem, we address both of these drawbacks: i) we invoke localized Rademacher complexity analysis to provide fast rate result which would be of the order of 1/n for VC or parametric function classes, and ii) we use more sophisticated data- tatttappattiionng technique called cross- fittinn, which allows the student to be trained using all of the available teacher data.
4.2 COMBATING TEACHER OVERFITTING WITH CROSS-FITTING
We now describe more sophisticated version data partitioning to make use of all data points in our student estimation, while at the same time not suffering from the sample complexity of the teacher's function space. This approach is referred to as cross-fitting (CF) in the semiparametric inference literature (see, e.g., Chernozhukov et al. (2018)):
1. Partition the dataset into B equally sized folds PI.....PB.
2. For each fold & E [B] estimate fit)) and (()) using all the out-of-fold data points.
3. Estimate f by minimizing the empirical loss:
= #rggminnezz 1 Ei= (x:
(Enhanced KD)
In other words, the nuisance estimates (,(00,,(0) that evaluated on the data points in fold when fitting the student in step 3, are estimated only using data points outside of Pt.
Theorem 5 (Enhanced KD analysis). Suppose fo belongs to convex set F. Let
B./B.C/BB 6n/B B + coy Blog(c)B/() / on the critical radius of the class
for universal constants co, c1 and dnjB upper bound
(( (2-))===-((00022 (2))) (z))))))))))))))))))))))) FfFr
for each E [B]. Let p(2) sup V (z:f p00 (z)) and assume that, with probability 1 for each t E [B], the loss Estes (z;0, (x)) is o-strongly convex in for each 2 and each g E G(i g(01) is uniformly bounded in - H. H). Moreover, suppose that the function class F satisfies the fg/t, ratio condition: sup folla.2
folla.4 <C. If the output of Enhanced KD, then,
with probability at least 1-5,
-
= -
( / C2 H22 (1u111++ (
+ -X)
+ +
(((1))(((()))))))) po) 12
The proof is found in App. G. Observe that, unlike Thm. the function classes ((p(t) ((()) in the Thm. 5 do not vary the teacher's model over P but rather evaluate p at the specific out-of-fod ofld estimates ((() and only vary f FE Since in practice the teacher's model can be quite complex, removing this dependence on the sample complexity of the teacher's function space can bring immense improvement with the critical radius of (p(t), g(t)) significantly smaller than that of & from Thm. 1.
For instance, suppose that the loss function (2: (2:f. (()) is L- L-Lipschitz with respect f and that F a VC-subgraph class with VC dimension d.F. Then the critical radius of the function class (( (at/19,,000 ( is of order / /d)llog(n)/ /n for any choice (f(t)) (+) (see, e.g., Foster & Syrgkanis, 2019, Sec. 4.2). However, under the same conditions, the critical radius of the teacher-student function class & in
In fact, under the Lipschitz condition alone and using contraction lemma arguments as in Foster & Syrgkanis (2019, Lem. 11), one can derive version of Thm. in which the upper bound depends only the critical radius of the function class (r(f- fo): e F. F,r 1). which solely depends the function space of the student.

---

Published as a conference paper at ICLR 2021
Thm. 1 will still depend on the teacher's function space. If is also VC-subgraph class with VC dimension dp >> dy, then the critical radius of & will be of the much larger order ddplog(n)/n.
We can also see in the bound of Thm. 5 the interplay between bias and variance introduced by y. In particular, the part of the bound that depends on 5:(0) can be further simplified as
/EJ0
n.C " (= TT ((0) (x) 112 (x))
-oo (X)
(3)
where the terms respectively encode the increase in variance and decrease in bias from employing loss correction. Notably, Thm. implies that CF without y-correction (i.e., 4000 0) sufficient
reduce student error due to teacher overfitting but may still be susceptible excessive student error due to teacher underfitting. These qualitative predictions accord with our experimental observations in Sec. and Fig. 5.
4.3 BIASED STOCHASTIC GRADIENT DESCENT ANALYSIS
When the set of candidate prediction rules fo is parameterized by a vector E ERd, we may alternatively fit 0 via stochastic gradient descent (SGD) (Robbins & Monro, 1951; Bottou & Bousquet, 2008) on the 7-corrected objective (n(J..).)) With a minibatch size I and a starting point Oo the parameter updates take the form
(4) Ideally, these updates would converge to minimizer of the ideal risk L(0:po) = LD (fo.po) Our next result shows that, the teacher ý is independent of (Wo) then the SGD updates (4) have excess ideal risk governed by a bias term ((f) and a variance term (5)) /n. Here, (0) represents directly the ideal
01+1 = 02 m (X,) V Vol, (W)))(X))))
for [n] [n].
the baseline stochastic gradient variance that would be incurred if SGD
risk C(O:po) rather than our surrogate risk. Our proof in App. H builds upon the biased SGD bounds of Ajalloeian & Stich (2020).
Theorem (Biased SGD analysis). Suppose that the loss ((0:po) is X-strongly smooth in 0. Define the bias and root-variance parameters
<(9) supser= V (Ti.. (ona--)
4) (p-(()))) po) 11,22
(9) =s Elll (X) -po(( (1)0 (p- (p- 11
for (0) & Eiel VarlV ((W: fo(X), Po(X)) the unbiased SGD variance. If
=o= L(oopp) [(0:pe then the iterates (0e Vi- ofe loss corrected algorithm satisfy minker B Ve
o( oltiyAr (G)) -
vn
If. in addition, [(0:p)) is p convex in 0, then the iterates satisfy (F)
-
O a(5)²
Similar to Thm. 5, the bound in Thm. 6. portrays the interplay of bias and variance as & ranges from to f (recall that ff is independent of fo for any Bregman loss). In particular, the part of the bound for strongly convex losses that depends on f can be further simplified to:
(5) This has a very intuitive form: the first term the impact of (X) on the variance, which is also related to the square of the noise of y, divided by the standard error scaling. The second controls how
E X
(1 (x)) -XX) (6(X)
(x)) ( (f
pa
f improves the bias introduced by the error in the teacher's p.
EXPERIMENTS
We complement our theoretical analysis with a pair of experiments demonstrating the practical benefits of cross-fitting and loss correction on six real-world classification tasks. Throughout, we use the SEL loss and report mean performance + 1 standard error across 5 independent runs. Code to replicate all experiments can be found at
https: / htt///gttttttt/te///////et cwm/mi wrwso http://www..tttteteeeeeeeeteette
and supplementary experimental details and results can be found in App. I.
7

---

Published as a conference paper at ICLR 2021
Selecting the loss correction matrix y Motivated by the analyses in Sec. 4, for each training point (z.y). we will select our correction matrix f(x) to balance bias and variance by minimizing a pointwise upper bound on the loss correction error (5) (ideally with closed-form solution to avoid excessive computational overhead). 'o eliminate dependence on the unobserved po, we observe that the bias term (I)) - = (f(z)=
(()))())) up to additive terms independent of 7. We introduce tunable hyperparameter a > to trade off between this bias bound and the variance term (5) and select (xx) ((a))=dia((v(()))) to minimize:
- po (x)) = ((19)..p11))
(v) ((x))) zz] +
x +1 p(z) -
Since the conditional expectation involves the unknown quantity Po. we estimate []]l(())( - ((=)) x]| with its sample [[v( ((.) This objective is quadratic in v(z) and thus has a closed-form solution. Given (2)), the student's loss-corrected objective is equivalent to a square loss with labels ((()))((((())))))) (x) (u (V-p(x)).
Tabular data. We first validate our KD enhancements on five real- world tabular datasets- FICO (FIC), StumbleUpon (Eve; Liu et al.. 2017), and Adult, Higgs, and MAGIC from Dheeru & Karra Taniskidou (2017) with random forest (Breiman, 2001) students and teachers. In Fig. 1a, examine the impact of varying student model capacity for a fixed high-capacity teacher with 500 trees on FICO. This setting lends itself to teacher overfitting, and find that cross-fitting consistently improves upon vanilla KD by to AUC percentage points. In Fig. lb we explore the impact of teacher underfitting by limiting the teacher's maximum tree depth on Adult. Here we observe consistent gains from loss correction with student performance exceeding even that of the teacher for smaller maximum tree depths. Analogous results for the remaining datasets can be found in App. I.1.
0.800
0.920
0.775
0.700
0.910
AUC
0.725
0.905
KD Student: 10 trees, max depth-
Student without KD
0.900
Cross-fr Student
0.700
0.865
Enhanced KD Student
0.675
Student
Teacher: 100 trees
0.600
Cross fit KD Student
Enhanced
0.885
0665
Teacher: trees
0.880
10
Student's number of trees
Teacher's max tree depth
(a) FICO dataset, when teacher overfits
(b) Adult dataset, when teacher underfits
Figure 1: For random forest students and teachers, cross-fitting improves student performance when the teacher overfits, while loss correction improves student performance when the teacher underfits.
Image data. We next validate our KD enhancements on the image classification dataset CIFAR- 10 (Krizhevsky & Hinton, 2009). We pair a residual network (ResNet-8) student with teacher networks of varying depths (ResNet-1 14200/2///0033//66 (He et al., 2016). It has been observed that larger and deeper teachers need not yield better students, as the teacher might overfit to the training set (Cho Hariharan, 2019; Müller et al., 2019). To induce this overfitting, we turn off data augmentation (random horizontal flipping and cropping). We compare students trained with Vanilla KD and Enhanced KD with and without loss correction in Fig. 2. We find that cross-fitting consistently reduces the effect of teacher overfitting with largest impact realized for the deepest models. This effect is most evident in the pooss-y test loss, where the Vanilla KD student incurs significantly larger loss than the cross-fitted student. For both accuracy and test loss, employing loss correction on top of cross-fitting provides an additional small performance boost.
Effect of the loss correction hyperparameter o. Our hyperparameter a controls the tradeoff' between bias and variance in loss correction. When a is very small, the objective is close to the vanilla KD objective. When a is large, the objective is closer to the Neyman-orthogomm loss. In Figure 3, we show the effect of varying a, with ResNet-8 as the student and ResNet-20 as the teacher, on the CIFAR- 10 dataset. Large values of a lead to high variance and thus lower test accuracy. Intermediate values of a improves on both the Vanilla KD objective, which corresponds to a 0 and on the orthogonal objective
'Balancing the bias and variance terms (3) Thm. 5 yields similar objective. An alternative estimate that performs slightly worse (1)=))(((((1()))))
8

---

Published as a conference paper at ICLR 2021
0.000
KD Student
Cross-itt KD Studintt
Studert
Test accuracy
Teacher
86-
Crass Teacher
0.500
0.475
82-
0.450
14
20
20
56
Teacher's network depth
Teacher's network depth
Figure 2: On CIFAR-10 with ResNet students and teachers, cross-fiting reduces the effect of teacher overfitting, and correction yields additional small performance boost. Here, the loss cross-entropy.
(a = DO). The test accuracy drops sharply beyond some threshold as the variance becomes too high (due the terms (5(x) diag pi(z) t (xr
causing training become unstable.
Entaanced KD Studintt
Entaanced KD Studint
0.83
0.82
0.75
Test loss
0.81
0.70
0.80
0.65
0.77
0.76
10
10
10-7
10"
10"
10-
10"
alpha
alpha
Figure On CIFAR-I0 ResNet students teachers, large values of loss correction hyperparameter (corresponding to the orthogonal loss correction) lead to large variance and training instability, while intermediate values improve upon cross-fit fit KD without loss correction (a (0). Here, the test loss is cross-entropy.
CONCLUSION
We developed new analysis of knowledge distillation under the lens of semiparametric inference. By framing the KD process as learning with plug-in estimation in the presence of nuisance, we obtained new generalization bounds for distillation and new lower bounds highlighting the susceptibility of KD to teacher overfitting and underfittin. To address these failure modes, we introduced two complementary KD enhancements cross-fitting and loss correction-whiee -which improve student performance both in theory and in practice. Past work has shown that augmenting the student training set with synthetic data from generative model (e.g.. generative adversarial network (Liu et al., 2018 or MUNGE (Bucila et al., 2006)) often leads improved student performance. natural next step to prove analogue of Thm. 5 for synthetic augmentation to understand when this strategy successfully mitigates the impact teacher overfitting. In addition, two tantalizing open questions are, first, whether other techniques from semiparametric inference, such as targeted maximum likelihood (Van Der Laan & Rubin, 2006), can be used to improve KD performance and, second, whether a semiparametric perspective explain the surprising success of self-distillatio (Furlanello et al., 2018) and noisy student training (Xie et al., 2020) through which students routinely outperform their teachers.
REFERENCES
Stumbleupon evergreen dataset. https: //t::://wwwww www.kkaaggg...ccaaoooooooo.oooca ctt.c//ccccc//o wttpe/////o///c////eee/eeeeeeeee stumbluumblee.....
eupon. eupon.

---

Published as a conference paper at ICLR 2021
FICO:
Explanable machine learning challenge.
http:://comuniyy.fficcooooo
Ahmad Ajalloeian and Sebastian U Stich. Analysis of sgd with biased gradient estimators, arXiv preprint arXiv:2008.00035 2020.
Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? Advances neural information processing systems, pp. 2654-2662, 2014.
Mikhail Belkin, Alexander Rakhlin, and Alexandre B Tsybakov. Does data interpolation contradict statistical optimality? In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1611-1619. PMLR, 2019.
George Bennett. Probability inequalities for the sum of independent random variables. Journal of the American Statistical Association, 57(297):33-455 1962.
Sergei Bernstein. The theory of probabilities. Gastehizdat Publishing House, 1946.
Léon Bottou and Olivier Bousquet. The tradeoffs of large scale learning. Advances neural information processing systems, pp. 161-168, 2008.
Leo Breiman. Random forests. Machine learning, (2,
Leo Breiman and Nong Shang. Born again trees. University of California, Berkeley, Berkeley, CA. Technical Report, 1:2, 1996.
Yuheng Bu, Weihao Gao, Shaofeng Zou, and Venugopal V Veeravalli. Information-theored under- standing of population risk improvement with model compression. In AAAI, pp. 3300-3337, 2020.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizz Model compression. In Proceedings the the Twelfth ACM SIGKDD International Conference Knowledge Discovery and Data Mining, Philadelphia, PA, USA, August 20-23, 2006, pp. 535-541, 2006. doi: 10.1145/11.50 150402. 150464.
Yevgen Chebotar and Austin Waters. Distilling knowledge from ensembles of neural networks for speech recognition. In Interspeech, pp. 1439-3443, 2016.
Wei-Chun Chen, Chia-Cho Che Chang, and Che-Rung Lee. Knowledge distillation with feature maps for image classification. Asian Conference on Computer Vision. pp. 200-215. Springer, 2018.
Xu Cheng, Zhefan Rao, Yilan Chen, and Quanshi Zhang. Explaining knowledge distillation by quantifying the knowledge. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12925 12935, 2020.
Victor Chernozhukov, Juan Carlos Escanciano, Hidehiko Ichimura, Whitney K. Newey, and James M. Robins. Locally Robust Semiparametric Estimation. arXiv e-prints, art. arXiv:1608 T0t.::6000000333.. July 2016.
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21(1):C1-CC 2018. doi: 10.1 1/ecti 12097. URL www.oonnineiibbrrr...
com/ /os//oi///// /aa///////// j. 12097.
Victor Chermozhukov, Whitney Newey, and Rahul Singh. De- Di-BBd Machine Learning of Global and Local Parameters Using Regularized Riesz Representers. arXiv e-prints, art. February 2018.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4794-4802, 2019.
Mark Craven and Jude W Shavlik. Extracting tree-structured representations of trained networks. In Advances in neural information processing procesini systems, pp. 24 24-30, 1996.
Dua Dheeru and Efi Karra Taniskidou. http://a http:///rchiieeee i ics uci wcc..ede edu/ml.
UCI machine learning repository, 2017. URL
10

---

Published as a conference paper at ICLR 2021
Rasool Fakoor, Jonas Mueller, Nick Erickson, Pratik Chaudhari, and Alexander J Smola. Fast, accurate, and simple models for tabular data via augmented distillation. arXiv preprint arXiv:2006.14284, 2020.
Dylan Foster and Vasilis Syrgkanis. Orthogonal statistical learning. urXiv preprint 1rX:v:900.0000000 2019.
Markus Freitag, Yaser Al-Onaizan, and Baskaran Sankaran. Ensemble distillation for neural machine translation. arXiv preprini urXiv:/702.0/4 2017.
Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. arXiv preprint arXiv:/805.047700 2018.
Yotam Gil, Yoav Chai, Or Gorodissky, and Jonathan Berant. White-to-black: Efficient distillation of black-box adversarial attacks. arXiv preprint arXiv:1904.02405, 2019.
Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3996-4003, 2020.
Jianping Gou, Baosheng Yu, Stephen John Maybank, and Dacheng Tao. Knowledge distillation: A survey. arXiv preprint arXiv:2006.05555, 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv: 02531, 2015.
Minghao Hu, Yuxing Peng. Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou. Attention-guica answer distillation for machine reading comprehension. arXiv preprint 1rXiv: 076444 2018.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint 1909. 10351, 2019.
Michael R Kosorok. Introduction to empirical processes and semiparametric inference. Springer Science Business Media, 2007.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Jinyu Li, Rui Zhao, Mii-Ting Huang, and Yifan Gong. Learning small-size dnn with output-distribution based criteria. In Fifteenth annual conference of the international speech communication association, 2014.
Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In Proceedings the ieee conference on computer vision and pattern recognition, pp. 6356-6364, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions pattern analysis and machine intelligence, 40(12):2935-2947) 2017.
Ruishan Liu, Nicolo Fusi, and Lester Mackey. Teacher-student compression with generative adversarial networks. arXiv preprint arXiv: /rXiv:
Yu Liu, Hantian Zhang, Luyuan Zeng, Wentao Wu, and Ce Zhang. MLBench: How good are machine learning clouds for binary classification tasks on structured data. ArXiv e-prints, 2017.
Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint 1rxiv:171..07535, 2017.
David Lopez-Pazz Pazz Léon Bottou, Bernhard Schölkopf. and Vladimir Vapnik. Unifying distillation and privileged information. arXiv preprint arXiv:/5/1.03644 2015.

---

Published as a conference paper at ICLR 2021
Liang Lu, Michelle Guo, and Steve Renals. Knowledge distillation for small-footpring highway networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4820-4824. IEEE, 2017.
Andreas Maurer and Massimiliano Pontil. Empirical bernstein bounds and sample variance penalization. arXiv preprint urXiv:0007.37440 2009.
Aditya Krishna Menon, Ankit Singh Rawat, Sashank J Reddi, Seungyeon Kim, and Sanjiv Kumar. Why distillation helps: a statistical perspective. arXiv preprint arXiv:2005.10419 2020.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L Bartleut. Self-distillation amplifies regularization in hilbert space. arXiv preprint arXivv2002.057/5, 2020.
Lili Mou, Ran Jia, Yan Xu, Ge Li, Lu Zhang, and Jin. Distilling word embeddings: An encoding approach. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 1977-1980, 2016.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems, pp. 4694 2019.
Elizbar A Nadaraya. On estimating regression. Theory of Probability Its Applications, 9(1): 141-142, 1964.
Ndapandula Nakashole and Raphael Flauger. Knowledge distillation for bilingual dictionary induction. In Proceedings of the 2017 conference on empirical methods in natural language processing, pp.
2497 2506, 2017.
Denis Nekipelov, Vira Semenova, and Vasilis Syrgkanis. Regularized Orthogonal Machine Learning for Nonlinear Semiparametric Models. arXiv e-prints, art. arXiv: 1eXXvv11800 18061800..0 006604423, June 2018.
Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al. Parallel wavenet: Fast high-fieelity speech synthesis. In International conference on machine learning, pp. 3118-998822 PMLR, 2018.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi-supervised knowledge transfer for deep learning from private training data. arXiv preprint 11X:v:1610.05755, 2016a.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. Distillation as a defense to adversarial perturbations against deep neural networks. In 2016 IEEE Symposium on Security and Privacy (SP), pp. 582-597, IEEE, 2016b.
Mary Phuong and Christoph Lampert. Towards understanding knowledge distillation. In International Conference on Machine Learning, pp. 11142-515,,22 2019.
Herbert Robbins and Sutton Monro. A stochastic approximation method. Math. Statist., 2203):400-407 09 1951.
Ann. URL
doi: 10. N0.221//amm/1117779955
https:/ doi. /tt.o/o.oooo.ooo//// 1214/a /tt////m//////////////////// hooms/11771n 1117772588..
Andrew Slavin Ross and Finale Doshi-Velez. Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients. arXiv preprint ariv: 1.09404, 2017.
inputgradientss
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.011000 2019.
Peng Shen, Xugang Lu, Sheng Li, and Hisashi Kawai. Feature representation of short utterances based on knowledge distillation for spoken language identification. In Interspeech, pp. 1813-1817, 2018.
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. 2020.
Sarah Tan, Rich Caruana, Giles Hooker, Paul Koch, and Albert Gordo. Learning global additive explanations for neural nets using model distillation. arXiv preprint arXiv: 1800008640, 2018.
12

---

Published as a conference paper at ICLR 2021
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv preprint arXiv:2002.03552, 2020.
Anastasios Tsiatis. Semiparcmettrc theory and missing data. Springer Science & Business Media, 2007.
Mark J Van Der Laan and Daniel Rubin. Targeted maximum likelihood learning. The international journal of biostatistics, 2(1), 2006.
Vladimir Vapnik and Rauf Izmailov. Learning using privileged information: similarity control and knowledge transfer. J. Mach. Learn. Res., (1L1):2222-04999 2015.
Martin J. Wainwright. Bigh-Dimensional Statistics: A Non-Asymptotic Viewpoint. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2019. doi: 10.1017/978 108627771.
Chong Wang, Xipeng Lan, and Yangang Zhang. Model distillation with knowledge transfer from face classification to alignment and verification. arXiv preprint arXiv: 2017.
Ji Wang, Weidong Bao, Lichao Sun, Xiaomin Zhu, Bokai Cao, and S Yu Philip. Private model compression via knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 1 1190-1199 1990,-19,,,001111
Shinji Watanabe, Takaaki Hori, Jonathan Le Roux, and John Hershey. Student-teacher network learning with enhanced features. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5275-5279. IEEE, 2017.
Geoffrey S Watson. Smooth regression analysis. Sankhya: The Indian Journal of Statistics, Series A. pp. 319-3559222 1964.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020.
13

---

Published as a conference paper at ICLR 2021
A EXTENDED LITERATURE REVIEW
We point the interested reader to Gou et al. (2020) for a sweeping survey of the many developments in knowledge distillation over the past half decade. In addition to the references discussing theoretical aspects of knowledge distillation provided in Sec. 1, we highlight here a number of empirical investigations of why distillation works. Cho & Hariharan (2019) show that larger teacher models do not necessarily improve the performance of student models as parsimonious student models are not able to mimic the teacher model. They suggest early stopping in training large teacher neural networks as means of regularizing. Cheng et al. (2020) demonstrate that when applied to image data, distillation allows the student neural net to learn multiple visual concepts simultaneously, while, when learning from raw data, neural networks learn concepts sequentially.
Knowledge distillation has also been used for adversarial attacks (Papernot et al., 2016b; Ross & Doshi-Velez, 2017; Gil et al., 2019; Goldblum et al., 2020), data security (Papernot et al., 2016a; Lopes et al., 2017: Wang et al., 2019), image processing (Li & Hoiem, 2017; Wang et al., 2017: Chen et al., 2018; Li ct al., 2017), natural language processing (Nakashole & Flauger, 2017; Mou et al., 2016; Hu et al., 2018; Freitag et al., 7), and speech processing (Chebotar & Waters, 2016; Lu et al., 2017; et al., 2017; Oord et al., 201 8; Shen et al., 2018).
Watanabe
B GLOSSARY
Table I: Glossary of notation
Notation
Definition
(e:f((X)Poo(X)) Population risk (f(p))
Loss functioon on random data point f(e(Z:f(X), (x](((:[(XX)p(XX))
Empirical risk L,, (f.))
Population optimal student model fo
frgmin argmingez<L.(fp) (f.ū)
Empirical optimal student model f
[II 1/9
V
Partial derivative of (2; with respect to the second input
+
V
Partial derivative of ((210,T)) with respect to the third input 8²
V Vn
J0, (=;0,,))
(x)
Eu- (Q1. (x)
R(oFF)
Localized Rademacher complexity of function class JF
6n
Critical radius
y-corrected loss (x:=:((x)))(())
((z:f(a),p(z))) + +0y (())()))T(()))
Population 7-risk (D())))) Empirical 7-risk (I,77)
(Z
C PROOF OF THM. 1: VANILLA DISTILLATION ANALYSIS
Introduce the shorthand Rademacher analysis of Foster & Syrgkanis (2019, Lem. 11) implies
(((x))) Since 8, upper bounds the critical radius of the function class G, the localized
- (fo))
00 1no -l. H8%,)
with probability at least - C. Moreover, by Cauchy-Sewhare
- 11 folll--folla.. foll2.4-
By the assumed 62/4, ratio condition we therefore have
<(f.pr) CH) f foll,, +.)
Plugging this bound into Lemma 4 (which holds irrespective of whether data re-use, sample splitting, or cross-fitting is employed) and applying the cechmetti--gemetriiee mean inequality yields
σ |f foll2 < 0
C
8
'ee apply Foster & Syrgkanis (2019, Lem. 11) with C. for g E G with 9" =0. Then we instantiate the concentration inequality for the choice g E f f0.0 E..
14

---

Published as a conference paper at ICLR 2021
D PROOF OF PROP. 2: IMPACT OF TEACHER UNDERFITTING ON VANILLA DISTILLATION
Suppose that Po does not vary with x and, for known e >> 0. belongs the set
P= (p pz(x) E [c,1], Vx E
j [A]].
As all quantities in this proof are independent of z, we will omit the dependence on whenever convenient.
Consider the constant teacher estimate p=
# T4X
Ve obtained via ridge regression with regularization streri S 1 and 8 1 n2i=1
A constant student prediction rule in
F= F= [ Vo ex
trained via Vanilla KD with SEL loss yields f(x) log((p)
Suppose that, unbeknownst to the teacher and student, the true po satisfies the more stringent condition po.j 2 2e for all j E [k]. Then the student satisfies fo -log(p)) log log 2 diag( ! (po p) No.0o (Po -00 No
(p--((-pu) minn(, 9 c))
(1+2)
=Yoopp App 1 (o) + min (0, 8 -1ttA-- (L+A/e) )) 27
Apo 18-pol
(1+X)
1+1
(1+2)
by the concavity of the logarithm and the choice A A 1. Since
P(IØ, Po. 12 0po,))
20 for 0 2(1- log( (t) npp. log( npo.3
K
by Bernstein's 's inequality (Bernstein, 1946), we have
P( f13, Il (Po- 2) P( fo. f (ee P(Apo.s 200 Pool. vj=l) ( 1-26
whenever
V log( ()+ 4 ne log( () 1
since limsup no+00 Io (Po 100-))1B,, = (-mimmmmmmm))) (1)))) with probability 1 whenever
Moreover,
m B, Po.y
with probability 1 by the law of the iterated logarithm,
22po., (1 po., (oglog(n)
12121 2loglog(n)
The choice
ne
A A=min
1,max
Zloglog(n)
2 log(
k
+
1 me log(E)
=0(
ne
ne V
now yields the first two advertised claims.
The final claim follows directly from Thm. 5 with B = 0(1 as 9(0) 8 (lc=0((BBk/n) by Wainwright (2019, Ex. 13.8).
T and the critical radius of G(j()), ((()) satisfies
E
PROOF OF PROP. 3: IMPACT OF TEACHER OVERFITTING ON VANILLA DISTILLATION
Suppose that Po has Lipschitz gradient and, for known <> 0, belongs to the set
P= (p: py p5(x)) E [e.1), Vx EX [k]].
Suppose moreover that X e Rd has Lebesgue density bounded away from 0 and oo and that e < El (X) (ox))'
4 (X))) for each
j. Consider the teacher estimates iÚ (x) max(e. (z)) for p the Nadaraya- Watson kernel smoothing estimator (Nadaraya, 1964; Watson, 1964)
p(x) à
Vi
if x=
Σ -z 1 (((-a()))
K((x xx)//) ((<-)a)))))))))))
otherwise
with kernel (xx) = -| 2 -11, a (0.d/2), and h = n- 11(444d) By Belkin et al. (2019, Thm. 1), the teacher satisfies B[llpo - 0 (4(4+d))
Now instantiate the notation of Thm. 1, and consider a student prediction rule trained to learn a constant prediction rule via Vanilla KD with the SEL loss and
F= f())=
for all x.x' E
(6)
Since f exactly interpolates the observed labels (i.e., f(x;)) = y:), the critical radius of the teacher-student function class G satisfies 8n- 2(1). Moreover, since the student only has access the teacher's training set probabilities, its estimate fxx) = Σm log (m) inconsistent for the optimal constant rule fz) E[]op(poXX))) as
fo.z -I) (x) = E Blog (ogg)ooo((oo
(X)) log.-ooommmaat E Po. (X (11 Po., (X)) + Po, (X))
Po., (Xx) (X) max (mmax(Y,
(X)
R
1- Po. (X)
Po. (X) (X) (X))
pa. (X)
15

---

Published as a conference paper at ICLR 2021
by Taylor's theorem with Lagrange remainder. This non-vanishing student error reflects the non non-vanishing critical radius d, of the composite sudent-teacher function class 9 defined in Thm. 1; since the student function class F has low complexity, the complexity of G is driven by the highly flexible interpolating teacher.
Next, instantiate the notation of Thm. 5, and consider a student prediction rule f trained via Enhanced KD with SEL loss, 4(0) =0, B 0 and F (6). The critical radius of ((i (700) satisfies 5n/8 = ( (kkBk//n) by Wainwright (2019, Ex. 13.8). Moreover, each cross-fitted teacher satisfies Eillpo inequalities, with probability at least 1 - 2/2,
- 4/((+d)) by Belkin et al. (2019, Thm. 1), so, by Chebyshev's and Jensen's
Pu p( |2 22. - Elpp /10 /
222 (1pp0 p(t) 1l2.2)/S
<1+ /1/ Billpo p(() |2221 = -O =2((+)))
for all
Therefore, Thm. implies that
13 |lf 11f- = + - (1)) ) (6(1) - -p0))12221
=0 + トー (diag( pre (5 12)2)
=0 + 1
B Et= -po ( 4/(a+d))
with probability at least --
PROOF OF LEMMA 4: ALGORITHM-AGNOSTIC ANALYSIS
First we define for any functional L(f) the Frechet derivative as:
When L is operator of the form: (x)(J(X))), then: -11000)--(/((())))1000)
By the o-strong convexity of LD. 6 we have that
Lo(f (.)) (
"I 211f- f1fffull2.2
Furthermore, our excess risk assumption and the optimality of fo give
Ill f < LD - -
excess risk f
(a) & -
20 optimality of fo
By Taylor's theorem with integral remainder.
EK(V& hWt.C//w.ccchhhhi-o//////ddddeeh
(7)
= -(p(x))-(<)) - ((0a())(((
whenever Vorlin is well-defined. We can now invoke the (7) and Cauchy-Schwand to obtain the bound
=
E((po(X)) (XT(((())))))))))xx))XX)))))))))
< (T)I(co((X--(x))) -1
S 110o0-1 ( 12222
Thus combining all the above inequalities:
llf f ( foll2.
2
'Notably this strong convexity assumption can be relaxed to F. (
16

---

Published as a conference paper at ICLR 2021
By an AM-GM inequality, for all a.b2 0: a.b 2 gb).. Applying this the product of norms on the RHS and re-arranging yields
1 fullā < (( + 1
(1(0-p)) po) (7.6.8
To get the final inequality, observe that:
(( (1.... (ofo:-~))ll = -
Moreover, by the boundedness of the third derivative, have:
((p--p)) po)
(To..P T
21, Elllp(X) Po (x =1
- po -
M 1
Combining all the above yields the final bound.
G PROOF OF THM. 5: CROSS-FITTED ERM ANALYSIS
Let E,, denote the empirical loss over the samples in the t-th fold and ( the nuisance functions used on the samples in the k-tth fold. For any E [K] and conditional on (,0,,, suppose that 6n upper bounds the critical radius of the function class G(D(t), (,(0),,)))),,,,,,,,))) then by Lemma 11 of Foster Syrgkanis (2019), if we denote with (z,(())=(()))))))())))))))))(( = (=)) w.p. 1-Ç:
[f ", ,((0)== - e (foz) Moreover, we have that by the definition of fross-fitted ERM:
(0,50")))))))))))))) ,(0)) (I.(f "0 ,4(0)) - ID( ,(0)) 00 (H 21 H6 H6
1
Ln (f 500 (000) -00)) (0.00,,0
Ln
B
Thus we have that w.p. 1 B-CB:
1
B
(000),,000))0000)),)) LL (00)) so HönIB. E lle.s Le -1..s--
1
1
B
B
Moreover, if we let (= (==)= sup (l( (a:))
then we have by Muuchy--Scwwarzinequuliyy
2:
|1 - l2,5 11225|114441111 flf fol 22,4
E ((X---)))))))7008)01(00)))))--)-
(1(p (x) || (((x))-(((x)))'
- Ilwllallf-ful +E
(Ill +E
[1(x-pt0<x)) [100===[[0x-p00000
|
1/1
1/1
T 8:00
2 11- fu
If we further assume that the function class satisfies an fg/ la condition that:
sup f 1-ffull22. full2,1 SC
/Ez ll. ful12.2
then w.p. 1-C:
Σ (( so [66m)BB/BB
1
0
||
((X))) follat 11
B
Applying 4 for any g(t) ,2(1) and averaging the final inequality we get:
i foll, S Σ (f (1(5p,000000 2 -6(9) ( -po)11,,)
We apply the lemma with L, = g and g E () ,2()) and g" =0. Then we instantiate the concentration incquality with --,,,-ffea "0(
17

---

Published as a conference paper at ICLR 2021
Plugging in the bound above Lemma 4 and applying the AM-GM inequality and Jensen's 's inequality, yields:
T f
Soll2,
o
en CnBBCIBC
||ll
B (II
B
fi [( -po) 1022
B
H
PROOF OF THM. 6: BIASED SGD ANALYSIS
Below, for any integer s, we define the operator norm of any vector e RR* and any tensor operating on R
||llop - ||ul|| and I|Tllop - sup |Toellop
o ==lll===
Recall the definition
=(W:0.p.)== VX (,,
Observe that since = can write for any 7: [(0:po)
Thus we also have that:
vH,y: V (...po)))
Given this observation, we can decompose the gradient that is used in our SGD algorithm into bias and variance component, when viewed from the perspective of biased SGD algorithm for the population oracle loss:
(WW:0pp.)) VgL(0:po)
P.)) - - (
b(0,p.))
n(W:0.p.))
The following two lemmas bound the gradient bias and noise terms.
Lemma 7 (Gradient bias). (erro/(VWVoWoWooooooo (o. 7, we have:
[ X
S M, then any parameter vector and functions p and
b(0xn)=B[oo(((x) (T (X) -xx))
Ve50 (T
( - (w- (w-Po)ll22) and
M V
[b(0p. Vofo (gfo.p (p-Po) 2.2
Proof By Taylor's thcorem with integral remainder and Lagrange remainder respectively the SGD bias for each parameter i
takes the form
b, = (ttp::/-BEB....../a0--///......
= ([(p(X)-oo(()) (X
Furthermore, operator norn assumption and SchwchySShwarziiige imply
M E
(b.(..))))l [B(r(())Po(( (gra.p
[o,_ M
2
< [[(((()))o(x)) (x),
2 felfel12.211 1po1...
Thus. by the triangle inequality and Jensen's 's inequality find that [|b(0.p.))la Vofo (7 (p (p-Po>ll22 and
M
[b((.p.))l22 Vefu (g/..p -)) po) 2.2
(p-Po))
Vofoll 1p
2
Lemma 8 (Gradient Variance). Define For any parameter and functions p and
/ B[)) (0) =
Vool (Tio. ( (p-po))122
18

---

Published as a conference paper at ICLR 2021
Proof
For each i [d], define the shorthand
A Va, (W: (X (V and
2 |
Vo. (7xx)))))(( (px -p(x))
(X) Vo,fa(x)"((xx)))
2 -2
for some convex combination p(X) of ((X) and (X).
We begin by bounding the target expectation using Cauchy-Schwarz
[W p..))
Var +o
#E|d
Σ #E|d
Var Vo (o
oo(0.p))2 2+ +
iE(d]
s 50(00m)) 2
Σ
Var Varr
iE(d]
I (0000
Var(A.)) +2
Va
Var A.]
vE(d
xE[4
*E|d]
(0o(0.po)
VarlA.])?
FEld]
We next employ the law of total variance rewrite the variance terms:
Var[A.]=
aE|d
iE[d]
E
(x
Σ
Var[Zi].
xE(d]
Finally, we control Var[Z:] using Cauchy-Schwarr Schwarz
Var[Z,] Vofo (Ti.. -1) (pp
[E|d]
The two claims of Thm. now follow from Theorems 2 and 3 of Ajallocian & Stich (2020) respectively, with the parameters 2 and ( inntantiatedd with quantities o² (7) and ((>) Lemmas and 8.
I EXPERIMENT DETAILS AND ADDITIONAL RESULTS
L.1 TABULAR DATA
We use cross-fittin fist--fittng with folds. The student trained using the SEL loss with clipped teacher class probabilities max (p(x),e) for e 10 The hyperparameter of the loss correction chosen by cross- cols-validation with 5 folds. We repeat the experiments 5 times to measure the mean and standard deviation.
For the overfitting experiment,
random forest with 500 trees the teacher and random forest with 1-40 trees as the student.
We also evaluate the impact of teacher underfitting by limiting the teacher's 's maximum tree depth (from to 20). Lower depth corresponds to greater underfitting. The teacher has 100 trees, and the student has 10 trees. For all of the datasets, loss correction successfully mitigates the teacher's 's underfitting and thus improves the student's performance. The effect most pronounced when the teacher underfits more heavily (has lower tree depth).
19

---

Published as a conference paper at ICLR 2021
We show the full results for all 5 of the datasets in Figs. 4 and 5.
0,00
333
Test Accurtyy
8.00
675
1.35
c3 Itdee
@ eet
3756
Druccd 60 Sh.ret
190
480
mardo Eo Maaee
121
number
Qtatert's rumber ofeee
Stadent's namber offteess
(a) Adult dataset
(b) FICO dataset
(c) Higgs dataset
0.90
6,22
0.00
0.54
10 Stdken:
suake Teather me
0.0
Teacher 550
Stadent's number trees
Shudent's number trees
(d) MAGIC dataset
(e) StumbleUpon dataset
Figure 4: Tabular random forest distillation with varying student complexity.
3321
E29
Mabue 10
Chitabrei 50 Lolien,
Cr Studer
3.6
Teatie
825
Tru
Statat depiin
310
Enhareed 10 Safer.
Teacher's max tree dopth
Teae het's thx tree deplt
Teacher's max tree desth
(a) Adult dataset
(b) FICO dataset
(c) Higgs dataset
CR25
Super whth
Stacent truu, duzth--
Stuicel witeet 1D
Emnancet 8D: Suddin
0:48
0:0
29 Teache''s thee depth
2
Teachers
(d) MAGIC dataset
(e) StumbleUpon dataset
Figure 5: Tabular random forest distillation with varying teacher complexity.
1.2 IMAGE DATA (CIFAR-10)
We use SGD with initial learning rate 0.1, momentum 0.9, and batch size 128 to train for 200 epochs. We use the standard learning decay schedule, where the learning rate is divided by epoch 60. 120, and 160. For loss correction, we select the value of the hyperparameter a that yields the highest accuracy on held-out validation set. For cross-fitting, we use 10 folds.
20


Reviews:
[{"review_id": "iWEYEsxQEYl", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper formulates knowledge distillation as a semi-parametric inference problem. Then, the paper adapts techniques from semi-parametric inference to analyze the error of a student model and improve the performance of knowledge distillation. The presentation is overall of good quality and it is relatively easy for me to follow the logic flow. That being said, there is still some room of improving the clarity of the paper.\n\nP1: Analyzing and improving knowledge distillation from a point view of semi-parametric inference is very interesting.  The adaptation of semi-parametric inference techniques such as cross-fitting and orthogonal machine learning is novel.\n\nC1: The related work is not sufficient. After reviewing some pioneer studies in knowledge distillation before 2015, this paper jumps to very recent studies in 2020.  As knowledge distillation is a hot research topic, there are a large number of related studies from 2016 to 2019. It is highly recommended for this paper to present the advance of knowledge distillation techniques from 2015 until now in a more comprehensive and organized way.\n\nC2: Some contributions of this paper, cross-fitting and orthogonal machine learning, are largely adapted from semi-parametric inference literature. The contribution that is not adapted from semi-parametric inference literature is the proposed correction that balances the bias and variance. The correction is a linear combination of a bias term and a variance term. It is not clear how the bias and variance terms are related to the theoretical analysis in previous sections. It would be better if this paper could provide a clearer justification of the bias and variance terms by the theoretical analysis.\n\nC3: The hyper-parameter \\alpha is introduced to balance the bias and variance of knowledge distillation in the experiment section. Since one of the key ideas of the paper is to balance the bias and variance, the hyper-parameter should be important in determining the accuracy of the student model. This paper should do some ablation studies to investigate how the hyper-parameter affects the performance of the proposed knowledge distillation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "hXqnLjSlp7", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The knowledge distillation (KD) approach is a two-step procedure: first train the teacher model on the labeled data and then train the student model using the predicted class probabilities from the teacher model. A key theoretical question about KD is whether and how much this two-step approach can improve on the one-step approach that trains the student model directly on the labeled data. This paper casts KD as a semiparametric inference problem by treating the optimal student model as the parameter of primary interest and the true class probabilities as the nuisance parameter. Building on the semiparametric framework, the paper makes two contributions: 1) develops theoretical guarantees for the vanilla KD algorithm; 2) proposes improved KD by using a first-order bias-corrected loss and a sample splitting procedure.\n\nOverall, I find the paper novel, well-written, and thought-provoking. It bridges the two directions of KD and semiparametric inference, allowing for the possibility of borrowing theoretical and methodological tools from semiparametric inference to analyze and improve the KD approach.\n\nOn the other hand, I think the paper somewhat oversells the semiparametric inference idea, since the theory presented in the paper (Theorem 1) does not have much semiparametric flavor. The central questions in semiparametric inference are the information bounds and semiparametric efficiency for the target parameter. Although Theorem 1 is useful, it does not directly address these key questions. For example, what is the best possible performance for the student model? In many classical semiparametric inference problems, we are able to construct a semiparametrically efficient estimator. Would this be possible for KD?\n\nMore comments:\n1) The paper seems to treat $f_0$ as infinite dimensional (page 3, 3rd paragraph of Section 3). However, classical semiparametric inference largely exploits the finite or low dimensionality of the target parameter to answer the questions mentioned above. In the KD framework, I still think the finite-dimensional case is more interesting and would provide more fundamental insights.\n\n2) There is a notable difference between the settings of KD and classical semiparametric inference: the target parameter $f_0$ in KD is ancillary in the sense that the data generating model depends only on the nuisance parameter $p_0$. Would this affect the applicability of semiparametric inference techniques?\n\n3) Page 4, line 11, “student” should be “teacher”.\n\nUpdate: I appreciate the authors’ response. The answer to my semiparametric efficiency question, however, is too terse and unclear. Theorems 1 and 3 provide only error bounds; I don’t see how they, together with the semiparametric efficiency of OLS, directly imply semiparametric efficiency for the student model. Semiparametric efficiency involves the optimal asymptotic variance. The authors might have confused the concept with rate optimality. Nevertheless, the paper is a nice contribution, and I will keep my rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "a_APlsswNm", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper under review presents a new approach to the theoretical understanding of konowledge distillation, an efficient technique that makes possible the training of small models given the predictions of a more complex model. The problem is recast as a problem in semi-supervised learning and several theorems are then drawn from this new viewpoint. \n\nBased on this re-interpretation of the problem, a nice analysis of vanilla knowledge distillation is given and a correction is derived from a Taylor expansion of the loss which is then carefully investigated as well. Finally a stochastic gradient method is studied using relevant bias and variance parameters.\n\nAll the proofs seem correct. The simulations results provide convincing illustrations of the findings.\n\nThe exposition follows a fast pace and is not very easy to comprehend for the non-expert. In particular a more detailed presentation of the semi-supervised learning background should be given in order to better substantiate the subsequent findings of this paper. \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "1x1k8UrufNS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\nThis paper gives generalization guarantees for vanilla knowledge distillation, where it identifies a large variance issue due to the teacher’s complexity in this guarantee, and then proposes alternative approaches to eliminate this issue. The framework it uses is inspired by semi-parametric methods. The main resulting algorithm has two key components: cross-fitting and a corrective surrogate loss. An SGD variant is also presented. Experiments illustrate some of the benefits of the cross-fitting approach.\n\n### Strengths\n+ The semi-parametric perspective and the attempt to have a bias-variance tradeoff as part of the knowledge distillation procedure are well-motivated and insightful.\n+  Lemma 2 is a neat technical building block and it has the potential of being useful beyond the scope of the paper.\n\n### Weaknesses\n- The paper lacks clarity in notation and presentation. In terms of notation, many expressions are overloaded (multiple forms of $\\gamma$ and completely different meanings for symbols $\\zeta$ and $\\mu$). In terms of clarity: important definitions are buried in the text instead of being highlighted (e.g. the corrective surrogate loss or the choice of, the definition of $q_p$ given only in an example), key steps are not always spelled out (e.g. the Bregman divergence simplifications, the choice of $\\hat \\gamma= q_{\\hat p}$ in the cross-fitting approach, or the use of the teacher in the SGD approach)\n- The motivation of the intermediate choices are not well-presented. For example, in transitioning to the corrective surrogate loss, a Taylor expansion of the ideal loss is used, but then the $\\gamma$-corrected loss is given without explanation. To me it seems to be motivated only by quadratic losses, but I’m sure more can be said by the authors.\n- Some of the arguments in favor of the paper are a bit underhanded. ~~For example, when discussing the drawbacks of vanilla distillation, the paper calls them *inherent*, though the only evidence is the upper bound provided. To make such a claim, a lower bound is needed.~~ [Edit: the author's lower bound is good.]\n- More importantly, the key contribution of the approach, the claim of eliminating the teacher complexity from the upper bounds is not quite crisp. Namely, Theorem 4 requires a upper bound on the critical radii of *uniformly over all* classes that could be empirically selected by the teacher. It’s true that the complexity of the union of these classes (as in Theorem 1) can still be larger than this uniform bound. But this is not the comparison that the paper makes when it expounds the advantage of the new bound. Instead, it compares the complexity of any *one* of these classes (paragraph at the end of Section 4.2, page 6) to the union, which is disingenuous.\n- Lastly, even though the corrective surrogate loss is touted as the key component of the approach, the experiments clearly show that it’s the cross-fitting that is ultimately regularizing the problem and reducing the dependence on the teacher’s complexity. But this isn’t analyzed in isolation in the paper. In general, the corrective loss only contributes modestly. The only exception seems in the Adult Dataset, but only when the teacher has *low* complexity, which is also not part of the message of the paper.\n\n### Overall\nI found that the paper takes an interesting approach to giving guarantees to KD methods, with several interesting technical contributions. However, the final message of the paper is muddied by the lack of clarity and an attempt to upsell the actual technical results. This is why at this point I would place the paper as borderline.\n\n_[Edit: I read the response of the authors. The lower bound for vanilla KD seems good. But the author still fail to adequately explain the gain of Thm 4 (now Thm 3). The requisite upper bound on the critical radius uniformly over all potential functions generated by the teacher means that the result of the theorem still has an implicit dependence on the teacher's complexity. For this reason, I will leave my recommendation the same.]_\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies knowledge distillation through the lens of semi-parametric inference. There has been a lot of work on knowledge distillation in the past, but, as the paper points out, most of it is heuristic or empirical in nature, and thus theoretical understanding on the subject is lacking. The reviewers generally agree that this paper makes a useful theoretical contribution towards a better understanding of knowledge distillation. However, the reviewers also raised some concerns, especially regarding the clarity of the text, and they felt that the paper might be overstating its contribution. Still, all reviewers recommend acceptance, and on balance the merits of the paper seem to outweigh the weaknesses, so I'd be happy to recommend acceptance.
Decision: Accept (Poster)