Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
Authors: Kenneth Borup
Lars Nørvang Andersen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Knowledge Distillation
Self Distillation
Kernel Ridge Regression
Statistical Learning
Theory

Abstract: Knowledge distillation is classically a procedure where a neural network is trained on the output of another network along with the original targets in order to transfer knowledge between the architectures. The special case of self-distillation, where the network architectures are identical, has been observed to improve generalization accuracy. In this paper, we consider an iterative variant of self-distillation in a kernel regression setting, in which successive steps incorporate both model outputs and the ground-truth targets. This allows us to provide the first theoretical results on the importance of using the weighted ground-truth targets in self-distillation. Our focus is on fitting nonlinear functions to training data with a weighted mean square error objective function suitable for distillation, subject to $\ell_2$ regularization of the model parameters. We show that any such function obtained with self-distillation can be calculated directly as a function of the initial fit, and that infinite distillation steps yields the same optimization problem as the original with amplified regularization. Furthermore, we provide a closed form solution for the optimal choice of weighting parameter at each step, and show how to efficiently estimate this weighting parameter for deep learning and significantly reduce the computational requirements compared to a grid search.

Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
Kenneth Borup
Lars N. Andersen
Department of Mathematics Aarhus University kemmethborup@math. au. dk
Department of Mathematics
Aarhus University larsa@math. au.
Abstract
Knowledge distillation is classically procedure where neural network is trained on the output of another network along with the original targets order to transfer knowledge between the architectures. The special case of self-distillation, where the network architectures are identical, has been observed to improve generalization accuracy. In this paper, consider iterative variant of self-distin aif-distillation in a kernel regression setting, which successive steps incorporate both model outputs and the ground-truth targets. This allows us provide the first theoretical results on the importance of using the weighted ground-truth targets self-distillation. Our focus is fitting nonlinear functions to training data with weighted mean square error objective function suitable for distillation, subject to &g regularization of the model parameters. We show that any such function obtained with self- distillation can be calculated directly as function of the initial fit, and that infinite distillation steps yields the same optimization problem the original with amplified regularization. Furthermore, we provide a closed form solution for the optimal choice of weighting parameter at each step, and show how to efficiently estimate this weighting parameter for deep learning and significantly reduce the computational requirements compared grid search.
Introduction
Knowledge distillation, most commonly known from Hinton al. (2015), a procedure transfer knowledge from one neural network (teacher) to another neural network (student) Often the student has fewer parameters than the teacher, and the procedure can be seen as model compression technique. Originally, the distillation procedure achieves the knowledge transfer by training the student network using the original training targets. denoted as ground-truth targets, as well softened distribution of logits from the (already trained and fixed) teacher network [ Since the popularization of knowledge distillation by |Hnton al. (2015), the idea of knowledge distillation has been extended to a variety of settings This paper will focus on the special case where the teacher and student are of identical architecture, called self-distillation, and where the aim is improve predictive performance, rather than compressing the model.
The idea self-distillation new targets for retraining the same model from scratch. We refer to this as step of self-distillation. and one can iterate this procedure for multiple distillation steps (see Figure T]. Empirically, has been shown that this procedure often generalizes better than the model trained merely on the
use outputs from a trained model together with the original targets as
Often knowledge distillation is also referred under the name Saucher-Studerr learning.
We will refer the weighted outputs the penultimate layer, i.e. pre-activation the last layer, as logits. See Section for a brief overview, see Wang and Yoon 2020 for more exhaustive survey
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

original targets, and achieves higher predictive performance on validation data, despite no additional information being provided during training (Furlanello et al., 2018 Ahn al., 2019 Yang al.) 2018).
X
Trsining
Distilation
Distillation
Figure 1: Illustration of self-distillation for two steps after the initial training, where we use the notation ((r) [(, (()) See Section for details.
Modern deep neural networks are often trained in the over-parameterized regime, where the amount of trainable parameters highly exceed the amount of training samples. Under simple first-order methods such gradient descent, such large networks
fit any target, but in order to generalize well, such overfitting is usually undesirable (Zhang et al., 2017 Nakkiran et al. 2020p. Thus, some type of regularization is typically imposed during training, in order to avoid overlitting. A common choice is to add an fz-regularization term to our objective function, which has been shown to perform comparably early-stopping gradient descent training (Yao et al. 2007). However, in the theoretical study of the over-parameterized regime, regularization is often overlooked, but recent results have shown connection between wide neural networks and kernel ridge regression through the Neural Tangent Kernel (NTK) (Lee et al. 2019 2020 Hu et al. (019). We briefly elaborate on this connection in Section which motivates our problem setup and connection deep learning in Section 5
Our Contributions
Through a theoretical analysis we show that
the solution any distillation step easily be calculated as function of the initial fit, and infinitely many steps of self-distin dif-distillation (with fixed distillation weight) correspond to solving the usual kernel ridge regression problem with specific amplified regularization parameter when the distillation weight is non-zero,
for fixed distillation weights, self-distillation amplifies the regularization at each distillation step, and the ground-truth targets dampen the sparsification and regularization of the self- distilled solutions, ensuring non-zero solutions for any number of distillation steps,
the optimal distillation weight has a closed form solution for kernel ridge regression, and can be estimated efficiently for neural networks compared to grid search.
Proofs of all our results can be found in Supplementary Material A and code reproduce
our illustrative example in Section 4.5] Experimental results in Section B can be found at github..om/Kenneehoorpppo disisiiiipsseeddddiiltttioooo distillation
2 Related Work
The idea of knowledge distillation dates back to Bucila et al. (2006), and was later brought to the deep learning setting by Ba and Caruana (2014) and more recently popularized by Hinton al. (2015) in the context of compressing neural networks. Since the original formulation, various extensions have been proposed. Some approaches focus on matching the teacher and student models on statistics other than the distribution of the logits, such as intermediate representations (Romero et al. (013). spacial attention maps (Zagoruyko and Komodakis 2019). Jacobians (Srinivas and Fleuret 2018). Gram
matrices (Yim et al. (017), or relational information between teacher outputs (Pakk et al. (019)).
Other extensions focus on developing the transfer procedure, such as self-distillation (Furlanello
et al. 2018), data-free distillation (Lopes et al. Naya,Naaya et al. 2019] Micaelli and Storkey 2019
Chen et al. 2019 Fang et 2019), data listillation (Radosavovic et al. 2018). residual know ledge
distillation (Goo et al. 2020). online distillation (Anil et al. 2018 contrastive distillation (Ahn
(ct al. 2019 Tian et al. 2020a).
'With slight differences. l2 regularization is often referred to as weight decay and ridge regularization in deep learning and statistical learning literature, respectively. See e.g. Loshchilov and Hutter (2019) for details.

---

The practical benefits of knowledge distillation have been proven countless of times in variety of settings, but the theoretical justification for knowledge distillation still highly absent. Hinton et al.) (2015) conjecture that the success of knowledge distillation should be attributed to the transfer of dark knowledge (e.g. inter-class relationships revealed in the soft labels). Müller et al. (2019); | Tang et al. (2020) support this conjecture, and argue that knowledge distillation is similar to performing adaptive label smoothing weighted by the teacher's confidence in the predictions. Dong et al. (2019) show the importance of early stopping when training over- werrpaarameterized neural networks for distillation purposes by arguing that neural networks tend to fit informative and simple patterns faster than noisy signals, and knowledge distillation utilizes these simple patterns for knowledge transfer. [Abnar et al. 2020) empirically investigate how knowledge distillation can transfer inductive biases between student and teacher models, and Gotmare al. (2019) empirically shows how the dark knowledge shared by the teacher mainly is disbursed to some of the deepest layers of the teacher.
To the best of our knowledge, few papers investigate knowledge distillation from rigorous theoretical point of view, and those that do, do so with strong assumptions on the setting. [huong and Lampert (2019) ignore the ground-truth targets during distillation and furthermore assume linear models. Mobahi et al. (2200) investigate self-distillation a Hilbert space setting with kernel ridge regression models where the teacher is trained on the ground-truth targets, and the student (and subsequent iterations) is only trained on the predictions from the teacher without access to the ground-truth targets. They show that deff-disiillation progressively limits the number of basis functions used to represent the solutions, thus eventually causing the solutions to underfit. In this paper, we build on the theoretical results of Mobahi et al. (2020), but we include the weighted ground-trute targets in the self-distillation procedure, where we allow the weight to depend on the self-distillation step, and show how this drastically affects the behavior and effect of self-distillation
3 Problem Setup
Notation Vectors and matrices are denoted by bold-faced letters; vectors column vectors by default, and for a vector a let [a]; be the i-th entry and for a matrix A let [A] Ali be the (i entry. Let I, denote the identity matrix of dimension n, [R] = {1, k). and let ||-I12 and | denote the f2-norm and the Frobenius norm, respectively. Finally, for function h R" IR and
X E _mxx,, we denote by h(X) the Raxd matrix of outcomes, where the i'th row of ((X) is the function applied the i' i'th row of X, i.e. [h(X)],, = h(x,).
Consider the training dataset D S Ed x I. and let X' {x (x,,) E D) and y = { (x, v) D) denote the inputs and targets, respectively. Let X = [xilke(n) E Ruxd be the matrix of inputs, y [yi the vector of targets, and X E pm y RT be the matrix and vector of validation inputs and targets, respectively. Given a feature map y Rd + V, where V has dimension D, denote by K x(X, X) [x(x,, x,) > R where x(),,))) (f(xx,, (x(x))), the symmetric kernel (Gram) matrix associated with the feature map yp
3.1 Self-Distillation Kernel Ridge Regressions
In order to avoid overfitting our training data, we will impose regularization term our weights, and thus investigate the kernel ridge regression functions f F mapping f X' - + J, to construct a solution which best approximates the true underlying data generating map and generalize well to new unseen data from this underlying map. We consider self-distillation in the kernel ridge regression setup; i.e. consider the (seff-distillation) objective function
diistill (f(X, ()x,8).1, = @
2 |lf(X. - y1 112
(1f(
v211
X 11912. 2
(1)
2
where a E [0, 11, A > 0. Y2 E RDTh and (X = *(X)33. The objective 1T is weighted sum
of two Mcan Square Error (MSE) objective functions with different target: and an ezrreuularization
In Supplementary Material E we relate our problem setup to Mobahi al. a (220)) and extend some of our results to constrained optimization setting with regularization functional in Hilbert space.
"Since the kernel trick makes the predictions depend only on inner products in the feature space, it is not a restriction if D is infinite. However, for ease of exposition assume D is finite.
II is straightforward to verify that minimizing | and the classic MSE objective with a weighted target, || are equivalent and that the
ie. _diseil (X, B), vI. 1/ [X 33- (ay (1 a)y²)l
objective functions are equal up the additive constant aaa- 1) y| y2
3

---

on the model weights. Minimization of w.I.t. /3 is straightforward and yields the minimizer
def argmin Cdistill (J(X, - (()) (K + AI,)) (ay1 + (1 - a)y2)
(2)
by Woodbury's 's matrix identity and definition of K. This solution can also be seen as a direct application of the Representer Theorem (Schölkopf et al. 2001). Let y(")) def y, i.e. the original
targets, and recursively define for the steps 7 2 1,
((T) del
argmin Edistill (). y(
(3)
(XX) (K+ (I,) (al (1-1 (1-a(t)) ((-)))(--1))
f(x. (1)) d8l B(x)T((T)
(4)
=x(x,X) (K + AI,)
(a (a (110)((1+00))+111+111111111)1)11
y _ 3(x,3)))
(5)
for fixed (1) E [0, and such is independent of (1)). Self- seff-disilllation treats the weighted average of the predictions, y()), from this initial model on X, and the ground-truth targets, y, as targets. This procedure is repeated as defined in
1]. Notice, the initial step (r 1) corresponds standard training by definition
33 and we obtain the self-distillatinn procedure as illustrated in Figure [] correspond to merely training on the predictions from
Note, the special cases a O and (7) the previous step, and only training on the original targets, respectively. Thus, (r(r) 1 is usually not of interest, as the solution is equal to a classical kernel ridge regression, and self-distillation plays no role in this scenario. We will often consider the special case of equal weights, (2) = ... = (T a, and if 0 this corresponds to the setting investigated in Mobahi et al. (2020) in slightly different setup. Thus, some of the following results can be seen as a generalization of Mobahi et al. (2020) to step-wise and non-zero a.
Main Results
In this section we present our main results for finitely and infinitely many distillation steps along with closed form solution for the optimal (T) as well as illustrative example highlighting the effect of the chosen sequence of (a)))) on the solutions.
4.1 Finite Self-Distillation Steps
Our first result, which follows from straightforward computations, states that the predictions obtained after any finite number of distillation steps can expressed directly as function of y and the kernel matrix K calculated at the initial fit (T 1).
Theorem 4.1. Let ( 3(T), and f(. f(.)(()) be defined as above. Fix a(?))...,,,!!! (T) E [0. 1). and
let r) det IT a1)), then for 1. we have that
y(()= Ea Oni + 1,5) (K(K+ + +I,+ (I,) T-i+1 +2(2,7) (K(K+ (Im) +((++An)))) alm)')))
y, (6)
f(x, B(r)) = s(x, ((1)) + (1- (xx ((1))
for any RRd, where ((5) the minimizer in 3 with a (T) 0.
Since (6) and are expressed only in terms of K, (K + (II,) -1 x(xx X), and y we are able to calculate the predictions for the training data as well as for any x E Rdd based merely on the initial fit (T = 1) without the need for any additional fits. Hence, despite the calculations of K, s(x, X), and especially (K + AI being (potentially) highly computationally demanding, when obtained, we can calculate any distillation step directly by the equations in Theorem 4.1 Furthermore, predictions at step 7 can be seen as a weighted combination of two classical ridge regression solutions, based on the original targets and the predicted targets from step 1. respectively. However, choosing

---

appropriate (*) for t = ....., 7 is non-trin non-trivial. We explore these dynamics in Section 4.3 and 4.4 First, we use Theorem to analyse the regularization that self-distillatton progressively impose on the solutions.
4.2 Effective Sparsification of Self-Distillation Solutions
We now show that we can represent the solutions weighted sum of basis functions, and that this basis sparsifies when we increase т, but also that the amount of sparsification depends on the choice was proved in Mobahi et al. (2020), and in particular, our (13) generalizes equation (47) in their paper.
of a. A similar sparsification result for the special case of fixed (7) 0 for
Using the spectral decomposition of the symmetric matrix we write K VDVT, where V Rax" is an orthogonal matrix with the eigenvectors of K as rows and D E I is non-negative diagonal matrix with the associated eigenvalues the diagonal. Inserting the diagonalization yields
K(K AL VDVT (VDV + AI,)
(L - VT,
(9)
where A > O. By straightforward calculations using and we have
(T) VB
where
(10)
B (
mi+ +22,T))', and A del D(D + AI,)
(11)
and A, B(7) R are diagonal matrices for any т. Furthermore, by (TO) the only part of the solution depending on r the diagonal matrix, B(r), and in the following we show how B
determines the effective sparsification of the solution f(-,3(r)).
Lemma 4.2. Let B(r), and A be defined above, and let B(0) del I,- Then can express ((r)
recursively as
B = A ( 11
(12)
and [B 1,, E [0, 1] is (strictly) decreasing in T for all k E [m] and T 2 iffa .... at) = a.
Similarly to (0). if we use Lemma 4.2 and Theorem 4.1 we can show that for any IR
f(x, ((7)) = k(x, X
TVD BB =x(x) (B(r) (r)
T
where
(13)
pxx) D
xxxxxX)) and V
Thus, the solution f(, ((r) can be represented as weighted sum of some basis functions, where the basis functions are the components of the orthogonally transformed and scaled basis p(x), and an orthogonally transformed vector of targets.
Now assume (2) ... = (T) = a for any 2 for remaining of this section. the following we show how the behaviour of (r), and in turn also the behaviour of f(- ((-)f())), with r dependent on the choice of a. Lemma 4.2 not only provides a recursive formula for B(r), but also shows that each diagonal element of B ( is in [0, 1] and strictly decreasing in т, which in turn implies that the self-distillation procedure progressively shrinks the coefficients of the basis functions. Using Lemma 4.2 we can now show, that not only does (r) decrease in т, smaller elements of B(r) shrink faster than larger elements for a 0, as we elaborate on below the theorem.
Theorem 4.3. For any pair of diagonals of D, i.e. dy and d,. where dy > dj, we have for all T 2 1,
1
B B
It
for 1
(14)
[
1
for a = 0,
5

---

and if we let sgn(-) denote the sign function
then E (0, 1) we have that
B B
lank
B
(
sgn
B
133
((1
[ ( K.k
[AJa.
[A] h AA]s (B(--1))
= sgn
B (
J.N - (Alss)
+1
0 (15)
B Je. is
If we consider a pair of diagonals of D, where dy > dj. then for a O. the fraction [B
strictly increasing in due to the r.h.s. of (14) inside the parenthesis being strictly larger than 1. Hence, the diagonals corresponding to smaller eigenvalues shrink faster than the larger ones as T increases. However, for E (0, 1) we not ensure this behaviour, but at step T predict the behaviour at step T + 1, by using (15). Thus, when we include the ground -ruth targets in our distillation procedure we do not consistently mcreasee the regularization with each distillation step. but can potentially obtain a solution which does not sparsify any further. We now turn our attention to the question of how to pick the (T))ss in an optimal manner, and find that it can be done if we relax the condition that the weights are restricted to the interval [0, 1].
able to
4.3 Closed Form Optimal Weighting Parameter
Recall, X E m is the matrix of validation inputs and y E I the vector of validation targets. If we allow (r) E I., we can find an optimal ( (which is a non-trivial function of X) at each step т. denoted by a (T) Here, optimal denotes the value for which the validation MSE is minimized. Note, a (T) is optimal for a single distillation step, but not necessarily so for multiple distillation steps, however we may consider (T) greedy estimate of the optimal value across multiple steps.
Theorem 4.4. Fix 2 2, X > 0 and (2)
(1 E RR then
(5l22 - -y(1) (5 -5())
(T) argmin oft)
y - )
(16)
| - ycull²
where ((1) = -( )) and 5/2 3(8
Since neither (x))) nor y:2 depend on the choice of (T), we can calculate (*()) recursively as presented in Algorithm where (+) has the closed form in (6)). In combination with the diagonalization results of Section 4.2 we can efficiently calculate the solutions. This should be compared to performing grid-search for with g equidistant values on [0, 1] in order to approximate the optimal a, which requires g(T - 1) + 1 model fits one uses the same a for each sequence of 2 2 steps (g--1 if is not fixed across distillation steps). However, by Algorithm D is sufficient to perform 2(T - 1) + 1 model fits, and obtain the exact optimal value at each step instead of an approximated value. In Section 5 we apply Algorithm to approximate (r) in a deep learning setting.
4.4 Infinite Number of Self-Distillation Steps
We now prove that if we were to perform an infinite number distillations steps (T + oo) with a fixed a (i.e. (2) problem, with an amplified regularization parameter (by a > if a > 0. Observe that, when = 0
(F) a) the solution would solve the classical kernel ridge regression
and + co, 66) and ( yield that the predictions y (*o) and f(x, (()) collapse to the zero-solution
for any x E P as expected from Mobahi et al. 2022)).
Note, we use the definition of sgn(-) where sgn del 0. "If (t)) [0. 1), the sign of either the first second of hecomes negative, indicating either too strong or weak regularization of the previous distillation step, and one might fear this affects distillation performance. However, simply clipping of (7)) to be in [0, 1] alleviates this, at the cost of requiring larger r.

---

Theorem 4.5. Let Letyl ,(,),, and f(. (() be defined as above, and a E (0, 11, then the following
limits hold
yloo) der
lim y(r) = K ( \ In 1-000
) y
(17)
a
f(x, glo0) daf
lim s(x, - as 3()) + (1 (x,,0)) T-200
where 17 corresponds to classical kernel ridge regression with amplified regularization parameter a/a, and let 7/(00) denote the kernel ridge regression weights associated with solving another kernel ridge regression on the targets ylxx with regularization parameter X. Furthermore, the
convergence lim, y(5) is of linear rate.
If a O, then by (9) and The-
Algorithm 1: Calculate 3 and a (1) for 2 2.
orem 4.5 we have that y
Dn d JJy and we shrink the eigenvectors with smallest eigen- values, corresponding the direc- tions with least variance, the most. Furthermore, a 0 the lim- iting solution is non-zero ker- nel ridge regression with regulariza- tion parameter X/a 2 X, causing the cigenvectors associated with the smallest cigenvalues to shrink even more than in the original solution.
d
Calculate ( from (3) (with any (1));;
Calculate y(1) = f(x, ((X,,())
for 2 to r do
Calculate 3 from and y a==0 = 1(X, P Solve:
a argmin ||s ()-((0((1 + +(1 a) nER Calculate ((0) from with a (();
end
Our results gives a theoretical explanation for why one should treat (T) as an adjustable hyperparam- eter to fine-tune the amount of regularization that self-distillation impose for particular problem, and that it can be chosen in an optimal way for kernel ridge regression. the following we provide an illustrative example, and in Section 5 we estimate the optimal weighting parameter for deep learning using an adaptation of Algorithm
4.5 Illustrative example
Consider the training dataset 'D where X' = {0, 0.1, ,0,,...,00. 0.9, }} and y = (sin(2 + 8 x E ), and is sampled from zero-mean Gaussian random variable with standard deviation 0.5. Let sp be the Radial Basis Function kernel, i.e. s(x,, x)) e->l|x;-x
where we choose y 80 and let 0.2 and consider the three cases; (a) a = 0, (b) a = 0.25, and (c) step-wise optimal a (r).
10
05-
05
00
00
-05-
-05-
-10
-10
31
100
(a) 0
(b) a = 0.25
(c) a (r)
Figure 2: Six steps of self-distillation with (a) zero limiting solution (dashed), (b) non-zero limiting solution (dashed), and (c) optimal step-wise a (7), Training examples represented with and in (c) we also plot a (T) with T.
As illustrated in Figure 2a for case (a), the regularization imposed by self-distillation initially improves the quality of the solution, but eventually overregularize and the solutions underfit the data, and will eventually converge to the zero-solution (see supplementary materials for the loss values). Using > (see Figure 2b). and more specifically a 0.25, reduce the imposed regularization and increases the stability of the distillation procedure; i.e. the solutions differ much less between each distillation step. This allows for more dense exploration of solutions during iterated distillation steps, where increasing reduces the difference between solutions from two consecutive steps, but
7

---

also reduces the space of possible solutions as the limit, f(., ((,)((x)), approaches the initial solution f(,, (() quickly 10 However, choosing the step-wise optimal a*(r) yields minuscule changes to the solution for > 2, and a single step of distillation is effectively enough. Furthermore, for т 2 3,
all (+(r) are approximately equal, and distillation procedure has reached an equilibrium
As expected from Lemma 4.2 and Theorem 4.3 Figure verifies that both in case (a) and (b), the diagonal of ( is decreasing and the diagonal coordinates corresponding to smaller eigenvalues shrink faster than those corresponding to larger eigenvalues in case (a). Without loss of generality we can assume di < d2
d,y, and for k I,,.. .... and any define R2
def B
B We expect R to be strietly increasing in for all & in case (a), but for case (b) we can make no such guarantee. Both of these properties are verified Figure
004
0.4-
(a) α
(b) a = 0.25
(c) a (7)
Figure 3: Diagonal of ((r) for T -,.... associated with Figure Note, the plots are overlaid,
but since the diagonal of (r) decrease in т. all values until convergence are visible. (a we expect and observe strictly decreasing values in r for all indices, until collapsing at O, but in (b) and (c) the values converge to a non-zero limit.
Finally, observe that in case (a), the values of (r)) shrink much faster than in case (b), and eventually collapse to all zeros, whereas the latter is nearly converged after six iterations. Furthermore, case (a) appear obtain more sparsified solution, the smallest coordinates effectively diminishes, which is not true for case (b). Furthermore, when directly comparing solutions from both cases with similar quality of fit, the solutions obtained with a = 0 usually has smaller coordinates in B(r) than those obtained with larger values of a.
130
144
13+
10
(a)
(b) 0.25
(c) (r)
Figure 4: Ratios, I2(T) of the ordered diagonal of B(7) for all r. In (a) we expect and observe strictly
increasing values in 7 for all but have no such guarantee in (b) or The x-axis corresponds to indices k = 1,..., 1.
5 Approximate Optimal Weighting Parameter for Deep Learning
The following experiment aim at empirically evaluating the theoretical analysis above in a simple deep learning setting. In (T6) we find (7) on closed form when f(-, ((7) is (self-distilled) kernel ridge regression. No closed form solution can be found for neural networks, but recent results show that (very) wide neural networks can seen as kernel ridge regression solutions with the neural
tangent kernel (Jacot et al. 2018 Arora al. 2019 Lee et al. 2019 2020
10 As expected by Theorem 4.5
experience fast convergence to the limit; usually less than 10 iterations
are sufficient to converge "If we clip a to be in 00, 1]. the a (r)) converges rather than 3.

---

Thus, inspired by 7 we propose to estimate (1))) for t = 2.... denoted by ((),, for neural network trained with self-distillation using an adapted Algorithm I network with vector of weights 0, and recursively for T 2 let ((T) be the weights solving
Let fun 0) IR" be a neural
(+)))
1 (T) +
argmin 2 [x, 0) Y
5 (X, 0) Y
+ 1011 (18)
2
F
with ( = (() and where Y E Raxp
Furthermore, let 0 be the weights associated with dal fan (X, 0 as well as (r def (X, ((7)) be the predictions on the validation input X. Then, following Algorithm | with |l-I|2 replaced by II-II and (18) rather than ! we can calculate the estimates a These estimates yield comparable predictive performance to the best fixed (7 (found with cone-con grid search), but only require one additional model fit per distillation step; i.e. 2(T 1) + 1 fits compared to g(T 1) + for grid search over y values. See Figure 5 for results and supplementary material for experimental details.
minimizing 18 with (+) 0, and Y!
a=0
5.1 Experiment
We perform self-distillation with ResNet-50 (He et al. 2016) networks CIFAR-10 Krizhevsky and Hinton 2009), with minor pre-processing and augmentations. The model is initialized randomly at each step and trained according to the above with either estimated optimal parameters, d(r), or fixed for all steps. We use the network weights from the last iteration of training each distillation step for the next step, irrespective of whether a better model occurred earlier in the training. Our models are trained for fixed 75 epochs and each experiment is repeated with different random seeds over 11 chains of distillation steps, corresponding to {0.0, 0.1, ..., 0.9} and G!T), with the first model initialized identically across all chains. The accuracy reported at the r'th step is based on comparing the training and validation predictions, Yr) and (X,)(r) with the original training and validation targets; Y and Y 14
0.96
031
0.1
0.0
0.8
022
0.02
0.5
0.00
076
077
Figure 5: Training and validation accuracy for five distillation steps with ResNet-50 models on CIFAR-10. Comparing fixed (!) for = 2..., - and estimating optimal weight with a(9) at each step. The experiment is repeated four times and the mean (and max/min in shaded) is reported.
Conclusion
In this paper, we provided theoretical arguments for the importance of weighting the teacher outputs with the ground-truth targets when performing self-distillation with kernel ridge regressions along with a closed form solution for the optimal weighting parameter. We proved how the solution at any (possibly infinite) distillation step can be calculated directly from the initial distillation step, and that self-distillation for an infinite number of steps corresponds to a classical kernel ridge regression solution with amplified regularization parameter. We showed both empirically and theoretically that
We treat class labels as p-dimensional one-hot encoded vectors and norm of the difference between the predicted class probabilities and the one-hot vectors. Note, we initialize the models equally across all for one experiment, but alter the seed for initialization between experiments.
"4The empirical experiments are constrained by the theoretical set-up and performed in a highly simple setting: using the weighted MSE loss from (18) (see supplementary materials for more details). Therefore, our accuracy measures be expected to be lower than for more fine-tuned training setups

---

the weighting parameter a determines the amount of regularization imposed by self-distillation, and empirically supported our results in a simple deep learning setting.
6.1 Future Research Directions
Interesting directions of future research are on rigorously connecting neural networks and kernel methods in a knowledge distillation setting, extend to other objective functions than MSE as well as including intermediate model statisties in the distillation procedure. Finally, a larger empirical study of the connection between the choice of a and the degree of overfitting is interesting as well.
A Acknowledgments and Disclosure of Funding
We would like to thank GenomeDK and Aarhus University for providing computational resources that contributed to these research results, Furthermore, we would like to thank Daniel Borup and Ragnhild Ø. Laursen for comments and discussion, well as Google Researcher Hossein Mobahi and Mchrdad Farajtabar (Deepmind) for clarifications on their experimental setup. We also thank the anonymous reviewers of the NeurlPS 2021 conference for their comments. Kenneth Borup is partly financed by Aarhus University Centre for Digitalisation, Big Data and Data Analytics (DIGIT).
References
Abnar, S., Dehghani, M., and Zuidema, W. (2020). Transferring Inductive Biases through Knowledge Distillation. arXiv preprint arXiv:2006.0055:
Ahn, S., Hu, S. X., Damianou, A., Lawrence, N. D., and Dai, Z. (2019). Variational information distillation for knowledge transfer. Proceedings of the IEEE Computer Society Conference on Computer Vision and Patlern Recognition, 9019-June:9155-916
Anil, R., Pereyra, G.,, Passos, A., Ormandi, R., Dahl, G. E., and Hinton, G. (2018). Large scale distributed neural network training through online distillation.
Arora, S., Du, S. S., Hu, W. Li, Z., Salakhutdinov, R., and Wang, R. (2019). On exact compu- tation with an infinitely wide neural net. Advances in Neural Information Processing Systems, 32(NeurIPS).
Ba, L. J. and Caruana, R. (2014). Do Deep Nets Really Need to be Deep? Advances in Neural Information Processing Systems, 3(amuary):2554-6662
Bucila, C., Caruana, R., and Niculescu-Mizii, A. (2006). Model Compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '06, pages 535-541, New York, NY, USA. ACM.
Chen, H., Wang, Y., Xu, C. C. C., Yang, Z., Liu, C., Shi, B., and Tian, Q. (2019). Data-free learning of student networks. In Proceedings of the IEEE International Conference on Computer Vision
Dong. B., Hou, J., Lu, Y., and Zhang, Z. (2019). Distillation approximately early stopping? harvesting dark knowledge utilizing anisotropic information retrieval for overparameterized neural network.
Falcon, W. (2019). PyTorch Lightning. github.com/Py TorchL hawphlihhhhnnnngytohhhhhhhgge
Fang, G., Song, J., Shen, C., Wang, X., Chen, D., and Song, M. (2019). Data-Free Adversarial Distillation. arXiv preprint arXiv: 191..11091.11
Furlanello, T., Lipton, Z. C., Tschannen, M., Itti, I.., and Anandkumar, A. (2018). Born Again Neural Networks. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1607-1616, Stockholmsmässan, Stockholm Sweden. PMLR.
Gao, M., Shen, Y., Li, Q., and Loy, C. C. (2020). Residual Knowledge Distillation. arXiv preprint arXiv:2002.09168,
Gotmare, A., Shirish Keskar, N., Xiong, C., and Socher, R. (2019). A closer look at deep learning heuristics: Learning rate restarts, warmup and distillation. 7th International Conference on Learning Representations, ICLR 2019.
10

---

He, K., Zhang, X., Ren, S., and Sun. J. (2016). Deep residual learning for image recognition. Pro- ceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2016-Decem:770-777
Hinton, G., Vinyals, O., and Dean, J. (2015). Distilling the Knowledge in Neural Network. arXiv preprint arXiv: 1oXXv::5003.05311
Hu, W. Li, Z., and Yu, D. (2019). Simple and Effective Regularization Methods for Training Noisily Labeled Data with Generalization Guarantee. 8th International Conference on Learning Representations, ICLR 2020.
Jacot, A., Gabriel, F., and Hongler, (2018). Neural tangent kernel: Convergence and generalization in neural networks. Advances in Neural Information Processing Systems, 2018- (8-Decmm(5):85)) 8580.
Krizhevsky, A. and Hinton, G. (2009). Learning multiple layers of features from tiny images.
Lee, J., Xiao, L.,, Schoenholz, S. S., Novak, Y. B. R., Sohl-Dickstein, J., and Pennington, (2019). Wide neural networks of any depth evolve as linear models under gradient descent. In Advances in Neural Information Processing Systems.
Lee, J. D., Shen, R., Song, Z., Wang. M., and Yu, Z. (2020). Generalized Leverage Score Sampling for Neural Networks. arXiv preprint VrXiv:2009.09829
Lopes, G., Fenu, S., and Starner T. (2017). Data-Free Knowledge Distillation for Deep Neural Networks. arXiv preprint arXiv:
Loshchilov, I. and Hutter, F. (2019). Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019.
Micaelli, P. and Storkey, A. J. (2019). Zero-shot Knowledge Transfer via Adversarial Belief Matching. In Wallach, H., Larochelle, H., Beygelzimer, A., dNtextquotesingle Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 32, pages 9551-955 9551-9561. Curran Associates, Inc.
Mobahi, H., Farajtabar, M., and Bartlett, L. (2020). Self-Distillation Amplifies Regularization in Hilbert Space. Advances Neural Information Processing Systems.
Müller, R., Kormblith, S., and Hinton, G. (2019), When Does Label Smoothing Help? In Advances in Neural Information Processing Systems.
Nakkiran, P., Barak, B., Kaplun, G., Sutskever, L., Bansal, Y., Yang, T., Barak, B., and Sutskever,
I. (2020). Deep Double Descent: Where Bigger Models and More Data Hurt. In International Conference on Learning Representations.
Nayak, G. K., Mopuri, K. R., Shaj, V., Babu, R. V., Chakraborty, A., Venkatesh Babu, R., and Chakraborty, A. (2019). Zero-shot knowledge distillation in deep networks. 36th International Conference Machine Learning, ICML 2019, 8019-Junc:3317-322.
Park, W., Kim, D., Lu, Y., and Cho, M. (2019). Relational Knowledge Distillation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
Phuong, M. and Lampert, C. H. (2019). Towards understanding knowledge distillation. 36th International Conference on Machine Learning, ICMI. 2019,
Radosavovic, I., Dollár, P., Girshick, R., Gkioxari, G., He, K., Dollar, P., Girshick, R., Gkioxari, G., and He, K. (2018). Data Distillation: Towards Omni- Sunii-Suerrvieed Learning. Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pages 4128.
Romero, A., Ballas, N., Kahou, S. E., Chassang, A., Gatta, C., and Bengio, Y. (2015). FitNets: Hints for thin deep nets. 3rd International Conference on Learning Representations, ICLR 2015 Conference Track Proceedings, pages 1-13,

---

Schölkopf. B., Herbrich, R., and Smola, A. J. (2001). A generalized representer theorem. In International conference on computational learning theory, pages 416-426. Springer.
Srinivas, S. and Fleuret, F. (2018), Knowledge transfer with jacobian matching. 35th International Conference Machine Learning, ICML 2018, 11:7515-7523.
Tang, J., Shivanna, R., Zhao, Z.., Lin, D., Singh, A., Chi, E. H., and Jain, S. (2020). Understanding and Improving Knowledge Distillation, arXiv preprint ar Xiv::202..03332.
Tian, Y., Krishnan, D., and Isola, P. (2020a). Contrastive Representation Distillation. In International Conference on Learning Representations, pages 1-19.
Tian, Y., Wang, Y., Krishnan, D., Tenenbaum, B., and Isola, (2020b). Rethinking Few-Shot Image Classification: Good Embedding Is All You Need?
Wang, L. and Yoon, K.-J (2020). Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks. arXiv preprini urXiv:2004.05937.
Yang, C., Xie, L., Qiao, S., and Yuille, A. (2018). Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students. arXiv preprint /85.v5180.005551
Yao, Y., Rosasco, I.., and Caponnetto, A. (2007). On Early Stopping in Gradient Descent Learning. Constructive Approximation, 26(2):289-315.
Yim, J., Joo, D., Bae, J., and Kim, J. (2017). A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7130-7138.
Zagoruyko, S. and Komodakis, N. (2019). Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. 5th International Conference Learning Representations, ICLR 2017 Conference Track Proceedings, pages 1-13.
Zhang, C., Recht, B., Bengio, S., Hardt, M., and Vinyals, O. (2017). Understanding deep learning requires rethinking generalization. 5th International Conference on Learning Representations, ICLR 2017- Conference Truck Proceedings.
12


Reviews:
[{"review_id": "oK_cTA5MmSs", "reviewer": "Reviewer_wK1m", "summary": "This paper proposed a variant of self distillation which incorporates both the model output and the ground truth targets. The author provided a detailed theoretical analysis about how fixed weighted ground truth restrain the regularization amplified by self distillation.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper clearly stated the motivation of introducing a weighted ground truth target while self distillation. The theoretical results (Theorem 4.3, 4.5) supported the argument. \nThe author also provided a close form solution for setting the step wise weight other than a fixed weight for all steps.\nThe estimation of the optimal weight provided practical usefulness for this method and guidance on how to adjust this hyperparameter efficiently other than grid search.\n\nHowever, my minor concern about this work are: \n\nthe main theoretical results are all derived under the setting of fixed weight for all distillation steps. While in practice the optimal weights are calculated at each step. \n\nIn section 4.2, the author argued by introducing the weight parameter, the self distillation process can potentially do not sparsify the basis. According to theorem 4.3, this potential heavily depends on the value of the weights and the weights are derived in a closed form manner at each step. It would be more rigorous if the author could provide some guarantees with probability of theorem 4.3 to show how likely the basis will not sparsify.\n\nOverall, I think this work proposed an inspiring potential method on addressing the evolution of basis problem carried out by Mobahi et al.(2020), along with some practical usage.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "MDkas_8BcYG", "reviewer": "Reviewer_qZC6", "summary": "This paper studies the theory of self-distillation in the kernel ridge regression setting. The authors extend the theoretical results in Mobahi et al. (2020) to further incorporate the ground-truth labels in the distillation objective and highlight that the ground-truth labels serve to dampen the sparsification and regularization of the self-distilled solutions.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides closed-form solutions of self-distillation in the kernel ridge regression setting. Overall, the entire paper is mostly well-written and easy to follow. It is a natural extension of the theoretical results in Mobahi et al. (2020), though the theoretical novelty is limited given the prior works (the theorems are not hard to derive based on the results of Mobahi et al. (2020)). See detailed comments below.\n\n\n1. After reading the paper, I am not convinced that incorporating the ground-truth label is essential during the self-distillation step to overcome underfitting. As you show in Theorem 4.5, the weighting parameter $\\alpha$ turns out to amplify the regularization parameter $\\lambda$. In this sense, we can also smartly tune $\\lambda$ on the validation set to achieve the same effect as tuning $\\alpha$. Therefore, I don't think that incorporating the ground-truth labels of training data are necessary. \n\n\n2. The way to estimate the value of $\\alpha$ in section 4.3 is not globally optimal, as it does not account for the future steps of self-distillation. I would instead call it a greedy estimation of $\\alpha$.\n\n\n3. For  ResNet-50 on CIFAR-10, why are the best training accuracy only 88% and the best test accuracy only 94%? In my experience, the training accuracy can always reach very close to 100%, and the test accuracy is around ~94%. Please check out this GitHub: https://github.com/kuangliu/pytorch-cifar. \n\n\n4. For Line 239-240, you mentioned that \"As illustrated in Figure 2a for case (a), the regularization imposed by self-distillation initially improves the quality of the solution\". To me, I don't see such a trend in Figure 2(a). The curves corresponding to $f_1, ..., f_{\\infty}$ become more underfitting as the index value increases. To better illustrate your argument, I would suggest the authors provide a figure of \"$\\tau$ vs. training loss\" (similar to Figure 5).\n\n\n5. It would be better to state the contribution in the introduction rather than in related works (Line 84-93). Also, the introductions to the notation (Line 96-101) are well-suited to be at the beginning of Section 3 Problem Setup.\n\n\n6. One minor correction to footnote 4, for some cases (e.g., using Adam optimizer), the equivalence between L2 regularization and weight decay does not hold. See Loshchilov et al. (2017) and Zhang et al. (2018).\n\n\nOverall, I think this paper is well-written and studies a critical problem in machine learning, considering that there are still relatively few works exploring the theories of knowledge distillation.  However, the theoretical novelty is very limited as most of the theorems are very natural extensions from the results in Mobahi et al. (2020). Also, I am not very convinced that incorporating the ground-truth label is necessary for self-distillation. Therefore, I vote for a weak rejection at the current stage. I am happy to increase my ratings if the authors can address my concerns properly.\n\nReferences:\n\n1. Loshchilov, Ilya, and Frank Hutter. \"Decoupled weight decay regularization.\" arXiv preprint arXiv:1711.05101 (2017).\n\n2. Zhang, Guodong, et al. \"Three mechanisms of weight decay regularization.\" ICLR 2018.\n\n\n====Updates=====\n\nThe authors' response resolved my concern. I will increase my score from 5 to 6. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitation of their work but did not discuss the negative societal impact. To discuss the negative societal impact, the authors can consider some cases where self-distillation can be applied to achieve some malicious goals.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "B11UN3Zn57Q", "reviewer": "Reviewer_ZxPW", "summary": "This paper presented a theoretical and empirical study of weighting the teacher outputs and the target outputs during self distillation. For an iterative self distillation process, a solution is shown to compute the fit at any step of the iteration given the first model. This was further related to classical kernel ridge regression with regularization.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The existing theory behind distillation is thin, and this paper makes a step toward a better grasp of it. I found the claims well supported by experiments and well explained in the text. I have not thoroughly checked the proofs in the supplementary material.\n\nI found the writing quality high and the exposition clear.\n\nThe findings and theory presented is likely significant in the community's effort to deepen our understanding of distillation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is adequate mention of the limitations of this work. The authors did not address the societal impact, and I agree with the assessment that it is not necessary for this work.", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hrs."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides an in-depth analysis of self-distillation in kernel regression setting, and studies the effect of using a weighted combination of ground truth labels and predictions made by the model, to define the next set of target values. It provides a closed form solution for the optimal choice of weighting parameter at each step, and shows how to efficiently estimate this weighting parameter for deep learning and significantly reduce the computational requirements compared to a grid search.

All reviewers find this paper interesting, and rate it in the accept zone. Some of the initial scores were lower at first, but after author's reply, reviewers found them convincing and increased the rating. In concordance with the reviewers, I find distillation an important tool with increasing popularity, and providing a solid understanding about this technique is of great interest to the community.

Decision: Accept (Poster)