Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation
Year: 2021

Abstract:
Knowledge distillation is classically a procedure where a neural network is trained on the output of another network along with the original targets in order to transfer knowledge between the architectures. The special case of self-distillation, where the network architectures are identical, has been observed to improve generalization accuracy. In this paper, we consider an iterative variant of self-distillation in a kernel regression setting, in which successive steps incorporate both model outputs and the ground-truth targets. This allows us to provide the first theoretical results on the importance of using the weighted ground-truth targets in self-distillation. Our focus is on fitting nonlinear functions to training data with a weighted mean square error objective function suitable for distillation, subject to `2 regularization of the model parameters. We show that any such function obtained with selfdistillation can be calculated directly as a function of the initial fit, and that infinite distillation steps yields the same optimization problem as the original with amplified regularization. Furthermore, we provide a closed form solution for the optimal choice of weighting parameter at each step, and show how to efficiently estimate this weighting parameter for deep learning and significantly reduce the computational requirements compared to a grid search.

heading: 1 Introduction
text: Knowledge distillation, most commonly known from Hinton et al. (2015), is a procedure to transfer knowledge from one neural network (teacher) to another neural network (student).1 Often the student has fewer parameters than the teacher, and the procedure can be seen as a model compression technique. Originally, the distillation procedure achieves the knowledge transfer by training the student network using the original training targets, denoted as ground-truth targets, as well as a softened distribution of logits from the (already trained and fixed) teacher network.2 Since the popularization of knowledge distillation by Hinton et al. (2015), the idea of knowledge distillation has been extended to a variety of settings.3 This paper will focus on the special case where the teacher and student are of identical architecture, called self-distillation, and where the aim is to improve predictive performance, rather than compressing the model.
The idea of self-distillation is to use outputs from a trained model together with the original targets as new targets for retraining the same model from scratch. We refer to this as one step of self-distillation, and one can iterate this procedure for multiple distillation steps (see Figure 1). Empirically, it has been shown that this procedure often generalizes better than the model trained merely on the
1Often knowledge distillation is also referred to under the name Teacher-Student learning. 2We will refer to the weighted outputs of the penultimate layer, i.e. pre-activation of the last layer, as logits. 3See Section 2 for a brief overview, or see Wang and Yoon (2020) for a more exhaustive survey
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
original targets, and achieves higher predictive performance on validation data, despite no additional information being provided during training (Furlanello et al., 2018; Ahn et al., 2019; Yang et al., 2018).
Modern deep neural networks are often trained in the over-parameterized regime, where the amount of trainable parameters highly exceed the amount of training samples. Under simple first-order methods such as gradient descent, such large networks can fit any target, but in order to generalize well, such overfitting is usually undesirable (Zhang et al., 2017; Nakkiran et al., 2020). Thus, some type of regularization is typically imposed during training, in order to avoid overfitting. A common choice is to add an `2-regularization4 term to our objective function, which has been shown to perform comparably to early-stopping gradient descent training (Yao et al., 2007). However, in the theoretical study of the over-parameterized regime, regularization is often overlooked, but recent results have shown a connection between wide neural networks and kernel ridge regression through the Neural Tangent Kernel (NTK) (Lee et al., 2019, 2020; Hu et al., 2019). We briefly elaborate on this connection in Section D, which motivates our problem setup and connection to deep learning in Section 5.
Our Contributions Through a theoretical analysis we show that
• the solution at any distillation step can easily be calculated as a function of the initial fit, and infinitely many steps of self-distillation (with fixed distillation weight) correspond to solving the usual kernel ridge regression problem with a specific amplified regularization parameter when the distillation weight is non-zero,
• for fixed distillation weights, self-distillation amplifies the regularization at each distillation step, and the ground-truth targets dampen the sparsification and regularization of the selfdistilled solutions, ensuring non-zero solutions for any number of distillation steps,
• the optimal distillation weight has a closed form solution for kernel ridge regression, and can be estimated efficiently for neural networks compared to a grid search.
Proofs of all our results can be found in Supplementary Material A, and code to reproduce our illustrative example in Section 4.5. Experimental results in Section B can be found at github.com/Kennethborup/self_distillation.

heading: 2 Related Work
text: The idea of knowledge distillation dates back to Bucila et al. (2006), and was later brought to the deep learning setting by Ba and Caruana (2014) and more recently popularized by Hinton et al. (2015) in the context of compressing neural networks. Since the original formulation, various extensions have been proposed. Some approaches focus on matching the teacher and student models on statistics other than the distribution of the logits, such as intermediate representations (Romero et al., 2015), spacial attention maps (Zagoruyko and Komodakis, 2019), Jacobians (Srinivas and Fleuret, 2018), Gram matrices (Yim et al., 2017), or relational information between teacher outputs (Park et al., 2019). Other extensions focus on developing the transfer procedure, such as self-distillation (Furlanello et al., 2018), data-free distillation (Lopes et al., 2017; Nayak et al., 2019; Micaelli and Storkey, 2019; Chen et al., 2019; Fang et al., 2019), data distillation (Radosavovic et al., 2018), residual knowledge distillation (Gao et al., 2020), online distillation (Anil et al., 2018) or contrastive distillation (Ahn et al., 2019; Tian et al., 2020a).
4With slight differences, `2 regularization is often referred to as weight decay and ridge regularization in deep learning and statistical learning literature, respectively. See e.g. Loshchilov and Hutter (2019) for details.
The practical benefits of knowledge distillation have been proven countless of times in a variety of settings, but the theoretical justification for knowledge distillation is still highly absent. Hinton et al. (2015) conjecture that the success of knowledge distillation should be attributed to the transfer of dark knowledge (e.g. inter-class relationships revealed in the soft labels). Müller et al. (2019); Tang et al. (2020) support this conjecture, and argue that knowledge distillation is similar to performing adaptive label smoothing weighted by the teacher’s confidence in the predictions. Dong et al. (2019) show the importance of early stopping when training over-parameterized neural networks for distillation purposes by arguing that neural networks tend to fit informative and simple patterns faster than noisy signals, and knowledge distillation utilizes these simple patterns for knowledge transfer. Abnar et al. (2020) empirically investigate how knowledge distillation can transfer inductive biases between student and teacher models, and Gotmare et al. (2019) empirically shows how the dark knowledge shared by the teacher mainly is disbursed to some of the deepest layers of the teacher.
To the best of our knowledge, few papers investigate knowledge distillation from a rigorous theoretical point of view, and those that do, do so with strong assumptions on the setting. Phuong and Lampert (2019) ignore the ground-truth targets during distillation and furthermore assume linear models. Mobahi et al. (2020) investigate self-distillation in a Hilbert space setting with kernel ridge regression models where the teacher is trained on the ground-truth targets, and the student (and subsequent iterations) is only trained on the predictions from the teacher without access to the ground-truth targets. They show that self-distillation progressively limits the number of basis functions used to represent the solutions, thus eventually causing the solutions to underfit. In this paper, we build on the theoretical results of Mobahi et al. (2020), but we include the weighted ground-truth targets in the self-distillation procedure, where we allow the weight to depend on the self-distillation step, and show how this drastically affects the behavior and effect of self-distillation.5

heading: 3 Problem Setup
text: Notation Vectors and matrices are denoted by bold-faced letters; vectors are column vectors by default, and for a vector a let [a]i be the i-th entry and for a matrix A let [A]i,j be the (i, j)-th entry. Let In denote the identity matrix of dimension n, [k] = {1, 2, . . . , k}, and let ‖·‖2 and ‖·‖F denote the `2-norm and the Frobenius norm, respectively. Finally, for a function h : Rn → Rd and X ∈ Rm×n, we denote by h(X) the Rm×d matrix of outcomes, where the i’th row of h(X) is the function applied to the i’th row of X, i.e. [h(X)]i,· = h(xi).
Consider the training dataset D ⊆ Rd ×R, and let X = {x | (x, y) ∈ D} and Y = {y | (x, y) ∈ D} denote the inputs and targets, respectively. Let X = [xi]i∈[n] ∈ Rn×d be the matrix of inputs, y = [yi]i∈[n] the vector of targets, and X̃ ∈ Rm×d, ỹ ∈ Rm be the matrix and vector of validation inputs and targets, respectively. Given a feature map ϕ : Rd → V , where V has dimension D, we denote by K = κ(X,X) = [κ(xi,xj)]ni,j=1 ∈ Rn×n, where κ(xi,xj) = 〈ϕ(xi), ϕ(xj)〉, the symmetric kernel (Gram) matrix associated with the feature map ϕ.6

heading: 3.1 Self-Distillation of Kernel Ridge Regressions
text: In order to avoid overfitting our training data, we will impose a regularization term on our weights, and thus investigate the kernel ridge regression functions f ∈ F mapping f : X → Y , to construct a solution which best approximates the true underlying data generating map and generalize well to new unseen data from this underlying map. We consider self-distillation in the kernel ridge regression setup; i.e. consider the (self-distillation) objective function
Ldistill(f(X,β),y1,y2) = α
2 ‖f(X,β)− y1‖22 + 1− α 2 ‖f(X,β)− y2‖22 + λ 2 ‖β‖22 , (1)
where α ∈ [0, 1], λ > 0, y1,y2 ∈ Rn and f(X,β) = ϕ(X)β. The objective in (1) is a weighted sum of two Mean Square Error (MSE) objective functions with different targets7 and an `2-regularization
5In Supplementary Material E we relate our problem setup to Mobahi et al. (2020) and extend some of our results to a constrained optimization setting with a regularization functional in Hilbert space.
6Since the kernel trick makes the predictions depend only on inner products in the feature space, it is not a restriction if D is infinite. However, for ease of exposition we assume D is finite.
7It is straightforward to verify that minimizing (1) and the classic MSE objective with a weighted target, i.e. L̃distill(f(X,β),y1,y2) = 12 ‖f(X,β)− (αy1 + (1− α)y2)‖ 2 2 + λ 2 ‖β‖22, are equivalent and that the objective functions are equal up to the additive constant α(α− 1) ‖y1 − y2‖22.
on the model weights. Minimization of (1) w.r.t. β is straightforward and yields the minimizer
β̂ def = argmin β Ldistill(f(X,β),y1,y2) = ϕ(X)ᵀ (K+ λIn)−1 (αy1 + (1− α)y2) (2)
by Woodbury’s matrix identity and definition of K. This solution can also be seen as a direct application of the Representer Theorem (Schölkopf et al., 2001). Let y(0) def= y, i.e. the original targets, and recursively define for the steps τ ≥ 1,
β̂(τ) def = argmin β Ldistill(f(X,β),y,y(τ−1)) (3)
= ϕ(X)ᵀ (K+ λIn) −1 ( α(τ)y + (1− α(τ))y(τ−1) ) ,
f(x, β̂(τ)) def = ϕ(x)ᵀβ̂(τ) (4) = κ(x,X)ᵀ (K+ λIn) −1 ( α(τ)y + (1− α(τ))y(τ−1) ) ,
y(τ) def = f(X, β̂(τ)), (5)
for fixed α(τ) ∈ [0, 1]. Notice, the initial step (τ = 1) corresponds to standard training by definition and as such is independent of α(1). Self-distillation treats the weighted average of the predictions, y(1), from this initial model on X, and the ground-truth targets, y, as targets. This procedure is repeated as defined in (3)-(5) and we obtain the self-distillation procedure as illustrated in Figure 1. Note, the special cases α(τ) = 0 and α(τ) = 1 correspond to merely training on the predictions from the previous step, and only training on the original targets, respectively. Thus, α(τ) = 1 is usually not of interest, as the solution is equal to a classical kernel ridge regression, and self-distillation plays no role in this scenario. We will often consider the special case of equal weights, α(2) = · · · = α(τ) = α, and if α = 0 this corresponds to the setting investigated in Mobahi et al. (2020) in a slightly different setup. Thus, some of the following results can be seen as a generalization of Mobahi et al. (2020) to step-wise and non-zero α.

heading: 4 Main Results
text: In this section we present our main results for finitely and infinitely many distillation steps along with a closed form solution for the optimal α(τ) as well as an illustrative example highlighting the effect of the chosen sequence of (α(t)) on the solutions.

heading: 4.1 Finite Self-Distillation Steps
text: Our first result, which follows from straightforward computations, states that the predictions obtained after any finite number of distillation steps can be expressed directly as a function of y and the kernel matrix K calculated at the initial fit (τ = 1).
Theorem 4.1. Let y(τ), β̂(τ), and f(·, β̂(τ)) be defined as above. Fix α(2), . . . , α(τ) ∈ [0, 1), and let η(i, τ) def= ∏τ j=i ( 1− α(j) ) , then for τ ≥ 1, we have that
y(τ) = ( τ∑ i=2 α(i)η(i+ 1, τ) ( K (K+ λIn) −1 )τ−i+1 + η(2, τ) ( K (K+ λIn) −1 )τ) y, (6)
f(x, β̂(τ)) = α(τ)f(x, β̂(1)) + (1− α(τ))f(x, β̂(τ)α=0) (7)
for any x ∈ Rd, where β̂(τ)α=0 is the minimizer in (3) with α(τ) = 0. Since (6) and (7) are expressed only in terms of K, (K + λIn)−1, κ(x,X), and y we are able to calculate the predictions for the training data as well as for any x ∈ Rd based merely on the initial fit (τ = 1) without the need for any additional fits. Hence, despite the calculations of K, κ(x,X), and especially (K+ λIn)−1 being (potentially) highly computationally demanding, when obtained, we can calculate any distillation step directly by the equations in Theorem 4.1. Furthermore, predictions at step τ can be seen as a weighted combination of two classical ridge regression solutions, based on the original targets and the predicted targets from step τ − 1, respectively. However, choosing
appropriate α(t) for t = 2, . . . , τ is non-trivial. We explore these dynamics in Section 4.3 and 4.4. First, we use Theorem 4.1 to analyse the regularization that self-distillation progressively impose on the solutions.

heading: 4.2 Effective Sparsification of Self-Distillation Solutions
text: We now show that we can represent the solutions as a weighted sum of basis functions, and that this basis sparsifies when we increase τ , but also that the amount of sparsification depends on the choice of α. A similar sparsification result for the special case of fixed α(τ) = 0 for τ ≥ 1 was proved in Mobahi et al. (2020), and in particular, our (13) generalizes equation (47) in their paper.
Using the spectral decomposition of the symmetric matrix K we write K = VDVᵀ, where V ∈ Rn×n is an orthogonal matrix with the eigenvectors of K as rows and D ∈ Rn×n is a non-negative diagonal matrix with the associated eigenvalues in the diagonal. Inserting the diagonalization yields
K(K+ λIn) −1 = VDVᵀ(VDVᵀ + λIn) −1 (8)
= VD (D+ λIn) −1 Vᵀ, (9)
where λ > 0. By straightforward calculations using (6) and (9) we have
y(τ) = VB(τ)Vᵀy, where (10)
B(τ) def = τ∑ i=2 α(i)η(i+ 1, τ)Aτ−i+1 + η(2, τ)Aτ , and A def= D(D+ λIn)−1, (11)
and A, B(τ) ∈ Rn×n are diagonal matrices for any τ . Furthermore, by (10) the only part of the solution depending on τ is the diagonal matrix, B(τ), and in the following we show how B(τ) determines the effective sparsification of the solution f(·, β̂(τ)).
Lemma 4.2. Let B(τ), and A be defined as above, and let B(0) def= In. Then we can express B(τ) recursively as
B(τ) = A ( (1− α(τ))B(τ−1) + α(τ)In ) , (12)
and [B(τ)]k,k ∈ [0, 1] is (strictly) decreasing in τ for all k ∈ [n] and τ ≥ 1 if α(2) = · · · = α(τ) = α. Similarly to (10), if we use Lemma 4.2 and Theorem 4.1, we can show that for any x ∈ Rp
f(x, β̂(τ)) = κ(x,X)ᵀVD−1B(τ)Vᵀy
= p(x)ᵀB(τ)z, where (13)
p(x) def = D−1Vᵀκ(x,X), and z def= Vᵀy.
Thus, the solution f(·, β̂(τ)) can be represented as a weighted sum of some basis functions, where the basis functions are the components of the orthogonally transformed and scaled basis p(x), and z is an orthogonally transformed vector of targets.
Now assume α(2) = · · · = α(τ) = α for any τ ≥ 2 for the remaining of this section. In the following we show how the behaviour of B(τ), and in turn also the behaviour of f(·, β̂(τ)), with τ is dependent on the choice of α. Lemma 4.2 not only provides a recursive formula for B(τ), but also shows that each diagonal element of B(τ) is in [0, 1] and is strictly decreasing in τ , which in turn implies that the self-distillation procedure progressively shrinks the coefficients of the basis functions. Using Lemma 4.2 we can now show, that not only does B(τ) decrease in τ , smaller elements of B(τ) shrink faster than larger elements for α = 0, as we elaborate on below the theorem.
Theorem 4.3. For any pair of diagonals of D, i.e. dk and dj , where dk > dj , we have for all τ ≥ 1,
[B(τ)]k,k [B(τ)]j,j =  1+ λdj 1+ λdk , for α = 1,( 1+ λdj 1+ λdk )τ , for α = 0,
(14)
and if we let sgn(·) denote the sign function8, then for α ∈ (0, 1) we have that
sgn ( [B(τ)]k,k [B(τ)]j,j − [B (τ−1)]k,k [B(τ−1)]j,j ) = sgn ((( [B(τ−1)]k,k [B(τ−1)]j,j − [A]k,k [A]j,j ) [A]j,j [B(τ−1)]k,k([A]k,k − [A]j,j) + 1 )−1 − α ) . (15)
If we consider a pair of diagonals of D, where dk > dj , then for α = 0, the fraction [B(τ)]k,k [B(τ)]j,j
is strictly increasing in τ , due to the r.h.s. of (14) inside the parenthesis being strictly larger than 1. Hence, the diagonals corresponding to smaller eigenvalues shrink faster than the larger ones as τ increases. However, for α ∈ (0, 1) we can not ensure this behaviour, but at step τ we are able to predict the behaviour at step τ + 1, by using (15). Thus, when we include the ground-truth targets in our distillation procedure we do not consistently increase the regularization with each distillation step, but can potentially obtain a solution which does not sparsify any further. We now turn our attention to the question of how to pick the α(τ)’s in an optimal manner, and find that it can be done if we relax the condition that the weights are restricted to the interval [0, 1].

heading: 4.3 Closed Form Optimal Weighting Parameter
text: Recall, X̃ ∈ Rm×d is the matrix of validation inputs and ỹ ∈ Rm the vector of validation targets. If we allow α(τ) ∈ R, we can find an optimal α(τ) (which is a non-trivial function of λ) at each step τ , denoted by α?(τ).9 Here, optimal denotes the value for which the validation MSE is minimized. Note, α?(τ) is optimal for a single distillation step, but not necessarily so for multiple distillation steps, however we may consider α?(τ) a greedy estimate of the optimal value across multiple steps.
Theorem 4.4. Fix τ ≥ 2, λ > 0 and α(2), . . . , α(τ−1) ∈ R, then
α?(τ) = argmin α(τ)∈R ∥∥∥ỹ − f(X̃, β̂(τ))∥∥∥2 2 = 1−
( ỹ (τ) α=0 − ỹ(1) )ᵀ ( ỹ − ỹ(1) ) ∥∥∥ỹ(τ)α=0 − ỹ(1)∥∥∥2
2
(16)
where ỹ(1) = f(X̃, β̂(1)), and ỹ(τ)α=0 = f(X̃, β̂ (τ) α=0).
Since neither ỹ(1) nor ỹ(τ)α=0 depend on the choice of α (τ), we can calculate α?(τ) recursively as presented in Algorithm 1, where α?(τ) has the closed form in (16). In combination with the diagonalization results of Section 4.2 we can efficiently calculate the solutions. This should be compared to performing grid-search for α with g equidistant values on [0, 1] in order to approximate the optimal α, which requires g(τ − 1) + 1 model fits if one uses the same α for each sequence of τ ≥ 2 steps (gτ−1 if α is not fixed across distillation steps). However, by Algorithm 1 it is sufficient to perform 2(τ − 1) + 1 model fits, and obtain the exact optimal value at each step instead of an approximated value. In Section 5 we apply Algorithm 1 to approximate α?(τ) in a deep learning setting.

heading: 4.4 Infinite Number of Self-Distillation Steps
text: We now prove that if we were to perform an infinite number of distillations steps (τ →∞) with a fixed α (i.e. α(2) = · · · = α(τ) = α) the solution would solve the classical kernel ridge regression problem, with an amplified regularization parameter (by α−1) if α > 0. Observe that, when α = 0 and τ →∞, (6) and (7) yield that the predictions y(∞) and f(x, β̂(∞)) collapse to the zero-solution for any x ∈ Rp as expected from Mobahi et al. (2020).
8Note, we use the definition of sgn(·) where sgn(0) def= 0. 9If α?(τ) /∈ [0, 1], the sign of either the first or second term of (1) becomes negative, indicating either too strong or weak regularization of the previous distillation step, and one might fear this affects distillation performance. However, simply clipping of α?(τ) to be in [0, 1] alleviates this, at the cost of requiring a larger τ .
Theorem 4.5. Let y(τ), β̂(τ), and f(·, β̂(τ)) be defined as above, and α ∈ (0, 1], then the following limits hold
y(∞) def = lim τ→∞ y(τ) = K
( K+ λ
α In
)−1 y (17)
f(x, β̂(∞)) def = lim τ→∞ f(x, β̂(τ)) = αf(x, β̂(1)) + (1− α)f(x, γ̂(∞))
where (17) corresponds to classical kernel ridge regression with amplified regularization parameter λ/α, and we let γ̂(∞) denote the kernel ridge regression weights associated with solving another kernel ridge regression on the targets y(∞) with regularization parameter λ. Furthermore, the convergence limτ→∞ y(τ) is of linear rate.
Algorithm 1: Calculate β̂(τ) and α?(τ) for τ ≥ 2. Calculate β̂(1) from (3) (with any α(1)); Calculate ỹ(1) = f(X̃, β̂(1)); for t = 2 to τ do
Calculate β̂(t)α=0 from (3) and ỹ (t) α=0 = f(X̃, β̂ (t) α=0); Solve:
α?(t) = argmin α∈R ∥∥∥ỹ − (αỹ(1) + (1− α)ỹ(t)α=0)∥∥∥2 2 ;
Calculate β̂(t) from (3) with α?(t); end
If α > 0, then by (9) and Theorem 4.5, we have that y(∞) =∑p j=1 vj dj dj+ λ α
vᵀj y and we shrink the eigenvectors with smallest eigenvalues, corresponding to the directions with least variance, the most. Furthermore, if α > 0 the limiting solution is a non-zero kernel ridge regression with regularization parameter λ/α ≥ λ, causing the eigenvectors associated with the smallest eigenvalues to shrink even more than in the original solution.
Our results gives a theoretical explanation for why one should treat α(τ) as an adjustable hyperparameter to fine-tune the amount of regularization that self-distillation impose for a particular problem, and that it can be chosen in an optimal way for kernel ridge regression. In the following we provide an illustrative example, and in Section 5 we estimate the optimal weighting parameter for deep learning using an adaptation of Algorithm 1.

heading: 4.5 Illustrative example
text: Consider the training dataset D where X = {0, 0.1, . . . , 0.9, 1} and Y = {sin(2πx) + ε | x ∈ X}, and ε is sampled from a zero-mean Gaussian random variable with standard deviation 0.5. Let ϕ be the Radial Basis Function kernel, i.e. κ(xi,xj) = e−γ‖xi−xj‖ 2 2 , where we choose γ = 180 , and let λ = 0.2 and consider the three cases; (a) α = 0, (b) α = 0.25, and (c) step-wise optimal α?(τ).
As illustrated in Figure 2a for case (a), the regularization imposed by self-distillation initially improves the quality of the solution, but eventually overregularize and the solutions underfit the data, and will eventually converge to the zero-solution (see supplementary materials for the loss values). Using α > 0 (see Figure 2b), and more specifically α = 0.25, reduce the imposed regularization and increases the stability of the distillation procedure; i.e. the solutions differ much less between each distillation step. This allows for a more dense exploration of solutions during iterated distillation steps, where increasing α reduces the difference between solutions from two consecutive steps, but
also reduces the space of possible solutions as the limit, f(·, β̂(∞)), approaches the initial solution f(·, β̂(1)) quickly.10. However, choosing the step-wise optimal α?(τ) yields minuscule changes to the solution for τ > 2, and a single step of distillation is effectively enough. Furthermore, for τ ≥ 3, all α?(τ) are approximately equal, and the distillation procedure has reached an equilibrium.11
As expected from Lemma 4.2 and Theorem 4.3, Figure 3 verifies that both in case (a) and (b), the diagonal of B(τ) is decreasing in τ and the diagonal coordinates corresponding to smaller eigenvalues shrink faster than those corresponding to larger eigenvalues in case (a). Without loss of generality we can assume d1 < d2 < · · · < dn, and for k = 1, . . . n−1 and any τ ≥ 1 define R(τ)k def = [B(τ)]k+1,k+1 [B(τ)]k,k
. We expect R(τ)k to be strictly increasing in τ for all k in case (a), but for case (b) we can make no such guarantee. Both of these properties are verified in Figure 4.
Finally, we observe that in case (a), the values of B(τ) shrink much faster than in case (b), and eventually collapse to all zeros, whereas the latter is nearly converged after six iterations. Furthermore, case (a) appear to obtain a more sparsified solution, as the smallest coordinates effectively diminishes, which is not true for case (b). Furthermore, when directly comparing solutions from both cases with similar quality of fit, the solutions obtained with α = 0 usually has smaller coordinates in B(τ) than those obtained with larger values of α.

heading: 5 Approximate Optimal Weighting Parameter for Deep Learning
text: The following experiment aim at empirically evaluating the theoretical analysis above in a simple deep learning setting. In (16) we find α?(τ) on closed form when f(·, β̂(τ)) is a (self-distilled) kernel ridge regression. No closed form solution can be found for neural networks, but recent results show that (very) wide neural networks can be seen as kernel ridge regression solutions with the neural tangent kernel (Jacot et al., 2018; Arora et al., 2019; Lee et al., 2019, 2020).
10As expected by Theorem 4.5, we experience a fast convergence to the limit; usually less than 10 iterations are sufficient to converge
11If we clip α?(τ) to be in [0, 1], the α?(τ) converges at τ = 4 rather than τ = 3.
Thus, inspired by (7) we propose to estimate α?(t) for t = 2, . . . , τ , denoted by α̂(t), for a neural network trained with self-distillation using an adapted Algorithm 1. Let fnn(·,θ) ∈ Rp be a neural network with vector of weights θ, and recursively for τ ≥ 1 let θ̂(τ) be the weights solving
argmin θ
α(τ)
2 ∥∥∥fnn(X,θ)−Y(1)∥∥∥2 F + 1− α(τ) 2 ∥∥∥fnn(X,θ)−Y(τ−1)∥∥∥2 F + λ 2 ‖θ‖22 , (18)
with α(τ) = α̂(τ) and where Y(τ) ∈ Rn×p.12 Furthermore, let θ̂(τ)α=0 be the weights associated with minimizing (18) with α(τ) = 0, and Ỹ(τ)α=0 def = fnn(X̃, θ̂ (τ) α=0) as well as Ỹ (τ) def= fnn(X̃, θ̂ (τ)) be the predictions on the validation input X̃. Then, following Algorithm 1 with ‖·‖2 replaced by ‖·‖F , and (18) rather than (1) we can calculate the estimates α̂(t). These estimates yield comparable predictive performance to the best fixed α(τ) (found with time-consuming grid search), but only require one additional model fit per distillation step; i.e. 2(τ − 1) + 1 fits compared to g(τ − 1) + 1 for a grid search over g values. See Figure 5 for results and supplementary material for experimental details.

heading: 5.1 Experiment
text: We perform self-distillation with ResNet-50 (He et al., 2016) networks on CIFAR-10 (Krizhevsky and Hinton, 2009), with minor pre-processing and augmentations. The model is initialized randomly at each step13 and trained according to the above with either estimated optimal parameters, α̂(τ), or fixed α for all steps. We use the network weights from the last iteration of training at each distillation step for the next step, irrespective of whether a better model occurred earlier in the training. Our models are trained for a fixed 75 epochs and each experiment is repeated with 4 different random seeds over 11 chains of distillation steps, corresponding to α ∈ {0.0, 0.1, . . . , 0.9} and α̂(τ), with the first model initialized identically across all chains. The accuracy reported at the τ ’th step is based on comparing the training and validation predictions, Y(τ) and f(X̃, β̂(τ)) with the original training and validation targets; Y and Ỹ.14

heading: 6 Conclusion
text: In this paper, we provided theoretical arguments for the importance of weighting the teacher outputs with the ground-truth targets when performing self-distillation with kernel ridge regressions along with a closed form solution for the optimal weighting parameter. We proved how the solution at any (possibly infinite) distillation step can be calculated directly from the initial distillation step, and that self-distillation for an infinite number of steps corresponds to a classical kernel ridge regression solution with amplified regularization parameter. We showed both empirically and theoretically that
12We treat class labels as p-dimensional one-hot encoded vectors and use norm of the difference between the predicted class probabilities and the one-hot vectors.
13Note, we initialize the models equally across all α for one experiment, but alter the seed for initialization between experiments.
14The empirical experiments are constrained by the theoretical set-up and performed in a highly simple setting; e.g. using the weighted MSE loss from (18) (see supplementary materials for more details). Therefore, our accuracy measures are to be expected to be lower than for more fine-tuned training setups.
the weighting parameter α determines the amount of regularization imposed by self-distillation, and empirically supported our results in a simple deep learning setting.

heading: 6.1 Future Research Directions
text: Interesting directions of future research are on rigorously connecting neural networks and kernel methods in a knowledge distillation setting, extend to other objective functions than MSE as well as including intermediate model statistics in the distillation procedure. Finally, a larger empirical study of the connection between the choice of α and the degree of overfitting is interesting as well.
Acknowledgments and Disclosure of Funding We would like to thank GenomeDK and Aarhus University for providing computational resources that contributed to these research results. Furthermore, we would like to thank Daniel Borup and Ragnhild Ø. Laursen for comments and discussion, as well as Google Researcher Hossein Mobahi and Mehrdad Farajtabar (Deepmind) for clarifications on their experimental setup. We also thank the anonymous reviewers of the NeurIPS 2021 conference for their comments. Kenneth Borup is partly financed by Aarhus University Centre for Digitalisation, Big Data and Data Analytics (DIGIT).

Review summaries:
This paper studies the theory of self-distillation in the kernel ridge regression setting. The authors extend the theoretical results in Mobahi et al. (2020) to further incorporate the ground-truth labels in the distillation objective and highlight that the ground-truth labels serve to dampen the sparsification and regularization of the self-distilled solutions.

This paper presented a theoretical and empirical study of weighting the teacher outputs and the target outputs during self distillation. For an iterative self distillation process, a solution is shown to compute the fit at any step of the iteration given the first model. This was further related to classical kernel ridge regression with regularization.