Feature Kernel Distillation
Authors: Bobby He
Mete Ozay
Venue: ICLR 2022 Poster (2022)
Keywords: Knowledge distillation
Neural Network (NN) Feature learning
ensembling NNs
Deep learning fundamentals
Image classification

Abstract: Trained Neural Networks (NNs) can be viewed as data-dependent kernel machines, with predictions determined by the inner product of last-layer representations across inputs, referred to as the feature kernel. We explore the relevance of the feature kernel for Knowledge Distillation (KD), using a mechanistic understanding of an NN’s optimisation process. We extend the theoretical analysis of Allen-Zhu & Li (2020) to show that a trained NN’s feature kernel is highly dependent on its parameter initialisation, which biases different initialisations of the same architecture to learn different data attributes in a multi-view data setting. This enables us to prove that KD using only pairwise feature kernel comparisons can improve NN test accuracy in such settings, with both single & ensemble teacher models, whereas standard training without KD fails to generalise. We further use our theory to motivate practical considerations for improving student generalisation when using distillation with feature kernels, which allows us to propose a novel approach: Feature Kernel Distillation (FKD). Finally, we experimentally corroborate our theory in the image classification setting, showing that FKD is amenable to ensemble distillation, can transfer knowledge across datasets, and outperforms both vanilla KD & other feature kernel based KD baselines across a range of standard architectures & datasets.

Published conference paper ICLR 2022
FEATURE KERNEL DISTILLATION
Bobby Hel.2i & Mete Ozay²
DDepartment of Statistics, University of Oxford
Samsung Research UK
ABSTRACT
Trained Neural Networks (NNs) can be viewed as data-dependent kernel ma- chines, with predictions determined by the inner product of last-layer represen- tations across inputs, referred to as the feature kernel. We explore the relevance of the feature kernel for Knowledge Distillation (KD), using mechanistic un- derstanding of an NN's optimisation process. We extend the theoretical analysis of Allen-Zhu & Li (2020) to show that trained NN's feature kernel is highly dependent on its parameter initialisation, which biases different initialisations learn different data attributes in multi-view data ting. This enables us to prove that KD using only pairwise feature kernel compar- isons improve NN test accuracy in such settings, with both single & ensemble teacher models, whereas standard training without KD fails to generalise. We further use our theory motivate practical considerations for improving student generalisation when using distillation with feature kernels, which allows us to pro- pose novel approach: Feature Kernel Distillation (FKD). Finally, we experimen- tally corroborate our theory in the image classification setting, showing that FKD is amenable to ensemble distillation, can transfer knowledge across datasets, and outperforms both vanilla KD & other feature kernel based KD baselines across range of standard architectures & datasets.
the same architecture
INTRODUCTION & BACKGROUND
A prevailing belief in the Deep Learning community is that feature learning, where data-dependent features are acquired during training, is crucial to explaining the empirical success of Neural Net- works (NNs) (Fort et al., 2020; Baratin et al., 2021). A comparison this regard is often made to kernel methods (Jacot et al., 2018), which can be thought of as feature selection methods from a fixed data-independent set of features. This separation has been caricaturised ture learning and kernel learning regimes (Chizat et al., 2019; Yang & Hu, 2020; Woodworth et al., 2020) of NN training. Though less amenable theoretical analysis compared to kernel regimes, feature learning regimes have the capability to capture more of the complex empirical phenomenon that
distinction between fea-
observe in NNs due parameter-space non-convexity, such i) how ensembling trained NNs differing solely in their independent parameter initialisations can lead to improvements in predictive accuracy & uncertainty (Lakshminarayanan ii) the effectiveness of knowledge distillation with both single & ensemble teacher models (Buciluă al., 2006; Hinton al., 2015). This implies that in order to understand ensembling & knowledge distillation (KD) in NNs, we need to understand the mechanisms of NN feature learning regimes.
al., 2017: Allen-Zhu & Li, 2020), or
Ensembling can loosely summarised as aggregating predictions from multiple models, used widely across machine learning (ML) to improve performance (Dietterich, 2000; Breiman, 2001). Conversely, knowledge distillation (KD), the idea of transferring knowledge from teacher model to student model, has gamered the most attention with NNs. Remarkably, via KD it is possible to
1 Researched during intersship
Samsung
Research
UK.
Correspondence
bobby, he@stats. ox. uk.

---

Published conference paper ICLR 2022
significantly improve single student's generalisation with knowledge from a teacher model, or an ensemble of teachers. This means that the single student's model has enough flexibility to generalise well (relative to the teacher), thus one must factor in the optimisation process (such as the parameter initialisation) in order to explain the mechanisms of ensembling and KD in NNs.
To describe KD, suppose we have N input-target (æt, E)R data pairs D={(ar; );) sampled i.i.d. (independent & identically distributed) from some distribution DD and a stu- dent NN architecture fx) Ws-hs(a; (2x 0s). Here hs(2 0I) is a student-specific feature extractor model (e.g. MLP, CNN, ResNet, or Transformer) with parameters Os. and Ws E x is a parameter matrix for the last layer. Assume also that we have loss: L( + (.fs(m.)),)) y:) which we seek minimise over 0s Ws in the hope that fs can generalise to unseen (z. 3) pairs. L is typically cross-entropy in the classification setting.
Vanilla KD (Hinton et al., 2015) distils knowledge from a trained teacher network fr(x)) (x. 0 to a student by regularising student is towards the teacher fy:
0s Ws) C(0s. Ws) + AKD
Σ N
1 L
N
fs(x.) fr(z)))
T
for temperature T>0 & regularisation Ax hyperparameters. Note, this is only valid if Cy=Cs- Following Hinton al. (2015), many methods have been proposed using different quirks of NNs to distil knowledge from teacher to student. A relevant line of work involves encouraging the student to match how similar/related the teacher views two inputs æ, x' to be (Passalis & Tefas, 2018; Tung & Mori, 2019; Park et al., 2019). These approaches have the benefit of being agnostic to teacher/student architectures & prediction spaces CT & Cs, but as of yet remain heuristically motivated. In this work, we explore such approaches under the more general framework of NN feature kernel (the kernel induced by the inner product of last-layer features h) learning, allowing us to provide the missing theoretical justification. Moreover,
use theoretical insights to introduce practical improvements for FKD in Section 4, which we show outperform these previous works in Section 5.
Allen-Zhu & Li (2020) provide the first theoretical exposition of the mechanisms by which vanilla KD and ensembling improve generalisation in NNs. To this end, the authors introduce the notion of multi-view data, which is when class in a multi-class classification problem has multiple identify- ing features/attributes. For example, an image of a car can be discerned by i) wheels, ii) windows, or iii) headlights. The key idea is that the NN parameter initialisation, and its random correlations with certain attributes, will bias the NN to learn only a subset of the entire set of attributes pertaining to a given class. When presented with single-view data lacking the llaas-identiffing attribute that the NN has learnt, the NN will not generalise. For example, an NN that has learnt to classify cars based on they have headlights will not generalise to a side-on image of a car that occludes headlights.
The implication then is that ensembling NNs works in part because independent parameter initiali- sations learn independent sets of attributes, so more data features will be learnt across the ensemble model. Moreover, it is argued that vanilla KD in NNs works because the features learnt by the teacher model (or models) are imparted to the student via soft teacher labels that capture ambiguity in given data input (such an image of a whose headlights look like the eyes of cat). This is fundamentally different to ensembling in strongly convex feature selection problems, such as linear or random features (Rahimi & Recht, 2007) models with regularisation. In such cases, different initialisations reach the same unique optimum, and additional noise must be added to ensure predic- tive diversity in the ensemble (Matthews et al., 2017). These analyses suggest that it is not possible to fully explain KD or ensembling in NNs without feature learning, thus motivating our study of Feature Kernel Distillation, where one performs KD on NN features directly.
Our contributions Feature learning can be thought of as when the feature kernel, induced by the inner product of last-layer representations in a NN, changes during training ( (Yang& & Hu, 2020), and kernel learning in NNs can be thought of as when this kernel is constant, In this work, we take
2

---

Published conference paper at ICLR 2022
feature learning perspective of knowledge distillation (KD). We first highlight the importance of the feature kemel by viewing trained NNs data-dependent kemel machines, & use this to motivate Feature Kerne Distillation (FKD). In FKD, we aim to ensure that the student's feature kernel is well suited for improved generalisation, using both the teacher's 's data-dependent feature kernel as well as an understanding of the student NN's optimisation process. In Section 3, we adapt the framework of Allen-Zhu & Li (2020) show that FKD offers the same generalisation benefits as found in vanilla KD in a multi-view data setting, and is further amenable to ensemble distillation. We then derive practical considerations from our insights in Section 4, improve FKD through an understanding of the NN's feature learning optimisation process, compared to previous methods which implicitly used the feature kernel for KD Finally, in Section 5, we provide experimental support that our theoretical claims extend to standard image classification settings, by: verifying that FKI is amenable to ensemble distillation; can transfer knowledge across datasets with different prediction spaces (unlike vanilla KD); and outperforms vanilla KD) & previous feature kernel based distillation methods over a range of architectures on CIFAR- 100 and ImageNet-IK.
2 MOTIVATION FOR FEATURE KERNEL DISTILLATION
One obvious limitation of vanilla KD that student fs and teacher fT need to share prediction spaces, i.e. Cs=Cr. In many situations, we may have teacher network trained on dataset with a different number of classes than the student's 's dataset, and it is not clear how one could apply vanilla knowledge distillation. One possibility could be to regularise directly in feature space by comparing hs and ht element-wise, but again this either requires same teacher-student last-layer sizes or additional projection layers.
Student A
A (2))
Distil
Teacher hT
(2%1) Pb)
Feature Kernel Distillation (FKD) from the feature ex- tractor of teacher A to that of student h.S
Figure 1:
To eschew such unnecessary complications, we take the perspective of NNs as data-dependent kernel machines. Define an NN's feature kernel to be:
Definition 1 (Feature Kernel). Suppose we have parameters O and last-layer NN feature extractor (1, 0):R + R. For two inputs Fi; B3 E Tkm,, the feature kernel is the kernel induced by the inner product of h(x:, 0) and h(x,, 0), that is: k(a;;,,) del (Max, h(,,( 0)).
At initialisation, it is well known that in the infinite NN-width limit, with appropriate scaling, the feature kernel converges almost surely to a deterministic kernel known as the Neural Network Gaussian Process (NNGP) kernel (Neal, 2012; Lee et al., 2018; Matthews et al., 2018; Yang. 2019). Yang & Hu (2020) show that there is a parameterisation-depeenddd feature learning regimes for infinite-wide during training. respectively. It has been widely demonstrated that a crucial component of the suc- cess of finite-width NNs is their ability to fiexibly learn features, and indeed the feature kernel, from data during training (Fort et al., 2020; Aitchison, 2020: Chen et al., 2020b; Maddox et al., 2021).
dichotomy between kernel & NNs, where the feature kernel k is constant or changes
To see the importance of the feature kernel, note that for fixed 0 with many common loss functions L. and some mild assumptions on strong convexity (which could be enforced e.g. with standard lg regularisation), the optimal W is uniquely determined and determines the entire predictive function (*). For example, with squared error, T. T.(f(x).)) y) = (=) - y and & regularisation strength À > 0, a trained NN is precisely kernel ridge regression with the data-dependent feature kernel k. whose job is to measure how similar different inputs are. Thus, all teacher knowledge is contained in its feature kernel, &T,, so the feature kernel can act as our primary distillation target, as depicted in Fig. 1. We show a corresponding result for cross-entropy loss in App. A.
Fig. 2 corroborates our claims. For ResNet20v1 (He et al., 2016) 'reference' model trained on CI- FARIO with cross entropy, we plot test class prediction confusion matrices between said model and:
3

---

Published conference paper ICLR 2022
i) a retrained version where all but the last-layer parameters are fixed (hence fully determined by the reference model's fea- ture kernel as per App. A), & ii) an independent model trained from a different initial- isation. As expected, there is significantly more disagreement across test predictions for mod- els with different initialisations than those which share feature kernels. This suggests: a) differ- ent initialisations bias the same architecture to learn different features, and b) the feature kernel (largely) determines a model's test predictions. Experimental details and breakdown of the predictive disagreements be found in App. F.
vs Retrained last layer
vs Insependent
Atternaee prediction
Azuenbou pozrcauuu
wattombble
prediction
prediction
Figure 2: CIFARIO test prediction confusion matrices between fixed reference model and model with: (left) retrained last layer, and (right) independent initialisation.
Having described the feature kernel as a central object in any NN, we use this to motivate our proposed FKD, where treat the teacher's feature kernel, A-,,, as key distillation target for the student's s feature kernel, A.S. Encouraging similarity across feature kernels shares useful features that the teacher has learnt with the student, which we theoretically show in Section 3.
We define the FKD student loss function via an additive regularisation term between feature kernels:
AAKD (0, W) C(O, W) + AED D(ke, kT)
where ARD > the regularisation strength, D is some (pseudo-)disse )distance over kernels, and the student feature kernel -. = ki is written to make explicit the dependence on student feature extrac- tor parameters We stress that Eq. (2) does require matching prediction (nor feature) spaces between teacher and student, allowing us to apply FKD across tasks, architectures, and datasets.
We consider Eq. (2) with to:
(kk, k kT) = 21 a0 H
(Ho x2) - TTT (x1, x1
with expectation approximated by an average over minibatch. In this work we choose = 2, so that vID gives the Frobenius norm of the difference in feature kernel gram matrices over batch.
THEORETICAL ANALYSIS FOR FKD
We now adapt the theoretical framework of Allen-Zhu & Li (2020), which is restricted to vanilla KD, to demonstrate the generalisation benefits of FKD over standard training. Note that FKD distils knowledge by comparing different data points, whereas vanilla KD compares single data point across classes: this core difference reflected throughout our analysis relative to Allen-Zhu & Li (2020). We first describe the multi-view data setting & CNN architecture recalling that standard training without KD fails to generalise well. We then provide our main theoretical result, Theorem 2, which shows that FKD improves student test performance. Though our thcoretical results are limited a specific scenario, inspired by real-world data (Allen-Zhi & Li, 2020), & NN architecture, yet rich enough to display the merits FKD. Moreover, we find in Section that our conclusions generalise to standard architectures & image datasets. In the interest of space & readability, we focus on providing intuition in this section, and fill in remaining details/proofs in the appendix.
consider, before
believe the setup consider is apt: is simple enough to be tractable,
Wee will sometimes drop the student S sub/superscript where obvious for elarity, like in Eq. (2). Any teacher specific object, e.g. Ae" will always have corresponding T sub/superscript.
4

---

Published conference paper ICLR 2022
Multi-view data. We consider the data classification problem introduced by Allen-Zhu & Li (2020), with C classes and inputs z with P patches each of dimension d, meaning For each class c, we suppose that there exist two attributes te,1; V,,2 E R.. For a belonging to class the attributes found in patches of æ will include v,,1 and 22,2, well out-of-class attributes { ( This denotes the multi-view nature of the data distribution. In the true view, which means that only one of ve,1 or 0e.2 is present æ when (æt y) is from class c. These will be the data for which standard training fails to generalise. A precise definition of multi-view data is presented in App. B.1.1. Allen-Zht Zhu & Li (2020) argue that this multi-viww setting provides a compelling proxy for standard image datasets such as CIFAR-10/10 Krizhevsky (2009).
(R")P.
random selection of
distribution D, we suppose that proportion of the data (æx, y) is single-
Intuition: FKD on multi-view data
Suppose have image classification task, with cat & car just two out of many classes. For the car class, 2e,1 could correspond to headlights, whilst 2222 could correspond to wheels. We would then expect Ve,1 also appear in patches of an input image. Zeat, corresponding to a cat with headlight-like eyes. Allen-Zhu & Li (2020) show that a single trained model is biased to learn exactly one of te,1 or 22,2, depending on its parameter initialisation. W.L.O.G, suppose that the student is biased to learn 22 & not tc.1. If the teacher model has learnt 2e,1, this means that the teacher model knows there a similarity between Zea & any car image, Tear. that displays headlights. Mathematically, we show that this corresponds to large value for ( Teaar) Our FKD regularisation forces the student to also have a large value for kg (Feat Fear). ensuring that attribute v is also learnt by the student network. Without distillation, student NN which has learnt 2 & not Ve,1 will not generalise to front-on images of cars that hide wheels.
Convolutional NN corresponding feature kernel. Like Allen-Zhu & Li (2020), for our theo- retical analyis we consider single hidden-layer convolutional NN (CNN) with sum-pooling." For each class c E [C]. we suppose that the CNN has m channels, giving Cm channels total. For channel r and class c, we suppose that we have weights Oer R This gives output for class c by
fa(
ReLLU((0ar) #:p))
where for ease of analysis ReLU ReLU-like but with continuous gradient, see App. B.2.
Before we consider FKD, we must first define the feature kernel for this CNN f. To do so, we recast
fxx==Wh((zz 0), where h(, 0)=RCCm and W CxCmm satisfying, for r E [m], c E [C], [C]:
h(,, 0 (e 1)m def Σ ReLU xp)), and W (o- Iym - 1fc= cd}.
(5)
Given that the feature kernel is k(x x²) del (h( h(r', 0)). we now have that:5
k(x,, x²)
ΣΣ Σ
ReLLU( #p)) (00,,
(6)
p-/==1
We first recall that standard training of the model f with gradient descent and cross entropy loss fails to generalise on half the proportion of data that is single-view.
It is straightforward extend to the of more than views per class if need be. 'ttpps: / /www. mi crosoftt com/ cow/coouo////ecc/ece www.e//sssaaaaaaeeeeeeeeeeeeeeae rehearcc/bll hwtphh//hhhhhhhhcchhhhhh og/threeeeeeeeeeeeeeeeee wtwttteeeeeeeeeeeeeeeeeeeee iisss-in-e deep-
ea rni moning-ensenn engembbmmmlll Te knowl edge- sistii i awwaaanddaaaaaaeeee self-di at illaiiinn/////
'It is straightforward to extend our analysis for max-pon max-poooing.
'The feature kernel defined in Eq. (6) corresponds to the Global Average Pooling CNN-GP kemel in Novak al. (2018) in the infinite-chanel limit, which captures intra-patch correlations unlike the vectorised CNN-GP, which corresponds to vectorising the spatial dimensions to give CmP rather than C'm channels.
5

---

Published conference paper ICLR 2022
Theorem (Standard training fails, Theorem of Allen-Zhu & (2020)). For sufficiently many classes C and channels E [polylog(C), C). with learning rate poly(Cy, training time
T poly(C), and multi-view data distribution (App. B.1.1), the trained model f(T*) satisfies with 9
probability least e (0Q(0))"())))
Training accuracy is perfect: For all (x,, y) E D, y = argmax, Se (x))
(T" (x)
Test accuracy bad but consistent: ((..)))> y + argmax, f(r") (x)) [0.49 0.51p].
Now we are ready to show that regularising the student model towards the teacher model's feature kernel, in FKD, improves test accuracy. We suppose teacher model is an ensemble of E models {fe 1em each with corresponding feature kernel ke, trained as standard on the same data with independent initialisations 0g. We average ke. over e [E] to obtain our teacher feature kernel:
k x') = E Σ ke (x,,
This is akin to concatenating all features in (he into a dim-diimniio feature vector, albeit without the additional computational baggage. We then have main theoretical result:
Theorem (FKD improves student generalisation and is better with larger ensemble). Given an ar- bitrary O. For any ensemble size E of teacher NNs trained as in Theorem I and sufficiently many classes C, for m polylog( C)), with learning rate n paay(Cy. and training time T polyiC). the ensemble teacher knowledge can be distilled into single student model f(T") uusing only teacher feature kernel kT, Eq. (7), such that with probability at least 1 -S0llog (C));
Training accuracy perfect: For all (a, y) E D, y = argmax, J! ((T"'()) (x).
Test accuracy good: P(..y)~ y # argmax, p(T") ((TT) (x)) s (5S + t)J..
Proof outline for Theorem 2. we first show, in Lemma 1, that single trained NN's feature kernel (which we defined Eq. (6)) can detect if two inputs share a data attribute that the NN has learnt due to its weight initialisation. We extend this result to an ensemble teacher in App. C.2, showing that the ensemble teacher feature kernel detects the union all data attributes learnt by individual [ke _ This simplifies our calculations when showing that our distillation regulariser, Eq. (3), is effective for improved student generalisation. The full proof can found App. C.
Intuition: FKD with ensemble of teachers
To parse Theorem 2, suppose we have a single teacher i.. E=1, Then, the test error is essentially
0.25μ in Theorem 2. The explanation is that both the student & teacher networks independently learn one of ( for each class c. Either vanilla KD (Theorem of Allen-Zhu & Li (2020)) feature kernel approach allow the student to learn the union of the independent attributes learnt by the student and the teacher, so for only a quarter of the single-view test data æ, will the student not have learnt the useful class attribute present in .... For general ensemble size E, the story is the same: the student & E teachers each independently learn one of the two useful attributes (ee for all E [C]. Distilling allows the student to learn the union of these attributes, which means that the student will fail on only ZहTT of the single-view data.
FKD IN PRACTICE
Next, we highlight practical considerations for implementing FKD derived from our theory. Pseudo- code and PyTorch-style code for our FKD implementation are given Algs. and respectively.

---

Published conference paper ICLR 2022
Correlation kernel.
We propose (ppo, as a regulariser in FKD instead of [Kk, KT), where: R3 (x, x²)
P=(2, def
and pT(x, def 1 E (e(,xz))
v (a
defines feature correlation kernel Pz. corresponding to feature kernel kz. for (0) The reason we use correlation kernels is that they normalise data, so that p(æ, x) Va. which zeros diagonal differences in Eq. (3): we hope FKD allows the student to learn from the teacher features shared between different inputs. Non-zero diagonal differences, like in Similarity-Preeerving (SP) KD (Tung Mori, 2019), encourage the student to learn noise
show in App. D.I, and we hypothesise that this contributes to the improved performance we observe of FKD over SP in Section 5. Moreover, this normalisation helps balance individual teacher's influence in ensemble teacher, and ensures that FKD does not need a temperature hyperparameter т, which produces soft labels in vanilla KD.
Feature regularisation. One downside using the cor- relation kernel is that our FKD regularisation. D(po. PT), becomes invariant to the scale of kg. For example, re-
FKD with FR FKD w.o. FR
aanogramm Density Histogr a
placing ko(ar, x²) with ((z))o x²), for any M (Rt) leaves the student correlation kernel un- changed. This may lead to degeneracies when training O, & large variations in k(ar, æ)) over may harm generalisa- tion (as evidenced by the fact that input normalisation is com- mon across MI, from linear models NNs). Moreover, our proof of Theorem 2 is not quantitative, in that we only show k0 x²) = (1) in the number of classes C' up to poly- logarithmic factor C, when xfズx share a data attribute Ver that has been learned by parameters O. These insights motivate us to regularise the student feature h during FKD training to control the norm of k<x, across inputs æ. We use an additive & regularisation
6.2
0.4
1.0
Normalised k(s, value
Figure 3: Histogram normalised fea- k(a..)
kernel values, mas,, xz)) over the CIFAR-100 test set.
I DE [1 1) E 25)) with regularisation strength >RR30, for each minibatch B
((rss, yb)] Fig. shows that using Feature Regularisation (FR) encourages a more even spread of k(x x) across inputs for a student VGG8 network trained with a VGG13 teacher model on CIFAR- 100. Corresponding plots for other architectures can found in Fig. Similar to Dauphin & Cubuk (2021), we find that FR improves generalisation & provide ablations in Section 5.
Algorithm Feature Kernel Distillation with SGD.
Require: Maximum number of iterations T", batch size B, learning rate y, teacher correlation kernel PT, FKD regularisation AXp > 0, Feature regularisation AFR > 0. Initialise student parameters 0o, Wo.
for iteration 1 = 0,... T do Sample minibatch (x. ) mmm
B
ltd. D.
2A Compute loss L Σe L(f(x (f) 2 B(B1)
Eit; (po, (x." (=,, x)))-p((a ax))
Add feature regularisation C = + ArB
De | 0.) (optional).
Update parameters oc+1 <- 0: - Vo W411 W. VWL.
end for return (0r- Wz-
EXPERIMENTS
Due to space concerns, App. F contains further experiments & any missing experimental details. Ensemble distillation. We first verify that larger ensemble teacher size, E, further im- proves FKD student performance suggested by Theorem 2. This is confirmed in Fig. 4, using VGG8 for all student & teacher networks on the CIFAR-100 dataset.
7

---

Published conference paper ICLR 2022
C:100 Test accuracy (9)
We also plot the test accuracy of the ensemble teacher across sizes E, whose predictive probabilities averaged over indi- vidual teachers, as well as the test accuracy of an undistilled stu- dent model. We see that FKD consistently outperforms vanilla KD, and both distillation methods outperform the teacher in the "self-distillation setting of E=1 (Furlanello et al., 2018; Zhang et al., 2019). Moreover, FKD allows a single student to match teacher performance when E-2, before positive but diminish- ing returns with larger E relative to the teacher ensemble.
KD
FKD
Teacher
Student
Teacher ensembile size
Figure 4: FKD as teacher ensemble size changes. Error bars denote 95% confidence for mean of 10 runs.
Dataset Transfer. We next show FKD can transfer knowl- edge across similar datasets. From a fixed VGG13 teacher network trained
Table Test accuracy (%) of FKD & baselines a dataset transfer dis- tillation setting. Error bars indicate 95% confidence for mean of 10 runs.
Dataset
Student
RKD
SP
FKD w.o. FR FKD
CIFAR- 100, we distil to student VGG8 NNs CIFAR-10, STL-10 & Tiny- ImageNet..... As no student dataset has 100 classes, unlike CIFAR-100, 100, it is not clear how one use vanilla KD (Hinton et al., 2015) in this case. We thus compare FKD other feature kernel based KD methods: Relational KD (RKD) (Park et al., 2019) & Similarity-Preserving (SP) KD (Tung& & Mori, 2019). In Table 1, we see that FKD without feature regularisation outper- forms both baselines across all datasets, and that feature regularisation (FR) further improves FKD performance, highlighting the benefit of our practical considerations in Section 4. The improved performance is particularly stark on STL-10 (which we downsize to 32x32 resolution), where FKD improves student performance by 2.75%. STL-10 well suited for FKD as has only 5K labeled in- puts but 100K unlabeled datapoints, which can be used in our feature kernel regulariser, D(po, PT).
C-100 - C-10
91.56
91.7 91.74.0.0
92 02.21
92.33
92.61
C-100 STL-10
72.69
g2.8
13.88
76.1710.00
75.44
C-100 Tiny-1
48.53
48.74g0.17
48.73g014
48.85
50.67.0:12
Table 2: CIFAR- 100 and ImageNet- I accuracies (%) comparing FKD with KD baselines, * denotes result from Tian et al. (2020); FKD uses the same teacher checkpoints provided by the authors," 6 with error bars denoting 95% confidence for the mean over students.
CIFAR-100
ImageNet-1K
Teacher
ResNet32244
VGG13
ResNet1224
ResNer50
RecNet34
Student
ResNet8x4
VGGS
ShuffleNerV
VGG8
ResNet18
Teacher#
79.42
74.64
79.42
79.34
73.26
Student
72.50
70.36
70.50
70.36
69.97
KD* (Hinton al., 2015)
23
72.98_0.17
74.072017
73.81
70.66
RKD* (Park et al., 2019)
71.90=0.10
71.48-0.04
72.28=0.84
71.50=0.00
NIA
SP (Tung Mori, 2019)
72.94_6.00
72.68_0.17
13.4820ar
73.3420.0
CRD (Tian et al., 2020)
75.5140.10
73.9440.10
75. 15.11
74.30
71.17
FKD w.o. FR FKD
74.89
73.08-0.16
74.66
73.99
70.84
75.3710.92
73.78go.17
75.00
74.61
71.23
Comparison on CIFAR-100 and ImageNet. Finally, we compare FKD to various knowledge distillation baselines CIFAR-100 and ImageNet, across selection of teacher/student architec- tures. We see in Table that FKD consistently outperforms: vanilla KD (Hinton et al., 2015), RKD (Park al., 2019), and SP (Tung & Mori, 2019). Moreover, FKD either matches the high-performing Contrastive Representational Distillation (Tian et al., 2020). We use the exact same teacher checkpoints used by Tian et al. (2020) and Chen al. (2021) for CIFAR-100 and Im- ageNet respectively to ensure fair comparison. We find, like in Table 1, that feature regularisation consistently improves FKD performance and that even without feature regularisation, FKD outper- forms all feature kernel based KD methods. This implies that using the correlation kernel to zero out diagonal differences, as described in Section 4, indeed helps improve student performance.
outperforms
"Apart from ImageNet-1 which used the pretrained ResNet34 from torchvision, like Chen al. (2021)
8

---

Published conference paper at ICLR 2022
RELATED WORK
NN Knowledge Distillation. Following Hinton et al. (2015), there has been much interest in ex- panding KD in NNs (Romero et al., 2014; Zagoruyko & Komodakis, 2016; Passalis Tefas, 2018; Zhang et al., 2018; Yu et al., 2019; Chen et al., 2020a; Tian et al., 2020). Most similar to FKD are Park et al. (2019); Tung & Mori (2019) who also use relations between inputs to distil knowledge (albeit not from the feature kernel learning perspective and without our theoretical justification), as well as Qian et al. (2020) who focus on reducing computational costs of full-batch kernel matrix operations. App. D highlights in more detail the differences of FKD compared to previous pairwise feature kernel based KD methods. Allen-Zhu & Li (2020) made the first theoretical connection us- ing the mechanisms of ensembling in NNs to explain the success of vanilla KD in NNs, which extend for feature kernel based KD.
Ensembling NNs. Ensembling NNs has long been studied for improving predictive accuracy (Hansen & Salamon, 1990; Krogh et al., 1995) with particular recent focus towards uncertainty quantification & Bayesian inference (Lakshminarayanan
al., 2017: Ovadia al., 2019; Zaidi et al.. 2020: Pearce et al.. 2020; He et al., 2020; Wilson & Izmailov, 2020; Wenzel et al., 2020; D' Angelo & Fortuin, 2021; Schut et al., 2021) and predictive diversity (Fort et al., 2019; D' D'Amour et al., 2020). On the topic of Bayesian inference, the feature kernel has also been studied under the name of Neural Linear Model (Riquelme et al., 2018; Ober & Rasmussen, 2019), and extensions treating features h inputs to standard Gaussian Process kernels are known by the name of Deep Kernel Learning (Wilson et al., 2016; Ober et al., 2021; van Amersfoort et al., 2021).
NN Feature learning. A recent flurry of work has focussed on characterising & understanding the importance of feature learning in NNs (Chizat et al., 2019; Fort et al., 2020; Baratin et al., 2021; Lee et al., 2020; Aitchison, 2020; Ghorbani et al., 2020). fuelled in part by the development that wide NNs become (Neural Tangent) Kernel machines in certain regimes (Jacot et al., 2018; Lee et al., 2019; Yang & Littwin, 2021). thus forgoing feature learning. The consensus in these works is that there are gaps between NTK theory & practical NNs that cannot be explained without feature learning. However, Yang & Hu (2020) proved that feature-learning is still possible with infinite- width NNs, and also that feature learning is equivalent to feature kernel learning in infinite-width NNs. This motivates our study of the feature kernel as key object for distillation. Regularising the feature kernel to the true target covariance kernel was suggested by Yoo et al. (2021).
CONCLUSION
We have theoretically shown that the feature kernel is valid object for Knowledge Distillation (KD) in Neural Networks (NNs) by extending the analysis of Allen-Zhu & Li (2020), which focused on vanilla KD (Hinton et al., 2015). Further, we used our thcoretical insights to motivate practical con- siderations when using feature kernels for distillation, such as using the feature correlation kernel & using feature regularisation, to improve on previous feature based KD methods. We term our ap- proach Feature Kernel Distillation (FKD), and note that FKD is more widely applicable than vanilla KD, as it benefits from being agnostic teacher and student prediction spaces. Experimentally. we have demonstrated that FKD is amenable to ensemble distillation as suggested by our theory, is able to transfer knowledge across similar datasets and that FKD outperforms vanilla KD & previous fea- ture kernel based KD methods across a variety of architectures on CIFAR-100 100, and ImageNet- IK.
Limitations & future work. Though feature learning is central to our results, we stress that there are still gaps between our theory & practice to understanding NN ensembling & KD, demonstrated by the divergence between ensemble teacher & FKD in Fig. 4 for larger ensemble size. This could be due to: the multi-view data setting not being able to capture the full complexity of real-world data, the role of hierarchical feature learning between layers in a deep NN, or the importance of mini- batching in stochastic gradient descent. Other future work could apply the multi-view data setting to analyse uncertainty quantification in NN ensembles, assess the impact of different FKD regulari- sation metrics in Eq. (3), or improve FKD further to compete with offe-of-theeaa KD methods.

---

Published conference paper ICLR 2022
We thank Emilien Dupont, Yee Whye Teh, Sheheryar Zaidi for helpful feedback this work.
Acknowledgements
REFERENCES
Laurence Aitchison. Why bigger is not always better: finite and infinite neural networks. In International Conference on Machine Learning, pp. 156-164. PMLR, 2020.
Zeyuan Alhen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning, arXiv preprint arXiv:2012.098/6, 2020.
Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoic, Pascal Vin- cent, and Simon Lacoste-Julien. Implicit regularization via neural feature alignment. Interna- tional Conference on Artificial Intelligence and Statistics, pp. 2269-2276 PMLR, 2021.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Pro- ceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535 501,-41,222 2006.
Defang Chen, Jian-Ping Ping Mei, Can Wang, Yan Feng, and Chun Chen. Online knowledge distillation with diverse peers. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 1430-3437, 2020a.
Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen. Cross- layer distillation with semantic calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7028-7036, 2021.
Shuxiao Chen, Hangfeng He, and Weijie Su. Label-aware neural tangent kerne:: Toward better gen- eralization and local elasticity. Advances in Neural Information Processing Systems, 33, 2020b.
Lénaïc Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in Neural Information Processing Systems, 32:2937-2944 2019.
Alexander D' Amour, Katherine Heller, Dan Moldovan, Ben Adlam, Babak Alipanahi, Alex Beu- tel, Christina Chen, Jonathan Deaton, Jacob Eisenstein. Matthew Hoffman, et al. Under- specification presents challenges for credibility in modern machine learning. arXiv preprint arXivv2011.03335,, 2020.
Francesco D' Angelo and Vincent Fortuin. Repulsive deep ensembles are bayesian. arXiv preprint arXiv:2106.11642, 2021.
Yann Dauphin and Ekin Dogus Cubuk. Deconstructing the regularization of batchnorm. In Interna- tional Conference on Learning Representations, 2021. URL https:// net/
forum? iimmmidd---- -in-2-dd=d----- X2F8 (2d-d-xxFF81-6 81Wg1.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv: 2019.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M Roy, and Surya Ganguli. Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. Advances in Neural Information Processing Systems, 33, 2020.
10

---

Published conference paper ICLR 2022
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference Machine Learning, pp. 1607-1616. PMLR, 2018.
Yan Gao, Titouan Parcollet, and Nicholas Lane. Distilling knowledge from ensembles of acoustic models for joint ctc-attention end-to-end speech recognition. volume abs/2005.09310 2020.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural
networks outperform kernel methods? arXiv preprint 2Xiv:20006114499 2020.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12(10):993-100 1001, 1990.
Bobby He, Balaji Lakshminarayanan, and Yee Whye Teh. Bayesian deep ensembles via the neu- ral tangent kernel. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1010-1022. Curran Asso- ciates, Inc., 2020. URL https::/pproceeeeteeet 2.112339922235438881111112222222
neurips ec wttpriissccppppppeeeeee cc/ppaper/220//11ee pdf.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02533 2015.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: convergence and gen- eralization in neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 8580-8589, 2018.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Anders Krogh, Jesper Vedelsby, al. Neural network ensembles, cross validation, and active learn- ing. Advances in neural information processing systems, 7:231-238, 1995.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic- tive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, 30, 2017.
Jaehoon Lee, Jascha Sohl-dickstein, Jeffrey Pennington, Roman Novak, Sam Schoenholz, and Yasaman Bahri. Deep Neural Networks as Gaussian Processes. In International Conference on Learning Representations, 2018.
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. Advances in neural information processing systems, 32:8572-8588 2019.
Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: an empirical study. Advances in Neural Information Processing Systems, 33, 2020.
Wesley Maddox, Shuai Tang, Pablo Moreno, Andrew Gordon Wilson, and Andreas Damianou. Fast adaptation with linearized neural networks. In International Conference on Artificial Intelligence and Statistics, pp. 2737-2745. PMLR, 2021.
11

---

Published conference paper ICLR 2022
Alexander G de G Matthews, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Sample-then- optimize posterior sampling for bayesian linear models. 2017.
Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian Process Behaviour in Wide Deep Neural Networks. International Conference on Learning Representations, volume 4, 2018.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & BBusiness Media, 2012.
Roman Novak, Lechao Xiao, Yasaman Bahri, Jachoon Lee, Greg Yang, Jiri Hron, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl-dickstein. Bayesian deep convolutional networks with many channels are gaussian processes. International Conference on Learning Representations, 2018.
Sebastian W Ober and Carl Edward Rasmussen. Benchmarking the neural linear model for regres- sion. arXiv preprint arXiv:1912.0844 2019.
Sebastian W Ober, Carl E Rasmussen, and Mark van der Wilk. The promises and pitfalls of deep kernel learning. arXiv preprint arXiv:2102.12108, 2021.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in Neural Information Processing Systems, 32: 18:1391--1400 2019.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 3967-3976, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 268-284, 2018.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Veto,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, nigh-peeffrmmacc deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'Ahh Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8024 8035. Curran Associates, Inc., 2019a. URL ttt wttp://pp//p//../////oo an- an-imperat i ve- htwprrrreeeeeee---eee-eerrrrrrre deepeecceeeeeeeeeeeeeeeeeee Serep--eeanne www.nnngeennnneeeeeeee librrry..rrrrrrrr .ddf.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019b.
Tim Pearce, Felix Leibfried, and Alexandra Brintrup. Uncertainty in neural networks: Approxi- mately bayesian ensembling. In International conference on artificial intelligence and statisties, pp. 234-244. PMLR, 2020.
Qi Qian, Hao Li, and Juhua Hu. Efficient kernel transfer in knowledge distillation. arXiv preprint arXiv:2009. 14416, 2020.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In Proceedings of the 20th International Conference on Neural Information Processing Systems, pp. 1177-1184, 2007.
12

---

Published conference paper ICLR 2022
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong. Ju- Ju-Chich Chou, Sung- Lin Yeh, Szu- Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. SpeechBrain: A general-purpose speech toolkit. 2021.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep bayesian bandits showdown: An empirical comparison of bayesian deep networks for thompson sampling. arXiv preprint Stfi::880209122, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
Lisa Schut, Edward Hu, Greg Yang, and Yarin Gal. Deep Ensemble Uncertainty Fails as Network Width Increases: Why, and How to Fix It. In Workshop on Uncertainty Robustness Deep Learning. 2021.
Xu Tan, Ren, Di He, Tao Qin, and Tie-Yan Liu. Multilingual neural machine translation with knowledge distillation. In International Conference Learning Representations, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In Interna- /
tional Conference Learning Representations, 2020. URL forum? i ddt.miiddssssssssS..
Frederick Tung and Greg Mori. Similarity-preseriing knowledge distillation. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 1365-1374, 2019.
Joost van Amersfoort, Lewis Smith, Andrew Jesson, Oscar Key, and Yarin Gal. On feature collapse and deep kernel learning for single forward pass uncertainty. arXiv preprint arXiv:2102. 1409, 2021.
Kees Van Den Doel, Uri M Ascher, and Eldad Haber. The lost honor of Ey-based regularization. In Large scale inverse problems, pp. 181-203. De Gruyter, 2013.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. arXiv preprint arXiv:2006. 13570, 2020.
Andrew Gordon Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. arXiv preprint wwXiv:2002.0879, 2020.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P Xing. Deep kernel learning. In Artificial intelligence and statistics, pp. 370-378, PMLR, 2016.
Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference Learning Theory, pp. 3673-3673. PMLR, 2020.
Greg Yang. Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architec- ture Gaussian Processes. arXiv preprint arXiv: /2478, 2019.
Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks, arXiv preprint MrXiv:2011.14522. 2020.
Greg Yang and Etai Littwin. Tensor programs iib: Architectural universality of neural tangent kernel training dynamics. arXiv preprint arXiv:2105.03703. 2021.
13

---

Published conference paper ICLR 2022
Boseon Yoo, Jiwoo Lee, Janghoon Ju, Seijun Chung, Soyeon Kim, and Jaesik Choi. Condi- tional temporal neural processes with covariance loss. In Marina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Pro- ceedings of Machine Learning Research, pp. 12051-12061. PMLR, 18-24 Jul 2021. URL
http:://proceddng........
http/yoollbbbb..
Lu Yu. Vacit Oguz Yazici, Xialei Liu, Joost van de Weijer, Yongmei Cheng, and Arnau Ramisa. Learning metrics from teachers: Compact networks for image embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2907-2916, 2019.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint urXiv:1612.0339 2016.
Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Whye Teh. Neu- ral ensemble search for uncertainty estimation and dataset shift. arXiv preprint arXiv:2006.0857. 085733 2020.
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 3713-3722, 2019.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4320-4328, 2018.
14

---

Published conference paper ICLR 2022
APPENDIX: FEATURE KERNEL DISTILLATION
A FEATURE KERNEL DEPENDENCE WITH
LOSS
Suppose, we have data points with fixed feature extractor h h(X) RR*x, trainable last layer weights W E R2xC and targets R"XC
In Section 2, described how with Mean Squared Error loss the trained NN is precisely kernel ridge regression using the feature kernel, when there is l regularisation on the last layer weights. Hence, the feature kernel exactly determines the predictions. We prove the same for cross entropy loss. That to say, we wish to show that given the feature kernel k, the predictive probabil- ities/logits at the optimal W are independent of both W* and h.
We define the loss, for some regularisation À 0 (purely to enforce strong convexity) by
L(W) = Σ CE(y,, IW
where E 1x is the 2h row of and
e
CE I) -log Eceloy e
(8)
cross entropy loss.
Proposition 1. Let feature extractor h(()), truining inputs X, training largets Y, l2 regularisation > all be fixed. Suppose we are given a test point x' with features W then the test predic- tion logits W optimal W can be expressed solely in terms of feature kernel evaluations k(,, (c.))=((((),)())))
Proof. If we differentiate C(W) and set to zero we get:
AW
his [L [} pt.]
(9)
where p E RaxC result applying softmax to each row of the logits h/W".
Let h2 denote the extracted features for training input æ;. Now recall that (h₂, h2) k( 2x)) is
the feature kernel evaluated at æ,, Zy. We multiply Eq. (9) by the test-data feature vector h' to give:
A(L'W", Σ k(x, 2;) = p.J]
(10)
but h'W" precisely the logits for test point v., and likewise p:, is the vector of probability predictions at training point i. Hence, we could solve Eq. (10) numerically for logits/predictive probabilities given only the feature kernel (without f(-) or W*), and we see that the feature kernel once again determines logit/prediction probabilities at the optimal last layer parameters like for squared error loss, albeit this time implicitly for cross entropy loss.
15

---

Published conference paper ICLR 2022
B SETUP FOR TRAINING ON MULTI-VIES DATA
Before we prove Theorem 2, which demonstrates the generalisation benefits of FKD theoretically, we need to recall the data and training setup of Allen-Zhu & Li (2020) for completeness and self- containedness. Unless otherwise stated, everything this section (App. B) is simplified version of the setup in Allen-Zhu & Li (2020). Our results hold in the more general version too, but present setup simplified setting here for readability and convenience, without sacrificing the key messages and intuitions: our focus is for theoretical and practical contributions to guide each other and align as much as possible, as opposed to c.g. maximising the generality our theory.
B. MULTI-VIEW DATA DISTRIBUTION
Recall that we consider a C'-class classification problem over P-patch inputs, where each patch has dimension d, so our inputs are described by (xx ...",....."" (Rd) For simplicity, we take P = C and d = poly(C) (C) for a large polynomial. Like Allen-Zhu & (2020), we use Ô, $2 to hide polylogarithmic factors in the number of classes, C, which we take to be sufficiently large.
We assume that each class [C] has exactly two attributes Ve,1, 02,2 E IRd, which are orthonormal for simplicity," such that:
V = (Ve,1, 2e22ee
is the set of all attributes.
B.1.1 DATA GENERATING MECHANISM
Let our data distribution for data pair (ær, y) ~ be defined as D JD, (1- (1)Dmm for multi- view & single-view distributions Dm & D. respectively, (x, y) DD are generated as follows:
1. Sample (C) uniformly at random.
2. Sample set V'(x) of attributes uniformly at random from {wer te each with probability and denote V(X) = (x) U (Ug,1, 0y,2) } as the set of attribute vectors used in data æ. We take = C..2
Intuition:
V(xx) correspond to the ambiguous attributes present in æ, such the cat whose eyes look like car headlights. These are crucial in our proofs as the FKD regularisation between ambiguous images ensures that the student learns attributes it would have oth- erwise missed.
For each v E Vxx), pick C, (where Cp global constant) many disjoint patches in [P] uniformly at random and denote this set as P.(x). Denote P(x) = UrEV(a) (o.):: all other patches & P(x)) will contain noise only.
4. If æ is single view, pick a value = e( (1,2) 2} uniformly at random. i corresponds to the attribute ! that is present in with v " missing.
5. For each (x) for some V(z):
Tp Ep +
Σ
appor + Ep
N'EV
As = (Cly(C) large polynomial, this isn't too far-fetched an assumption.
16

---

Published conference paper ICLR 2022
where Op,'' [0, ट represents feature noise and Sp o(0,ooīaa) is independent
random noise, with pp Vd The coefficients 2p 2 0 satisfy:
(a) If x is multi view, When (1y,1, vy.2),
E (pp 3x 2 (1,2) (peP,(e) (p) =0=1
(11)
Intuition:
These conditions ensure that both attributes for class y are equally likely to learnt, averaged over different random initialisations of parameters.
When ((z),
Eper.(a) up 0.4 Σ EpEP. 2pp O(1)
(12)
(b) If is single view,
When y=
Σ EpEPi(a) 2p =
When w= y=
Σ (P))(() 2p = C --2.
When (xx),
EpEPo(a) 2p =r
where r = PCl
Intuition:
This is where the single view name comes from, as 0.2 << is barely present in æ.
we see that v y,3-
6. For each p E (z))
Zp = Σ
apporo + Ep
N'EN
for feature noise Opp. [0, T|x] and Ep NO0 Cd Ia) is independent random noise.
Intuition:
One think of the feature noise setting Oppe 0 Vp, v' for simplicity. But the general formulation above renders the problem unlearnable by linear classifier, as the maximum permissible feature noise across patches dominates the minimum possible signal: ब C >
Remark It possible to allow more relaxed assumptions, e.g. on Zp, as in Allen-Zhu Li (2020).
Training data Recall we have D = pD, + (1-4)D,,, so that proportion (1- - 11) of the data is multi-view. Our training data, D, is N independent samples from D. Letting D D, U D. denote the split into multi and single view training data. We μ poly(C) that each label appears at least (11) in D...
1 and we suppose [N] C².2 so
17

---

Published conference paper ICLR 2022
ReLU
ReLU
0.4
0.2
0.0
-0.4
-0.2
0.0
Figure 5: Comparison between ReLU & ReLU, for 0.2.
B.2
SMOOTHED RELU CNN
Our theoretical analysis considers single hidden layer CNN with sum-pooling, f. such that for class c:
Sa(ar) ΣΣ (0.r,(( Ve E [C].
For threshold (C)) define the smoothed ReLU function as:
if S 0 i E [0, e] = e
ReLU(=))
which has continuous monotonic gradient denoted as ReLU
B.3
HYPERPARAMETER VALUES IN SETUP
Hyperparameter values used in the theoretical setup are given in Table 3.
B.4 STANDARD TRAINING
We define our empirical loss by:
L Σ L(f(x.). N NEIN]
where L is the cross entropy loss defined in Eq. (8). We randomly initialise parameters iid, @o crr (O, o for of = J.
Standard training in the theoretical analysis comprises of full-batch gradient descent on the empirical loss with learning rate 17 S pay(er and for poly(C) iterations.
C PROOF OF THEOREM 2
We restate Theorem 2, where recall E denotes the teacher ensemble size and C is the number of classes:
18

---

Published conference paper ICLR 2022
Table 3: Hyperparameter values used in our theoretical analysis, corresponding to the setup of Allen-Zhu & Li (2020). (*) denotes undefined in our presentation, but appearing in Allen-ZMi & Li (2020), because the hyperparameter takes only one value in this work. We note that m is restricted to he Polylog(C) in the setting of Theorem 2.
Hyperparameter
Description
Value(s)
N
Training set size Proportion of single-view data
C
Poly(C)
Input patch dimension
poly(C)
P
Number of patches per input
C
m
Number of channels per class Out-of-class attribute sparsity
[polylog(C),C) C]
C
Parameter initialisation standard deviation (std)
C-0.3
dp
Input patch additive noise std
Vdpolylog(C)
r
Out-of-class attribute strength in single- view data.
polylog(C)
ReLU threshold
dpolyloog())
q (*)
ReLU mid-section exponent
p (*)
In-class weaker attribute strength in single-view data C
Theorem (FKD improves student generalisation and is better with larger ensemble). Given an ar- bitrary O. For any ensemble size E of teacher NNs trained as in Theorem I and sufficiently many classes C, for m = polylog (C), with learning rate 1i S polytyy, and training time T poly(C) the ensemble teacher knowledge can be distilled into single student model f(T") uusin only teacher feature kernel kT, Eq. (7), such that with probability at least 1 -Allogg (C));
Training accuracy is perfect: For all (x, y) E DD y = argmax, fo {(T") (x))
Test accuracy good: P(e.))n ly + argmax, _!("') (x)) (x)) S (म E)
Outline of proof of Theorem 2
1. We first analyse the feature kernel for single trained model, in App. C.I.
2. We then extend our results ensembled teacher model feature kernel in App. C.2.
3. We next outline our kernel distillation training scheme and provide key result concerning how the student's 's parameters become increasingly correlated with the teacher's ''s learnt views in Apps. C.3 and C.4
4. We combine all threads to prove the final result in App. C.5.
C.1 FEATURE KERNEL FOR SINGLE TRAINED MODEL
To prove Theorem 2, first calculate the feature kernel kt for a single trained model with trained parameters 0* from initialisation 0 The point of this exercise is to show that the feature kernel k. is able detect whether two inputs x, x' share common attribute which in the subset of attributes that was learnt by the trained parameters 0*, due to random correlation with initialised parameters @" This is formally shown in Lemma
19

---

Published conference paper ICLR 2022
Recall definition of the feature kernel At for our CNN architecture:
k x')
2e ReLU (eLU(((an) (0.r:
c=1 p=ppp'=
Let us first make few more useful definitions. For 1 = 1,2, we have: Definition 2.
(o,( def Llved E V(x)}
Σ
Zp.
pEPvo.1(2)
Intuition:
(z) is scalar constant describing the strength, æ of the presence of attribute Ve. (where may either correspond to true class or to different class; the latter has the effect of producing 'soft labels' when using vanilla knowledge distillation).
Definition 3.
Σ (00 ( v. rE(m)
Definition 4.
T def
[10 c,n Uo.))
rE/m]
Intuition:
T2,; 22, both parameter-deppndeent, data-independent scalars that describe the amount that attribute Ve, has been 'learnt' by the network parameters. Ye appears in the feature kernel whereas t appears in the function predictions Se directly, for our particular architecture in Eq. (4) (Allen-Zhu & Li, 2020).
Lemma 1. We have, for x. x! D
(z) + 0( (ar, x.) polylog( (C) + Ō
km (ax x0)
Σ
C
(e.D:.e.3-1)MMM
where
Vel V(x) n V(x') and (c,3-1) MH the number of shared attributes of x and x', that are also members of the set of attributes learni by the single trained model with parameters 0", and M is defined in Fact A.e below.
Proof. We have
C
P
P
ke (xx
= Σ cm
ReLU
cr. pp))
Σ
ReLU
pm1
First, define the hidden-layer activations for class and channel r to be:
Definition 5.
c,, (2:) del
ReLU((B (,,),,,)))
p=1
20

---

Published conference paper ICLR 2022
We know from Theorem C.2. of Allen-Zhu Li (2020) that, for multi-view æ D' and some [C]:
Fact A. For every p Pr.,(() and for 1 1, 2. we have: (02r.. xp) =0 2a,12p (
Fact A.b For every p e (((x) (P... (x) U Po.. (z)), we have 10 Tpp) S Ō(0o)
Fact A.C For every p E [P]] ((t), we have 10 xp)) S Ō vc
Fact A.d For every r E [m] M every 1 [2]. holds that (0 val) S Ö(ro), where:
M def
E [m] EI E [2] (0 vcl)
0 log C)
))
max rE[m]
(
c. mc))) s.
Note from Proposition B.1 of Allen-Zhu & Li (2020), that mo with probability at least 1 - e -0log (C)),
def
M = O(log"
Intuition:
M denotes the key channels in [m]] which have "won the lottery' and are relevant for class in that in the C' + do limit the predictions for fe are the same as if one forgets the other channels, as shown in Allen-Zhu Li (2020).
Fact A.e For every p E Poe, (x) and E [m]. if 3-1) E M, then we have: (0z,,, xp)| S Ō(0o), where:
M def
(c,1) E [C] x [2]
rE(m] c,rt vo,1il+ 2 (1+ log²
max (
max (000 rE(m]
c,Ty Uc(3-1)1+
Intuition:
M denotes the data attributes Ve, which are more likely to be learnt by the NN pa- rameters (compared their fellow class attributes ve,3-i) because of correlations of the initial parameters with such attributes. So, Fact A.e is saying that if (c, - 1) E M, then the attribute (c, I) is not learnt at all during standard single model training.
Thus,
2 Σ 1=1
Ver Ver(())
(vea E V(x)}
ReL.U((0 crr:
(r. zp))
(13)
PEP (w)
Σ
ReLo((((oo))
(14)
peP(m) (P...())) (x))
Σ
ReLU On ))
(15)
Pc (P(P)(((ar)
First, note that (Poy (x)) = Cp constant Ve, From Fact A.b, Eq. (14) can easily seen to be Ö( = 0(8-188) of 1 and C and likewise Eq. (15) be to be O(())))) = 3(C-2) as P = C², by Fact A.c. We note here that summing these equations over m and C' will be bounded above by O( (po)iio((3)) ).
21

---

Published conference paper ICLR 2022
Now, let's consider Eq. (13), Let notation 1,,1, Vo,2 denote either de,1 or No,2 some set S, and ve,1, N,,2 & S denote neither 2e,1 nor V2,2 S.
There are three cases to consider:
1. If 1, Vo.2 & (ax), then Eq. (13) zero.
2. Else if 31 E [2] such that var e V(x) and (c,3-1) -3-1)
M, then by Fact A.e, have
Eq. (13) ((C-))
Intuition:
This setting is when only attribute (c. 1) appears in but attribute (c, - 1) was domi- nant initialisation so (c, 1) has been learnt by 0"
3. Else, we have that:
2 $1 1fea E v(x)}
Eq. (13) -
Σ (e)
1=1
Putting this all together with Fact A.a, we see that
V (x)) Σ I((e,) (MM(,,) 1=1
Σ
ReLU vo.) -((oo) + (T.))
pEPoc..
Now, if & MD (i.e. neuron r not dominant initialisation), and (c, 1), 2) æ:), the by Fact A.d, we have that V Ve, (1)
On the other hand, recall = polylo[[C] and mg = OC)o And also, recall that if 2 > p. then ReLU(2) = O()). Moreover, by Claim C.11 of Allen-Zhu & Li (2020) we know that Br' such that (0) c,pr, Ve,1r) = [( Hence, for any r E M. we know that either:
= 1. Ver (x) = Σi= 1((c,1) e sm(x, ()) cri ve, (z) + +Ō or
2. c.r (x: = 1(c, 1), (c. 2) E sa x) 0(0)) - 0 (ctz).
Moreover, V (x) Ō(1) Vr, by e.g. Lemma C.21 of Allen-Zhu & Li (2020). And so it can be seen, for small enough we have:
V (x) (xx
2
1f(c, - (1
10 cr v +
polylog )]
+ +ō
CT.
1=1
Σ
[f(c, 3 1) (
[r + O
polylog(C)) )] Ō
CT.
1=1
22

---

Published conference paper ICLR 2022
where we also use Fact A.e above, such that is not possible for both (0 voa)) + and (02r1 Vc,3-1)+ to be large. 8 Note also from e.g. the proof of Allen-Zhu & Li (2020) Theorem 1, that if (c,3 & M. then max, (02, Ve,1)
(())
Thus, we see that the contribution to & from the m class-c channels/neurons is:
ke
V (c)
(16)
r=1
(L(1)
if (c, E (M)(()))))))
(17)
10
else
Summing & over [C] completes the proof of the lemma.
We now use Lemma to analyse h and p. (xx, x')
(m. (art,a))) a00))
First we look T Recall that (0 ve,1) = O(1 for all E [C],1 e [2],r e [m]. and likewise
so is M More specifically,
If (c,1) E M, then we have from the proof of Theorem 1 in Allen-Zhu & Li (2020) that Ô(1)
(o ve2)+ 2 Q(log(C)), and so max,-(0 c.r: voa)+
If neither (c, 1) nor (c,2) are in M, then Claim C.10 of Allen-Zhu & Li (2020) shows us that both max, (0 vo,1) +, max, 2 Vo22)* Ô(1)
Combining these facts, we have that:
r2, = +(1)
- &
and so from Lemma 1, we have:
SM x¹)
k, (xxx,
Σ
r (x) + 0
+ +(C-0.8)
(18)
polylog C?)
(c.))(e.331)k -DEM
S.M (2 x') polylog C)
=ô(1)
Σ
(x.( Zo, 0
+ ((C-88)
(19)
(c.))((.3-))MM
=(ss((2z, ( SM x')
+ 0.88
(20)
polylog(C)
Finally, we arrive at an expression for the correlation kernel p. trained single model by
(x 6 8M (x. x²)
1
S. ( ) (1 + O polylog(C)) )) + Ō (
(21)
where we define that (x((a) (a xx) is the number of attributes in that have been learnt by the trained network 0", and is s(1 = o(1) with high probability.
Unless neither (c, 1) nor (c, M for given but that only occurs in o(C) classes, and does change the order e.g. 8.M (xr, x²) which is what we really care about.
23

---

Published conference paper ICLR 2022
Intuition:
Compare in Eq. (21) to the 'soft' probability labels p E I of Zhu & Li (2020) (Claim F.4) with temperature t (c)"
c2,1 or 22 is V(x))
(x) = s(m)
0
else
where s(x) is the number of indices E [C] such that 1,,1 or V/ is in V(x)). Note, the setting of Allen-Zhu & Li (2020) is with large f2(1) ensemble, every attribute is learnt (akin M being empty for us). In the case that .M- = { being empty, if x & x' and they share at least feature, then from App. B.1.1 with high probability they will share exactly
feature, that SM x') 1. Moreover, s(x) = s s (a (x)==s((()) hence we see that Eq. (21) matches roughly with p (at), but without the need for lemperature hyperparameter.
We see that vanilla KD learns new attributes by comparing a single data point æ between classes, and giving larger target labels to the classes where ambiguous attributes learnt by the teacher are present in æc On the other hand, Ps gives higher values data pairs æ, x'' that share attributes that have been learnt by the trained model, and we will see later this is how FKD learns attributes the student.
C.2 ENSEMBLED TEACHER
To summarise what we have done so far in App. C.I, we have seen Eqs. (20) and (21) that is possible, for single trained model 0", to simplify both the feature kernel k. (xx, æ²) and correlation kernel P. (x, x') in terms of the number of shared attributes between æ, x' which are also learnt by the trained model. The set of attributes learnt by the single trained model is captured by the set
M = {
[2]
max 100 c,r vo.,)|+ 2 (1 log"(m)
max (00 e.r, Vo, +++
rE(m)
(m)
rE(m)
where Oo was the random parameter initialisation for 0" From Fact A.e, we know that if (c,3-M)EM, then the attribute úe,t has not been learnt by the network.
Consider now an ensemble of F = (11) independently trained networks, (0)) with an averaged feature kernel:
k (xc, xx'
k (ac x²).
Suppose [ denotes the corresponding independent parameter initialisations. Then, for each E [E]. let define:
cirs m))+ }.
M, del (c.)) [2] max ((0..) rE[m] crry
vo 2 (1 log (og)(m)
max (0 rE[m)
Note that these Me are completely independent sets due to the independent initialisations, and also by Proposition B.2 of Allen-Zhu & Li (2020), we know that
P((c,1) 1) or (c, e Mo] 2 -1- o(1)
Vc [C], [E].
Therefore,
C2 Mal 2 C(1 - op))
Ve e [F].
24

---

Published conference paper ICLR 2022
Moreover, Eq. (11) & Proposition B.2 of Allen-Zhu & Li (2020) tell us that each the two attributes 1,,1, 2, equally likely to be in Me (and so learnt in the multi-view setup), This means that:
Mel 2
C(1 C(1-0(1)))
Define My = =Ne=1 MM. From Eq. (19) and the definition of ky, we see that:
SM7 (ar, x²))
= (r( x=);))))=11 (1) Σ
1[(,3 - 1) (z) O polylog(C) + 0
0.8)
(c.)
(22)
= b (a, x²)) + O SMy (x,
((C-0.8)
(23)
polylog(C))
where for the reader's convenience, we redefine:
8 8M7 8M, x') = [(e,() vaat E V(x)) n V(x') and 1c, -31-1 & Mr).
Intuition:
We see that only for those attributes (c. l) such that (c, - 1) E My does the ensembled teacher kt miss the fact that we should have a strong 6(1) kernel value between æ, x'. This is when (V(x) n V(x') = {Vc,1} is non-empty (or in other words, when SMy (a, > (V(())) and hence there should be large keree value (f
So we see that only [MT| of the attributes are not learnt by the teacher, which fraction LMr 20 = 5t (1- o(1) of all the attributes. These missed attributes are where the श test error in Theorem 2 comes from (teacher ensemble of size E, and plus for the attributes learnt from the student's 's initialisation too).
What's more, we can decompose the teacher's feature kernel ky E. TT into contributions AT from each class c, like in Eq. (16). From Eq. (17), we see that the contribution to the teacher's feature kernel from class, for x';
(1) 0 (
if (c, E (x,
T x (x.x')
(24)
else,
is able to decipher between whether or not x, x' share an attribute from class e for all attributes apart from those (c,1) such that (c, - 1) MT.
C.3 TRAINING SCHEME FOR FKD
We note at this point that we are morally done in terms of proving Theorem 2, with Eqs. (23) and (24), our key results telling that the (ensemble) teacher kernel by identify when two in- puts share common attributes that have been learnt by the (ensemble) teacher, and more specifically that k. can do when said common attribute is from class c.
What remains is repackaging of the proof techniques of Allen-Zhu & Li (2020) (particularly for their Theorem 4 regarding seff-distillation), that knowledge distillation (this time only using feature kernels instead of temperature-scalle scaled logits, and with explicit dependence on teacher ensemble size) can improve generalisation performance of student.
25

---

Published conference paper ICLR 2022
For convenience, the theoretical analysis of Allen-Zhu & Li (2020) introduces some slight discrep- ancies between the actual practical weight updates of vanilla KD Hinton al. (2015), i.c. the gradients of:
1 N
Σ
f(ai) fT( L
5
7
and the weight updates in their theoretical exposition. Namely,
1. The authors assume that temperature-depentn threshold caps the logits to give soft labels:
emin(r"sc(a))) (c()))1)
Pe (x) EjE(o) e
2. The authors truncate the negative part of the gradient of the KD regularisation to only encourage logits to increase not decrease, with weight updates for Ogr on input æ:
-00' def 0t e.r o.
ox l x Vo C + IN
1 Σ (vT(x)) pō: pT:((a)) (a)) Vo,, fc(x))
where pr. are the temperature-scaled teacher labels.
3. The authors scale the output of both student and teacher models by a (polylogarithmic) factor, in order to ensure that both reach the threshold to give soft labels in Item 1. above.
4. Self-distillation (Furlanello et al., 2018: Zhang et al., 2019) distils single teacher and a single student of same architecture into the student, like an ensemble of size (stu- dent+teacher). Allen-Zhu & Li (2020) modify the training scheme for their theoretical analysis of self-distillation so that the student is first trained on its own in order separate learning its own attributes/fcatures from those of the teacher. Our analysis covers similar scheme.
These modifications are justified in that they make the theoretical analysis more convenient, whilst illustrating the main mechanisms by which KD works, which is to share 'dark knowledge' that is held in the teacher (in the form the multi-view attributes that have been acquired by the teacher due to its parameter initialisation), with the student.
In the same vein, we now introduce some modifications to the practical implementation of FKD we propose in Alg. 1 to aid our theoretical analysis, and describe the main mechanisms by which FKD works, corroborating our initial analyses in Scction and App. A about how the feature kernel is crucial object in any NN and captures all the 'dark knowledge' that a teacher network could possess in the multi-view data setting.
It is likely possible to extend our proof of Theorem with different modifications/traninn schemes, but given that the focus this work is to introduce FKD as principled alternative to vanilla KD with certain advantages such as prediction-space independence, and that the multi-view setting consider is a plausible simplification of real world data (as demonstrated in Allen-Zhu & Li (2020)), leave this future work. We stress that any simplifications the update rule in Alg. for this section can be efficiently computed, only requiring access to pairwise evaluations of the student and teacher feature kernels, if need be.
Modified training regime for FKD
1. We first suppose that the student is trained as standard (as in App. B.4) for T; poly(C)) steps, and learns its own subset of attributes Ms, dependent on its initialisation 00, before being trained with the FKD objective:
"
26

---

Published conference paper ICLR 2022
Intuition:
This mirrors the celf-distillation setup of Allen-Zhu Li (2020) Theorem The idea being that the student first learns Ms before picking up the other attributes that the teacher has access to.
2. For a given feature kernel we threshold the feature kernel &o based on value, to define modification & such that
{
if k(x x²) 2 1
k(x, w²)
m2
else
Intuition:
This condition delineates between the setting where æ, x' share common attributes learnt by student parameters 0 in the initial phase of training (i.e. delineates between whether M,, (z, x') nonempty or empty).
To see this: note that if Vo, (ax) V(x') and (c,3 1) & Ms then we know from Allen-Zhu & Li (2020) that Th 2 ((log(C)).
Hence max, (02, Ve.)) 2 22 2(Iog="(C)) (C)) the number of active neurons my |M
O(log*)), and so it's easy to see that for large enough m we have k(x,, x.) 2 m² via Lemma 1.
On the other hand, if [Vo,] V(Φ) n V(x') and (c, 3 - ()EMs then from Lemma we know that k(ær, x²) = ((C-0.8) < ...
3. Similar to Allen-Zhu & Li (2020), we also truncate our FKD regularisation to only en- courage kernel values to increase, and not decrease. For any input pair æi 22 we have parameter update:
- 22) (21,
ko (x1, x2))
Σ JE(1.2}
Ver(
1 (x3
(25)
where recall
(x) def
ReLU (0.,; ap)) so that Vo.. V (or(x)) = > ReLU ((0 )2))ap
and
(,, x²) del Σ cerr (w) (a,r
were defined in Definition and Eq. (16).
Intuition:
If the loss was
(ke( x2) - To? (x1, x2))²
then the gradient with respect Oer would be:
(k. 22) k-7 (æ1, 22))
Σ
(er( Vo., V (er(((3))))
JE(1.2}
the only differences with Eq. (25) are truncating (1o( - 1i (x1, and also the thresholding to obtain I.
27

---

Published conference paper ICLR 2022
To summarise, after training the student on its own for Ti steps (such that we are in the setting of Theorem 1) to reach parameters 0 we update for T2 poly(C) steps as (hiding S subscript):
A0 A0, 1/R ar (11, ki (x1, x2))
Σ
ver (2,)) (err (26)
JE(1.2]
C.4 FEATURE CORRELATION GROWTHS
We now seek analyse what extent the attributes (Ve,)e, learnt during our T2 FKD training steps. The central objects describing how much Var has been learnt by parameters 0 are:
$ def
Σ 10 (e,n..
and
def
c.t
rE[m]
[E(2)
as well as (x)) as defined above.
Intuition:
Pe,t is a data-independent quantity that reflects the strength of correlation with feature Ve,t by parameters 0. On the other hand, (a.)(ar) is data-dependent quantity that reflects the activation of channel for class with input æ.
Also recall that
(,)(())) def 1 (ree E ((x)
Zp
pEPre..
and define (a,),( (which convenient for calculating the size of gradient updates for 0z,r): Definition 6.
Va,,((=) def 1e V(x)
Σ pEPP..2(m)
ReLU
Like how Lemma simplified the feature kernel in terms of data-dependent Zz,( and data- independent Y Te.,, we have result from Allen-Zhu & Li (2020) to simplify function predictions fa in terms of (xx) and D..1:
Claim (Claim F.7 from Allen-Zhu & Li (2020)). For every < Ti + T2. every e E [C], every (x, y) D (or every test sample (x, y ~ D with probability - -i(wee' (C)))
fx(x) = Σ ( x (x)) + O
polylog(C))
LE[2
We also have the following facts from Allen-Zhu & Li (2020) regarding the correlation of gradient (x) with Ve, for (x. E D. [2] and r E [m]:
Vo.,, Vf V
Claim (c.f. Claim F.6 of Allen-Zhh -hu & Li (2020)). For every t < Ti + T2. for every (z, y) D,
every e [C]. r E [m] and E [2]:
Va,,, Va,2 E V(x),, then (Vo.,, Va, ver) 2 (Ven)(a)- Ölo
(Vo... Var val) S (I E (a))) O(C=)))
For every # e, 1(- (-Vo,,, The V (x), aal) 0
28

---

Published conference paper ICLR 2022
Disclaimer Technically, Allen-Zhu & (2020) only show Claims and for 1 S Ti and one would need to similar proof techniques (such their inductive hypothesis F.1) to show the case for 2j < t 2 T2, which we skip for conciseness.
We now study the growth of the student's Le,t for those (c, () which have been learnt by the teacher but not the student:
Lemma 2 (Correlation Growth for attributes learnt by teacher). For every E [C].1 E [2]. T2 2
t Ti, such that (c, 3 - 1) & Mr, suppose & S 2m then we have:
ns² 2 &e 52(
(+e
ReLU (o)))
ol
Proof. For any E r r e [mm, E [2]. we have from Claim 2*
(A0', c,r. (c,))=nE, ns a2 D2 (Rg x2) (a x2))
Σ
5E(1,2}
Note that / S poly(C) suppose that both 1 22 multi-view data. Using Claim 2, have that
(AO' cnr ve,)) 2 n 2i Do (bg (m11 022) ka x2))
Σ V Verr(at)VVern(ss)))))
SE(1.2}
Let r argmax [(mm] (0 vo1)3, such that definitely vo)) (o.) (1)))))) because m polylog(C)..
But if a is multi-view and vell E V(x) such that Epep #/ (11, and also by Fact A.a we have that:
Var,(m)) 2 2(1) ReL.U verl) 2 RReLU (0',)) V (x) > Σ ReLU c. ve,) Ō(oo)) 2 2( )
pEP
Now, we have assumed that (c,3 - 1) & Mr. such that the teacher model has learnt attribute (c, 2) and satisfies k. (ax, x') = 1 when Ve,1 E V(()) Va=') (for large enough polylogarithmic m)).
Also, it is simple to see that when Ver E ((x) n V(x') v0,3-1 & V(xr) n V(x'), for large enough m, that of, < 2' implies that ko( x') < m by Lemma 1, and so ka x') = O, i.e. the student has not (yet) learnt Ve.!-
So see there two more conditions that must be satisfied in order for =1>00 0:
(ET (I1, -
1. Ne,t ((x1) n V(x2) so that R.T (æ1, 222) =
2. Vo,311 & V(x1) O (xx2) so that ko x2) =
o(1) Going back to App. 1.1.1, we know that these conditions occur with probability & (1)) for independently sampled æı, 02 ~ D. Finally, putting everything together we have that:
s15"" 10 -ReLU (#'))
(0'+1, vea) + (0) vo.))*
C2
29

---

Published conference paper ICLR 2022
summing over r' [m]]] and noting (A0 crr' Vo.) 2 0 Vr', up to small error (as og pol((C) a large polynomial), gives us our result.
Lemma 2 immediately gives the following corollaries, because Φ2) (((o)) from Allen-Zhu & Li (2020) Induction Hypothesis F.1.g and ReLU is increasing.
Corollary 1. Define iteration threshold T- θ noto?! (c,3 & My we have:
a (( then for every (c, () such that
$ T
4m
But likewise, we can also bound the growth of Det
Lemma 3. If (c. - e MT, once & 2 C):
loglog(C) (C)
no longer gets updated (for large
Proof. Recall Definitions and 4 that - veal and r -
Erempll +ca)]++
Hence by Cauchy-Schwarz we have:
loglog(C) m²
2
b S mY
r
0.42m²
for large enough C. Thus, if 32 are both multi-view and Va,1 E V(xi) n (ar2), by Lemma 1 we must have
ko x2)
m
It's also not difficult check that any other possible setting for æ 22, and V(x11) n ((x2)) will lead to (k? (x1, (x11 (x11))))) + = 0. and hence
(K? (x), ko (2))
V0 322
C.5 WRAPPING UP PROOF OF THEOREM 2
Proof. We are now ready to wrap up proof. Recall from the proof of Theorem I in Allen-Zhu & Li (2020), that after the initial phase of T, steps of student training on own:
t > I(log(C) Ve E [C],
and more specifically:
If (c,3 & Ms. then
4 > A(log(C))
This gives us perfect test accuracy on the multi-view data, and 50% accuracy on the single- view data, so 0.5y test accuracy overall without distillation, as per Theorem 1.
30

---

Published conference paper ICLR 2022
Moreover, we have that if (e, - 1) & Mr and (c,3 - 1) E Ms. then by Corollary and Lemma 3:
loglog(C) 2 T1+T2 c,.
2
m
v Vloglog(C) 12) +Tg
4m
We see that this change in $ Ti +7 after FKD training is much smaller than ((log(C)) and c.
so the student after FKD training still has perfect multi-view accuracy, as well as correct predictions on any single-view data that possess the attributes learnt in the initial phase of training.
On the other hand, for single-view data, we know that if we have data point æ, y and attribute Ve,1 E V()), such that * then EpePo Ep = I O polying(Cy as defined App. B.I.I.
Hence for small enough I' (< J.) we have that if (c, -) & My and the single view data x is of class e with [(xr) = then we have correct prediction, as per Claim
Combining these means that we have correct prediction for any single-view data of class c, and [(x)) 1 such that 3c 1) & Myn.s
By the independence of these sets we have that [Mr n M. (2 (2-5)A(1 - o(1) this means we have test error less than (2-E-1 + E)μ for any e 0, for large enough C as required.
D DIFFERENCES BETWEEN FKD & OTHER FEATURE KERNEL BASED KD METHODS
In this section, we highlight how our FKD approach overcomes some of the shortcomings of previ- ous feature kernel based KD methods which only use pairwise evaluations of the feature kernel: SP (Tung& & Mori, 2019) and RKD (Park et al., 2019). One advantage of FKD relative to these previous works is that have shown FKD is amenable to ensemble distillation. Morcover, it goes without saying that Feature Regularisation, which arises naturally thanks to our feature kernel learning per- spective in Section 4, already significant departure that improves FKD relative SP & RKD. However, even without FR we observe in Section that FKD outperforms both RKD & SP across different datasets and architectures, which warrants explanation.
D.1 IMPORTANCE ZERO DIAGONAL DIFFERENCES: SP (TUNG MORI, 2019)
First, consider diagonal kernel differences, S (x. x) - A (x, x) for fixed æ, and motivate using zero diagonal differences, which not present in SP (Tung Mori, 2019) but is in FKD thanks to of the correlation kernel. Fig. 6 displays this comparison between FKD & SP graphically.
Intuition: Downside of non-zero diagonal differences
The key intuition, which we detail below using our theoretical setup, is that non-zero diagonal differences k(æ,, æ) AT (xe, x>) jf encourage the student learn noise in input æ. compared to when we have zero diagonal differences k(x, x) kT (xx x = 0. In the latter case, we only have non-zero differences for k(xxx') kT (x, x') where + x'
31

---

Published conference paper ICLR 2022
(K. - x2 2 for SP
(K. K. ,2) for FKD
64
0.6
Figure 6: Comparison of (normalised) squared differences in ka.a* k(r, x²) between student S & teacher Т, across minibatch of size 64 of CIFAR-100 training data, for SP (left) and FKD (right). We that whercas FKD has zero diagonal differences, SP largely dominated by non-zero diagonal differences. Note there is a slight abuse of notation here, in that we plot squared differences in normalised kernels, so that FKD uses the correlation kernel and SP uses now-normalisation (Tung & Mori, 2019).
Diagonal updates Consider our parameter update Eq. (25) when σ1 22 x: didn't 't have zero diagonal differences and instead Kg (x, x) ki (xx x) 1, then:
A (x. x)) 21/ e.r (x)
Σ ReLU ((0 Fpp) D.p
Now suppose e = 1o,1 E V(x). For each E (x)), recall (from App. B.1.1) that:
xp = 2p0 Ep:
where we assume zero feature noise for simplicity. We then see that (c.f. Claim C.13 of Allen-Zhu & (2020)):
(A (a, x), Ep) = u (z) ((0,,,, + +0(
Vd
as (v, Ep) = 0 V with high probability.
But at the same time (by e.g. Claim F.6 of Allen-Zhu & Li (2020)):
(AO x), u) =
and note that that (z) < ((1) by definition. Moreover, in order for the student network to learn attribute ve,1, then eventually must satisfy max, (0ar v) Ö(L) from Allen-Zhu & Li (0x,(
(2020). Thus, for o small enough, if argmax, (0,, (0er,,)) v), we have ReLU ((0c,,, (0,,,:pp)) = 1.
Non-diagonal updates
On the other hand, if æ j ar and we have E V( 0 ((x2):
A (Ii, xz2) = 77 Σ Ver (at
Σ
ReLU (3-m)))3
1=1,2
p=1
and so w = x and Ep denotes the random noise in 1t then:
(21, θ(7) V. (2
R ReLU (10 x1,))) + +
32

---

Published conference paper ICLR 2022
as 2. Sp) = 0( (7)). But this time we have
(x1, v) = n (21)
+ (a.
We see that whereas in diagonal updates ( x) the increments for (A0er æ), Ep) and (A æ), v) are 1:1, for -on-diagonal updates e0,. they are 1:2 respectively.
Thus, the parameter updates for Ozr with non-zero diagonal differences in feature kernels are more likely to learn noise, Sp. compared to our zero diagonal updates which rely only on non-diagonal A0,,((r1, 32) for 21 * Z2- This is why we zero out diagonal differences for FKD, using the feature correlation matrix in practice.
D.2 PROBLEM OF HOMOGENEOUS NNS IN RKD (PARK AL., 2019)
The distance-wise version of RKD (Park et al., 2019) is follows: for æ, x', calculate (a x') = (x,, 0T) - la 0T) where recall ht is the last-layer teacher feature ex- tractor. Likewise, we also calculate s/s (ær, x²) (a,, 0s) hi h 0s) The RKD loss adds AKDLE [(t'T(x x²) shs (æ. x' =²] to the student's training loss." 9 While RKD (Park et al., 2019) does ensure zero diagonal differences, i.c that // (2, x) ( x) 0, it suffers from a related issue, due to the homogeneity of NNs that use ReLU nonlinearity, which is ubiquitous in image classification tasks.
Suppose we take and define x' = Mxx for some M >> 0. For example, think of taking cat image and multiplying all the pixel values by M. For ReLU (C)NNs without bias parameters, we have that h(x, 0) is 1-homogeneous: h(a'', 0) Mh(ar, 0). This means that is likely (depending the norms the features hs and lr)) that we will have thy Mæ) - w (r. Ma) * O. But a cat image multiplied by some scalar . still image, hence RKD runs the same problems as in App. D.I of learning noise in æ.
On the other hand for FKD: correlation kernel ps(æ, Mx) (x))) hence ps( Mx)-pr(a4 Mx)=0 Vo E R
E PYTORCH-STYLE PSEUDOCODE FOR FKD
In Alg. provide py Torch-style Paszke et al. (2019b) pseudocode for the distillation and feature regularisation losses FKD. We note that FKD only requires pairwise computations of feature (correlations) kernels. This alleviates the need for matrix mullipiicatioonnnversion operations with batch-by-batch size matrices, which is beneficial for scalability.
F EXPERIMENTAL DETAILS AND FURTHER RESULTS
F.1 FIG. 2: PREDICTIVE DISAGREEMENT ACROSS INDEPENDENT INITIALISATIONS RETRAINED LAST LAYER
All models are ResNet20y1 trained with standard hyperparameters:
160 epochs training time with batch size 128 and learning rate 0.1 which is decayed by factor of 10 after epochs 80 and 120.
SGD optimiser with momentum 0.9 and weight decay of 0.0001.
We not consider angle-wise RKD loss here, but there will similar issues due to homogeneity.
33

---

Published conference paper ICLR 2022
Algorithm PyTorch-sppp style pseudocode for Feature Kernel Distillation (FKD).
Batch
L FKD:
FKD zegularisation strength.
L. PR:
Feature regularisation strength.
D_a:
Student feature dimension.
# b_t:
Teacher feature dimension.
Student
# f_t:
Teacher features x D_t
matrix-matrix multiplication
# Compute atudent feature correlation kernel matrix 8_0 mm mm f_s.T)
diag iny sqrt torch.diag. wtt.at
pooo(-122)
a k diag inv ingt * e.k. k_k_ inv__irttt....aa
reshape 1) BBBx 1 1Bxx
s.k d_k_iiagiin inv_g_tt a_k s_k diag inrggiiyy_err...._
# Compute teacher feature correlation kernel matrix t_c with
no_gra(();
t.k mm ft).))
t.k diag inv inv_aqrt torch. diag(t k) (ox(1222)
t.k k_k_giag_t inv sqrt t.k diag inv sqet. reshapp (-1, 1) B x t_c t_k_diag ing_
t_k_ anv_sarr.."
distil loss = ((t s.c) pow (2)(2)) mean
feat_reg_lose (f_s.pow (2)) m)aa
# FRD loss to be added to supervised loss
loss fkd FKD distil loss L. feat_reg_ loss
CIFAR10 data is normalised in each channel such that the training data is zero mean and unit standard deviation. Random crops and horizontal flips used data augmentation.
All models are initialised with Kaiming initialisation He al. (2015).
Out of 10000 test points, the predictive disagreements between a 'reference' model and either: in- dependent initialisations (top row) retrained last layens (bottom row) are depicted in Table 4. We see two clear trends. First, the retrained last layer has much fewer disagreements with the ref- erence model than an independent initialisation model, highlighting the importance of the feature kernel. Secondly, the vast majority of disagreements between independent initialisations are where one of the models is correct. This reinforces works because different initialisations bias the models to capture different useful features, and hence ensemble distillation (via feature kernels) can improve student performance.
intuition/theoretien analysis that ensembling NN
Table 4: Breakdown of predictive disagreements between reference and alternate models over 10000 CIFARIO points, terms of which model (if any) correct. Mean + standard deviations over independent initialisations for top row, and over independent reference models for bottom row. All models achieved between 8.0%-8. 8.5% test error.
Alternate model
Reference correct
Alternate correct
Neither correct
Total disagreement
Independent Init
350 +: 14.6
383 15.9
124 4 3.7
857 +: 29.8
Retrained LL
30 + 2.9
35 = 1.7
15 5.4
80 + 4.1
F.2 FIG. 3: ADDITIONAL FEATURE KERNEL HISTOGRAMS
In Fig. 7, we provide additional plots to Fig. 3 that depict the difference in distribution (over æ) of feature kernel values k(x, .), for FKD with and without Feature Regularisation (FR).
34

---

Published conference paper ICLR 2022
ResNet32x4 ShuffieNetV
ResNet32x4
ResNet50 VGGB
15-
FKD with FR
Histogram ramm Density
FKD w.o. FR
3-
9.4 Normalised Mar, value
48 h2
6-4
Normalised a(r,)) value
Normalised a().a) value
Figure 7: Comparison of normalised k= æ) values between FKD with & without Feature Regularisation (FR), across different Teacher-tStudent architectures, on CIFAR-I00 100 test set. We see, like Fig. 3 that FR encourages a more even distribution of k( x) across æ, for all architectures.
Negative hypothesis. We originally hypothesised that FR could benefit FKD, in addition to bal- ancing the distribution of k( æ), as it could reduce the sparsity in the NN last-layer representa- tion activations, which is consistent with the proof' of Theorems I and Indeed, the NN predic- tions dominated by select few neurons who 'have on account of being most correlated with one of the attributes {Fed [V<))eEp))cc(() at random initial- isation. In our analysis, as few as O(log"(m) (m) out of neurons could be inactive. From our feature kernel learning perspective, this seems like a highly undesirable phenomenon, because k(z, x') = (h( 0), (<z:0)) = CC (x, (x', 0) only has useful contributions from a dominant minority of E [Cm], then we are utilising the full capacity of the model. FR seemed appropriate to reduce this sparsity as la regularisation is known to promote non-sparse solutions (Van Den Doel al., 2013). However, we experimentally found the opposite to our hypothesis: that FR trained FKD student had more inactive neurons relative to FKD students trained without FR. This again highlights that, while (we believe) our results in this work provide compelling evidence to highlight the validity of feature kernel based distillation, there are still gaps between our theory and practice, and further questions to be answered in future work.
the lottery' (Allen-Zhu & Li, 2020)
F.3 FIG. 4: ENSEMBLE DISTILLATION
All individual VGG8 networks that made up the teacher ensemble were trained using the default training regime from Tian al. (2020), with independent parameter initialisations. Indeed, all of our experiments in Section used Tian et al. (2020)'s excellent open open-source PyTorch codebase (Paszke al., 2019a).
For student networks, we used the training regime for vanilla KD from Tian et al. (2020) for all ensemble sizes. For FKD, we used the hyperparameters from our ResNet50--VGG8 experiment in Table 2 for all ensemble sizes.
F,4 TABLE 1: DATASET TRANSFER
The VGG13 teacher checkpoint is provided by Tian al. (2020). For both CIFAR-10 and STL- 10, all student networks are trained for 160 epochs with batch size 64 using SGD with momentum, with learning rate decays at epochs 80, 120, 150. The student trained without KD used default hyperparameters from Tian et al. (2020), which are indeed strong hyperparameters for standard training. For FKD, RKD (Park et al., 2019) and SP (Tung & Mori, 2019), we tuned the learning
https: / / thub. com Hooo/ Long Rttpetttoooonngnnnneeeeeeeeeeeee Retnnng//eDDisstleeeee
35

---

Published conference paper ICLR 2022
rate, learning rate decay, and KD regularisation strength ÅKn on a labeled validation set of size 5000 for CIFAR-10 10 and 1000 for STL-10, before retraining using best hyperparameters the full training(tumlabllee)) dataset. We also tuned the FR regularisation strength. AyR for FKD when FR was used. All RKD and SP hyperparameters were tuned in large window around their default values from Tian et al. (2020), which were all author recommended. For FKD, we allowed ARD to range in [1,1000], and XFR range in [0,20]. All hyperparameters sweeps were conducted using Bayes search.
For STL-10, we used batch size of 512 for all KD methods' regularisation terms, compared to 64 for the standard cross-entropy loss. This was due to the fact that STL-10 has only 5K labeled datapoints, and we wanted ensure that the student used as much of the unlabeled data as possible for each feature-kernel based KD method's 's additional regularisation term during 160 epochs of training. 512 batch size was the maximum power of before we ran into memory issues on a 11GB VRAM GPU, which occured for the RKD method.
Both CIFAR-10 10 and STL-10 data are normalised in each channel such that the training data is zero mean and unit standard deviation. Random crops and horizontal flips used as data augmentation. STL-10 images are downsized from 96x96 to 32x32 resolution.
F.5 TABLE 2: CIFAR-100 AND IMAGENET COMPARISON
CIFAR-100. All networks were trained for 240 epochs with batch size 64, with learning rate decay at epochs 150, 180, 210 using SGD with momentum. All teacher networks use the exact same checkpoints provided by Tian et al. (2020). Learning rate, learning rate decay, ÅKD. and ÅIR (when used) were tuned as in App. F.4 on a validation of size 5000. All other hyperparameters were set to the default values used by Tian et al. (2020). The CIFAR- 100 data is normalised in each channel such that the training data is zero mean and unit standard deviation, with random crops and horizontal flips used for data augmentation. All results provided denote the test accuracy the end of the 240 epochs of training.
ImageNet. The ImageNet dataset (ILSVRC-2012) consists of about 1.3 million training images and 50,000 validation images from 1.000 classes. Each training image is extracted as randomly sampled 224x224 crop or horizontal flip without any padding operation. All teacher networks use the exact same checkpoints as provided by Chen et al. (2021). The initial learning rate is 0.1 and divided by 10 at 30 and 60 of the total 90 training epochs. We set the mini-batch size to 256 and the weight decay to 10-4, AKD. and ÀFR (when used) were tuned as in App. F.4 on validation of size 5000 except using Bayes search. All results are reported in a single trial. All other hyperparameters were set to the default values used by Chen et al. (2021). All results provided denote the Top-1 test accuracy (%). Accuracy of baselines were reported in Tian et al. (2020).
F.6 SENSITIVITY TO ARD
In Fig. 8, we plot the sensitivity of FKD to the strength of the distillation regularisation Ago in Eq. (2) for the V experiment on CIFAR-100. We see that a well tuned \ςd(Rs 300 here) is important for best student generalisation. Feature regularisation APR 20 in Fig. 8.
F.7 TABLE 5: ANALYSES IN NEURAL MACHINE TRANSLATION
In this section, We performed analyses for a neural machine translation (NMT) task proposed by Tan al. (2019). In the analyses, we could only obtain data for En-De (from English to German) translation since links to the datasets for other languages are broken. Therefore, we employed a self-distillation method on a pre-trained English model for En-De translation as follows:
36

---

Published conference paper ICLR 2022
VGG13 - VGG8 on CIFAR-100
74
Test accuracy (%)
73
72
71
67
10a
10a
10a
10*
FFK Regularisation Strength
Figure 8: Comparison of normalised k(az, .) values between FKD with & without Feature Regularisation (FR), across different Teacher-tStudent architectures, on CIFAR-100 test set. We see, like in Fig. 3 that FR encourages a more even distribution of k(x,, æ)) across æ, for all architectures.
We train single teacher transformer model the IWSLT dataset for English (Tan et al., 2019).
We perform seff-distillation on the teacher model for En-De translation (Tan et al., 2019).
We did not search for optimal hyperparameters, and used default parameters of the code provided by the authors of Tan et al. (2019). The results are given in Table 5.
Table 5: BLEU of the teacher model of (Tan et al., 2019) (Teacher), self-distillation of (Tan et al., 2019) (SD), SD with KD of (Hinton et al., 2015), SD with FKD, and SD with FKD loss obtained by replacing distillation loss (2) of (Tan et al., 2019) with FKD, in En De neural machine translation tasks.
Teacher (Tan al., 2019) SD (Tin et al., 2019)
with SD with FKD with FKD loss
27.32
27.49
27.51
27.64
27.79
We first note that, we adapted vanilla KD and our FKD for sequential data in the KD loss (2) of (Tan al., 2019) in this task. More precisely, probabilities, and added these loss functions to the KD loss (eq 2 of (Tan et al., 2019)) in KD and FKD. In the results, aggregating vanilla KD with the KD loss (eq 2 of (Tan et al., 2019)) improved accuracy from 27.49 to 27.51. However, FKD further boosted BLEU to 27.64. We then replaced KD loss (Eq. of (Tan et al., 2019)) with FKD for training. Remarkably, FKD further boosted the BLEU to 27.79. These results suggest that the proposed FKD can be applied in NMT tasks, successfully. We hope that these results will motivate researchers to employ FKD in various different NL.P tasks including but not limited to multilingual NMT, named entity recognition and question answering.
first computed vanilla KD and FKD token
F.8 TABLE 6: ANALYSES IN AUTOMATIC SPEECH RECOGNITION
In this section, we used a CRDNN model (VGG + LSTM,GRU.LGGRU+ DNN) on the TIMIT dataset. In this experiment, used distillation approach proposed by Gao et al. (2020) for ASR tasks as follows:
We train single teacher model on the TIMIT dataset Ravanelli et al. (2021).
37

---

Published conference paper ICLR 2022
We perform self-distillation on teacher Gao et al. (2020).
We did not search for optimal hyperparameters, and used default parameters of the Speech-
Brain Library.
In this task, replacing CTC/NLL distillation losses with KD (Hinton et al., 2015) did not converge. Additional investigation with hyperparameter search is needed. We used phoneme error rate (PER) to measure accuracy of models.
Table 6: Phoneme emoor rate (PER) of methods in automatic speech recognition tasks.
Teacher
Distilled Teacher (Gao et al., 2020) KD (Hinton et al., 2015)
FKD
13.26
12.80
12.86
12.59
The results are given Table 6. Similar to the NMT task, we adapted vanilla KD and our FKD for sequential data follows: We first computed vanilla KD and FKD loss functionn token probabilities, and then added to the total loss (eq 7 of Gao et al. (2020)). In the analyses, Vanilla KD (Hinton al., 2015) increased the PER from 12.80 to 12.86. However, FKD further improved the PER from 12.80 to 12.59. In this task, training models by replacing CTC/NLL distillation losses (eq 4 or 5 of Gao al. (2020)) with KD (Hinton al., 2015) and FKD did not converge. In conclusion, these results propound that FKD can be applied for different tasks, i.c., image classification, NMT and ASR, boosting accuracy of baseline distillation methods. We hope that these initial results will motivate researchers in different communities (computer vision, NLP, and ASR) to further expound and apply FKD in additional sub-tasks.
38


Reviews:
[{"review_id": "yz5jqn9wzRF", "reviewer": "Reviewer_68WG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem investigated in this paper is an interesting one. The approach of trying to regularise to match the kernels of the teacher rather than temperature scaled logits is also quite interesting. The experimental results shows that the method is competitive in various settings. The theory is interesting though it seems to be a fairly straightforward extension of Allen-Zhu and Li. \n\nThe one weakness of the paper is the lack of concordance between the theory and the practice. The experiments show that FKD is quite competitive even without ensembles. But the theory does not explained why. It was not clear to me how Feature Regularisation would effect the theory.", "summary_of_the_paper": "In this paper the authors consider extending the knowledge distillation (KD) framework to cases where the student and the teacher do not share prediction spaces. To this end they propose Feature Kernel Distillation (FKD), where one views the last layer weights of a neural network as a data dependent feature kernel. Instead of regularising the distance between temperature scaled logits corresponding to each data point of the student and the teacher in vanilla Knowledge Distillation, the authors instead regularise based on the distance between feature kernels of a pair of data points (inner product of weights of the pre-final layer of the NN). \n\nUnder a simplified setting, viewing data as being multi-view the authors show a setting where classical knowledge distillation would not do well, but a FKD based knowledge distillation using an ensemble of teachers does well (the teacher kernel between two data points is the average teacher kernel across the ensemble). The authors then propose using a correlation kernel instead of raw kernels and feature regularisation to spread out the kernel values.\n\nThe authors proceed to empirically demonstrate  that this approach helps in dataset transfer between datasets of different number of classes. They also show that using an ensemble of teachers enables FKD to improve performance (but KD also improves performance).", "summary_of_the_review": "The paper examines an interesting problem. The Feature Kernel Distillation is an interesting algorithm. The experiments seem to indicate that it is competitive in a variety of settings. The theory is reasonable interesting as well. The paper is well written as well. Hence I recommend acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iKOvZkylrc9", "reviewer": "Reviewer_DKJu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n- The proposed FKD is novel, to my knowledge. Especially, it can apply to the setting where the teacher's output dimension is different from that of the student, while the traditional KD is not applicable. \n- The authors provided theoretical justifications for their FKD under a carefully designed setup, which I believe intuitively makes sense.\n- The entire paper is well-organized and the authors provided detailed and intuitive explanations when they introduce the theorems and derivations, which makes it easier for me to follow the paper.\n\nCons:\n\n- The technical novelty is very limited. I roughly went through the appendix, and I found most of the derivations are based on the results in  Allen-Zhu & Li (2020). There were no novel proof techniques presented.\n\n- It's unclear for me, how does FKD compares to KD theoretically. If I understand correctly, FKD, KD, and ensembles are all able to deal with the multi-view setting, where standard training will fail to generalize. So, what's the advantage of FKD over KD in theory? I think some clarifications on this are essential.\n\n- Why there is an $\\epsilon$ in theorem 2? I don't understand where it comes from. Can you explain?\n\n\nMinors:\n\nIn Definition 2 in Appendix C.1, page 18 & the second equation in Appendix C.4, page 27, $\\mathcal{P}\\_{v, l}$ should be $\\mathcal{P}\\_{v_{c,l}}$. \n\n\nOverall, I believe the proposed FKD is a novel and (potentially) practically-useful algorithm. However, I think the theoretical results in the paper do not provide any new insights on KD. So, I would suggest the authors move the theoretical results into the appendix, and focus on providing more empirical results, e.g., NLP tasks. I believe this will better sell the paper, and also expose the paper to a broader audience. ", "summary_of_the_paper": "This paper proposed a new knowledge distillation framework, feature kernel distillation (FKD), by matching the student network's feature kernel and that of the teacher network's feature kernel. The authors extended the theoretical results in Allen-Zhu & Li (2020) to demonstrate the advantages of their framework over standard NN training. The theoretical results are built upon a synthetic dataset consisting of multi-view and single-view data, where standard training will fail to generalize while FKD will generalize well. Empirically, the authors (loosely) showed the consistency between the empirical behavior of FKD and the theoretical predictions, though the empirical setting is fairly different from the theoretical setting. In addition, on several image datasets, FKD can outperform other KD baselines in terms of test accuracy.", "summary_of_the_review": "I like the idea of FKD that matches the student network's feature kernel with that of the teacher network in KD. This can apply to situations where the teacher network and student network have different output dimensions. However, the theoretical novelty is limited, and also the empirical evaluations are not thorough (only on some very small datasets). In summary, I vote for a weak acceptance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "84xw_VGNiaY", "reviewer": "Reviewer_h8ud", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Authors nicely motivate their work and draw connections with recent theoretical findings. However, I have several concerns regarding this paper:\n\n1) Authors claim that \"We further use our theory to motivate practical considerations for improving student generalisation when using distillation with feature kernels, which allows us to propose a novel approach: Feature Kernel Distillation (FKD).\" In other words, this work does not only provide a theoretical discussion on feature kernel distillation, but also it is claimed that a novel feature distillation method is proposed. However, there is a vast amount of works that employ similar kernels (even if not explicitly drawing connections with kernels) and employ divergence metrics between the kernels for performing distillation. More specifically, the angle-wise loss used in RKD as proposed in [1] is probably exactly the same (or very similar ) as the formulation used in (3), PKT [2] employs a KL divergence loss based on similar cosine-based kernels (ref to Section 3.1 on correlation kernel, which seems like a cosine-based formulation), while even the work in [3] is highly related, since they proposed MDS-based distillation.\n\nOnly [1] is briefly discussed and included in the experimental setup. However, I think authors should very carefully discuss similarities among the proposed method and the related ones.\n\n[1] Park, Wonpyo, et al. \"Relational knowledge distillation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n[2] Passalis, Nikolaos, and Anastasios Tefas. \"Learning deep representations with probabilistic knowledge transfer.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n\n[3] Yu, Lu, et al. \"Learning metrics from teachers: Compact networks for image embedding.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n\n2) Furthermore, authors seem to also ignore a very large portion of the recent literature on ensemble-based online distillation methods:\n\n[4] Zhang, Ying, et al. \"Deep mutual learning.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.\nLan, Xu, Xiatian Zhu, and Shaogang Gong. \"Knowledge distillation by on-the-fly native ensemble.\", NIPS2018\n\n[5] Chen, Defang, et al. \"Online knowledge distillation with diverse peers.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.\n\nAuthors heavily discuss topics related to ensemble distillation, so I think that recent works on the topics should be discussed and - most importantly - included in the experimental evaluation.\n\n3) On the theoretical side, I am not sure that similarities can indeed provide useful knowledge in datasets with many classes. For example, would the authors expect their method to scale when 1000 classes are used? What kind of kernel should be employed in this case? In my experience, such methods tend to degenerate when the number of the classes go up, since the discriminative power of most kernels goes down. Therefore authors should better define the application area of the proposed method and/or provide relevant experiments. \n\n4) It is unclear if the proposed method is expected to work when activation functions other than ReLU are used. \n\n5) Is the proposed method only applicable when performing distillation from the last fully connected layer?", "summary_of_the_paper": "In this paper the authors consider neural networks as data-dependent kernel machines and propose applying a distillation method directly on the pairwise kernel matrix of the models. Authors extend their setting into ensemble settings, examining both some theoretical aspects of the process, building upon the work of Allen-Zhu & Li (2020), as well experimentally. More specifically, authors provide experiments on ensemble distillation on CIFAR-100, as well as on dataset knowledge transfer on CIFAR-100 to STL-10 and CIFAR-10.", "summary_of_the_review": "Overall, this is a well written paper, easy to follow and the proposed method is - mostly - sound and backed up with some theoretical evidence. However, given the amount of literature ignored, together with the high similarity of the proposed method with many existing ones, I am very skeptical of this paper at this current form. Also, authors should provide more in depth experiments to evaluate how the employed kernel behaves when more classes are used, as well as when there are architecture changes. In other works, the scope and application area of the method should be clearly defined and possible limitation clearly discussed. As a minor point, I feel that some claims are quite bold, e.g., \"This enables us to *prove* that KD using only pairwise feature kernel comparisons can improve NN performance in such settings\". ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2YVRxXgXbz_", "reviewer": "Reviewer_NGSC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper. The paper is clearly written. Though I did not follow the detailed proof in the appendix, the brief description of the proof in the main paper was helpful to understand the idea behind the theorem. \n\nThe following are minor questions:\nIn eq(4), sum-pooling is used. In practice, however, max-pooling is also widely employed. Does the theoretical analysis also work for the max-pooling? \nIn numerical experiments, how was the regularization parameter lambda_KD determined?\nWhen the genuine ReLU is used instead of tilde{ReLU}, can one obtain almost the same numerical results? \n", "summary_of_the_paper": "There are several approaches in knowledge distillation for neural networks. This paper studies kernel-based knowledge distillation for multi-view data set. The authors revealed that the assembled feature map obtained from the ensemble teacher network effectively improves students' generalization ability. Some numerical results are presented to show the practical effectiveness of the learning method based on theoretical findings. \n", "summary_of_the_review": "This is an interesting paper. The paper is clearly written. So my recommendation is to accept this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a borderline paper. 
This paper proposed feature kernel distillation (FKD), a new distillation framework, by matching the kernels obtained from the networks of student and the  teacher.  Theoretical justification is provided by  extending the results of Allen-Zhu and Li(2020)(ALi20 hereafter). Empirical results show superiority of FKD over vanilla KD on several datasets.
There is however concern that the technical novelty is limited and  incremental, an opinion shared by DKJu, and 68WG, compared to ALi20. Reviewer DKJu suggests that the authors could highlight those results which are not straightforward extensions of ALi20. Another important point of concern is that the paper may have some Overstated claims.  The authors clarified that the language of the claims be suitably edited. In this regard Reviewer h8ud have some specific suggestions which should be easy to incorporate. 

In view of additional experiments conducted and detailed discussion during rebuttal addressed some of the concerns of the reviewers.
If accepted, the final version, should include most of the discussion and additional experiments.
Decision: Accept (Poster)