Year: 2021

Abstract:
Trained Neural Networks (NNs) can be viewed as data-dependent kernel machines, with predictions determined by the inner product of last-layer representations across inputs, referred to as the feature kernel. We explore the relevance of the feature kernel for Knowledge Distillation (KD), using a mechanistic understanding of an NN’s optimisation process. We extend the theoretical analysis of Allen-Zhu & Li (2020) to show that a trained NN’s feature kernel is highly dependent on its parameter initialisation, which biases different initialisations of the same architecture to learn different data attributes in a multi-view data setting. This enables us to prove that KD using only pairwise feature kernel comparisons can improve NN performance in such settings, with both single & ensemble teacher models, whereas standard training without KD fails to generalise. We further use our theory to motivate practical considerations for improving student generalisation when using distillation with feature kernels, which allows us to propose a novel approach: Feature Kernel Distillation (FKD). Finally, we experimentally corroborate our theory in the image classification setting, showing that FKD is amenable to ensemble distillation, can transfer knowledge across datasets, and outperforms both vanilla KD & other feature kernel based KD baselines across a range of standard architectures & datasets.

heading: 1 INTRODUCTION & BACKGROUND
text: A prevailing belief in the Deep Learning community is that feature learning, where data-dependent features are acquired during training, is crucial to explaining the empirical success of Neural Networks (NNs) (Fort et al., 2020; Baratin et al., 2021). A comparison in this regard is often made to kernel methods (Jacot et al., 2018), which can be thought of as feature selection methods from a fixed data-independent set of features. This separation has been caricaturised as a distinction between feature learning and kernel learning regimes (Chizat et al., 2019; Yang & Hu, 2020; Woodworth et al., 2020) of NN training. Though less amenable to theoretical analysis compared to kernel regimes, feature learning regimes have the capability to capture more of the complex empirical phenomenon that one can observe in NNs due to parameter-space non-convexity, such as: i) how ensembling trained NNs differing solely in their independent parameter initialisations can lead to improvements in predictive accuracy & uncertainty (Lakshminarayanan et al., 2017; Allen-Zhu & Li, 2020), or ii) the effectiveness of knowledge distillation with both single & ensemble teacher models (Buciluǎ et al., 2006; Hinton et al., 2015). This implies that in order to understand ensembling & knowledge distillation (KD) in NNs, we need to understand the mechanisms of NN feature learning regimes.
Ensembling can be loosely summarised as aggregating predictions from multiple models, & is used widely across machine learning (ML) to improve performance (Dietterich, 2000; Breiman, 2001). Conversely, knowledge distillation, the idea of transferring knowledge from a teacher model to a student model, is more elaborate as described below, & has garnered the most attention with NNs.
Suppose we have a set of input-target (x,y) ∈ Rd×RCS data pairs D̂ = {(xi,yi)}Ni=1 sampled i.i.d. (identically and independently distributed) from some distribution D, and a student NN architecture
fS(x) = WS ·hS(x,θS). Here hS(x,θS) ∈ Rm×1 is a student-specific feature extractor model (e.g. MLP, CNN, ResNet, or Transformer) that takes input x & parameters θS , and WS ∈ RCS×m is a matrix of parameters for a linear output layer. Assume also that we have a loss function:
L(θS ,WS) = 1
N N∑ i=1 L(fS(xi),yi),
which we seek to minimise over θS & WS in the hope that fS can generalise to unseen (x,y) pairs. L is typically the cross-entropy loss in the classification setting.
In vanilla KD (Hinton et al., 2015), we distill knowledge from a trained teacher network fT (x)=WT ·hT (x,θT ) ∈ RCT to a student by augmenting the loss with a regularisation term towards the teacher’s temperature-scaled logits:
L̃(θS ,WS) = L(θS ,WS) + λKD 1
N N∑ i=1 L (fS(xi) τ , fT (xi) τ ) , (1)
for temperature τ>0 & regularisation λKD>0 hyperparameters. Note, this is only valid if CT =CS . Following Hinton et al. (2015), many methods have been proposed using different quirks of NNs to distill knowledge from teacher to student. A relevant line of work involves encouraging the student to match how similar/related the teacher views two inputs x,x′ to be (Tung & Mori, 2019; Park et al., 2019). These approaches have the benefit of being agnostic to teacher/student architectures & prediction spaces CT & CS , but as of yet remain heuristically motivated. In this work, we explore such approaches under the more general framework of NN feature kernel (the kernel induced by the inner product of last-layer features h) learning, allowing us to provide the missing theoretical justification. Moreover, our feature kernel learning perspective enables us to derive implementation considerations in Section 3.1, which we show outperform these previous works in Section 4.
A remarkable property one observes in NN Knowledge Distillation is that it is possible to distill and improve a single student’s generalisation with a more powerful teacher, or an ensemble of teachers. This means that the single student’s model has enough flexibility to generalise well (relative to the teacher), so that one must factor in the optimisation process (such as the parameter initialisation) in order to explain the mechanisms of ensembling and KD in NNs. Allen-Zhu & Li (2020) provided the first theoretical exposition of the mechanisms by which vanilla KD and ensembling improve generalisation in NNs. To this end, the authors introduce the idea of multi-view data, which is when a class in a multi-class classification problem has multiple identifying features/attributes. For example, an image of a car can be discerned by i) wheels, ii) windows, or iii) headlights.
A key idea from Allen-Zhu & Li (2020) is that the parameter initialisation of an NN, and its random correlations with certain attributes, will bias the NN to learn only a subset of the entire set of attributes pertaining to a given class. When presented with single-view data lacking the classidentifying attribute that the NN has learnt, the NN will not generalise. For example, an NN that has learnt to classify cars based on if they have headlights or not will not generalise to a side-on image of a car that occludes headlights.
Allen-Zhu & Li (2020) suggest then that ensembling NNs works in part because independent parameter initialisations learn independent sets of attributes, so more data features will be learnt by the ensemble model. This is fundamentally different to ensembling in strongly convex feature selection problems, such as linear or random features (Rahimi & Recht, 2007) models with `2 regularisation. In such cases, different initialisations reach the same unique optimum, and additional noise must be added to ensure predictive diversity in the ensemble (Matthews et al., 2017). Moreover, Allen-Zhu & Li (2020) claim that vanilla KD in NNs works because the features learnt by the teacher model (or models) are imparted to the student via soft teacher labels that capture ambiguity in a given data input (such as an image of a car whose headlights look like the eyes of a cat). These analyses highlight the fact that without feature learning, it is not possible to fully explain how vanilla KD, variants of KD, or ensembling work in NNs.
Our contributions Feature learning can be thought of as when the feature kernel, induced by the inner product of last-layer representations in a NN, changes during training (Yang & Hu, 2020), and kernel learning in NNs can be thought of as when this kernel is constant. In this work, we take a feature learning perspective of knowledge distillation (KD). We first highlight the importance of the feature kernel by viewing trained NNs as data-dependent kernel machines, & use this to motivate Feature Kernel Distillation (FKD). In FKD, our objective is to ensure that the student’s feature kernel is well suited for improved generalisation, using both the teacher’s data-dependent feature kernel as well as an understanding of the student NN’s optimisation process. In Section 3, we extend the framework of Allen-Zhu & Li (2020) to show that FKD offers the same generalisation benefits as found in vanilla KD in a multi-view data setting, and is further amenable to ensemble distillation. We then derive practical considerations from our theoretical insights, highlighting the benefit of motivating FKD through an understanding of an NN’s feature learning optimisation process, compared to previous methods which implicitly considered the feature kernel for KD. Finally, in Section 4, we provide experimental support that our theoretical claims extend to standard image classification settings, by: verifying that FKD is amenable to ensemble distillation; can transfer knowledge across datasets with different prediction spaces (unlike vanilla KD); and outperforms vanilla KD & previous feature kernel based distillation methods over a range of architectures on CIFAR-100.

heading: 2 MOTIVATION FOR FEATURE KERNEL DISTILLATION
text: One obvious limitation of vanilla KD is that we require student fS and teacher fT to share prediction spaces, i.e. CS=CT . In many situations, we may have a teacher network trained on a dataset with a different number of classes than the student’s dataset, and it is not clear how one could apply vanilla knowledge distillation. One possibility could be to regularise
directly in feature space by comparing hS and hT element-wise, but again this either requires same teacher-student last-layer sizes or additional projection layers.
To eschew such unnecessary complications, we take the perspective of NNs as data-dependent kernel machines. Define an NN’s feature kernel to be: Definition 1 (Feature Kernel). Suppose we have parameters θ and last-layer NN feature extractor h(·,θ):Rd 7→ Rm. For two inputs xi,xj ∈ Rm, the feature kernel, k, is the kernel induced by the inner product of h(xi,θ) and h(xj ,θ):
k(xi,xj) def = 〈h(xi,θ), h(xj ,θ)〉.
At initialisation, it is well known that in the infinite NN-width limit, with appropriate scaling, the feature kernel k converges almost surely to a deterministic kernel known as the Neural Network Gaussian Process (NNGP) kernel (Neal, 2012; Lee et al., 2018; Matthews et al., 2018; Yang, 2019). Yang & Hu (2020) show that there is a parameterisation-dependent dichotomy between kernel & feature learning regimes for infinite-width NNs, where the feature kernel k is constant or changes during training, respectively. It has been widely demonstrated that a crucial component of the success of finite-width NNs is their ability to flexibly learn features, and indeed the feature kernel, from data during training (Fort et al., 2020; Aitchison, 2020; Chen et al., 2020; Maddox et al., 2021).
To see the importance of the feature kernel, note that for a fixed θ with many common loss functions L, and some mild assumptions on strong convexity (which could be enforced e.g. with standard `2 regularisation), the optimal W is uniquely determined and k determines the entire predictive function f∗(·). For example, with squared error, L(f(x),y) = ‖f(x)− y‖22, and `2 regularisation
strength λ > 0, a trained NN is precisely kernel ridge regression with the data-dependent feature kernel k, whose job is to measure how similar different inputs are. Thus, all teacher knowledge is contained in its feature kernel, kT , so the feature kernel can act as our primary distillation target, as depicted in Fig. 1. We show a corresponding result for cross-entropy loss in Appendix A.
Fig. 2 corroborates our claims. For a given ResNet20v1 (He et al., 2016) ‘reference’ model trained on CIFAR10 with cross entropy, we plot test class prediction confusion matrices in log-scale between said model and: i) a retrained version where we fix all but the last-layer parameters (hence fully determined by the reference model’s feature kernel as per Appendix A) & ii) an independent model trained from a different initialisation. We see, as expected, that there is significantly more disagree-
ment across test predictions for models with different initialisations than those which share feature kernels. This highlights the fact that: a) different initialisations bias the same architecture to learn different features, and b) the feature kernel (largely) determines a model’s test predictions. Experimental details and a breakdown of the predictive disagreements can be found in Appendix F.
Having described the feature kernel as a central object in any NN, we use this to motivate our proposed FKD, where we treat the student’s feature kernel, kS , as a key distillation target using the teacher’s feature kernel, kT . Encouraging similarity across feature kernels shares useful features that the teacher has learnt with the student, which we theoretically show in Section 3.
We define the FKD student loss function via an additive regularisation term between feature kernels:1
LλKD(θ,W ) def= L(θ,W ) + λKD · D(kθ, kT ) (2) where λKD > 0 is the regularisation strength, D is some (pseudo-)distance over kernels, and the student feature kernel k = kθ is written to make explicit the dependence on student feature extractor parameters θ. We stress that Eq. (2) does not require matching prediction spaces between teacher and student, allowing us to apply FKD across tasks and datasets.
We train θ using a stochastic optimisation algorithm such as stochastic gradient descent. For that reason, we require D to be amenable to stochastic mini-batching, therefore, in this work we choose
D(kθ, kT ) = E x1,x2 i.i.d.∼D̂
[( kθ(x1,x2)− kT (x1,x2) )2] , (3)
where we can approximate the expectation with an average of pairwise differences in a minibatch.

heading: 3 THEORETICAL ANALYSIS
text: We now extend the theoretical framework of Allen-Zhu & Li (2020), which is restricted to vanilla KD, to demonstrate the generalisation benefits of our FKD over standard training in a multi-view data setting, and use our theory to motivate practical considerations when using feature kernel distillation. We first describe the mutli-view data setting & CNN architecture we consider, before recalling that standard training without KD fails to generalise well. We then provide our main theoretical result, Theorem 2, which shows that FKD can improve student test performance. Though
1We will sometimes drop the student S sub/superscript where obvious for clarity, like in Eq. (2). Any teacher specific object, e.g. kT , will always have corresponding T sub/superscript, so there should be no confusion.
our theoretical results are limited to a specific scenario, inspired by real-world data (Allen-Zhu & Li, 2020), & NN architecture, we believe the setup we consider is apt: it is simple enough to be tractable, yet rich enough to display the merits of FKD. Moreover, we find in Section 4 that our conclusions generalise to standard architectures & image datasets. In the interest of space & readability, we focus on providing intuition in this section, and fill in remaining details/proofs in the appendix.
Multi-view data. We consider the data classification problem introduced by Allen-Zhu & Li (2020), with C classes and inputs x with P patches each of dimension d, meaning x ∈ (Rd)P . For each class c, we suppose that there exist two attributes vc,1, vc,2 ∈ Rd. For x belonging to class c, the attributes found in patches of x will include vc,1 and vc,2, as well as a random selection of out-of-class attributes {vc′,l}c′ 6=c,l∈[2].2 This denotes the multi-view nature of the data distribution. In the true data-generating distribution D, we suppose that a proportion µ of the data (x,y) is single-view, which means that only one of vc,1 or vc,2 is present in x when (x,y) is from class c. These will be the data for which standard training fails to generalise. A precise definition of multi-view data is presented in Appendix B.1.1. Allen-Zhu & Li (2020) argue that this multi-view setting provides a compelling proxy for standard image datasets such as CIFAR-10/100.3
Intuition: FKD on multi-view data Suppose we have an image classification task, with cat & car just two out of many classes. For the car class, vc,1 could correspond to headlights, whilst vc,2 could correspond to wheels. We would then expect vc,1 to also appear in patches of an input image, xcat, corresponding to a cat with headlight-like eyes. Allen-Zhu & Li (2020) show that a single trained model is biased to learn exactly one of vc,1 or vc,2, depending on its parameter initialisation. W.L.O.G, suppose that the student is biased to learn vc,2 & not vc,1. If the teacher model has learnt vc,1, this means that the teacher model knows there is a similarity between xcat & any car image, xcar, that displays headlights. Mathematically, we show that this corresponds to a large value for kT (xcat,xcar). Our FKD regularisation forces the student to also have a large value for kS(xcat,xcar), ensuring that attribute vc,1 is also learnt by the student network. Without distillation, a student NN which has learnt vc,2 & not vc,1 will not generalise to front-on images of cars that hide wheels.
Convolutional NN & corresponding feature kernel. Like Allen-Zhu & Li (2020), for our theoretical analyis we consider a single hidden-layer convolutional NN (CNN) with sum-pooling. For each class c ∈ [C], we suppose that the CNN has m channels, giving Cm channels in total. For channel r and class c, we suppose that we have weights θc,r ∈ Rd. This gives output for class c by
fc(x) def = m∑ r=1 P∑ p=1 R̃eLU(〈θc,r,xp〉) (4)
where for ease of analysis R̃eLU is ReLU-like but with continuous gradient, see Appendix B.2.
To define the feature kernel for this CNN f , we recast f(x)=W ·h(x,θ), where h(x,θ)∈RCm and W∈RC×Cm satisfying, for r ∈ [m], c ∈ [C], c′ ∈ [C]:
h(x,θ)r+(c−1)m def = P∑ p=1 R̃eLU(〈θc,r,xp〉), and Wc′,r+(c−1)m def = 1{c = c′}. (5)
Given that the feature kernel is k(x,x′) def= 〈h(x,θ), h(x′,θ)〉, we now have that:4
k(x,x′) = C∑ c=1 m∑ r=1 P∑ p,p′=1 R̃eLU(〈θc,r,xp〉) · R̃eLU(〈θc,r,x′p′〉). (6)
2It is straightforward to extend to the case of more than two views per class if need be. 3https://www.microsoft.com/en-us/research/blog/three-mysteries-in-deep-
learning-ensemble-knowledge-distillation-and-self-distillation/
We first recall that standard training of the model f with gradient descent and cross entropy loss fails to generalise on half the µ proportion of data that is single-view. Theorem 1 (Standard training fails, Theorem 1 of Allen-Zhu & Li (2020)). For sufficiently many classes C and channels m ∈ [polylog(C), C], with learning rate η ≤ 1poly(C) , training time T ∗ = poly(C)η , and multi-view data distribution (Appendix B.1.1), the trained model f (T∗) satisfies with probability at least 1− e−Ω(log2(C)):
• Training accuracy is perfect: For all (x,y) ∈ D̂, y = argmaxc f (T∗) c (x). • Test accuracy is bad but consistent: P(x,y)∼D [ y 6= argmaxc f (T∗) c (x) ] ∈ [0.49µ, 0.51µ].
Now we are ready to show that regularising the student model towards the teacher model’s feature kernel, as in FKD, improves test accuracy. We suppose our teacher model is an ensemble of E ≥ 1 models {fe}Ee=1, each with corresponding feature kernel ke, trained as standard on the same data with independent initialisations θe0. We average ke over e ∈ [E] to obtain our teacher feature kernel:
kT (x,x ′) =
1
E E∑ e=1 ke(x,x ′). (7)
This is akin to concatenating all features in {he}Ee=1 into a ECm-dimensional feature vector, albeit without the additional computational baggage. We then have our main theoretical result: Theorem 2 (FKD improves student generalisation and is better with larger ensemble). Given an arbitrary > 0. For any ensemble sizeE of teacher NNs trained as in Theorem 1 and sufficiently many classes C, for m = polylog(C), with learning rate η ≤ 1poly(C) , and training time T ∗ = poly(C)η , the ensemble teacher knowledge can be distilled into a single student model f (T ∗) using only teacher feature kernel kT , Eq. (7), such that with probability at least 1− e−Ω(log 2(C)):
• Training accuracy is perfect: For all (x,y) ∈ D̂, y = argmaxc f (T∗) c (x). • Test accuracy is good: P(x,y)∼D [ y 6= argmaxc f (T∗) c (x) ] ≤ ( 1 2E+1 + )µ.
To give an outline for the proof of Theorem 2, we first show, in Lemma 1, that a single trained NN’s feature kernel is able to detect if two inputs share a data attribute that has been learnt by said NN. We then extend this result to an ensemble teacher in Appendix C.2, showing that the ensemble teacher feature kernel detects the union of all data attributes learnt by individual {ke}Ee=1. This simplifies our calculations when showing that our distillation regulariser, Eq. (3), is effective for improved student generalisation. The full proof of Theorem 2 can be found in Appendix C.
Intuition: FKD with ensemble of teachers To parse Theorem 2, suppose that ensemble size E=1. Then, test error is essentially bounded by 0.25µ in Theorem 2. The explanation is that both the student and teacher networks independently learn exactly one of {vc,l}2l=1 for each class c. Either vanilla KD (Theorem 4 of Allen-Zhu & Li (2020)) or our feature kernel approach allow the student to learn the union of the independent attributes learnt by the student and the teacher, so for only a quarter of the single-view test data xs will the student not have learnt the useful class attribute present in xs. For general ensemble size E, the story is the same: the student & E teachers each independently learn one of the two useful attributes {vc,l}l∈2 for all c ∈ [C]. Distilling allows the student to learn the union of these attributes, which means that the student will fail on only 1
2E+1 of the single-view data.
4The feature kernel defined in Eq. (6) corresponds to the Global Average Pooling CNN-GP kernel in Novak et al. (2018) in the infinite-channel limit, which captures intra-patch correlations unlike the vectorised CNN-GP, which corresponds to vectorising the spatial dimensions to give CmP rather than Cm channels.

heading: 3.1 PRACTICAL CONSIDERATIONS
text: Next, we highlight practical considerations for implementing FKD derived from our theory.
Correlation kernel. We propose D(ρθ, ρT ) as a regulariser in FKD instead of D(kθ, kT ), where:
ρz(x,x ′) def = kz(x,x ′)√
kz(x,x)kz(x′,x′) , and ρT (x,x′) def =
1
E E∑ e=1 ρe(x,x ′)
defines feature correlation kernel ρz , corresponding to feature kernel kz , for z∈[E]∪{θ}. The reason we use correlation kernels is that they normalise data, so that ρ(x,x)=1 ∀x, which zeros diagonal differences in Eq. (3): we hope FKD allows the student to learn from the teacher features shared between different inputs. Non-zero diagonal differences, like in Similarity-Preserving KD (Tung & Mori, 2019), encourage the student to learn noise as we show in Appendix D.1, and we hypothesise that this contributes to the improved performance we observe of FKD over SP in Section 4. Moreover, this normalisation helps balance individual teacher’s influence in an ensemble teacher, and ensures that FKD does not need a temperature hyperparameter τ , which produces soft labels in vanilla KD. Pseudo-code for our practical FKD implementation is shown in Appendix E.
Feature regularisation. One downside to using the correlation kernel is that our FKD regularisation, D(ρθ, ρT ), becomes invariant to the scale of kθ. For example, replacing kθ(x,x′) with √ M(x)M(x′)kθ(x,x
′), for any function M(x):Rd 7→R+, leaves the student correlation kernel unchanged. This may lead to degeneracies when training θ, & large variations in k(x,x) over xmay lead to worse generalisation (as evidenced by the fact that input normalisation is common practice in ML, from linear models to NNs). Moreover, our proof of Theorem 2 is not quantitative, in the sense that we only show that kθ(x,x′) = Θ̃(1) in the number of classes C up to polylogarithmic factor in C, when x 6=x′ share a data attribute vc,l that has been learned by parameters θ. These insights motivate us to regularise the student feature h during FKD training to control the norm of k(x,x) across inputs x. We use an additive `2 regularisation λFR 1B ∑B b=1 ‖h(xb,θ)‖
2 2 = λFR 1 B ∑B b=1 k(xb,xb) with regularisation strength λFR > 0, for each minibatch {(xb,yb)}Bb=1. Fig. 3 shows that using Feature Regularisation (FR) encourages a more even spread of k(x,x) across inputs for a student VGG8 network trained with a VGG13 teacher model on CIFAR-100. Corresponding plots for other architectures can be found in Fig. 7. Similar to Dauphin & Cubuk (2021), we find that FR improves generalisation performance & provide ablations in Section 4.
4 EXPERIMENTS
Due to space concerns, Appendix F contains all experimental details not discussed in this section.
Ensemble distillation. We first verify that increasing ensemble teacher size, E, can allow FKD to improve student performance further as suggested by Theorem 2. This is confirmed in Fig. 4, using VGG8 for all student & teacher networks on the CIFAR-100 dataset. For each E, we also plot the test accuracy of the ensemble teacher of sizeE, whose predictive probabilities are averaged over individual teachers, as well as the test
accuracy of a single student model. We see that FKD consistently outperforms vanilla KD, and both distillation methods outperform the teacher in the ‘self-distillation’ setting of E=1 (Furlanello et al., 2018; Zhang et al., 2019). Moreover, FKD even allows a single student to match teacher performance when E=2, before positive but diminishing returns with increased E relative to the teacher ensemble. As discussed in Section 6, this suggests that there are still gaps between our theory and KD in practice.
Dataset Transfer. We next show that FKD is able to transfer useful knowledge across similar datasets. From a fixed VGG13 teacher network trained on CIFAR-100, we distil to a student VGG8 network on both CIFAR-10 & STL-10. As both student datasets have 10 classes compared to 100 for CIFAR-100, it is not clear how one should use vanilla KD (Hinton et al., 2015) in this setting. We thus compare FKD to other feature kernel based KD methods: Relational KD (RKD) (Park et al., 2019) & Similarity-Preserving (SP) KD (Tung & Mori, 2019). In Table 1, we see that FKD without feature regularisation outperforms both baselines, and that feature regularisation (FR) further improves FKD performance, highlighting the benefit of our practical considerations in Section 3.1. The improved performance is particularly stark on STL-10 (which we downsize to 32x32 resolution), where FKD improves student performance by 2.75%. STL-10 is well suited for FKD as it has only 5K labeled inputs but 100K unlabeled datapoints, which can be used in our feature kernel regulariser, D(ρθ, ρT ).
Comparison on CIFAR100. We next compare FKD to various knowledge distillation baselines on CIFAR100, across a selection of teacher/student architectures. We see in Table 2 that FKD consistently outperforms: vanilla KD (Hinton et al., 2015), RKD (Park et al., 2019), and SP (Tung & Mori, 2019). Moreover, FKD either matches or outperforms the high-performing Contrastive Representational Distillation (Tian et al., 2020). We use the exact same teacher checkpoints provided by Tian et al. (2020) to ensure fair comparison. We find, like in Table 1, that feature regularisation consistently improves FKD performance and that even without feature regularisation, FKD significantly outperforms all feature kernel based KD methods. This implies that using the correlation kernel to zero out diagonal differences, as described in Section 3.1, indeed helps improve student performance.
5As found here: https://github.com/HobbitLong/RepDistiller

heading: 5 RELATED WORK
text: NN Knowledge Distillation. Following the seminal work of Hinton et al. (2015), there has been much interest in expanding and improving the performance of KD in NNs (Romero et al., 2014; Zagoruyko & Komodakis, 2016; Tian et al., 2020). Most similar to our work are Park et al. (2019); Tung & Mori (2019) who also use relations between inputs to distil knowledge (albeit not from the feature kernel learning perspective and without our theoretical justification/practical considerations), as well as Qian et al. (2020) who focus on reducing computational costs of expensive kernel matrix operations, whereas we show that FKD works with only pairwise interactions between inputs. Appendix D highlights some differences of FKD compared to previous pairwise feature kernel based KD methods in more detail. Allen-Zhu & Li (2020) made the first theoretical connection using the mechanisms of ensembling in NNs to explain the success of vanilla KD in NNs, which we extend for feature kernel based KD. Ensembling NNs. Ensembling NNs has long been studied for improving predictive accuracy (Hansen & Salamon, 1990; Krogh et al., 1995) with particular recent focus towards uncertainty quantification & Bayesian inference (Lakshminarayanan et al., 2017; Ovadia et al., 2019; Zaidi et al., 2020; Pearce et al., 2020; He et al., 2020; Wilson & Izmailov, 2020; Wenzel et al., 2020; D’Angelo & Fortuin, 2021; Schut et al., 2021) and predictive diversity (Fort et al., 2019; D’Amour et al., 2020). On the topic of Bayesian inference, the feature kernel has also been studied under the name of Neural Linear Model (Riquelme et al., 2018; Ober & Rasmussen, 2019), and extensions treating features h as inputs to standard Gaussian Process kernels are known by the name of Deep Kernel Learning (Wilson et al., 2016; Ober et al., 2021; van Amersfoort et al., 2021). NN Feature learning. A recent flurry of work has focussed on characterising and understanding the importance of feature learning in NNs (Chizat et al., 2019; Fort et al., 2020; Baratin et al., 2021; Lee et al., 2020; Aitchison, 2020; Ghorbani et al., 2020), fuelled in part by the development that wide NNs become (Neural Tangent) Kernel machines in certain regimes (Jacot et al., 2018; Lee et al., 2019; Yang & Littwin, 2021). The consensus among these works is that there are gaps between the NTK framework and practical NNs that cannot be explained without feature learning. However, Yang & Hu (2020) proved that feature-learning is still possible with infinite-width NNs, and also showed that feature learning is equivalent to feature kernel learning in infinite-width NNs. This motivates our study of the feature kernel as a key object for distillation. Regularising the feature kernel to the true target covariance kernel was suggested by Yoo et al. (2021).

heading: 6 CONCLUSION
text: We have theoretically shown that the feature kernel is a valid object for Knowledge Distillation (KD) in Neural Networks (NNs), by extending the analysis of Allen-Zhu & Li (2020) in a multiview data setting. Further, we used an understanding of a student NN’s optimisation process to motivate practical considerations when using feature kernels for distillation. These include using the feature correlation kernel & using feature regularisation, and we term our approach Feature Kernel Distillation (FKD). Experimentally, we have demonstrated that FKD is amenable to ensemble distillation as suggested by our theory, is able to transfer knowledge across similar datasets and that FKD outperforms vanilla KD & other feature kernel based KD methods across a variety of architectures of CIFAR100. Limitations & future work. Though feature learning is crucial to our results, we stress that there are still gaps between our theory and practice to understanding NN ensembling & KD, as demonstrated by the divergence between ensemble teacher & FKD in Fig. 4 for larger ensemble size. This could be due to a number of reasons, such as the multi-view data setting not being able to capture the full complexity of real-world data, the role of hierarchical feature learning between layers in a deep NN, or the importance of minibatching in stochastic gradient descent. Other future work could involve using the multi-view data setting to analyse the benefit of ensembling NNs for uncertainty quantification, or improving FKD further to compete with state-of-the-art KD methods.

heading: APPENDIX: FEATURE KERNEL DISTILLATION

heading: A FEATURE KERNEL DEPENDENCE WITH CROSS-ENTROPY LOSS
text: Suppose, we have n data points with fixed feature extractor h = h(X) ∈ Rn×p, trainable last layer weights W ∈ Rp×C and targets Y ∈ Rn×C . In Section 2, we described how with Mean Squared Error loss the trained NN is precisely kernel ridge regression using the feature kernel, when there is `2 regularisation on the last layer weights. Hence, the feature kernel exactly determines the predictions. We now prove the same for cross entropy loss. That is to say, we wish to show that given the feature kernel k, the predictive probabilities/logits at the optimal W ∗ are independent of both W ∗ and h.
We define the loss, for some regularisation λ > 0 (purely to enforce strong convexity) by
L(W ) = n∑ i=1 CE(yi, hiW ) + λ 2 ‖W‖22 ,
where hi ∈ R1×p is the ith row of h and
CE(y, f) = −log e fy∑
c∈[C] e fc , (8)
is cross entropy loss.
Proposition 1. Let feature extractor h(·), training inputs X , training targets Y , `2 regularisation λ > 0 all be fixed. Suppose we are given a test point x′ with features h′, then the test prediction logits h′W ∗ at optimal W ∗ can be expressed solely in terms of feature kernel evaluations k(·, ·)=〈h(·), h(·)〉.
Proof. If we differentiate L(W ) and set to zero we get:
λW ∗j,l = n∑ i=1 hi,j [ 1{yi = l} − pi,l ] (9)
where p ∈ Rn×C is the result of applying softmax to each row of the logits h′W ∗. Let hi denote the extracted features for training input xi. Now recall that 〈hi, hj〉 = k(xi,xj) is the feature kernel evaluated at xi,xj . We multiply Eq. (9) by the test-data feature vector h′ to give:
λ(h′W ∗)l = n∑ i=1 k(x∗,xi) [ 1{yi = l} − pi,l ] (10)
but h′W ∗ are precisely the logits for test point x∗, and likewise pi,l is the vector of probability predictions at training point i. Hence, we could solve Eq. (10) numerically for logits/predictive probabilities given only the feature kernel (without h(·) or W ∗), and we see that the feature kernel once again determines logit/prediction probabilities at the optimal last layer parameters like for squared error loss, albeit this time implicitly for cross entropy loss.

heading: B SETUP FOR TRAINING ON MULTI-VIEW DATA
text: Before we prove Theorem 2, we need to recall the data and training setup of Allen-Zhu & Li (2020) for completeness and self-containedness. Unless otherwise stated, everything in this section (Appendix B) is a simplified version of the setup in Allen-Zhu & Li (2020). Our results should hold in the more general version too, but we present the setup in a simplified setting here for readability and convenience, without sacrificing the key messages and intuitions.

heading: B.1 MULTI-VIEW DATA DISTRIBUTION
text: Recall that we consider a C-class classification problem over P -patch inputs, where each patch has dimension d, so our inputs are described by x = (x1, . . . ,xP ) ∈ (Rd)P . For simplicity, we take P = C2 and d = poly(C) for a large polynomial. Like Allen-Zhu & Li (2020), we use Õ, Θ̃, Ω̃ to hide polylogarithmic factors in the number of classes, C, which we take to be sufficiently large.
We assume that each class c ∈ [C] has exactly two attributes vc,1, vc,2 ∈ Rd, which are orthonormal for simplicity,6 such that:
V = {vc,1, vc,2}c∈[C] is the set of all attributes.7

heading: B.1.1 DATA GENERATING MECHANISM
text: Let our data distribution for a data pair (x,y) ∼ D be defined as D = µDs + (1−µ)Dm, for multiview & single-view distributions Dm & Ds respectively. (x,y) ∼ D are generated as follows:
1. Sample y ∈ [C] uniformly at random 2. Sample a set V ′(x) of attributes uniformly at random from {vc′,1, vc′,2}c′ 6=y each with
probability sC and denote V(X) = V ′(x) ∪ {vy,1, vy,2} as the set of attribute vectors used in data x. We take s = C0.2. Intuition:
V ′(x) correspond to the ambiguous attributes present in x, such as the cat whose eyes look like car headlights. These V ′ are crucial in our proofs as the FKD regularisation between ambiguous images ensures that the student learns attributes it would have otherwise missed.
For each v ∈ V(x), pick Cp (where Cp is a global constant) many disjoint patches in [P ] uniformly at random and denote this set as Pv(x). Denote P(x) = ∪v∈V(x)Pv(x): all other patches p 6∈ P(x) will contain noise only.
3.4 If x is single view, pick a value l̂ = l̂(x)∈{1, 2} uniformly at random. l̂ corresponds to the attribute vy,l̂ that is present in x, with vy,3−l̂ missing.
5. For each p ∈ Pv(x) for some v ∈ V(x): xp = zpv + ξp
where ξp ∼ N (0, σ2pId) is independent random noise, with σp= 1√dpolylog(C) . The coefficients zp ≥ 0 satisfy:
6As d = poly(C) for a large polynomial, this isn’t too far-fetched an assumption. 7Throughout, we refer to the input patch features vc,l as attributes in this multi-view data setup, and the NN
last-layer activations/representations as features to avoid confusion.
(a) If x is multi view, • When v ∈ {vy,1, vy,2}, {∑
p∈Pv(x) zp ∈ [1, 2)∑ p∈Pv(x) z 4 p = 1
(11)
Intuition: These conditions ensure that both attributes for class y are equally likely to be learnt, averaged over different random initialisations of parameters.
• When v ∈ V ′(x), {∑ p∈Pv(x) zp = 0.4∑ p∈Pv(x) z 4 p = Θ(1)
(12)
(b) If x is single view, • When v=vy,l̂,
∑ p∈Pv(x) zp = 1
• When v=vy,3−l̂, ∑ p∈Pv(x) zp = C −0.2
• When v ∈ V ′(x), ∑ p∈Pv(x) zp = Γ
where Γ = 1polylog(C) Intuition:
This is where the single view name comes from, as C−0.2 1 we see that vy,3−l̂ is barely present in x.
6. For each p ∈ [P ]\Pv(x): xp = ξp
where ξp ∼ N (0, 1CdId) is independent random noise.
Remark It is possible to allow more relaxed assumptions, e.g. on zp, as in Allen-Zhu & Li (2020).
Training data Recall we have D = µDs + (1− µ)Dm, so that a proportion (1− µ) of the data is multi-view. Our training data, D̂, is N independent samples from D. Letting D̂ = D̂m ∪ D̂s denote the split into multi and single view training data. We let µ = 1poly(C) and we suppose |N | = C1.2 µ so that each label c appears at least Ω(1) in D̂s.

heading: B.2 SMOOTHED RELU CNN
text: Our theoretical analysis considers a single hidden layer CNN with sum-pooling, f , such that for class c:
fc(x) = m∑ r=1 P∑ p=1 R̃eLU(〈θc,r,xp〉), ∀c ∈ [C].
For a threshold % = 1polylog(C) , we define the smoothed ReLU function as:
R̃eLU(z) =  0 if z ≤ 0 z4
4%3 if z ∈ [0, %] z − 34% if z ≥ %
which has continuous monotonic gradient denoted as R̃eLU ′ .

heading: B.3 HYPERPARAMETER VALUES IN SETUP
text: Hyperparameter values used in the theoretical setup are given in Table 3.

heading: B.4 STANDARD TRAINING
text: We define our empirical loss by
L = 1 N ∑ i∈[N ] L(f(xi),yi)
where L is the cross entropy loss defined in Eq. (8). We randomly initialise parameters θ0c,r
i.i.d.∼ N (0, σ20) for σ20 = 1C . Standard training in the theoretical analysis comprises of full-batch gradient descent on the empirical loss L with learning rate η ≤ 1poly(C) and for T = poly(C) η iterations.

heading: C PROOF OF THEOREM 2
text: We restate Theorem 2, where recall E denotes the teacher ensemble size and C is the number of classes:
Theorem 2 (FKD improves student generalisation and is better with larger ensemble). Given an arbitrary > 0. For any ensemble sizeE of teacher NNs trained as in Theorem 1 and sufficiently many classes C, for m = polylog(C), with learning rate η ≤ 1poly(C) , and training time T ∗ = poly(C)η , the ensemble teacher knowledge can be distilled into a single student model f (T ∗) using only teacher feature kernel kT , Eq. (7), such that with probability at least 1− e−Ω(log 2(C)):
• Training accuracy is perfect: For all (x,y) ∈ D̂, y = argmaxc f (T∗) c (x). • Test accuracy is good: P(x,y)∼D [ y 6= argmaxc f (T∗) c (x) ] ≤ ( 1 2E+1 + )µ.

heading: Outline of proof of Theorem 2
text: 1. We first analyse the feature kernel k for a single trained model, in Appendix C.1.
2. We then extend our results to an ensembled teacher model feature kernel in Appendix C.2.
3. We next outline our kernel distillation training scheme and provide a key result concerning how the student’s parameters become increasingly correlated with the teacher’s learnt views in Appendices C.3 and C.4
4. We combine all threads to prove the final result in Appendix C.5.

heading: C.1 FEATURE KERNEL FOR A SINGLE TRAINED MODEL
text: To prove Theorem 2, we first calculate the feature kernel k∗ for a single trained model with trained parameters θ∗ from initialisation θ0. The point of this exercise is to show that the feature kernel k∗ is able to detect whether two inputs x,x′ share a common attribute which is in the subset of attributes that was learnt by the trained parameters θ∗, due to random correlation with initialised parameters θ0. This is formally shown in Lemma 1.
Recall the definition of the feature kernel k for our CNN architecture:
k(x,x′) = C∑ c=1 m∑ r=1 P∑ p,p′=1 R̃eLU(〈θc,r,xp〉) · R̃eLU(〈θc,r,x′p′〉)
Let us first make a few more useful definitions. For l = 1, 2, we have:
Definition 2. Zc,l(x) def = 1{vc,l ∈ V(x)} ∑ p∈Pv,l(x) zp.
Intuition:
Zc,l(x) is a scalar constant describing the strength, in x, of the presence of attribute vc,l (where c may either correspond to true class y or to a different class; the latter has the effect of producing ‘soft labels’ when using knowledge distillation).
Definition 3. Φ∗c,l def = ∑ r∈[m] [〈θtc,r, vc,l〉]+.
Definition 4. Υ∗c,l def = ∑ r∈[m] [〈θ∗c,r, vc,l〉]+ 2 .
Intuition: Υ∗c,l,Φ ∗ c,l are both parameter-dependent, data-independent scalars that describe the amount that attribute vc,l has been ‘learnt’ by the network parameters. Υ∗c,l appears in the feature kernel whereas Φ∗c,l appears in the function predictions fc directly, for our particular architecture in Eq. (4) (Allen-Zhu & Li, 2020).
Lemma 1. We have, for x,x′ ∼ D̂m,
k∗(x,x ′) = ∑ (c,l):(c,3−l)/∈M Υ∗c,lZc,l(x)Zc,l(x ′)±O ( sM(x,x′) polylog(C) ) + Õ( 1 C0.8 )
where sM(x,x
′) = |{(c, l) : vc,l ∈ V(x) ∩ V(x′) and (c, 3− l) /∈M}| is the number of shared attributes of x and x′, that are also members of the set of attributes learnt by the single trained model with parameters θ∗, andM is defined in Fact A.e.
Proof. We have
k∗(x,x ′) = C∑ c=1 m∑ r=1 [ P∑ p=1 R̃eLU(〈θ∗c,r,xp〉) · P∑ p′=1 R̃eLU(〈θ∗c,r,x′p′〉) ] .
First, define the hidden-layer activations for class c and channel r to be:
Definition 5.
Ψc,r(x) def = P∑ p=1 R̃eLU(〈θ∗c,r,xp〉).
We know from Theorem C.2. of Allen-Zhu & Li (2020) that, for multi-view x ∈ D̂m and some c ∈ [C]:
Fact A.a For every p ∈ Pvc,l(x) and for l = 1, 2, we have: 〈θ∗c,r,xp〉 = 〈θ∗c,r, vc,l〉zp ± õ(σ0)
Fact A.b For every p ∈ P(x)\(Pvc,1(x) ∪ Pvc,2(x)), we have |〈θ∗c,r,xp〉| ≤ Õ(σ0)
Fact A.c For every p ∈ [P ]\P(x), we have |〈θ∗c,r,xp〉| ≤ Õ( σ0√C )
Fact A.d For every r ∈ [m]\M0c , every l ∈ [2], it holds that 〈θ∗c,r, vc,l〉 ≤ Õ(σ0), where:
M0c def =
{ r ∈ [m] ∣∣∣∣ ∃l ∈ [2] : 〈θ0c,r, vc,l〉 ≥ (1−O( 1log(C) )) · maxr∈[m][〈θ0c,r, vc,l〉]+ } .
Note from Proposition B.1 of Allen-Zhu & Li (2020), that m0 def = |M0c | = O(log5(C)) with probability at least 1− e−Ω(log5(C)).
Intuition:
M0c denotes the key channels in [m] which have ‘won the lottery’ and are relevant for class c, in that in the C → ∞ limit the predictions for fc are the same as if one forgets the other channels, as shown in Allen-Zhu & Li (2020).
Fact A.e For every p ∈ Pvc,l(x) and r ∈ [m], if (c, 3− l) ∈M, then we have: |〈θ∗c,r, xp〉| ≤ Õ(σ0), where:
M def= { (c, l) ∈ [C]× [2] ∣∣∣∣ max r∈[m] [〈θ0c,r, vc,l〉]+ ≥ ( 1 +
1
log2(m) ) · max r∈[m] [〈θ0c,r, vc,3−l〉]+ }
Intuition: M denotes the attributes vc,l which are more likely to be learnt by the NN parameters (compared to their fellow class attributes vc,3−l) because of correlations of the initial parameters with such attributes. So, Fact A.e is saying that if (c, 3− l) ∈ M, then the attribute (c, l) is not learnt at all during standard single model training.
Thus,
Ψc,r(x) = 2∑ l=1 1{vc,l ∈ V(x)} · ∑
p∈Pvc,l (x)
R̃eLU ( 〈θ∗c,r,xp〉 ) (13)
+ ∑
p∈P(x)\ ⋃
l{Pvc,l (x)}
R̃eLU ( Õ(σ0) ) (14)
+ ∑
p∈[P ]\P(x)
R̃eLU ( Õ(
σ0√ C
) ) . (15)
First, note that |Pvc′,l′ (x)| = Cp is constant ∀c ′, l′. From Fact A.b, Eq. (14) can be easily seen to be Õ(σ40s) = Õ(C −1.8) as σ20 = 1 C and s = C
0.2, and likewise Eq. (15) can be seen to be Õ(( σ0√
C )4P ) = Õ(C−2) as P = C2, by Fact A.c. Summing these equations over m and C will be
bounded above by O( 1polylog(C) ).
Now, let’s consider Eq. (13). Let notation vc,1, vc,2 ∈ S denote either vc,1 or vc,2 ∈ S for some set S, and vc,1, vc,2 /∈ S denote neither vc,1 nor vc,2 ∈ S. There are three cases to consider:
1. If vc,1, vc,2 /∈ V(x), then Eq. (13) is zero.
2. Else if ∃l ∈ [2] such that vc,l ∈ V(x) and (c, 3 − l) ∈ M, then by Fact A.e, we have Eq. (13) is Õ(σ40)=Õ(C −2).
Intuition:
This setting is when only attribute (c, l) appears in x, but attribute (c, 3− l) was dominant at initialisation so (c, l) has not been learnt by θ∗.
3. Else, we have that:
Eq. (13) = 2∑ l=1 1{vc,l ∈ V(x)} · ∑
p∈Pvc,l (x)
R̃eLU ( 〈θ∗c,r,xp〉 ) .
Putting this all together with Fact A.a, we see that
Ψc,r(x) = 2∑ l=1 1{(c, l) ∈ sM(x,x)} ∑
p∈Pvc,l (x)
R̃eLU ( 〈θ∗c,r, vc,l〉zp + õ(σ0) ) + Õ( 1
C1.8 ).
Now if r /∈M0c and (c, 1), (c, 2)∈sM(x,x), by Fact A.d, we have that Ψc,r(x)=Õ(C−1.8).
On the other hand, recall % = 1polylog(C) and m0 = O(log 5(C)). And also, recall that if z > %, then R̃eLU(z) = z + O(%). Moreover, by Claim C.11 of Allen-Zhu & Li (2020) we know that ∃r′, l′ such that 〈θ∗c,r′ , vc,l′〉 = Ω( 1m0 ). Hence, for any r ∈M 0 c , we know that either:
1. Ψc,r(x) = ∑2 l=1 1{(c, l) ∈ sM(x,x)} ( 〈θ∗c,r, vc,l〉Zc,l(x)±O(%) ) + Õ( 1C1.8 ), or
2. Ψc,r(x) = 1{(c, 1), (c, 2) ∈ sM(x,x)}O(%) + Õ( 1C1.8 ).
Moreover, Ψc,r(x) = Õ(1) ∀r, by e.g. Lemma C.21 of Allen-Zhu & Li (2020). And so it’s easy to see, for % small enough we have:
m∑ r=1 Ψc,r(x)Ψc,r(x ′)
= 2∑ l=1 1{(c, 3− l) /∈M}Zc,l(x)Zc,l(x′)[ m∑ r=1 [ 〈θ∗c,r, vc,l〉]+ 2 ±O( 1 polylog(C) ) ] + Õ( 1 C1.8 )
= 2∑ l=1 1{(c, 3− l) /∈M}Zc,l(x)Zc,l(x′)[Υ∗c,l ±O( 1 polylog(C) ) ] + Õ( 1 C1.8 )
where we also use Fact A.e above, such that it is not possible for both 〈θ∗c,r, vc,l〉+ and 〈θ∗c,r, vc,3−l〉+ to be large.8 Note also from e.g. the proof of Allen-Zhu & Li (2020) Theorem 1, that if (c, 3− l) /∈M, then maxr〈θ∗c,r, vc,l〉 = Θ̃(1). Thus, we see that the contribution to k from the m class c channels is:
kc(x,x ′) def = m∑ r=1 Ψc,r(x)Ψc,r(x ′) (16)
= { Θ̃(1) if (c, 1), (c, 2) ∈ sM(x,x′) Õ( 1C1.8 ) else
(17)
Summing kc over [C] completes the proof of the lemma.
We now use Lemma 1 to analyse k∗ and ρ∗(x,x′) = k∗(x,x ′)√ k∗(x,x)k∗(x′,x′) .
8Unless neither (c, 1) nor (c, 2) ∈ M for a given c, but that only occurs in o(C) classes, and does not change the order of e.g. sM(x,x′) which is what we really care about.
First we look at Υ∗c,l. Recall that 〈θ∗c,r, vc,l〉 = Õ(1) for all c ∈ [C], l ∈ [2], r ∈ [m], and likewise so isM0c . More specifically,
• If (c, l) ∈ M, then we have from the proof of Theorem 1 in Allen-Zhu & Li (2020) that∑ r〈θ∗c,r, vc,l〉+ ≥ Ω(log(C)), and so maxr〈θ∗c,r, vc,l〉+ = Θ̃(1).
• If neither (c, 1) nor (c, 2) are inM, then Claim C.10 of Allen-Zhu & Li (2020) shows us that both maxr〈θ∗c,r, vc,1〉+,maxr〈θ∗c,r, vc,2〉+ are Θ̃(1).
Combining these facts, we have that:
Υ∗c,l = Θ̃(1) if (c, 3− l) /∈M
and so from Lemma 1, we have:
k∗(x,x ′) = ∑ (c,l):(c,3−l)/∈M Υ∗c,lZc,l(x)Zc,l(x ′)±O ( sM(x,x′) polylog(C) ) + Õ(C−0.8) (18)
= Θ̃(1) ∑
(c,l):(c,3−l)/∈M
Zc,l(x)Zc,l(x ′)±O ( sM(x,x′) polylog(C) ) + Õ(C−0.8) (19)
= Θ̃(sM(x,x ′))±O ( sM(x,x′) polylog(C) ) + Õ(C−0.8). (20)
Finally, we arrive at an expression for the correlation kernel ρ∗ of a trained single model by
ρ∗(x,x ′) = Θ̃ ( sM(x,x′)√ sM(x)sM(x′) )( 1 +O(
1 polylog(C) ) ) + Õ( 1 C0.8 ) (21)
where we define that sM(x) = sM(x,x) is the number of attributes in x that have been learnt by the trained network θ∗, and is s(1± o(1)) with high probability.
Intuition:
Compare ρ∗ in Eq. (21) to the ‘soft’ probability labels pτ ∈ RC of Allen-Zhu & Li (2020) (Claim F.4) with temperature τ = 1log2(C) :
pτc (x) =
{ 1
s(x) if vc,1 or vc,2 is in V(x) 0 else
where s(x) is the number of indices c ∈ [C] such that vc,1 or vc,2 is in V(x). Note, the setting of Allen-Zhu & Li (2020) is with a large Ω̃(1) ensemble, so every attribute is learnt (akin toM being empty for us). In the case thatM = {} being empty, if x 6= x′ and they share at least one feature, then from Appendix B.1.1 with high probability they will share exactly one feature, so that sM(x,x′) = 1. Moreover, s(x) = sM(x)≈sM(x′), hence we see that Eq. (21) matches roughly with pτc (x), but without the need for a temperature hyperparameter.
We see that vanilla KD learns new attributes by comparing a single data point x between classes, and giving larger target labels to the classes where ambiguous attributes learnt by the teacher (all attributes in this case) are present in x. On the other hand, ρ∗ gives higher values to data pairs x,x′ that share attributes that have been learnt by the trained model, and as we will see later this is how FKD learns new attributes in the student.

heading: C.2 ENSEMBLED TEACHER
text: To summarise what we have done so far in this section, we have seen in Eqs. (20) and (21) that it is possible, for a single trained model θ∗, to simplify both the feature kernel k∗(x,x′) and correlation kernel ρ∗(x,x′) in terms of the number of shared attributes between x,x′ which are also learnt by the trained model. The set of attributes learnt by the trained model is captured by the set
M = { (c, l) ∈ [C]× [2] ∣∣∣∣ max r∈[m] [〈θ0c,r, vc,l〉]+ ≥ ( 1 +
1
log2(m) ) · max r∈[m] [〈θ0c,r, vc,3−l〉]+ }
where θ0 was the random parameter initialisation for θ∗. From Fact A.e, we know that if (c, 3−l)∈M, then the attribute vc,l has not been learnt by the network.
Consider now an ensemble ofE = Θ(1) independently trained networks, {θ∗e}Ee=1, with an averaged feature kernel:
kT (x,x ′) =
1
E E∑ e=1 k∗e(x,x ′).
Suppose {θe,0}Ee=1 denotes the corresponding independent parameter initialisations. Then, for each e ∈ [E], let us define:
Me def = { (c, l) ∈ [C]× [2] ∣∣∣∣ max r∈[m] [〈θe,0c,r , vc,l〉]+ ≥ ( 1 +
1
log2(m) ) · max r∈[m] [〈θe,0c,r , vc,3−l〉]+ } .
Note that theseMe are completely independent sets due to the independent initialisations, and also by Proposition B.2 of Allen-Zhu & Li (2020), we know that
P [ (c, 1) or (c, 2) ∈Me ] ≥ 1− o(1) ∀c ∈ [C], e ∈ [E].
Therefore, C ≥ |Me| ≥ C(1− op(1)) ∀e ∈ [E].
Moreover, Eq. (11) & Proposition B.2 of Allen-Zhu & Li (2020) tell us that each of the two attributes vc,1, vc,2 are equally likely to be inMe (and so learnt in the multi-view setup), This means that:
| E⋂ e=1 Me| = 1 2E−1 C(1− op(1)).
DefineMT = ⋂E e=1Me. From Eq. (19) and the definition of kT , we see that:
kT (x,x ′) = Θ̃(1) ∑ (c,l) 1{(c, 3− l) /∈MT }Zc,l(x)Zc,l(x′)±O (sMT (x,x′) polylog(C) ) + Õ(C−0.8)
(22)
= Θ̃(sMT (x,x ′))±O (sMT (x,x′) polylog(C) ) + Õ(C−0.8) (23)
where for the reader’s convenience, we redefine:
sMT (x,x ′) = {(c, l) : vc,l ∈ V(x) ∩ V(x′) and (c, 3− l) /∈MT }.
Intuition:
We see that only for those attributes (c, l) such that (c, 3− l) ∈MT does the ensembled teacher kT miss the fact that we should have a strong Θ̃(1) kernel value between x,x′. This is when |V(x) ∩ V(x′)| = {vc,l} is non-empty (or in other words, when sMT (x,x′) 6=|V(x)∩V(x′)|), and hence there should be a large kernel value kT (x,x′). So we see that only |MT | of the attributes are not learnt by the teacher, which is a fraction |MT |
2C = 1 2E (1− o(1)) of all the attributes. These missed attributes are where the 1 2E+1 test error in Theorem 2 comes from (teacher ensemble of size E, and plus 1 for the attributes learnt from the student’s initialisation too). What’s more, we can decompose the teacher’s feature kernel kT = ∑ c k T c into contributions k T c from each class c, like in Eq. (16). From Eq. (17), we see that the contribution to the teacher’s feature kernel from class , for x 6= x′;
kTc (x,x ′) = { Θ̃(1) if (c, 1), (c, 2) ∈ sMT (x,x′) Õ( 1C1.8 ) else,
(24)
is able to decipher between whether or not x,x′ share an attribute from class c for all attributes apart from those (c, l) such that (c, 3− l) ∈MT .

heading: C.3 TRAINING SCHEME FOR FKD
text: We note at this point that we are morally done in terms of proving Theorem 2, with Eqs. (23) and (24), our key results telling us that the (ensemble) teacher kernel kT can identify when two inputs share common attributes that have been learnt by the (ensemble) teacher, and more specifically that kTc can do so when said common attribute is from class c.
What remains is a repackaging of the proof techniques of Allen-Zhu & Li (2020) (particularly for their Theorem 4 regarding self-distillation), that knowledge distillation (this time only using feature kernels instead of temperature-scaled logits, and with explicit dependence on teacher ensemble size) can improve generalisation performance of a student.
For convenience, the theoretical analysis of Allen-Zhu & Li (2020) introduces some slight discrepancies between the actual practical weight updates of vanilla KD Hinton et al. (2015), i.e. the gradients of:
L̃ = L+ λ 1 N ∑ i L( f(xi) τ , fT (xi) τ ),
and the weight updates in their theoretical exposition. Namely,
1. The authors assume that a temperature-dependent threshold caps the logits to give soft labels:
pτc (x) = emin{τ 2fc(x),1}/τ∑ j∈[C] e min{τ2fj(x),1}/τ .
2. The authors truncate the negative part of the gradient of the KD regularisation to only encourage logits to increase not decrease, with weight updates for θc,r on input x:
−∆θtc,r def = θtc,r − θt+1c,r ∝ ∇θc,rL+ η
1
N ∑ i ( pτc (x)− pτ,Tc (x) )−∇θc,rfc(x) where pτ,T are the temperature-scaled teacher labels.
3. The authors scale the output of both student and teacher models by a (polylogarithmic) factor, in order to ensure that both reach the threshold to give soft labels in Item 1. above.
4. Self-distillation (Furlanello et al., 2018; Zhang et al., 2019) distils a single teacher and a single student of same architecture into the student, like a size 2 ensemble. Allen-Zhu & Li (2020) modify the training scheme for their theoretical analysis of self-distillation so that the student is first trained on its own in order seperate learning its own attributes/features from those of the teacher. Our analysis covers a similar scheme.
These modifications are justified in that they make the theoretical analysis more convenient, whilst illustrating the main mechanisms by which KD works, which is to share ‘dark knowledge’ that is held in the teacher (in the form of the multi-view attributes that have been acquired by the teacher due to its parameter initialisation), with the student.
In the same vein, we now introduce some modifications to the practical implementation of FKD we propose in Algorithm 1 to aid our theoretical analysis, and describe the main mechanisms by which FKD works, corroborating our initial analyses in Section 2 and Appendix A about how the feature kernel is a crucial object in any NN and captures all the ‘dark knowledge’ that a teacher network could possess in the multi-view data setting.
It is likely possible to extend our proof of Theorem 2 with different modifications/training schemes, but given that the focus of this work is to introduce FKD as a principled alternative to vanilla KD with certain advantages such as prediction-space independence, and that the multi-view setting we consider is a plausible simplification of real world data (as demonstrated in Allen-Zhu & Li (2020)), we leave this to future work. We stress that any simplifications to the update rule in Algorithm 1 for this section can be efficiently computed, only requiring access to the student and teacher feature kernels, if need be.

heading: Modified training regime for FKD
text: 1. We first suppose that the student is trained as standard (as in Appendix B.4) for T1 = poly(C) η
steps, and learns its own subset of attributesMS , dependent on its initialisation θ0s , before being trained with the FKD objective:
Intuition: This mirrors the self-distillation setup of Allen-Zhu & Li (2020) Theorem 4. The idea being that the student first learns MS before picking up the other attributes that the teacher has access to.
2. For a given feature kernel k, we threshold the feature kernel k based on value, to define a modification k̃ such that
k̃(x,x′) = { 1 if k(x,x′) ≥ 1m2 0 else
Intuition:
This condition delineates between the setting where x,x′ share common attributes learnt by student parameters θT1 in the initial phase of training (i.e. whether sMs(x,x
′) nonempty or empty).
To see this: note that if vc,l ∈ V(x) ∩ V(x′) and (c, 3 − l) /∈ MS then we know from Allen-Zhu & Li (2020) that ΦT1c,l ≥ Ω(log(C)). Hence maxr〈θ∗c,r, vc,l〉 ≥ Ω(log−4(C)) as the number of active neuronsm0 = |M0c | = O(log5C), and so it’s easy to see that for large enough m we have k(x,x′) ≥ 1m2 via Lemma 1. On the other hand, if {vc,l} = V(x) ∩ V(x′) and (c, 3 − l)∈MS then from Lemma 1 we know that k(x,x′) = Õ(C−0.8) 1m2 .
3. Similar to Allen-Zhu & Li (2020), we also truncate our FKD regularisation to only encourage kernel values to increase, and not decrease. For any input pair x1,x2, we have parameter update:
−∆θc,r(x1,x2) ∝ ( k̃c(x1,x2)− k̃Tc (x1,x2) )−[ ∑ j∈{1,2} Ψc,r(xj)∇θc,rΨc,r(x3−j) ]
(25)
where recall
Ψc,r(x) def = P∑ p=1 R̃eLU(〈θc,r,xp〉) so that ∇θc,rΨc,r(x) = P∑ p=1 R̃eLU ′ (〈θc,r,xp〉)xp
and
kc(x,x ′) def = m∑ r=1 Ψc,r(x)Ψc,r(x ′)
were defined in Definition 5 and Eq. (16).
Intuition: If the loss was
(kc(x1,x2)− kTc (x1,x2))2
then the gradient with respect to θc,r would be:( kc(x1,x2)− kTc (x1,x2) )[ ∑ j∈{1,2} Ψc,r(xj)∇θc,rΨc,r(x3−j) ]
so the only differences with Eq. (25) are truncating ( k̃c(x1,x2) − k̃Tc (x1,x2) )− and also the thresholding to obtain k̃.
To summarise, after training the student on its own for T1 steps (such that we are in the setting of Theorem 1) to reach parameters θT1s , we update for T2 = poly(C) η steps as (hiding S subscript):
∆θtc,r=− ηEx1,x2∼D̂2 [( k̃c(x1,x2)−k̃Tc (x1,x2) )− ∑ j∈{1,2} Ψc,r(xj)∇θc,rΨc,r(x3−j) ] (26)

heading: C.4 FEATURE CORRELATION GROWTHS
text: We now seek to analyse to what extent the attributes {vc,l}c,l are learnt during our T2 FKD training steps. The central objects describing how much vc,l has been learnt by parameters θ are:
Φtc,l def = ∑ r∈[m] [〈θtc,r, vc,l〉]+ and Φtc def = ∑ l∈[2] Φtc,l
Intuition: Φc,l is a data-independent quantity that reflects the strength of correlation with feature vc,l by parameters θ. On the other hand, Ψc,r(x) is a data-dependent quantity that reflects the activation of channel r for class c with input x.
Also recall that Zc,l(x) def = 1{vc,l ∈ V(x)} ∑ p∈Pv,l(x) zp
and define Vc,r,l(x) (which is convenient for calculating the size of gradient updates for θc,r): Definition 6.
Vc,r,l(x) def = 1{vc,l ∈ V(x)} ∑ p∈Pvc,l(x) R̃eLU ′ (〈θc,r,xp〉)zp
Like how Lemma 1 simplified the feature kernel in terms of data-dependent Zc,l(x) and dataindependent Υc,l, we have a result from Allen-Zhu & Li (2020) to simplify function predictions fc in terms of Zc,l(x) and Φc,l: Claim 1 (Claim F.7 from Allen-Zhu & Li (2020)). For every t ≤ T1 + T2, every c ∈ [C], every (x,y) ∈ D̂ (or every test sample (x,y) ∼ D with probability 1− elog2(C)):
f tc(x) = ∑ l∈[2] ( Φtc,l × Ztc,l(x) ) ±O( 1 polylog(C) )
We also have the following facts from Allen-Zhu & Li (2020) regarding the correlation of gradient ∇θc,rΨtc,r(x) with vc,l for (x,y) ∈ D̂, l ∈ [2] and r ∈ [m]:
Claim 2 (c.f. Claim F.6 of Allen-Zhu & Li (2020)). For every t ≤ T1 + T2, for every (x,y) ∈ D̂, every c ∈ [C], r ∈ [m] and l ∈ [2]:
• If vc,1, vc,2 ∈ V(x), then 〈∇θc,rΨtc,r(x), vc,l〉 ≥ ( Vc,r,l(x)− Õ(σpP ) ) • 〈∇θc,rΨtc,r, vc,l〉 ≤ ( 1{(vc,l ∈ V(x)}Vc,r,l(x) + Õ(C−2)
) • For every i 6= c, |〈−∇θc,rΨtc,r(x), vi,l〉| ≤ Õ(C−1.5)
Disclaimer Technically, Allen-Zhu & Li (2020) only show Claims 1 and 2 for t ≤ T1 and one would need to use similar proof techniques (such as their inductive hypothesis F.1) to show the case for T1 ≤ t ≤ T2, which we skip for conciseness. We now study the growth of the student’s Φc,l, for those (c, l) which have been learnt by the teacher but not the student:
Lemma 2 (Correlation Growth for attributes learnt by teacher). For every c ∈ [C], l ∈ [2], T2 ≥ t ≥ T1, such that (c, 3− l) /∈MT , suppose Φtc,l ≤ 12m , then we have:
Φt+1c,l ≥ Φ t c,l + Ω̃(
ηs2 C2 ) · Φtc,l 4 · R̃eLU ′ (Φtc,l)
Proof. For any c ∈ [C], r ∈ [m], l ∈ [2], we have 〈∆θtc,r, vc,l〉=ηEx1,x2∼D̂2 [( k̃Tc (x1,x2)− k̃c(x1,x2)
)+[ ∑ j∈{1,2} Ψc,r(xj) ( Vc,r,l(x3−j)−Õ(σpP ) )]] Note that as µ ≤ 1poly(C) , we can suppose that both x1,x2 are multi-view data. Using Claim 2, we have that
〈∆θtc,r, vc,l〉 ≥ ηEx1,x2∼D̂2 [( k̃Tc (x1,x2)− k̃c(x1,x2) )+[ ∑ j∈{1,2} Ψc,r(xj) ( Vc,r,l(x3−j)−Õ(σpP ) )]] Let r = argmaxr′∈[m]{〈θtc,r′ , vc,l〉}, such that definitely 〈θtc,r, vc,l〉≥Ω̃(Φtc,l) because m = polylog(C).
But if x is multi-view and vc,l ∈ V(x) such that ∑ p∈Pvc,l
z4p = Θ(1), and also by Fact A.a we have that:
Vc,r,l(x) ≥ Ω(1) · R̃eLU ′( 〈θtc,r, vc,l〉 ) ≥ Ω̃ ( R̃eLU ′ (Φtc,l) ) Ψc,r(x) ≥
∑ p∈Pvc,l R̃eLU(〈θtc,r, vc,l〉zp − õ(σ0)) ≥ Ω̃(Φtc,l 4 )
Now, we have assumed that (c, 3− l) /∈ MT , such that the teacher model has learnt attribute (c, l) and satisfies k̃Tc (x,x
′) = 1 when vc,l ∈ V(x) ∩ V(x′) (for large enough polylogarithmic m). Also, it is simple to see that when vc,l ∈ V(x) ∩ V(x′) & vc,3−l /∈ V(x) ∩ V(x′), for large enough m, that Φtc,l ≤ 12m implies that kc(x,x ′) ≤ 1m2 by Lemma 1, and so k̃c(x,x ′) = 0, i.e. the student has not (yet) learnt vc,l . So we see there are two more conditions that must be satisfied in order for( k̃Tc (x1,x2)−k̃c(x1,x2) )+ =1 > 0:
1. vc,l ∈ V(x1) ∩ V(x2) so that k̃Tc (x1,x2) = 1
2. vc,3−l /∈ V(x1) ∩ V(x2) so that k̃c(x1,x2) = 0.
Going back to the Appendix B.1.1, we know that these conditions occur with probability s 2
C2 (1 − o(1)) for independently sampled x1,x2 ∼ D̂. Finally, putting everything together we have that:
〈θt+1c,r , vc,l〉+ − 〈θt+1c,r , vc,l〉+ ≥ Ω̃( ηs2
C2 )Φtc,l
4 · R̃eLU ′ (Φtc,l)
summing over r′ ∈ [m] and noting 〈∆θtc,r′ , vc,l〉 ≥ 0 up to small error (as σpP = 1poly(C) for a large polynomial) gives us our result.
Lemma 2 immediately gives us the following corollaries, because ΦT1i,l ≥ Ω̃(σ0) and R̃eLU ′
is increasing.
Corollary 1. Define iteration threshold T2 = Θ̃( C 2 ηs2σ70 ) = Θ̃(C 5.1
η ), then for every (c, l) such that (c, 3− l) /∈MT we have:
ΦT1+T2c,l ≥ 1
4m
But likewise, we can also bound the growth of Φc,l Lemma 3. If (c, 3 − l) ∈ MS\MT , once Φtc,l ≥ √ loglog(C) m , it no longer gets updated (for large C): Proof. Recall Definitions 3 and 4 that Φ∗c,l= ∑ r∈[m][〈θtc,r, vc,l〉]+ and Υtc,l= ∑ r∈[m][〈θtc,r, vc,l〉]+ 2
Hence by Cauchy-Schwarz we have:
Φtc,l 2 ≤ mΥtc,l =⇒ Υtc,l ≥ loglog(C) m2 ≥ 2 0.42m2
for large enough C. Thus, if x1, x2 are both multi-view and vc,l ∈ V(x1)∩V(x2), by Lemma 1 we must have
kc(x1,x2) ≥ 1
m2
It’s also not difficult to check that any other possible setting for x1,x2, and V(x1) ∩ V(x2) will lead to ( k̃Tc (x1,x2)− k̃c(x1,x2) )+ = 0, and hence(
k̃Tc (x1,x2)− k̃c(x1,x2) )+ = 0 ∀x1,x2

heading: C.5 WRAPPING UP PROOF OF THEOREM 2
text: Proof. We are now ready to wrap up our proof. Recall from the proof of Theorem 1 in Allen-Zhu & Li (2020), that after the initial phase of T1 steps of student training on its own:
ΦT1c ≥ Ω(log(C)) ∀c ∈ [C],
and more specifically:
• If (c, 3− l) /∈MS , then ΦT1c,l ≥ Ω(log(C))
This gives us perfect test accuracy on the multi-view data, and 50% accuracy on the singleview data, so 0.5µ test accuracy overall, as per Theorem 1.
• Moreover, we have that if (c, 3− l) /∈ MT and (c, 3− l) ∈ MS , then by Corollary 1 and Lemma 3, then: √
loglog(C) m ≥ ΦT1+T2c,l ≥ 1 4m
=⇒ √
loglog(C) ≥ ΦT1+T2c,l ≥ 1
4m
We see that this change in ΦT1+T2c,l after FKD training is much smaller than Ω(log(C)) and so the student after FKD training still has perfect multi-view accuracy, as well as half accuracy on single-view data from the (approximately) half attributes learnt inMS . On the other hand, for single-view data, we know that if we have data point x, y and attribute vc,l ∈ V(x), such that c 6= y, then ∑ p∈Pvc,l (x) zp = Γ = O( 1
polylog(C) ), as defined in Appendix B.1.1.
Hence for small enough Γ ( 1m ) we have that if (c, 3− l) /∈MT and the single view data x is of class c with l̂(x) = l then we have correct prediction, as per Claim 1.
Combining these means that we have correct prediction for any single-view data x of class c, and l̂(x) = l such that (c, 3− l) /∈MT ∩Ms. By the independence of these sets we have that |MT ∩Ms| = (2−E)k(1 − o(1)) this means we have test error less than (2−E−1 + )µ for any > 0, for large enough C as required.

heading: D DIFFERENCES BETWEEN FKD & OTHER FEATURE KERNEL BASED KD METHODS
text: In this section, we highlight how our FKD approach overcomes some of the shortcomings of previous feature kernel based KD methods which only use pairwise evaluations of the feature kernel: SP (Tung & Mori, 2019) and RKD (Park et al., 2019). One advantage of FKD relative to these previous works is that we have shown FKD is amenable to ensemble distillation. Moreover, it goes without saying that Feature Regularisation, which arises naturally thanks to our feature kernel learning perspective in Section 3.1, is already a significant departure that improves FKD relative to SP & RKD. However, even without FR we observe in Section 4 that FKD outperforms both RKD & SP across different datasets and architectures, which warrants explanation.
D.1 IMPORTANCE OF ZERO DIAGONAL DIFFERENCES: SP (TUNG & MORI, 2019)
First, we consider diagonal kernel differences, k(x,x) − kT (x,x) for fixed x, and motivate using zero diagonal differences, which is not present in SP (Tung & Mori, 2019) but is in FKD thanks to our use of the correlation kernel. Fig. 6 displays this comparison between FKD & SP graphically.
Intuition: Downside of non-zero diagonal differences
The key intuition, which we detail below using our theoretical setup, is that non-zero diagonal differences k(x,x)− kT (x,x) 6= 0 encourages the student to learn noise in input x, compared to when we have zero diagonal differences k(x,x) − kT (x,x) = 0. In the later case, we only have non-zero differences for k(x,x′)− kT (x,x′) where x 6= x′.
Consider our parameter update Eq. (25) when x1 = x2 = x: If we didn’t have zero diagonal differences and instead k̃c(x,x)− k̃Tc (x,x) = −1, then:
∆θc,r(x,x) = 2ηΨc,r(x) P∑ p=1 R̃eLU ′ (〈θc,r,xp〉)xp,
Now suppose v = vc,1 ∈ V(x). For each p ∈ Pv(x), recall (from Appendix B.1.1) that:
xp = zpv + ξp,
We then see that (c.f. Claim C.13 of Allen-Zhu & Li (2020)):
〈∆θc,r(x,x), ξp〉 = 2Θ̃(η)Ψc,r(x)R̃eLU ′ (〈θc,r,xp〉) +O( 1√ d ),
as 〈v, ξp〉 = O( 1√d ) But at the same time (by e.g. Claim F.6 of Allen-Zhu & Li (2020)):
〈∆θc,r(x,x), v〉 = 2ηΨc,r(x)Vc,r,1(x)(1± o(1))
and note that that Vc,r,1(x) ≤ Θ(1) by definition. Moreover, in order for the student network to learn attribute vc,1, then eventually it must satisfy maxr′〈θc,r′ , v〉 = Õ(1) from Allen-Zhu & Li (2020). Thus, for % small enough, if r = argmaxr′〈θc,r′ , v〉, we have R̃eLU ′ (〈θc,r,xp〉) = 1.
On the other hand, if x1 6= x2 and we have v ∈ V(x1) ∩ V(x2):
∆θc,r(x1,x2) = η ∑ j=1,2 Ψc,r(xj) P∑ p=1 R̃eLU ′ (〈θc,r,x3−j,p〉)x3−j,p,
and so if x1 = x then
〈∆θc,r(x1,x2), ξp〉 = Θ̃(η)Ψc,r(x2)R̃eLU ′ (〈θc,r,x1,p〉) +O( 1√ d ),
as 〈x2,p, ξp〉 = O( 1√d ). But this time we have 〈∆θc,r(x1,x2), v〉 = η ( Ψc,r(x1)Vc,r,1(x2) + Ψc,r(x2)Vc,r,1(x1) ) (1± o(1)),
We see that whereas in diagonal updates ∆θc,r(x,x) the increments for 〈∆θc,r(x,x), ξp〉 and 〈∆θc,r(x,x), v〉 are 1:1, for non-diagonal updates ∆θc,r(x1,x2) they are 1:2 respectively. Thus, the parameter updates for θc,r with non-zero diagonal differences in feature kernels are more likely to learn noise, ξp, compared to our zero diagonal updates which rely only on non-diagonal ∆θc,r(x1,x2) for x1 6= x2. This is why we zero out diagonal differences for FKD, using the feature correlation matrix in practice.

heading: D.2 PROBLEM OF HOMOGENEOUS NNS IN RKD (PARK ET AL., 2019)
text: The distance-wise version of RKD (Park et al., 2019) is as follows: for x,x′, we calculate ψT (x,x
′) = ‖hT (x,θT )− hT (x′,θT )‖2, where recall hT is the last-layer teacher feature extractor. Likewise, we also calculate ψS(x,x′) = ‖hS(x,θS)− hS(x′,θS)‖2. The RKD loss adds λKDEx,x′ [(ψT (x,x′)− ψS(x,x′))2] to the student’s training loss.9 While RKD (Park et al., 2019) does ensure zero diagonal differences, i.e. that ψT (x,x) − ψS(x,x) = 0, it suffers from a related issue, due to the homogeneity of NNs that use ReLU nonlinearity, which is ubiquitous in image classification tasks.
Suppose we take x and define x′ = Mx for some M > 0. For example, think of taking a cat image and multiplying all the pixel values by M . For ReLU (C)NNs without bias parameters, we have that h(x,θ) is 1-homogeneous: h(x′,θ) = Mh(x,θ). This means that it is likely we will have ψT (x,Mx) − ψS(x,Mx) 6= 0. But a cat image multiplied by some scalar M is still a cat image, hence RKD runs into the same problems as in Appendix D.1 of learning noise in x.
On the other hand for FKD: correlation kernel ρS(x,Mx)=ρT (x,Mx)=1, hence ρS(x,Mx)−ρT (x,Mx)=0 ∀x ∈ Rd,M > 0.

heading: E PSEUDOCODE FOR PRACTICAL FKD ALGORITHM
text: Algorithm 1 Feature Kernel Distillation with SGD. Require: Maximum number of iterations T ∗, batch size B, learning rate η, teacher correlation kernel ρT ,
FKD regularisation λKD > 0, Feature regularisation λFR > 0. Initialise student parameters θ0,W0. for iteration t = 0, . . . , T ∗ do
Sample minibatch (xBi , y B i ) B i=1 i.i.d.∼ D̂. Compute loss L = 1
B ∑B i=1 L(f(x B i ), y B i )+ 2λKD B(B−1) ∑B i 6=j ( ρθt(x B i ,x B j )−ρT (xBi ,xBj ) )2. Optionally, add feature regularisation L = L+ λFR
B ∑B i=1 ∥∥h(xBi ,θt)∥∥22 . Update parameters θt+1 ← θt − η∇θL, Wt+1 ←Wt − η∇WL.
end for return {θT∗ ,WT∗}

heading: F EXPERIMENTAL DETAILS AND FURTHER RESULTS

heading: F.1 FIG. 2: PREDICTIVE DISAGREEMENT ACROSS INDEPENDENT INITIALISATIONS VS RETRAINED LAST LAYER
text: All models are ResNet20v1 trained with standard hyperparameters:
• 160 epochs training time with batch size 128 and learning rate 0.1 which is decayed by a factor of 10 after epochs 80 and 120.
• SGD optimiser with momentum 0.9 and weight decay of 0.0001.
• CIFAR10 data is normalised in each channel such that the training data is zero mean and unit standard deviation. Random crops and horizontal flips used as data augmentation.
• All models are initialised with Kaiming initialisation He et al. (2015).
9We do not consider the angle-wise RKD loss here, but there will be similar issues due to homogeneity.
Out of 10000 test points, the predictive disagreements between a ‘reference’ model and either: independent initialisations (top row) or retrained last layers (bottom row) are depicted in Table 4. We see two clear trends. First, the retrained last layer has much fewer disagreements with the reference model than an independent initialisation model, highlighting the importance of the feature kernel. Secondly, the vast majority of disagreements between independent initialisations are where one of the models is correct. This reinforces our intuition/theoretical analysis that ensembling NN works because different initialisations bias the models to capture different useful features, and hence ensemble distillation (via feature kernels) can improve student performance.

heading: F.2 FIG. 3: ADDITIONAL FEATURE KERNEL HISTOGRAMS
text: In Fig. 7, we provide additional plots to Fig. 3 that depict the difference in distribution (over x) of feature kernel values k(x,x), for FKD with and without Feature Regularisation (FR).
Negative hypothesis. We originally hypothesised that FR could benefit FKD, in addition to balancing the distribution of k(x,x), as it could reduce the sparsity in the NN last-layer representation activations, which is consistent with the proofs of Theorems 1 and 2. Indeed, the NN predictions are dominated by a select few neurons who ‘have won the lottery’ (Allen-Zhu & Li, 2020) on account of being most correlated with one of the attributes {vc;l}l∈[2],c∈[C] at random initialisation. In our analysis, as few as O(log5(m) out of m neurons could be inactive. From our feature kernel learning perspective, this seems like a highly undesirable phenomenon, because if k(x,x′) = 〈h(x;θ), h(x′;θ)〉 = ∑Cm r=1 hr(x,θ)hr(x
′,θ) only has useful contributions from a dominant minority of r ∈ [Cm], then we are not utilising the full capacity of the model. FR seemed appropriate to reduce this sparsity as `2 regularisation is known to promote non-sparse solutions (Van Den Doel et al., 2013). However, we experimentally found the opposite to our hypothesis: that
FR trained FKD student had more inactive neurons relative to FKD students trained without FR. This again highlights that, while (we believe) our results in this work provide compelling evidence to highlight the validity of feature kernel based distillation, there are still gaps between our theory and practice, and further questions to be answered in future work.

heading: F.3 FIG. 4: ENSEMBLE DISTILLATION
text: All individual VGG8 networks that made up the teacher ensemble were trained using the default training regime from Tian et al. (2020), with independent parameter initialisations. Indeed, all of our experiments in Section 4 used Tian et al. (2020)’s excellent open-source PyTorch codebase (Paszke et al., 2019).10
For student networks, we used the training regime for vanilla KD from Tian et al. (2020) for all ensemble sizes. For FKD, we used the hyperparameters from our ResNet50→VGG8 experiment in Table 2 for all ensemble sizes.

heading: F.4 TABLE 1: DATASET TRANSFER
text: The VGG13 teacher checkpoint is provided by Tian et al. (2020). For both CIFAR-10 and STL10, all student networks are trained for 160 epochs with batch size 64 using SGD with momentum, with learning rate decays at epochs 80, 120, 150. The student trained without KD used default hyperparameters from Tian et al. (2020), which are indeed strong hyperparameters for standard training. For FKD, RKD (Park et al., 2019) and SP (Tung & Mori, 2019), we tuned the learning rate, learning rate decay, and KD regularisation strength λKD on a labeled validation set of size 5000 for CIFAR-10 and 1000 for STL-10, before retraining using best hyperparameters on the full training(+unlabeled) dataset. We also tuned the FR regularisation strength, λFR for FKD when FR was used. All RKD and SP hyperparameters were tuned in a large window around their default values from Tian et al. (2020), which were all author recommended. For FKD, we allowed λKD to range in [1,1000], and λFR to range in [0,20]. All hyperparameters sweeps were conducted using Bayes search in Weights & Biases (Biewald, 2020).
For STL-10, we used a batch size of 512 for all KD methods’ regularisation terms, compared to 64 for the standard cross-entropy loss. This was due to the fact that STL-10 has only 5K labeled datapoints, and we wanted to ensure that the student used as much of the unlabeled data as possible for each feature-kernel based KD method’s additional regularisation term during 160 epochs of training. 512 batch size was the maximum power of 2 before we ran into memory issues on a 11GB VRAM GPU, which occured for the RKD method.
Both CIFAR-10 and STL-10 data are normalised in each channel such that the training data is zero mean and unit standard deviation. Random crops and horizontal flips used as data augmentation. STL-10 images are downsized from 96x96 to 32x32 resolution.

heading: F.5 TABLE 2: CIFAR-100 COMPARISON
text: All networks were trained for 240 epochs with batch size 64, with learning rate decay at epochs 150, 180, 210 using SGD with momentum. All teacher networks use the exact same checkpoints as provided by Tian et al. (2020). Learning rate, learning rate decay, λKD, and λFR (when used) were tuned as in Appendix F.4 on a validation of size 5000. All other hyperparameters were set to the default values used by Tian et al. (2020). The CIFAR-100 data is normalised in each channel such that the training data is zero mean and unit standard deviation, with random crops and horizontal flips used for data augmentation. All results provided denote the test set accuracy at the end of the 240 epochs of training.
10https://github.com/HobbitLong/RepDistiller

Review summaries:
In this paper the authors consider neural networks as data-dependent kernel machines and propose applying a distillation method directly on the pairwise kernel matrix of the models. Authors extend their setting into ensemble settings, examining both some theoretical aspects of the process, building upon the work of Allen-Zhu & Li (2020), as well experimentally. More specifically, authors provide experiments on ensemble distillation on CIFAR-100, as well as on dataset knowledge transfer on CIFAR-100 to STL-10 and CIFAR-10.

This paper proposed a new knowledge distillation framework, feature kernel distillation (FKD), by matching the student network's feature kernel and that of the teacher network's feature kernel. The authors extended the theoretical results in Allen-Zhu & Li (2020) to demonstrate the advantages of their framework over standard NN training. The theoretical results are built upon a synthetic dataset consisting of multi-view and single-view data, where standard training will fail to generalize while FKD will generalize well. Empirically, the authors (loosely) showed the consistency between the empirical behavior of FKD and the theoretical predictions, though the empirical setting is fairly different from the theoretical setting. In addition, on several image datasets, FKD can outperform other KD baselines in terms of test accuracy.

In this paper the authors consider extending the knowledge distillation (KD) framework to cases where the student and the teacher do not share prediction spaces. To this end they propose Feature Kernel Distillation (FKD), where one views the last layer weights of a neural network as a data dependent feature kernel. Instead of regularising the distance between temperature scaled logits corresponding to each data point of the student and the teacher in vanilla Knowledge Distillation, the authors instead regularise based on the distance between feature kernels of a pair of data points (inner product of weights of the pre-final layer of the NN).   Under a simplified setting, viewing data as being multi-view the authors show a setting where classical knowledge distillation would not do well, but a FKD based knowledge distillation using an ensemble of teachers does well (the teacher kernel between two data points is the average teacher kernel across the ensemble). The authors then propose using a correlation kernel instead of raw kernels and feature regularisation to spread out the kernel values.  The authors proceed to empirically demonstrate  that this approach helps in dataset transfer between datasets of different number of classes. They also show that using an ensemble of teachers enables FKD to improve performance (but KD also improves performance).

There are several approaches in knowledge distillation for neural networks. This paper studies kernel-based knowledge distillation for multi-view data set. The authors revealed that the assembled feature map obtained from the ensemble teacher network effectively improves students' generalization ability. Some numerical results are presented to show the practical effectiveness of the learning method based on theoretical findings.