Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
Authors: Jiaming Lv
Haoyuan Yang
Peihua Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Knowledge Distillation; Wasserstein Distance; Image Classification; Object Detection
Primary area: machine_vision
TL;DR: We propose novel knowledge distillation methods based on Wasserstein Distance, which outperforms predominant KL divergence based ones and other state-of-the-art competitors.

Abstract: Since pioneering work of Hinton et al., knowledge distillation based on Kullback-Leibler Divergence (KL-Div) has been  predominant, and recently its variants have achieved compelling performance.  However, KL-Div  only compares probabilities of the corresponding category between the teacher and student while lacking a mechanism for cross-category comparison.  Besides, KL-Div is problematic when applied to intermediate layers, as it cannot handle non-overlapping distributions and is unaware of geometry of the underlying manifold. To address these downsides, we propose a methodology of Wasserstein Distance (WD) based knowledge distillation. Specifically, we propose  a logit distillation method called WKD-L based on discrete WD, which performs cross-category comparison of  probabilities and thus can explicitly leverage  rich interrelations among categories.  Moreover, we introduce a feature distillation method called WKD-F, which  uses a parametric method for modeling feature distributions and adopts continuous WD  for transferring knowledge from  intermediate layers. Comprehensive evaluations on image classification and object detection  have shown  (1) for logit distillation WKD-L outperforms very strong KL-Div variants; (2) for feature distillation WKD-F is superior to the KL-Div counterparts and state-of-the-art competitors.

Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
Jiaming Lv", Haoyuan Yang", Peihua Lit
Dalian University of Technology
w.l...dd..... edu. en, yanghaoyuan@mail dlut.edu. enu peihual i@dlut.............. edu.
Abstract
Since pioneering work of Hinton et al., knowledge distillation based on Kullback- Leibler Divergence (KI.-Div) has been predominant, and recently its variants have achieved compelling performance. However, KL-Div only compares probabilities the corresponding category between the teacher and student while lacking a mechanism for cross-category comparison. Besides, KL-Div is problematic when applied to intermediate layers, as it cannot handle non-overlapping distributions and is unaware of geometry of the underlying manifold. To address these down- sides, we propose a methodology of Wasserstein Distance (WD) based knowledge distillation. Specifically, we propose logit distillation method called WKD-L based on discrete WD, which performs cross-category comparison of probabilities and thus can explicitly leverage rich interrelations among categories. Moreover, we introduce a feature distillation method called WKD-F, which uses a parametric method for modeling feature distributions and adopts continuous WD for transfer- ring knowledge from intermediate layers. Comprehensive evaluations on image classification and object detection have shown (1) for logit distillation WKD-L outperforms very strong KL-Div variants; (2) for feature distillation WKD-F is superior to the KL-Div counterparts and state-of- the art competitors. The source code is available at http://peihuall.or//KKK
Introduction
Knowledge distillation (KD) aims transfer knowledge from high-performance teacher model with large capacity to lightweight student model. In the past years, it has attracted ever increasing interest and made great advance deep learning, enjoying widespread applications in visual recog- nition and object detection, among others [I]. In their pioneering work, Hinton et al. [2] introduce Kullback- Leibler divergence (KL-Div) for knowledge distillation, where the prediction of category probabilities of the student is constrained to be similar to that of the teacher. Since then, KL-Div has been predominant in logit distillation and recently its variants 3 have achieved compelling performance. In addition, such logit distillation methods are complementary to many state-of-the-an methods that transfer knowledge from intermediate layers 6 7..
Despite the great success, KL.-Div has two downsides that hinder fully transferring of the teacher's knowledge. First, KL.-Dv only compares the probabilities of the corresponding category between the teacher and student, lacking a mechanism to perform cross-category comparison. However, real-world categories exhibit varying degrees of visual resemblance, e.g., mammal species like dog and wolf look more similar to each other while visually very distinct from artifact such as car and bicycle. Deep neural networks (DNNs) can distinguish thousands of categories 9 and thus
'Tese authors contributed equally to this work and share first authorship. Corresponding author. The work was supported by National Natural Science Foundation of China (6247 (62471083, 61971086).
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Unfortunately. unable to Loght disillattiloaoooon (WKD-L)
destleditillttoonn (WKD+FF
Welt
Taxher
U
DiscrteWW
Conficasas WD
Student 5
C.om
100g
Welf
(a) Real-world categories exhibit rich in- terrelations (IRs) in feature space, e.g.. dog is near other mammal while far from artifact like car. We quantify pairwise IRs as feature similarities among cate- gories. Best viewed by zooming in.
(b) For logit distillation, discrete WD performs cross-category com-- parison by exploiting pairwise IRs, contrast KL-Div that a catogory-to-catege
measure and lacks a mechanism to use such IRs (cf. Figure 2). For feature distillation, we use Gaussians for distribution modeling and continuous WD for knowledge transfer.
Our methodology of Wasserstein Distance (WD) based knowledge distillation. To effectively exploit rich category interrelations (a), we propose discrete WD based logit distillation (WKD)LL) (b) that matches predicted distributions between the teacher and student. Besides, we introduce feature distillation method based on continuous WD (WKD-F) (b), where we let student mimic parametric feature distributions of the teacher. In (a), features of 100 categories are displayed by the corresponding images as per their 2D embeddings obtained by I-SNE; refer to Section A. for details on this visualization.
Figure 1:
well-informed of such complex relations among categories, as shown in Figure Ta due its aitegory-to-cttegor nature, the classical KD [2] and its variants 3 explicitly utilize this rich cross-category knowledge.
4
Secondly, KL-Div is problematic for distilling knowledge from intermediate layers. Deep features of an image are generally high-dime nigh-dimensional and of small size, so being populated very sparsely in the feature space [10 Chap. 2]. This not only makes non-parametric density estimation (e.g., histogram) that KL-Div requires infeasible due to curse of dimensionality. but also leads to non-overlapping discrete distributions that KL-Div fails to deal with [1I] One may turn to parametric, continuous methods (e.g., Gaussian) for modeling feature distributions. However, KL-Div and its variants have limited ability for measuring dis-similarity between continuous distributions, it is not metric [ and is unaware of geometric structure of the underlying manifold 13
The Wasserstein distance (WD) [4], also called Earth Mover's 's Distance (EMD) optimal transport, has the potential to address the limitations of KL-Div. The WD between two probability distributions is generally defined as the minimal cost to transform one distribution to the other. Several works have made exploration by using WD for knowledge transfer from intermediate layers [15] [6]. Specifically, they measure the dis-similarity of mini-batce of images between the teacher and student based on discrete WD, which concerns comparison across instances in soft manner, failing to make use of relations across categories. Moreover, they mainly quest for non- -or-mmraettiic method for modeling distributions, behind in performance state-of-the-arr KL-Div based methods.
To address these problems, we propose a methodology of Wasserstein distance based knowledge distillation, which we call WKD. This methodology is applicable to logits (WKD-L) as well as to intermediate layers (WKD-F) as shown in Figure lb In WKD-L, between the predicted probabilities of the teacher and student using discrete WD for knowledge transfer. In this way, perform cross-category comparison that effectively leverages interrelations (IRs) among categories, in stark contrast to seagorr-to-categoon comparison in the classical KL-Div. We propose to use Centered Kernel Alignment (CKA) [[7 18 to quantify category IRs, which measures the similarity of features between any pair of categories.
minimize the discrepancy
For WKD-F, we introduce WD into intermediate layers to condense knowledge from features. Unlike the logits, there is no class probability involved in the intermediate layers. Therefore, we let the student directly match the feature distributions of the teacher. As the dimensions of DNN features

---

high, the non paraamettric methods (e.g.. histogram) are infeasible due to curse of dimensionality [10] Chap. 2], we choose parametric methods for modeling distributions. Specifically, we utilize one of the most widely used continuous distributions (i.e., Gaussian), which is of maximal entropy given 1st- and 2nd-moments estimated from features [19] Chap. 1]. WD between Gaussians can be computed in closed form and is a Riemannian metric on the underlying manifold [20].
We summarize our contributions in the following.
We present discrete WD based logit distillation method (WKD-L)). It can leverage rich interrela- tions among classes via cross-category comparisons between predicted probabilities the teacher and student, overcoming the downside of category-to-cattgory KL divergence.
We introduce continuous WD into intermediate layers for feature distillation (WKD-F). It can ef- fectively leverage geometric structure of the Riemannian space of Gaussians, better than geometry- unaware Kl.-divergence.
On both image classification and object detection tasks, WKD-L per-rperform better than very strong KL-Div based logit distillation methods, while WKD-F is supervisor to the KL-Div counterparts and competitors of feature distillation. Their combination further improves the performance.
2
WD for Knowledge Transfer
Given a pre-trained, high-performance teacher model TJ our task is train lightweight student model S that can distill knowledge from the teacher. As such, supervisions of the student are from both the ground truth label with the cross entropy loss and from the teacher with distillation losses to be described in the next two sections.
2.1 Discrete WD for Logit Distillation
Interrelations (IRs) among categories. As shown in Figures la and real-world categories exhibit complex topological relations in the feature space. For instance, mammal species are nearer each other while being far away from artifact or food. Moreover, features of the same category cluster and form distribution while neighboring categories have overlapping features and cannot be fully separated. As such, we propose to quantify category IRs based on CKA [18], which is a normalized Hilbert-Schmidi Independence Criterion (HSIC) that models statistical relations of two sets features by mapping them into Reproducing Kernel Hilbert Space (RKHS) [21].
Given a set of b training examples of category C. we compute a matrix X; Ruxb where the k-th column indicates the feature of example k that output from the DNN's penultimate layer. Then we compute a kernel matrix K, E R with some positive definite kernel, e.g., a linear kernel for which K, =xx?'X, X, where T' indicates matrix transpose. Besides the linear kernel, we can choose other kernels such polynomial kernel and RBF kernel (ef. Section for details). The IR between C, and C, is defined as:
HSIC(C,,C) C,) V HSSCCC,, C.) CIS)CIC,,C,
IR(C,, C,) =
HSICIC,,C) = (b- 112 tr(K
(I)
Here H I-I- is the centering matrix where indicates the identity matrix and 1 indicates an all-one vector; indicates matrix trace. IR(C,, C3) E [0, 1] is invariant to isotropic scaling and orthogonal transformation. Note that the cost to compute the IRs can be neglected since we only need compute them once beforehand. As the teacher is more knowledgeable, interrelations using the teacher model, which is indicated by IR" C,).
compute category
Besides CKA, cosine similarity between the prototypes of two categories can also be used to quantify IRs. In practice, the prototype of one category can be computed as the average of the features of the category's ' examples. Alternatively, the weight vectors associated with the softmax classifier of a DNN model can be regarded prototypes of individual categories (22].
Loss function. Given an input image (instance), we z = [z]] T be the corresponding logits of a DNN model where S, & {1,... nn indicates the index of i-th category. The predicted category probability p [ps] is computed via the softmax function with temperature i.e.,
p. (2); A exp(zx/T\/ exp(2//T). We denote by p" and pd the predicted category
3

---

probabilities of the teacher and student models, respectively. The classical KD 2 is an instance- wise method, which measures the discrepancy between p and pd given the same input image:
DKL(p"
(pppp [|0)= E, pi log /p.)
(2)
Logits of teacher
Cr
Dog
Walf
KL-Div (2) only compares predicted probabilities correspond- ing to the same category between the teacher and student, es- sentially short of mechanism to perform cross-category parison, as shown in Figure 2 Though during gradient back- propagation probability of one category affects probabilities of other categories due to the softmax function, this implicit effect insignificant and, above all, cannol explicitly exploit rich knowledge of pairwise interrelations as described in (
KL Divergenne
of
studen
In contrast to KL-Div, WD performs cross-category compar- ison and thus naturally makes use of category interrelations, shown in Figure Tb (left). We formulate discrete WD as entropy regularized linear programming [23]:
Car
Dog
Wall
Figure 2: KL-Div cannot perform cross-category comparison. Com- pare to WD in Figure Th (left).
DW ,p')
min Σe, 4:5
Cigli naij log 46
Σ Tis - s.t. 4ij 20,
01 カー i,j,Sn,
where Gy and 46 respectively indicate transport cost per mass and the transport amount while moving probability mass from P.. to pf: is regularization parameter. We define the cost Cij by converting the similarity measure IRs distance measure according to the commonly used Gaussian kernel [10 Chap. 6), i.e. Ci degree of sharpening IR. The smaller IR" (C,, Cj) in the feature space, the less cost is needed for transport between the two categories. As such, the loss function of WKD-L is
exp(-xtt
TR" (Ci, C,))), where is a parameter that can control the
LWKD-L= DwwDp' (p'.').
Recent work 3 discloses target probability (i.e., the probability of the target category) and the non- target ones play different roles: the former concerns the difficulty of training examples while the latter containing prominent "dark knowledge". It has been shown that this separation helpful to balance their roles and improves greatly over the classical KD 3 4.. Inspired by them, also consider similar separation strategy. Let be index of target category and z = [ ER-1 iE S, \ {t} be the teacher's 's logits of non-target categories. We normalize z,, as previously, obtaining the teacher's non-target probabilities P [p In this case, loss functions consist of two terms:
LWKD-L= (.kD-L=-DDD(PP..P.)) ft, L. -==-o(z)) log o (2
(5)
where \ is the weight.
2.2 Continuous WD for Feature Distillation
As the features output from intermediate layers of DNN are of high dimension and small size, the non-parametric methods, e.g., histogram and kernel density estimation, infeasible. Therefore, use one of the widely used parametric methods (i.c.. Gaussian) for distribution modeling.
Feature distribution modeling. Given input image, let us consider feature maps output by some intermediate layer of a DNN model, whose spatial height, width and channel number are h, w and 1, respectively. We reshape the feature maps to matrix F E | where i-th column f, E R' indicates a spatial feature. For these features, we estimate the Ist-moment μ 1/2 Σ, and the 2nd-moment Σ (f, 4f(f, 14)². We model feature distribution of the input image by a Gaussian with mean vector and covariance matrix Σ as its parameters:
h x and the
1 2 1/2 exp( exp(- (f-w) (f- (f-H)),
(u, Σ
(6)
where indicates matrix determinant.

---

We estimate Gaussian distribution of the teacher directly from its backbone network. For the student, as in previous arts 24 25 26], a projector is used to transform the features so that they are compatible in size with the features of the teacher. Then the transformed features produced by the projector are used to compute the student's is distribution. We select Gaussian for distribution modeling as it is of maximal entropy for given the 1st- and 2nd-moments
[99 Chap. 1] and has closed form WD that is
a Riemannian metric (20).
Loss Function. Let Gaussian NO (u',, Σ²) be feature distribution of the teacher. Similarly, we denote by 2 the student's distribution. The continuous WD between the two Gaussians is defined as
(AD. inf / -1--- dr
JR
where fd and & are Gaussian variables and || indicates Euclidean distance; the joint distribution q is constrained to have marginals and A Minimization of Eq. 7 leads to the following closed form distance 14]:
DWD DWD(N,,0 (N,,, = mean (n², 1 + D Dene (E'. )
(8)
Here Drean (n², 22)) = -131 and Drov ( Σの) = ur(x Σ 2(
) Σ )))+
where superscript indicates matrix square root. As the covariance matrices estimated from high- dimensional features are often ill-conditioned [27]. we add a small positive number (le-5) to the diagonals. We also consider diagonal covariance matrices, for which we have Deov (Σ", '5,)
||8" -5'1" where 5t is vector of standard variances formed by the square roots of the diagonals of ET We later compare Gaussian (Full) and Gaussian (Diag) which have full and diagonal covariance matrices, respectively. To balance role of the mean and covariance, we introduce a mean-cov ratio and define the loss as
LWKD-F= =-00FF=DD.. mean (H )+0)+ +Dov Deov EJ ΣS}
We can use the strategy of spatial pyramid pooling 128 29 6 to enhance representation ability. Specifically, we partition the feature maps into a x k spatial grid, compute a Gaussian for each cell of the grid and then match per cell the Gaussians of the teacher and student.
KL-Div (30 and symmetric KL-Div (i.e., Jeffreys divergence) [11. both having closed form expres- sions for Gaussians (32,, can be used for knowledge transfer. However, they are not metrics [12). unaware of the geometry of the space of Gaussians 13 that is a Riemannian space. Conversely, DWD is a Riemannian metric that measures the intrinsic distance (0). Note that DeNet [3 proposes a metric between Gaussians that leverages the geometry based on Lie group, which can be used to define distillation loss. Besides Gaussian, one can also use Laplace and exponential distributions for modeling feature distributions. Finally, though histogram or kernel density estimation are infeasible, one can still model feature distribution with probability mass function (PMF) and accordingly use discrete WD to define the distillation loss. Details on these methods can be found in Section A.2
3 Related Works
We summarize KD methods related to ours and show their connections and differences in Table
KL-Div based knowledge distillation. Zhao et al. (3] disclose the classical KD loss 2 a coupled formulation that limits its performance, and thereby propose a decoupled formulation (DKD) that consists of a binary logit loss for the target category and a multi-class logit loss for all non-target categories. Yang al. [4] propose normalized KD (NKD) method, which decomposes the classical KD loss into a combination of the target loss (like the widely used cross-entropy loss) and the loss of normalized non-target predictions. WTTM (5) introduces Rényi entropy regularizer without temperature scaling for student. In spite of competitive performance, they cannot explicitly exploit relations among categories. By contrast, our Wasserstein distance (WD) based method can perform crtes-cat comparison and thus exploit rich category interrelations.
WD based knowledge distillation. The existing KD methods founded on WD 15 16 mainly
concern cross-instance matching for feature distillation, as shown Figure (left). Chen et al. [[5] propose a Wasserstein Contrastive Representation Distillation (WCoRD) framework which involves
5

---

Logi Disillation
Fanur Disiilltionn
WC&8D/ EMD-IPOT
NET LOKD.C
Mechod
Distsbution
Dis similarity
Intervlationn
Distiiution
similarty
Ricmamian Metric
Tracher J
DKD NKD 4
Disertte
XL, divergenes
WTTM 1
Diurrie
Matual Informotion
Disncte
Wisscratii
Disunne
2nd mod-mmmmmnt
Probennus noem
LMD-IPUT
Diserete
Wieazreiin
Diszance
2nd End-mooeen
WKD (ours)
Disrcte
Wassersiein Disiunce
Conciuous
(Gauveian)
Wassersiela
Divaras
V
Spuild
Fmbesius
ICKD
2nd-moment
Charnel
Probealus
x
Stadent
Student 8
VID
Diurrte
Informstion
Mutual
Concinnous
Information
Mutral
Figure 3: Diagrams of WCoRD /EMD+IPOT and NST/ICKD-C.
Table 1: Comparison with related works.
a global and a local contrastive loss. The former loss minimizes mutual information (via dual form of WD) between the distributions of the teacher and student; the latter loss minimizes Wasserstein distance for the penultimate layer only, where the set of features of the mini-batch images are matched between the teacher and student. Lohit et al. [16] independently propose a similar cross-instance matching method called IMD+IPOT, in which knowledge is transferred from all intermediate layers and discrete WD is computed via inexact proximal optimal transport algorithm 34 The differences of our work from them are twofold: (1) They fail to leverage category interrelations which our WKD-L can make full use of: (2) They are concerned with cross-instance matching based on discrete WD, while our WKD-F involves instance-wise matching with continuous WD.
Other arts based statistical modeling. NST (35] is among the first formalize feature distilla- tion as a distribution matching problem, in which the student mimics the distributions of the teacher based Maximum Mean Discrepancy (MMD). They show that the 2nd-order polynomial kernel performs best among the candidate kernels of MMD, and that the activation-based attention transfer (AT) (36] is special case of NST. Yang al. (37 propose a novel loss function, which transfers the statistics learned by the student back to the teacher based on adaptive instance normalization. Liu et al. 66] propose inter-channel correlations (ICKD-C) model feature diversity and homology for better knowledge transfer. Both NST and ICKD-C can be regarded as Frobenius norm based distribution modeling via 2nd-moment along spatial- and channel-dimension, respectively, as shown in Figure (right). However, they fail to utilize the geometric structure of the 2nd- moment matrices, which are symmetric positive definite (SPD) and form Riemannian space [38 39.. Ahn et [40] introduce variational information distillation (VID) based on mutual information. VID assumes that feature distributions are Gaussians, and its loss boils down to mean square loss (i.e., FitNet (24) Gaussians are further assumed to have unit variance.
4 Experiments
We evaluate WKD for image classification ImageNet [41 and CIFAR- 42
Also, evaluate the effectiveness of WKD on self- enolennow distillation (Self-KD). Further, we extend WKD to object detection and conduct experiment MS-COCO 43). We train and test models with PyTorch framework (44, using a PC with an Intel Core 19- 13900K CPU and GeForce RTX 4090 GPUs.
4.1 Experiment Setting
Image classification. ImageNet [4] contains 1,000 categories with 1.28M images for training, 50K images for validation and 100K for testing. In accordance with [25], we train the models for 100 epochs using SGD optimizer with a batch size of 256, a momentum of 0.9 and a weight decay of 1c-4. The initial learning rate is 0.1, divided by 10 at the 30th, 60th and 90th epochs, respectively. We use random resized crop and random horizontal clip for data augmentation. For WKD-L, we use POT library [45] for solving discrete WD with =0005 and 9 iterations. For WKD-F, the projector has bottleneck structure, i.e., 1 Convolution (Conv) and a 3 Conv both with 256 filters followed by 1 x Conv with BN and ReLU to match the size of teacher's 's feature maps.

---

Dis similarity
Target- -No target
Top-1
4
Dis-similanie KL-Div 2
IR Method
Top-1
w/o
71.03
71.03
KL-Div
Lincar kernel
72.49
+1.46
w/
71.70
+0.67
CKA
Polynomial kemel
72.
+1.07
w/
71.96
+0.93
WD
RDF kernel
72.30
+1.27
w/o
72.04
+1.01
Classilicr weight
72.19
+1.16
WD
Cosine
72.49
+1.46
Class centroid
72.02
+0.99
(a) WD versus KL-Div.
(b) Different interrelation (IR) modelings.
Table 2: Ablation analysis of WKD-L for image classification (Acc, %) on ImageNet.
CIFAR-100 100 [42] contains 60K images of 32 x 32 pixels from 100 categories with 50K for training and 10K for testing. Following the setting of OFA [46]. conduct experiments across Convolutional Neural Networks (CNNs) and vision transformers. Additional experiments within CNN architectures are given in Table 77 of Section C.6 The images
upsampled the resolution of 224x224. All models are trained for 300 epochs with a batch size of 512 and a cosine annealing schedule. For CNN-based students, we SGD optimizer with an initial learning rate of 2.5e-2 and weight decay of 2c-3. For transformer-baut 2.5c-4 and weight decay of 2e-3.
students, we use A optimizer with an initial learning rate of
Object detection. MS-COCO 44 a commonly used benchmark for object detection that contains 80 categories. Following the common practice, images for training and 5K images for validation. As in ReviewKD [29]. we employ the framework of Faster-RCNT [47 with Feature Pyramid Network (FPN) 48 Detectron2 platform 44). As in
standard split of COCO 2017 with 1 118K
previous arts 50 29 51, we use the detection models officially trained and released as the teachers, while training the student models whose backbones are initialized with the weights pre-trained on ImageNet. The student networks are trained in 180K iterations with a batch size of 8; the initial learning rate 0.01, decayed by factor of 0.1 at 120K and 160K iterations.
4.2 Dissection of WD-based Knowledge Distillation
We analyze key components of WKD-L and WKD-F ImageNet. We adopt ResNet34 as teacher and ResNet18 as a student (i.e., setting (a)), whose Top-1 Accuracies are 73.31 % and 69.75%, respectively. See Section C.7 for analysis on hyper-parameters, e.g., temperature and weight.
4.2.1 Ablation of WKD-L
How WD performs against KL-Div? We compare WD to KL-Div with (w/) and without (w/o) separation of target probability in Table 2a For the case of without separation, WD (w/o) improves over KL-Div (wlo) by 1.0%; for the case of with separation, WD (w/) outperforms non-trivially KL-Div (w/) based DKD and NKD. The comparison above clearly shows that (1) WD performs better than KL-Div in both cases and (2) the separation also matters for WD. As such, WD with separation of target probability is used across the paper.
How to model category interrelations (IRs)? Table 2b compares two methods for IR modeling, i... CKA and cosine. For the former, we assess different kernels while for the latter we evaluate prototypes with classifier weights or class centroids. We note all WD-based methods perform much better than the baseline of KL-Div. Overall, IR with CKA performs better than IR with cosine, indicating it has better capability to represent similarity among categories. For IR with CKA, RBF kernel is better than polynomial kernel, while linear kernel is the best so is used throughout.
4.2.2 Ablation of WKD-F
Full covariance matrix or diagonal one? As shown in Table 3a (3rd and 4th rows), for Gaussian (Full), WD performs better than De (33) which suggests that the former metric is more appropriate for feature distillation. When using WD, Gaussian (Diag) (5th row) produces higher accuracy than Gaussian (Full). We conjecture the reason is that high dimensionality of features makes estimation of full covariance matrices not robust [52] : in contrast, for Gaussians (Diag). we only need to estimate 1D variances for univariate data of single dimension. Besides, Gaussian (Diag) much more efficient than Gaussian (Full). So use Gaussians (Diag) throughout the paper.
7

---

Distribution
Dissimilaiit
Top-1
Method
WD
Strategy
Mitching
Top-1
4
PiNNa
Frohesims
FiLNe: 24]
Instance-wise
70.53
Caussinn (Full)
DeNet
WD
72.23 +1.70
72.37 +1.84
WCORD L5
EMD-IPOT 16
Discrete
Cruss instanceeee
70.46 -0.07
71.49 +0.96
WD
72,50 +1.97
WKD-F
Continaous Instance-wise
72.50 +1.97
Gaussian (Diag)
KL.Div
71.25 +1.22
Parametric
Sym KL.
71.93 +1.40
(b) Different matching strategies.
Laplace
Ki..Dvv
71,38 10.85
Exponential
KI.Div
70.14 -0.39
Stage
Top-1
Sputial 1st mement
Euclidean
70,56 +0.05
Coav 4x Coev 5x 1x1 1x11 x
FüNet 24
70.53
Spatial 2nd momemt
Frobenins
71.14 +0.61
71.52
+0.99
Channel Iel-moment
Euclickan
Channel momest
Frobenius
WT
72.40 +1.87
Non-parametric PMP
WD
71.57 +1.01
72.44 +1.91
(a) Feature distribution modeling.
(c) Distillation position and grid scheme.
Table 3: Ablation analysis of WKD-F for image classification (Acc, %) on ImageNet.
How to model distributions? In Table 3aj we compare different parametric methods for knowledge distillation, including Gaussian, Laplace, exponential distribution, well separate -st-moment and 2md-moment. Note that NST [35] adopts spatial 1st- and 2nd-moment while ICKD-C 6 uses channel mommmmmnnt. Besides, we compare to con-parametric method based PMF.
For Gaussians (Diag), KL-Div and symmetric (Sym) KL-Div produce similar accuracies which are both lower than WD. The reason may be that KL-related divergences are not intrinsic distances, failing to exploit geometric structure of the manifold of Gaussians. For statistical moments, we see that channel-wise moments perform better than spatial-wise ones. For channel-wist representations, 1 st-moment outperforms 2nd-moment, suggesting that the mean plays a more important role. At last, the non-parametric method of PMF underperforms the parametric method of Gaussians.
Besides Gaussian (Diag), We can also use univariate Laplace or exponential functions to model distributions of each component of features. For them KL-Div can be computed in closed-form [53] but WD an unsolved problem. When using KL-Div, Gaussian (Diag) achieves better performance than both Laplace and exponential distributions, highlighting Gaussian as more suitable option among these parametric alternatives. Further, the Gaussian (Diag) combined with WD yields superior performance compared to KL-Div, suggesting advantage of the Riemannian metric.
Instance-wise cross-instance matching? Our WKD-F is an instance-wise matching method based on continuous WD, while WCoRD and EMD+IPOT concern cross-instance matching for a mini-batch of images based on discrete WD. As seen Table 3b WCoRD produces an accuracy much higher than EMD+IPOT, which may be attributed to its extra global contrast loss based on mutual information; WKD-F outperforms the two counterparts by a large margin of 1.0%, which suggests the ad advattgge of our strategy. Note that our WKD-F runs remarkably faster than the two counterparts that rely on optimization algorithm solve discrete WD.
Distillation position and grid scheme. We evaluate in Table the effect of position at which we perform distribution matching and that of different grid schemes. From the 3rd and 4th rows, that the last stage of Conv _5 performs much better than Conv _x_ indicating high-level features are more suitable for knowledge transfer. Comparing the 4th and 5th rows, we 2 2 2 grid does not improve over x 1 grid. Lastly, combination of features of Conv 4x and Conv _5x bring no further gains. Therefore, we use features of Conv_Sx and 1 X 1 grid for classification on ImageNet.
4.3 Image Classification on ImageNet
Table compares to existing works in two settings. Setting (a) involves homogeneous architecture, where the teacher and student networks are ResNet34 and ResNet18 18 9. respectively; setting (b) concerns heterogeneous architecture, in which we set the teacher as ResNet50 and the student as MobileNetV1 [57]. Refer Section for hyper-parameters in Settings (a) and (b).
For logit distillation, we compare our WKD-L with KD 2 DKD 3 NKD 4. CTKD 54 and WTTM (5] Our WKD-L performs better than the classical KD and all its variants in both settings. Particularly, our WKD-L outperforms WTTM, a very strong variant of KD, which additionally introduces sample-adaptive weighting method. This suggests Wasserstein distance that performs
8

---

Logit
Logit F'eature
Seoting
KD
DKD 1
NKD
CTKD
WITM
WKD-L
(ours)
FINN 24
CRD
KD
Revicw
CAT
WXD-F
(oun)
KD 25
CRD+
DPK
FCED
Zemi 56
KD-
WKD-L+
WKD F
Tup 73.31 69.75
71.08
71.20
71.51
72.19
72.49
70.53
71.11
71.36
72.50
71.38
72.25
72.17
72.76
(2)
5
90.41
90.47
90.75
89.57
90.51
90455
90.49
90.46
91.08
Top-1 76.16 68.87
70.50
72.05
72.58
73.00
73.17
70.26
71.37
72.56
72.24
73.12
73.26
73.26
73.02
73.69
Top-5 92.86 88.76 89,50
91.05
91.32
90.14
90.4
91.00
91.13
91.39
91,17
91.24
91.05
91.63
Table 4: Image classification results (Acc, %) on ImageNet. In setting (a), the teacher (J) and student (8) are ResNet34 and ResNet18, respectively, while setting (b) consists of teacher of ResNet50 and a student of MobileNetV1. We refer to Table 10 in Section C.4 for additional comparison to competitors with different setups.
cross-category comparison is superior to category-to-category KI.-Div. For feature distillation, we compare to FitNet [24]. CRD [25 ReviewKD [29] and CAT (55). Our WKD-F improves ReviewKD,
previous top-performer, by ~0.9% in the setting (a) and ~0.6% in the setting (b) in terms of top-1
accuracy; this comparison indicates that, for knowledge transfer, matching of Gaussian distributions is better than matching of features. Finally, combination of WKD-L and WKD-F further improves and outperforms strong competitors, including CRD+KD (25), DPK 77. FCFD 8 and KD-Zero [56).
More results of combination about WKD-L or WKD-F can be found in Table 9
Table 5 compares in Setting (a) latency of different meth- ods with a batch size of 256 using a GeForce RTX 4090. For logit distillation, the latency of WKD-L is 11.3 times larger than KL-Div based methods, due to the optimization procedure solve discrete WD. WKD-F has latency par with KI.-Div based methods, while running 1=.6 faster than ReviewKD and 1.2 faster than EMD+IPOT; this is because WKD-F only involves mean vectors and variance vectors, leading negligibly additional cost. Finally, com- bination of WKD-L and WKD-F has larger latency but better performance than ICKD-C, and meanwhile more efficient than state-of-the of-the--rtt art FCFD.
Strategy
Method
Top-1 (%)
Params
(M)
Latency
(ms)
KD 2
71.03
215
Logit
NKD 2T
71.96
214
WKI3- (K))-L((urr))
72.40
2501
ReviewKD
71.61
7.2
349
Feature
EMD-IPOT 16
70:46
0.25
258
WKI-F (Ours)
72.50
0.81
2017
Logit
FCFD
8
72.25
5.98
303
ICKD-C
6
72.19
Feature
WKD-L+
72.76
WKD-P (ours)
Table 5: Training latency on ImageNet.
4.4 Image Classification on CIFAR-100
We evaluate WKD in the settings where the teacher is a CNN and the student Transformer or vice versa. We use CNN models including ResNet (RN) [9] MobileNetV2 (MNV2) (58) and
ConvNeXt [59], as well as vision transformers that involve VIT 60 DeiT [6], and Swin Trans-
former 62 The setting of hyper-parameters can be found in Section C.5
and OFA 46 As shown in T WKD-L consistently outperforms state-of-the-as OFA for transferring knowledge from Transformers to CNNs vice versa. For feature distillation, we compare to FitNet [24], CC [64]
For logit distillation, compare WKD-L KD 21. DKD B. DIST
63
RKD 655 and CRD 25 WKD-F ranks first across the board; notably, it significantly outperforms the previous best competitors by 2.1% 1%-3.4% in four out of five settings. We attribute superiority of WKD-F to our distribution modeling and matching strategies, i.e., Gaussians and Wasserstein distance. We posit that, for knowledge transfer across CNNs and Transformers that yield very distinct features [46.. WKD-F is more suitable than raw feature comparisons as in FitNet and CRD.
Logit
Feature
Teacher (Ace)
Stuckat (Ace)
KD 21
DKD 3
DIST
OFA H6
WKD-L (ours)
FitNet X1
CC
RKD
CRD 25
WKD-F (ours)
68
[1
65
Transformer CCNN
Swin-T (89.26)
RNI8 (74.01)
78.74
80.26
77.75
80.54
81.42:0.22
78.87
74.19
74,11
77.63
81.57=00.22
VIT-S (92.04)
RN18 (74.01)
77.26
78.
76.49
80.15
80.11-0021
77.71
74.26
73.72
36.60
81.12::024
VIT.S (92.04)
(73.673
72.77
69.80
72.54
78.45
19.04
73.54
70.67
78.14
79.11
CNN- Transformer
CouxNeXI-T (88 DeiT-T (68.00) 00)
72.99
74.60
73.55
75.76
76.11=20118
60.78
68.01
69.79
65.94
73,27:0.22
ConvNeXi-T (8841)
Swin.P (72.63)
76.44
76.80
76.41
78.32
78.
24.06
72.63
71.73
67.09
74.50-20.13
Table 6: Image classification results (Top-1 1 Acc, %) on CIFAR-
100 across CNNs and Transformers.

---

4.5 Self-Knowledge Distillation on ImageNet
We implement our WKD the framework of Born-Agiin Network (BAN) [66] for self-knowledge
distillation (Self-KD), Specifically, we first train an initial model Sy using ground truth labels. Then we distill, using WKD-L, the knowledge of So into a student model S with the same architecture as So- For the sake of simplicity, we do not perform multi-generatie distillation, such as training student model S2 with Si as the teacher, etc.
We conduct experiments with ResNet18 on Ima- geNet, where the nyper-parameters are consistent with those in Setting (a). As shown in Table 7 BAN achieves competitive accuracy that is comparable state-of-the the-art results. Our method achieves the best result, outperforming BAN by 0.9% in Top-1 accu- racy and the second- best (i.e.. USKD) by 0.6%. This comparison demonstrates that our WKD can well generalize to self-knowledge distillation.
Method
SclFKDD
Dis-simalbrity
Top
Stundard train
NA
69.75
TF-KD [6]
KL-Div
70.14
FRSKD [8
KL-Div
70.17
Zipf's KD 69
KL-Div
70.30
USKD 4
KL-Div
70.75
BAN 66
KI-Div 70.50
WKD-L
WD 71.38
Table 7: Sclf-KD results (Acc, %) on Ima- geNet with ResNet18.
4.6 Object Detection on MS-COCO
We extend WKD to object detection in the framework of Faster-RCNN [47] For WKD-L, we the classification branch in the detection head for logit distillation. For WKD-F, we transfer knowledge from features straightly fed to the classification branch, i.e., features output by the RolAlign layer, and choose a 4x4 spatial grid for computing Gaussians. Implementation details, ablation of key components, and extra experiments are given in Section K oo Appendix.
We compare with existing methods in two set- tings, as shown in Table 8 In RN101- setting, the teacher is ResNet101 and the student is ResNet18: 18; in RN50-MMV2, the teacher and student are ResNet50 and MobileNetV2 [58]. respectively. For logit distillation, significantly outperforms the classical KD 2
RNI01 -SNN88
RNSI-MNVY
Faster RCNN-FFN
APrs
AP50
APTs
Teacher
42.04
45.88
61.02
43.81
Straloga
Student
33.26
48.87
30.90
KD
33.97
51,66
36.62
30.13
50.28
31.35
WKD-L
Logit
DKD
35.05
56.60
37.54
32.31
53.77
34.01
WKD-L (KDsL(Ouss)
35.24
56.73
37.91
32.48
53.85
34.21
and is slightly better than DKD 3. For feature
FiNe
24
34.13
54.16
36.71
30.20
49.80
31.69
PGH
50
35.44
55.51
38.17
31.16
50.68
32,92
distillation, we compare with FitNet, FGFI [50,
Featune
ICD
51
35.00
56.02
38.75
32.88
52.56
34.93
ICD (51 and ReviewKD 29 ; our WKD-F im-
Review KD
20
36.75
56.72
34.00
33.71
53.15
36.13
proves ReviewKD, the previous top feature dis- tillation performer, by non-trivial margin in both settings. Finally, by combining WKD-L and WKD-F, we achieve performance better than DKD+ReviewKD [3. When additional bounding-box regression used for knowl- edge transfer, our WKD-L+WKD-4 improves further, outperforming previous state-of-the-art FCFD g
WKT-F (Ours)
37.21
57.32
40.15
34.47
54.67
36.85
DKD+
Review KD
3
37.01
57.53
39.85
34.35
51.89
36.61
Logit
WKD-L+
37.49
37.28
WKD-F (ours)
FCFD
37.37
57.60
40.34
34.97
55.04
37.51
WKD-L
WKD-F! (ours)
37.79
41.08
35.48
55.21
38.45
Table 8: Object detection results on MS-COCO.
f Additional bounding- regression is used.
5 Conclusion
The Wasserstein distance (WD) has shown evident advantages over KL-Div in several fields such as generative models [T] However, in knowledge distillation, KL.-Div still dominant and it is unclear whether WD will outperform. We argue that carlier attempts on knowledge distillation based on WD fail to unleash the potential of this metric. Hence, we propose a novel methodology of WD-based knowledge distillation, which can transfer knowledge from both logits and features. Extensive experiments have demonstrated that discrete WD is very promising alternative of predominant KL- Div in logit distillation, and that continuous WD can achieve compelling performance for transferring knowledge from intermediate layers. Nevertheless, our methods have limitations. Specifically, WKD-L is more expensive than KL-Div based logit distillation methods, while WKD-F assumes features follow Gaussian distribution. We refer to Section in Appendix for detailed discussion limitations and future research. Finally, we hope our work can shed light on the promise of WD and inspire further interest in this metric in knowledge distillation.
10

---

References
[1] Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129())1789-1811 2021.
[2] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.000 02,3,, 2015.
[3] Borui Zhao, Quan Cui, Renjie Song, Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11953-11962, 2022.
[4] Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. From knowledge distillation to self- knowledge distillation: A unified approach with normalized loss and customized soft labels. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17185- 17194, 2023.
[5] Kaixiang Zheng and EN-HUI YANG. Knowledge distillation based on transformed teacher matching. In International Conference on Learning Represenlations, 2024.
[6] Li Liu, Qingle Huang, Sihao Lin, Hongwei Xie, Bing Wang, Xiaojun Chang, and Xiaodan Liang. Exploring inter-channel correlation for diversity-presert knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8271-8280, 2021.
[7] Martin Zong, Zengyu Qiu, Xinzhu Ma, Kunlin Yang. Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. In International Conference on Learning Representations, 2023.
[8] Dongyang Liu, Mcina Kan, Shiguang Shan, and CHEN Xilin. Function-consistent feature distillation. In International Conference Learning Representations, 2023.
[9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[10] Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistial Learning. Springer, 2009.
[11] Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pages 214-223, 2017.
[12] Karim Abou-Moustafa and Frank Ferrie. A note on metric properties for some divergence measures: The Gaussian case. In Asian Conference on Machine Learning, pages 1-15, 2012.
[13] Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aäron van den Oord, Sergey Levine, and Pierre Sermanet. Wasserstein dependency measure for representation learning. In Advances in Neural Information Processing Systems, pages 15578-15588, 2019.
[14] Gabriel Peyré and Marco Cuturi. Computational optimal transport: With applications to data science. Foundations and Trends in Machine Learning, 1(5-6))3555607, 2019.
[15] Liqun Chen, Dong Wang, Zhe Gan, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasser- stein contrastive representation distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16296-16305, 2021.
[16] Suhas Lohit and Michael Jones. Model compression using optimal transport. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2764-2773, 2022.
[17] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. International Conference on Machine Learning, pages 3519-3529, 2019.

---

[18] Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kernels based on centered alignment. The Journal of Machine Learning Research, 13(28):795-88) 2012.
[19] Christopher Bishop. Pattern Recognition and Machine Learning. Springer, 2006.
[20] Luigi Malago, Luigi Montrucchio, and Giovanni Pistone. Wasserstein Riemannian geometry of Gaussian densities. Information Geometry, 1(2):137-179, 2018.
[21] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International Conference on Algorithmic Learning Theory, pages 63-77, 2005.
[22] Vardan Papyan, Xuemei Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 1 117(40):24652-246) 2020.
[23] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Advances in Neural Information Processing Systems, pages 2292-2300, 2013.
[24] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In International Conference on Learning Representations, 2015.
[25] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2020.
[26] Yudong Chen, Sen Wang, Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. In Advances Neural Information Processing Systems, pages 12084-12095, 2022.
[27] Olivier Ledoit and Michael Wolf. Journal of Multivariate Analysis, 88(2):365 411, 2004.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. In European Conference on Computer Vision, pages 346-361, 2014.
[29] Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Distilling knowledge via knowl- edge review. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5008-5017, 2021.
[30] Solomon Kullback and Richard A Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1):79-86, 1951.
[31] Harold Jeffreys. An invariant form for the prior probability in estimation problems. Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, 186(1007):453- 461, 1946.
[32] Shaohua Kevin Zhou and Rama Chellappa. From sample similarity to ensemble similarity: probabilistic distance measures in reproducing kernel hilbert space. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(6):917-925 2006.
[33] Qilong Wang, Jiangtao Xie, Wangmeng Zuo, Lei Zhang, and Peihua Li. Deep CNNs meet global covariance pooling: Better representation and generalization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(8):2582-259, 2021.
[34] Yujia Xie, Xiangfeng Wang, Ruijia Wang, and Hongyuan Zha. A fast proximal point method for computing exact Wasserstein distance. In Uncertainty in Artificial Intelligence, pages 433-453, 2020.
[35] Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint arXiv:1707.0122. 2017.
12

---

[36] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. International Conference on Learning Representations, 2017.
[37] Jing Yang, Brais Martinez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via adaptive instance normalization. arXiv preprint urXiv:2003. 04289, 2020.
[38] Rajendra Bhatia. Positive Definite Matrices. Princeton University Press, 2015.
[39] Mehrtash Harandi, Mathieu Salzmann, and Richard Hartley. Dimensionality reduction on SPD manifolds: The emergence of geometry-aware methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(1):48- 62, 2018.
[40] Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D. Lawrence, and Zhenwen Dai. Vari- ational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163-9171, 2019.
[41] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[42] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[43] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference Computer Vision, pages 740-755, 2014.
[44] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 2019.
[45] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélie Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. POT: Python optimal transport. The Journal of Machine Learning Research, 22(78):1-8, 2021.
[46] Zhiwei Hao, Jianyuan Guo, Kai Han, Yehui Tang, Han Hu, Yunhe Wang. and Chang Xu. One-for-all: Bridge the gap between heterogeneous architectures in knowledge distillation. In Advances in Neural Information Processing Systems, volume 36, pages 79570-79582, 2023.
[47] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-tin time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(6):117-1119, 2017.
[48] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 21 2117-2125, 2017.
[49] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo, and Ross Girshick. Detectron2. https: trub.huu..c. http://aaccbooreeeerrcceeettttee
2019.
[50] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- grained feature imitation. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 4933 4942, 2019.
[51] Zijian Kang, Peizhen Zhang, Xiangyu Zhang, Jian Sun, and Nanning Zheng. Instance- conditional knowledge distillation for object detection. In Adyances Neural Information Processing Systems, pages 16468-16480, 2021.
13

---

[52] Eunho Yang. Aurelie Lozano, and Pradeep Ravikumar. Elementary estimators for sparse covariance matrices and other structured moments. In International Conference on Machine Learning, pages 397-405, 2014.
[53] Manuel Gil. On rényi divergence measures for continuous alphabet sources. Master thesis, 2011.
[54] Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1504-1512, 2023..
[55] Ziyao Guo, Haonan Yan, Hui Li, and Xiaodong Lin. Class attention transfer based knowledge distillation. In Croceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1188-118777 2023.
[56] Lujun Li, Peijie Dong, Anggeng Li, Zimian Wei, and Ya Yang. Kd-zero: Evolving knowledge distiller for any teacher-student pairs. Advances in Neural Information Processing Systems, pages 69490-69504, 2023.
[57] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang. Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint urXiv:1704.0444 048611, 2017.
[58] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4510-4520, 2018
[59] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1196-11986. 2022.
[60] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference Learning Representations, 2020.
[61] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data- duta-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pages 10347-10357 PMLR, 2021.
[62] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[63] Tao Huang, Shan You, Fei Wang. Chen Qian, and Chang Xu. Knowledge distillation from stronger teacher. In Advances Neural Information Processing Systems, pages 33716-33727, 2022.
[64] Baoyun Peng, Xiao Jin, Jiaheng L Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5007-5016, 2019.
[65] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019.
[66] Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International conference on machine learning, pages 1607-1616. PMLR, 2018.
[67] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEFF/CVE cenffrececo on computer vision and pattern recognition, pages 3903-3911, 2020. 2020.
14

---

[68] Mingi Ji, Seungjae Shin, Seunghyun Hwang, Gibeom Park, and I1-Chul Moon. Refine myself by teaching myself: Feature refinement via self- knowlede distillation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10664-10673, 2021.
[69] Jiajun Liang, Linze Li, Zhaodong Bing, Borui Zhao, Yao Tang, Bo Lin, and Haoqiang Fan. Efficient one pass self-distillation with zipf's label smoothing. In European conference computer vision, pages 104-119. Springer, 2022.
[70] Jason Altschuler, Jonathan Niles-Weed, and Philippe Rigollet. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In Advances in Neural Information Processing Systems, volume 30, 2017.
[71] Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. In European Conference on Computer Vision, pages 53-69, 2022.
[72] Tao Huang. Yuan Zhang, Mingkai Zheng, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge diffusion for distillation. In Advances in Neural Information Processing Systems, pages 65299-6531 2023.
[73] Ying Jin, Jiaqi Wang, and Dahua Lin. Multi-level logit distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 24276-24285, 2023.
[74] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[75] Zhiwei Hao, Jianyuan Guo, Kai Han, Han Hu, Chang Xu, and Yunhe Wang. Revisit the power of vanilla knowledge distillation: from small scale to large scale. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 10170-10183. Curran Associates, Inc., 2023.
the [76] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings the British Machine Vision Conference, 2016.
[77] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[78] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6848-6856, 2018
[79] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference Computer Vision, pages 618-626, 2017.
[80] Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren. Distance-iou loss: Faster and better learning for bounding box regression. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12993-13000, 2020.
[81] Li Yuan, Francis EH Tay, Guilin Li, Tao Wang. and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3902-3910, 2020.
[82] Tom Brown, Benjamin Mann, Nick Ryder, and et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc., 2020.
[83] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774. 2023.
[84] Utkarsh Ojha, Yuheng Li, Anirudh Sundara Rajan, Yingyu Liang, and Yong Jae Lee. What knowledge gets distilled in knowledge distillation? In Advances Neural Information Process- ing Systems, 2023.
15

---

A Implementation Details on WKD
A.1 Interrelations (IRs) among Category for WKD-L
Visualization of category interrelations. We select in random fifty images from each of one hundred categories randomly chosen on the training set of ImageNet. Then we feed them to pre-trained ResNet50 model and extract from the penultimate layer features that are projected to 2D space using t-SNE. The 2D embedding points are shown in Figure 4a where different categories are indicated by different colors. For intuitive understanding, inspired by karpathy we display features by the corresponding images at their nearest 2D embedding locations in Figure 4b It can be seen that the categories exhibit complex topological relations (distances) in the feature space, e.g., mammal species are nearer each other while far from artifact or food. The relations encode abundant information and are beneficial for knowledge distillation. In addition, the features of the same category cluster and form (unknown) distribution that often overlaps with those of neighboring categories. The observation suggests that it is more desirable to model the interrelations with statistical method.
Artifact
Animal
(a) Features are projected to 2D space using ISNE. Different categories indicated by different colors.
(b) Features are displayed by the corresponding im- ages their nearest 2D embedding locations.
Figure 4: Visualization of interrelations among 100 categories in feature space. The categories exhibit complex topological relations, where features of the same category cluster and form distribution that often overlaps with those of neighboring categories.
Quantization of IRs with CKA. We propose to use CKA for modeling category IRs it can effectively characterize similarity of deep representations [17]. CKA is normalized HSIC [18] that measures statistical dependence of random variables (features) by mapping them into RKHS with some positive definite kernels. Recall that, for category C, we have a matrix X; E uxb of u-dimensional features of b training images. As such, for the commonly used kernels,
have
X X, the polynomial kernel KFoly (X 1)* where k (,, 4), and RBF kernel Kibf exp( Med(D) where the bandwidth a (0.2,0.4. 0.6} and D; sym 2X X, For matrix A, Med(A) denotes the median of all of its entries, diag(A) (A) denotes a vector formed by its diagonals, and (A)aym = (A+AT).
linear kernel K!inn
D.
Quantization of IRs with cosine similarity. Besides CKA, cosine similarity between the prototypes of two categories is used to quantify category interrelations. The prototype of a category can be naturally computed as feature centroid of this category's training examples, i.e., X; X,11
Alternatively, the weight vectors associated with the softmax classifier can be used as prototypes (2). Specifically, if the weight matrix of the last FC layer is W Ruux where is number of total classes, then its i- th column w; can be regarded as the prototype of category C₁ i.e., X, W;.
https://cs..o wtt.s//fordd..upppoo//////aaaaaa
16

---

A.2 Distributions Modeling for WKD-F
Recall that, for an input image, we have 3D feature maps output by some layer of DNN, whose spatial height, width and channel number are h, w and I, respectively. We reshape the feature maps matrix F E R where m h w; we denote the i-th column as a feature E R'' while the j-th row (after transpose) as feature f3 E R". In WKD-F, we estimate channel Ist-moment Iu R' and End-moment Σ E R
m C. E
E TM m i=1 E (G-
(10)
which are used to construct a parametric Gaussian (H, Σ). For measuring difference between Gaussians, we use Wasserstein distance (WD) that is a Riemannian metric. We prefer Gaussians (Diag) that have diagonal covariances to Gaussians (Full) that have full covariances, as they are much more efficient and have better performance as shown in Table 3a
DeNet Different from WD, Wang al. 33] propose method called GeDeNet, which considers Lie group of the space of Gaussians and embeds a Gaussian into the space of SPD matrices:
un
N (n Σ
μ
(11)
1
Thus the difference between two Gaussians is defined as the Euclidean distance of the Gaussian embeddings of the teacher and student models.
ICKD-C and NST. ICKD-C 66 uses channel 2nd moment for exploring inter-channel correla- tions of features, i.e., M > f.f." Instead of channel-wise statistics, NST [35] estimates raw spatial Ist-moment पï E R and raw spatial 2nd-moment M IR for describing distributions:
Σ j=1
M Σ fff..
(12)
KL-Div between Gaussians (Diag). Let 4= M1..... .m] be mean and 8= [...... 8.] be variance of Gaussians (Diag). In this case, both KL-Div and symmetric KL-Div have closed forms:
T
DK I
2 log
1,
(13)
2
1 E, (n) -
8T 2
Dsym (L''
+(
2
)
2-2
87
0
Here T and denote the teacher and student, respectively.
KI.-Div between Laplace distributions. [J,,... "T)" E Rd are statistically independent. Then the Laplace distribution of f is T(,, v) 1
We assume components of feature =
II 2v, exp(- exp(-16-=ml)) (اايا ), where is the mean and v= [p1...., .1] T vi
is the scale parameter. The
KL-Div takes the following form [53]:
InT u 1/
DRL (L' (111L) = Σ, log
exp(
(14)
V.
KL-Div between exponential distributions.
Under independence assumption, the exponential
distribution of feature f is E(B) II, 3: (xp()B:ft), where B3 = [3i,... .B]] is the rate parameter. For exponential distributions, the KL-Div can be written as [53]:
S 38
DKL (ET 1ES) E, log
1.
(15)
B
Non-parametric PMD. Though non parameerii methods, e.g., histogram kernel density esti- mation, are infeasible due to curse of dimensionality, we can still use probability mass function (PMF) for distribution modeling. Specifically, given a set of features i, 1, ..., m, the PMF of the features of the form:
m,
Pr = Σ i-1
Pr. (f.) Pr: =
1
(16)
17

---

where (f::) denotes Kronecker function that is equal to one if fff=f, ff; and zero otherwise. Let p/ /P/ be the PMFs of the teacher/studens We can use discrete WD measure their discrepancy, i.c.. DWD (P7 p the transport cost Cij between two features f and fç is computed as 1 c-cos f f5))
Note that KL-Div is inapplicable to PMF as it cannot handle non-overlapping distributions (11].
It is worth noting that (1) the space of Gaussians is Riemannian space, which WD is intrinsic distance [20] while KL-Div or its symmetric version are not [[12] and thus are unaware of the
geometry [33;; (2) the space of either channel spatial 2nd moments is manifold of symmetric,
positive definite matrices rather than a Euclidean space [38 39,, so the Frobenius norm is not an
intrinsic distance 35 6
and fails exploit the geometric structure of the manifold.
B
Computational Complexity of WKD
The logit-based WKD-I. is formulated as entropy regularized linear programming, which can be solved fastly by Sinkhorn algorithm [23]. Let be the dimension of the predicted logits, the complexity of WKD-L can be written O(Dn² log n) [70]. Here D ||C|| is constant, where ||C||> indicates the infinity norm of the transportation cost matrix = [Ctyle and e > 0 indicates prescribed error. In contrast, the computational complexity of KL-Div is (nz.. Despite its high complexity, WKD-L can be computed efficiently as Sinkhorn algorithm is highly suitable for parallel computation GPU [23]. For feature-base based WKD-F, the dominant cost lies in computation of means and variances. Given a set of m features f, of I-dimension, the means can be computed by global average pooling that takes (mll) time; the complexity of variances is also Omml), as it can be obtained by dlement-wise square operations followed by global average pooling.
C Extra Experiment on Image Classification
C.1 More Ablation Analysis of WKD
We adopt the setting (a) for ablation on ImageNet, in which the teacher is ResNet34 and the student ResNet18.
Alype-pparameter of WKD-L. The total loss function of WKD-L consists of the cross-entropy loss LCE, the target loss L, and WD-based logit loss LWKD-L- Following (3 4), we set the weights of the two former losses to across the paper. As such, our syperr-prameters includes the temperature т, the weight À of LWKD-L. the sharpening parameter that controls smoothness of IRs for transport cost in WD, and the regularization parameter 1) (set to 0.05) in discrete WD. As simultaneous optimization of them computationally infeasible, we first analyze the effect of temperature by
72.5
72.5
72.0
72.0
Acr
71.5
71.5
71.0
0 71.0-
70.5
WKD4 (0um)
70.5-
WKD4 (Dun)
70.0
70.0-
0.5
1.0
1.5
2.0
Jemperature
Sharpening parameter
73.0
73.0
72.5
72.5-
Top Acc (96)
72.0
72.0
Acc
70.5
WKD- (0urs)
70.5-
W904 (Ours)
KD
KD
70.0
70.0
10
20
30
60
0.05
0.10
0.15
020
Weight
Regularizing
Figure 5: Analysis of hyper-parameter of WKD-L on ImageNet.
18

---

fixing the sharpening parameter and weight to and 30, respectively. As seen in Figure S (top left), across all temperatures WKD-L performs better than the baseline of KD [2] by large margins while achieving the highest accuracy when the temperature is 2. Next, we fix the temperature and weight, analyzing the effect of sharpening parameter. From Figure (top right) we see that performance variation is not large against the sharpening parameter and the best performance is obtained when it is equal to 1. Subsequenll. by fixing the best temperature and sharpening parameters, we evaluate the weight of WKD-L. As Figure (bottom left) shows, its accuracy varies smoothly against the weight and achieves the best accuracy when the weight is 30. Finally, Figure 5 (botiom right) illustrates that the performance varies smoothly wide range; for consistency, we set 7 to 0.05 in all experiments throughout this paper.
function of the regularization parameter 7, over reasonably
Sarraetassmeeers of WKD-F. The loss function of WKD-F method contains the cross cross-entropy loss LCE and WD-based feature loss LWKD-F- As in previous work, we set the weight of the cross-entropy loss to 1. Therefore, there are two hyper-parameters, i.e., the mean-cov ratio 7 and the weight of LWKD-r- r- We first analyze the effect of the former by fixing the latter to 2e-2. Table 6 (left) shows the performance of the mean-cov ratio WKD-F is clearly better than the baseline of FitNet (24]: the best accuracy is obtained when its value is 2, which suggests that the means play more important role than the covariances. Next, we fix the mean-cov ratio and evaluate the effect of the weight. As seen in Table (right), WKD-F is much better than the baseline of FitNet across all weights and achieves the best performance when the weight is 0.02.
function of the mean-cov ratio. We see that in the whole range
73.0
73.0
72.5
72.5-
mo0 Acc (96)
E 72.0
72.0
71.5
71.5-
71.0
71.0
70.5
70.5
70.0
0.02
0.04
0.06
0.10
Mean-cov ratio
Weight
Figure 6: Analysis of hyper-parametens WKD-F on ImageNet.
C.2 Summary of Slyper-parametess on ImageNet
The setting (a) involves homogeneous architecture, where the teacher and student networks are ResNet34 and ResNet18, respectively. For WKD-L, the weights of LCE, L. and LWKD-LL are 1, I and 30, respectively; the temperature 7 and sharping parameter k k= For WKD-F, we set the weights of Lce and LWKD-F to 1 and 2e-2, respectively. We adopt features from Conv5 x and Gaussian (Diag) with mean-cov ratio 2= The setting (b) concerns heterogeneous architecture, in which the teacher is ResNet50 and the student is MobileNetV1. In this setting, the weight of L set to 25 while that of LWKD-F F 1e-3, and the remaining hyper- syper-parameters are identical to those in the setting (a).
C.3 More Experiments on Combination of WKD-L or WKD-F
How WKD complements other KD methods? We combine WKD-F with a state-of-the-art logit- based knowledge distillation (i.e., NKD), and combine WKD-L with a state-of-the-art feature-based method (i.c., ReviewKD). The results are shown in Table Da and Table 9b] respectively. We can see that NKD+WKD-F improves over individual NKD and W WKD-F, which suggests our WKD-Fi is complementary to NKD. Notably, NKD+WKD is slightly inferior to WKD-L+WKD-F, addition, WKD-L+ KD improves over ReviewKY KD but underperforms WKD-L. We conjecture that, due to the large gap (~~ 0.9%) between ReviewKD and WKD-L, the combination hurts WKD-L.
Will separation of target probability in WKD-L still help when combined with WKD-F? To answer this question, we integrate WKD-L without separation into WKD-F, which is called WKD-L (w/o)+WKD-F From Table 9c we can see that it is slightly inferior to WKD-L (w/)+WKDFF in which the separation scheme is used in WKD-L.. This suggests that the benefit due to the separation scheme decreases but still persists in the logit+feature approach.
19

---

Method
Top-1
Top-5
Method
Top-1
Top
Method
Target Bonget---Noon T--No--tume target
Top-1
Top-5
NKD
71.66
WKD-L
72.49
90.75
WKD-L
wlo
72.04
90.42
WKD-F
72.50
91,00
ReviewKD
20
71.61
90.51
WKD-L
w/
72.49
90.75
NKDdWKDDFF
72.68
91.08
ReviewkD
72.32
90.63
WKD-F
72.50
91.00
WKD-L
72.49
90.75
WKD-L
72.49
90.75
WKD.L+WKD-F
w/o
72.70
91.07
WKD.L+WKD.F
72.76
91.08
WKD.L+WKD.F
72.76
91,08
WKD.L-WKD.F
72.76
91.08
(a)
Combination of NKD
(b) Combination WKD-L
(c) Effect of separation scheme on WKD-
and WKD-F
and ReviewKD
L+WKD-F.
Table Analysis on different combinations of logit and feature distillation methods.
C.4 Comparison to Competitors with Different Setups
In Tof the main paper, we conduct comparison in the ordinary setting, which is formalized in
CRD [25] and adopted by most methods. However, there exist some works which adopt Settings (a)
and (b) but with non-trivially different setups, e.g., DIST (63), NKD [4], MGD [71], and DifIKD (72). Specifically,, their students and/or teachers have performance higher than those in the ordinary
settings, which make them somewhat advantageous when comparing with the metric of Top-1 accuracy. For a fair comparison, we propose to use gains of the distilled student over the vanilla student in Top-1 accuracy, which is denoted by A They can more faithfully indicate how much knowledge the student has learned from the teacher. The comparison results are shown in Table [T]
For logit distillation, WKD-L outperforms NKD, a strong KI.-Div based variant, by 0.68% and 0.93% in setting (a) and setting (b), respectively. Instead of KL.-Div, DIST matches predicted probabilities based on Pearson correlation coefficients, which exploits both instance-wise and cross-instance knowledge, in contrast to WKD-L that uses instance-wiss wise knowledge only. Nevertheless, WKD-L performs better than it by 0.43% in setting (a) and large margin of 1.19% in setting (b). For feature distillation, we compare to MGD that randomly masks the student's features and then forces the student to generate the teacher's features. The accuracies of -KD-F are over 1.0% higher than those of MGD both settings. Regarding logit+feature distillation, we contrast with MGD+WSLD and DiffKD. DiffKD denoises the student features through a diffusion model trained by teacher features whose computation cost is high. WKD-L+*KKD-F surpasses DiffKD by 0.55% % and 1.33% in setting (a) and setting (b), respectively.
Setting (o):
ResNer34
Nes
Setting (b):
ResNer50 MobileNetV
Strategy
Method
Teacher
Vanilla student
Distilled
Teacher
Vanilla student
Distilled student
student
NKD 4
73.62
69.90
71,96
+2.06
76.55
69.21
72.58
+3.37
Logit
DIST 5
73.31
69.76
72.07
+2.31
76.16
70.13
73.24
+3.11
WKD--I (curs)
73.31
69.75
72.49
42.74
36.16
68.87
73.17
+4.30
MGD ?
73.62
71.58
+1.68
76.55
72:35
+3.14
Feature
W WKD-F (pur)
73.31
69.75
72.50
+2.75
76.16
68.87
73.12
+4.25
MGD&WSLD 211
73.62
69.90
71.80
+1.90
76.55
69.21
72.59
+3.38
Logit
DHiKD
73.31
+2.46
76.16
70.13
+3.49
Feature
WKD-L+WKD-F (ours)
73.31
69.75
72.76
+3.01
76.16
68.87
73.69
+4.82
Table 10: Image classification results (Top-1 Acc, %) ImageNet between WKD and the competitors with different setups. Red numbers indicate the teacher/student model has non-trivially higher represents the gains of the
performance than the commonly used ones formalized in CRD distilled student over the vanilla student.
25
Additionally, MLKD (7) makes use of a stronger image augmentation, i.e., RandAugment [74] (cf. their implementation) for improving performance. Vanilla KD (75) studies the great potential of vanilla KD in a very different setting, in which optimizer with much longer epochs, diverse and very strong augmentations along with more regularization methods are used. In contrast, we and many of the state-of ofate-of-the-art the-art methods follow the ordinary setting formalized in CRD (25).
20

---

C,5 Summary of seyper-parameters for across CNNs and Transformers on CIFAR-100
For fair comparison, we follow OFA 46 and separately tune the hyper-parametert for different setrings. For WKD-L, we set the temperature to 2 and the sharpening parameter to 1, while searching the optimal weight of LWKD- in [50, 200] with a step of 50. For WKD-F, the projector simply a linear layer. We set mean-cov ratio to 2, and perform grid search for the weight of LWKD-F F in (1,2,4, 4, 8) x 1e-2. The spatial grid for computing Gaussians (Diag) set to xx1. We report average accuracy and standard deviation after three runs in Table 6 the results of all competing methods are duplicated from OFA.
C.6 Knowledge Distillation within CNN Architectures on CIFAR-100
[}] Experimental Setup. We follow the setting of CRD 25 where the networks cover ResNet
Wide-ResNet (WRN) (76). VGG 777,, MobileNetV2
58 and ShuffleNetV1 (SNVI) 78]. All
models are trained for 240 epochs via the SGD optimizer with batch size of 64, momentum of 0.9 and a weight decay of 0.0005. The initial learning rate is 0.01 for MobileNetV2 and ShuffleNetV] and 0.05 for the remaining networks, divided by 10 at the 150th, 180th, and 210th epochs, respectively, As in CRD 255, the projector is a 1 X 1 Conv or 4 4 transposed Conv both with BN and ReLU.
For fair comparison, we follow the practice of DKD 13,, CAT (55), ReviewKD 129. FCFD ( and WTTM (5%, i.e., tuning eyper-parametery separately for different architectures. For WKD-L. we perform grid search for the weight of & LWKD-1. in [50, 800] with step of 50, the temperature in (4,8) and the sharpening parameter e in {0.5, 13. For WKD-F, we perform grid search for the weight of LWKD-F in {1, 2, ................................ 50} ) x 1e-2 and mean-cov ratio in {2,3.... . 8). The spatial grid for computing Gaussians (Diag) is searched {1 2xx1,2x2,44 4x4. We report the average accuracy and standard deviation of our method after three runs.
Results. The comparison results are shown in Table For logit distillation, WKD-L outperforms
the classical KD by large margins in all settings, and surpasses the competitors in 5 out of 6 settings across homogeneous and heterogeneous architectures. For feature distillation, WKD-F invariably achieves better results than EMD-based counterparts (i.e.. WCoRD and EMD+IPOT) and 2nd- moment based counterpart (i.e., NST); meanwhile, WKD-F also very competitive. compared to other tott--ofthh-arr methods. For logit+feature distillation, WKDDL+WKKD settings across the board. Overall, our methods have comparable or lower standard deviation, as opposed to the competing methods, which suggests that our method is statistically robust.
ranks first in 4 out of
Homogeneous
Heterogeneous
Teacher
WRN40-2
RN32x4
VGG13
WRN4D-2
VGG13
RN50
Student
WRN40-1
RN844
VGG8
SNVI
MNV2
MNV2
Shategy
Teacher
75.61
79.42
74.64
75.61
74.64
79.34
Student
71.98
72.50
70.36
70.50
64.60
64.60
KD
:
73.54.00.20
73.33.10.25
72.98.10.19
74.83.10.17
67.37.0.32
10.32
DIST 63
74.73:10.24
76.3111:.19
68.66.10.23
74.81
76.32
76.70
Logit
76.35
74.86
70.67
WTTM
S
74.58
76.06
74.44
75.42
69.16
69.59
WKD-L LXDD-LBO (Ours)
74.84:10.32
76.53-10.14
75.09:10.13
76.72:1:0.09
7..21:0022
71.10::0116
FidNet
72.24:10.24
73.50:1:0288
71.02:10.31
73.73:10.32
6..14:0.50
63.161:0.47
40
73.30:20.13
73.09:10.21
71.23:10.06
73.61:1:.12
65.56:0.42
67.57:10.28
CRD A5
74.14+0:22
75.51 18
73.9440.22
76.05-0.114
60.732042
69.11::0.28
ReviewKl KD
70
75.09
75.63
74.84
77.14
70.37
69.89
Feature
CAT 55
74,82
76.91
74.65
77.35
69.13
71,36
NST 35
72.24=50222
73.30-E0.728
71.53
31.33H0.1
74.39720.25
"S..
16266=226
64.
64.96.0.77
WCoRD
74.73
75.95
74.55
76.32
69.47
EMD+IPOT
74.19
72.80
WKD-F (Ours)
75.02+0006
76.77:40.26
75.02:+0.18
77.36:10.19
70.34=0.15
71887:0.35
DPK x
75.27
74.96
Logit
FCFD 3
75.46
76.62
75.22
77.99
70.65
71.00
Feature
DiffKD 72
74.09:10.09
76.72-10.15
09.21::0:27
ICKD>C
74.63
75.48
73.88
WKD-L+
WKD-F
75.35=00.13
77.28:1:024
75.25:20.15
77.50-20.32
70.6=00.10
71.71=00.28
Table 11: Image classification results (Top-1 Acc, %) on CIFAR- 100 within CNN architectures.
21

---

D Visualization
D.1 Visualization of Teacher-Sttuent Discrepancies
Following 25 3 we visualize the difference of correlation matrices of student and teacher log-
its. Figure 7a shows visualization results in settings CIFAR-100 42]. In setting of 4 the teacher and student are ResNet32x4 and ResNet8x4, respectively; the setting VGGI3 consists of teacher of VGG13 and a student of VGG8. On the whole, in both settings the colors of WKD-L are much lighter than those of KD [2], which indicates WKD-L produces correlations matrices more similar to the teacher than KD. As the differences correlation matrices capture inter-class correlations (25 smaller differences of WKD-L suggest better-informel cross-category relations than KL divergence can be learned.
In addition, we visualize whether WKD-F can learn distributions that are more similar to the teacher. high-level features output from the last Conv layer of network model for
To that end, computing distributions, as they encode the most discriminative information for classification. For each validation image of the i-th category, we compute WD between feature distributions of the teacher and student. Hence, for n categories each with k validation images, we obtain a n X matrix of distribution matching. Figure 7b shows visualization of two settings on CIFAR-100. We can see that overall WKD-F demonstrates smaller discrepancies with the teacher than FitNet 24 it can better mimic the teacher's 's distributions.
suggesting
KD
WKD-L
WKD
ResNet32x4-Resshe R
VGGI3
(a) Differences between correlation matrices student and teacher logits.
FiBNet
WXD-F
FiNett
WKD-F
ResNat32x4-ResNettkt
VGG13-VGGG8
(b) Differences between distributions of teacher and featurreaatee.
Visualization of teacher-student discrepancies for WKD-L (a) and WKD-F (b). Darker color indicates larger difference.
Figure 7:
D.2 Visualization of Class Activation Maps (CAMs)
We use Grad-CAM 79 to compute class activation maps using features output from the last Conv layer. Figure g shows, for three example images. CAMs of different models, including the teacher, vanilla student, distilled models by KD, WKD-L, FitNet and WKD-F. It can seen that WKD-L and WKD-F have more similar CAMs with the teacher than KD and FitNet, and localize more accurately the important regions of objects. The comparison suggests that WKD-L and WKD-F can learn features with better representation capability.
22

---

Logit distillation
Feature distillation
Input
Teacher
Vanilla Student
WKD-L (Ours(
FitNet
WKD-F (Ours)
Figure 8; Visualization of different models via Grad-CAM.
E
Extra Experiment on Object Detection
is much more The framework of RaNN 47 with Feature Pyramid Network (FPN) 48
complicated than that of image classification. It contains the backbone network, Region Proposal Network (RPN), Feature Pyramid Network (FPN), and detection head consisting of a classification branch and localization branch. For feature distillation, besides RolAlign layer, FPN can also be used for knowledge transfer 29
E.1 Implementation details COCO
discrete WD to match the probabilities predicted by the classification branches of the teacher and student. We set the temperature = and sharpening parameter к x= 2, and set all the weights of LCE. L, and LWKD-L are the same as RN101-18 -R18 but that set to 1. For WKD-F, we transfer knowledge from features straightly fed to the classification branch, i.e., features output by the RolAlign layer. We let RolAlign generate a high resolution of x8x15 18 feature maps to exploit more features, and choose a 4 x4 spatial grid for computing Gaussians. For both settings, set the weight of LcE to 1, and that of LWKD-F we use a projector of x Conv with BN.
For WKD-L,
for RN101 RNN18. All hyper-parameterss of RN50-MMV?
to 5e-3 with mean-cov ratio = 2; as in FCFD S
E.2 Ablation Analysis of WKD-F for Object Detection
We analyze key component of WKD-F specific to object detection on MS-COCO as the teacher and 8 88 as the student.
4 with ResNet101
Where to distill features: RolAlign or FPN? Among five stages (P2-P6) of FPN, we select single with spatial grid 16 for extracting Gaussians; this option produces best result among the candidates, and combination of multiple stages brings us no benefit. We compare to the bascline FitNet [24] and the results are shown in Table [2a When distilling features FPN, WKD-F significantly outperforms FitNet (over 2% mAP). By moving the distillation position to RolAlign layer, the performance both methods improve, while WKD-F still performs better than FitNet by non-trivial margin.
How spatial size of RolAlign features affect performance? In RaNN, the RolAlign layer outputs standard 7 x 7 feature maps. To exploit more spatial information, we let RolAlign layer output feature maps of higher resolution. Table shows effect of size of feature maps on performance. can be seen that when the spatial size enlarges mAP increases accordingly, and the mAP tends to saturate if the size as large as 28 28. The result suggests that larger of RolAlign features benefit feature distillation.
23

---

Method
Position of feutures
mAP
APso
AP75
Size of maps
mAP
AP50
AP75
RCNN-IPN
RNIOI -RNI8
APaa
APrs
FRNNet
WKD-F (ours)
FPN
36.40 56.52
34.13
54.16
39.64
36.71
7x7
36.94
57.03
40.01
RCFD
37.37
57.60
40.34
FiaNet
36.45
56.93
39.72
18x
37.21
$7.32
40.15
+WKD-L-WKD
37.66
58.01
40.66
WKD-F (ours)
RolAlign
37.21
57.32 40.15
28
37.15
57.23
40.11
ReviowDD
36.75
56.72
34,00
(b) Effect of size of RolAlign feature maps.
+WKD-L-WKD.F
37.50
57.79 40.38
(a) Effect of position of feature distilla- tion." Reproduced by
Table 13: Our WKD bene- fits state-of-the KD meth- ods for object detection.
Table 12:
Ablation analysis of feature distillation (i.e., WKD-F) for object detection on MS-COCO.
E.3 Integration with State-of-the-art KD Methods
We assess whether our methodology is complementary two state-of-the-an methods, i.c. ReviewKD and FCFD. To this end, for ReviewKD, we add the losses of WKD-L and WKD-F into the original loss functions; the weights of our two losses and nyper-parameters are same as those in Section F. For FCFD, we replace respectively the original KD) loss and feature distillation loss (i.e., FitNet loss) by the losses of WKD-L and WKD-F whose weights are identical those specified in the main paper. The results shown in Table 13 We can that, by integrating our methods, both FCFD and Re improve by non-trivial margins, which indicates that our methodology parallel to them and thus can enhance their performance.
E,4 Implementation Details on BB Regression for KD
Besides common logit distillation and feature distillation, FCFD 8 additionally uses BB regression for knowledge transfer and achieves state-of-the-ar performance. Here we also introduce it into our methodology. Specifically, for each proposal from RPN of the student [47]. both student and teacher perform BB regression, predicting separately a localization offset vector (LOV) o, from which we obtain predicted bounding box B of the target class. Then the distillation loss is written as
LBB C2 _o) +ELoo ,B²))..
(17)
where L2 indicates the square of Fuclidean distance between the student's LOV vector o and the teacher's 's one o²T. We use the Distance-loU loss LDIoU defined by Zheng et al. (80), which measures the Intersection over Union (loU) between two bounding-bow boxes 8 and B with a penalty term. The constant & is to balance the two losses and is set to 20 throughout.
F Limitations and Future Research
The cost of our WKD-L is higher than KL-Div based methods due to the regularized linear program- ming (23]. However, it is affordable as shown in Table 5 and will benefit from advance of faster algorithms for solving WD [5). Potentially, WKD-L can generalize be a label smoothing regular- ization method. Specifically, besides the CE loss, one introduces an additional loss that measures WD rather than KL-Div between the logits and uniform distribution [81) while computing IRs using embeddings of textual category names as 'prototypes' via ready-made LLMs [82]. Besides BAN based Self-KD in Section 4.3] is promising to further generalize WKD-L for teacher- ffrce knowledge distillation, e.g., by using customized soft labels and IRs based on weights of the softmax classifier.
Our WKD-F models distribution of features with Gaussians. As deep features of DNNs are generally of high-ding high-dimension and small-size, accurate estimation of covariance matrices difficult [52]. Therefore, exploration of robust and efficient methods for estimating Gaussian may further improve the performance of WKD-F. Besides, the Gaussians may be sub-optimal for modeling feature distributions. To the best of our knowledge, what distributions deep features may exactly follow an open problem. interesting to study other parametric distributions and the corresponding dis disiliiilasssies for knowledge distillation.
24

---

G Broader Impact
We address limitations of Kullback-Leiberr Divergence for knowledge distillation (KD), and can obtain stronger, lightweight models suitable for resource-limited devices. Our methodology is very promising, readily applicable to variety of visual tasks, e.g., image classification and object detection. We hope our work sheds light on importance of WD and inspires future exploration of it in the field of knowledge distillation.
With breakthroughs of large-scale pre-training, multimodal large language models (LLMs) like GPT- 4 (83 have excelled in many visual tasks. Our methodology be potentially applied transfer knowledge from LLMs to smaller ones for specific visual or language tasks, allowing for improved performance while preserving fast inference cost. Consequenty, researchers as well as practitioners can more easily benefit from advanced technologies of LLMs, facilitating their widespread use and broader accessibility.
Currently, the theoretical quest for KD is limited. Due to the black-box nature of the distillation process, the student distilled using methods may inevitably inherit harmful biases from the teacher [84). Moreover, the reduction in deployment cost may lead to more potential harms of model abuse. This highlights the necessity for more widespread efforts regulate the of artificial intelligence techniques including knowledge distillation.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly reflect the contributions and scope of the paper in the abstract and Section | and verify our main claims in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

Justification:
Our paper contains no theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided in-depth implementation details and experimental settings reproduce experimental results in Sections
C.5
C.6
and
E
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
27

---

Answer: [Yes]
Justification:
The source code available at http://peihualloor/WWDDD
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ Caleaas//oSSSSuubisseeee httgs/cccoobbblccgggnoooo/o/eeee
seipaiisssooooooyyyo. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present all the training and test details in Sections 4.1
C.6
and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide the mean accuracy and standard deviation over three runs on CIFAR-100 in Table and Table I
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
28

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the computational resources used for our experiments in Section and the training latency our method Table 5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: All our research strictly adheres to the NeurIPS Code Ethics, and we have discussed the potential societal impact in Section G
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed this in Section
G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
29

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All our experiments conducted open- source datasets. Therefore, paper does not poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the data, code and models used in the paper, and all the assets we utilized are open-source.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
30

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This issue is not applicable paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "7EuTP9GghE", "reviewer": "Reviewer_ScyW", "summary": "The paper proposes the utilization of Wasserstein Distance based distillation as opposed to KLD, as is common in practice. This is because the latter does not facilitate cross-category comparisons. Both logit and feature based variants have been proposed. The comparisons have been shown for both classification as well as detection.", "questions": "None", "limitations": "The cost expensiveness has been mentioned.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Though straightforward look at the empirical gains are not massive, the method is theoretically sound, and the paper is easy to understand. The experiments are decently presented in my opinion. The paper provides a fresh perspective of WD in distillation.", "weaknesses": "I would have been happier to see if changing the divergence led to more significant boosts, if at all possible (basis the premise of providing cross category comparisons).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ScyW,\n\nThank you for reviewing our paper and for providing constructive and insightful comments. Particularly, we appreciate your positive comments on novelty (\"a fresh perspective of WD in distillation\"), on soundness (\"theoretically sound\"), and presentation \\& contribution (\"good\"). We hope our detailed responses can address your concern, and could you please consider increasing your rating accordingly.\n\n> ### I would have been happier to see if changing the divergence led to more significant boosts, if at all possible (basis the premise of providing cross category comparisons).\n\nThanks for your concern. \n\nOur WKD provides a novel viewpoint of WD in knowledge distillation (KD), which have shown very competitive performance, compared to pre-dominant KL-Div based knowledge distillation methods. Considering that WD is overlooked and rarely studied in KD, we believe that **the promising results of our work can inspire great interest and the follow-up works have potentials to yield larger performance boosts over the KL-Div based ones.** \n\nIt is worth mentioning that **the performance improvement of WKD is non-trivial in the *commonly used settings*** which has long been studied. For example, in the setting (a) on ImageNet, for logit distillation WTTM (SOTA of 2024) improves NKD (SOTA of 2023) by 0.23 percentage points (PPs) while the latter surpasses DKD (SOTA of 2022) by 0.26 PPs; in contrast, WKD-L outperforms WTTM by 0.30 PPs; for feature distillation our WKD-F surpasses the SOTA method of ReviewKD by 0.89 PPs. Finally, we note that **our WKD considerably outperforms the competitors *in the new setting of across CNN and vision transformers***, as shown in Table 9(a) in Section C.2 of Appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "033EDSruuY", "reviewer": "Reviewer_Zit9", "summary": "The paper introduces a novel methodology for knowledge distillation using Wasserstein Distance (WD) instead of the traditional Kullback-Leibler Divergence (KL-Div). The proposed methods include a logit distillation approach (WKD-L) that leverages cross-category comparisons and a feature distillation method (WKD-F) that models feature distributions parametrically. The authors demonstrate that their methods outperform strong KL-Div variants on image classification and object detection tasks.", "questions": "* Why did you choose Gaussian distributions for feature modeling in the context of WD? Are there other parametric methods that you considered, and how would they compare in terms of performance and feasibility?\n* How does the computational complexity of your proposed WD-based methods compare to traditional KL-Div based methods? Can you provide a detailed analysis or empirical comparison of the computational costs involved?", "limitations": "* please check the questions and weaknesses.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The introduction of WD in knowledge distillation provides a fresh perspective and addresses the limitations of KL-Div, particularly in terms of cross-category comparisons and handling non-overlapping distributions.\n* The use of parametric methods for feature distribution modeling, specifically Gaussian distributions, is innovative and effectively leverages the geometric structure of the data.", "weaknesses": "* The paper suffers from several writing issues, including grammatical errors and unclear explanations, making it difficult to follow the arguments and methodology at times.\n* Some of the assumptions made for the application of WD, particularly the choice of Gaussian distributions for feature modeling, may not hold universally. Further justification or exploration of alternative parametric methods would strengthen the work.\n* The computational complexity of implementing WD-based methods, especially in large-scale scenarios, is not adequately addressed. A comparison of computational costs between WD and KL-Div would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Zit9,\n\nThank you for reviewing our paper and for providing constructive and insightful comments. We are grateful for your positive comments on novelty and contribution of WD based methods, such as \"a fresh perspective and addresses the limitations of KL-Div\" and \"is innovative and effectively leverages the geometric structure of data\". We hope our detailed responses can address your concerns, and could you please consider increasing your rating accordingly.\n\n> ### Why did you choose Gaussian distributions for feature modeling in the context of WD? Are there other parametric methods that you considered, and how would they compare in terms of performance and feasibility?\n\nThanks for your concerns.  \n\nKindly note that, **we have discussed why we choose Gaussians on Lines 63--68 in Section 1 and Lines 154--157 in Section 2.2.** Specifically, Gaussian is of maximal entropy for given 1st-moment (mean) and 2nd-moments (covariance) estimated from random samples (features) among the family of parametric distributions. Namely, it has maximum uncertainty with the least prior and thus is more general than other candidates. In addition, WD between Gaussians is a Riemannian metric with closed-form solution; in contrast, WDs between other parametric distributions are open problems, as far as we know.\n\n**We have compared to other parametric methods in Table 3 (a) and Lines 267--276**, including *separate spatial 1st- or 2nd-moments [35]* as well as *separate channel 1st- or  2nd-moments [6]*. We note that both [35] and [6] fail to consider the geometry of the underlying manifold; kindly refer to Lines 205--218 and Table 1 for our discussion. The experiments show that our method outperform them by large margins with insignificant increase of computation (See Table 5).  \n\n**Following your suggestion, we consider additional parametric distributions** including Laplace and exponential distributions. We use univariate Laplace or exponential distribution for modeling each component of features. For them KL-Div can be computed in closed-form [Ref1] but WD is an unsolved problem. For Gaussian (Diag) we use univariate Gaussian for each feature component. We experiment with the same setting as in Table 3 (a) and the comparison is shown in the table below. We firstly note that, *with KL-Div*, Gaussian is better than both Laplace and exponential functions, which suggests Gaussian is a better option among the competing parametric distributions. Furthermore, Gaussian with WD outperforms Gaussian with KL-Div, which we attribute to the fact that WD can effectively exploit geometry of the underlying manifold but KL-Div cannot. \n\n|Distribution |$\\mid$|Dis-similarity|$\\mid$|Top-1|\n|:-:|:-:|:-:|:-:|:-:|\n|Laplace|$\\mid$|KL-Div|$\\mid$|71.38|\n|Exponential|$\\mid$|KL-Div|$\\mid$|70.14|\n|Gaussian (Diag)|$\\mid$|KL-Div|$\\mid$|71.75|\n||\n|Gaussian (Diag)|$\\mid$|WD|$\\mid$|**72.50**|\n||\n\n[Ref1] M. Gil. On Rényi divergence measures for continuous alphabet sources. Master's thesis\n, 2011.\n\n> ### How does the computational complexity of your proposed WD-based methods compare to traditional KL-Div based methods? Can you provide a detailed analysis or empirical comparison of the computational costs involved?\n\nThanks for the comments.\n\nKindly note that, **we have provided an empirical comparison of the computational cost in Table 5 and Lines 306--317.**  Compared to traditional KL-Div, our WKD-L, the logit-based distillation method, has somewhat higher cost due to the regularized linear programming; this limitation is discussed in Section F. Meanwhile, our WKD-F, the feature-based distillation method, has comparable cost with traditional KL-Div.    \n\n**Following your suggestion, we further conduct theoretical analysis.** The logit-based WKD-L is formulated as an entropy regularized linear programming, which can be solved by fast Sinkhorn algorithm [23]. Let $n$ be the dimension of the predicted logits, the computational complexity of WKD-L can be written as is $O(Dn^2 \\log n) $ [Ref2]. Here  $D=\\|\\|C\\|\\|\\_{\\infty}^{3}\\epsilon$ is a constant, where $\\|\\|C\\|\\|\\_{\\infty}$ indicates the infinity norm of the transportation cost matrix $C = [c_{ij}]$, and $\\epsilon>0$ indicates a prescribed error. In contrast, the computational complexity of KL-Div is $O(n)$. Despite its high complexity, WKD-L can be computed efficiently as Sinkhorn algorithm is highly suitable for parallel computation on GPU [23]. For feature-based WKD-F, the dominant cost is due to computation of means and variances. Give a set of $m$ features $\\mathbf{f}_{i}$ of $l$-dimension, the means can be computed by global average pooling (GAP) that takes $O(ml)$ time; the complexity of variances is also $O(ml)$ which can be implemented by element-wise square operations followed by a GAP. \n\n[Ref2] J. Altschuler, et al. Near-linear time approximation algorithms for optimal transport via sinkhorn iteration. In NeurIPS, 2017.\n\n\n> ### Some of the assumptions made for the application of WD, particularly the choice of Gaussian distributions for feature modeling, may not hold universally.\n\nThanks for the comment. \n\nWe agree that the Gaussian distributions may not hold universally. **Kindly note that this limitation has been discussed in Section F.** Specifically, what distributions deep features may exactly follow is an open problem, which is rarely studied in deep learning to our best knowledge. As such, Gaussian may be sub-optimal for modeling distributions of DNN features, and it is interesting to explore other parametric distributions.  In addition, the Wasserstein distances between parametric distributions other than Gaussian are open questions in probability and statistics and worth future research.\n\n> ### The paper suffers from several writing issues, including grammatical errors and unclear explanations, making it difficult to follow the arguments and methodology at times.\n\nThanks for your comment. We’ll further polish our manuscript, carefully addressing writing issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0gSsFnvXqM", "reviewer": "Reviewer_5GY4", "summary": "This paper proposes a Wasserstein distance based knowledge distillation method for both the logit distillation and feature distillation settings. The logit based version uses discrete WD to model the discrepancy beteween the prediction probabilites of student and teacher networks. It further uses the separation of target probability to improve performance. The feature based version minimizes the continuous WD between the patch features of an image from the student and teacher networks, under the assumption that they form Gaussian distributions. The covariance term is simplifed to its diagnoals to further improve performance. Comapred to recent baslines, the proposed method shows superior performance on ImageNet classification and COCO object detection.", "questions": "1. Could you explain the statement at line 120 \"this implicit effect is insignificant\"? \n2. Line 38. \"... features of an image are ... small size\". What does the \"small size\" of an image referes to?\n3. Are the student networks randomly initialized or initialized from a trained weights for baseline methods (line 239)?\n4. Line 172 \"... partition the feature maps into a kxk spatial grid\", and line 289 \"use ... 1x1 grid for classification\". There is no grid setting for detection in the main text and can mislead readers to think grid is not used at all.", "limitations": "The authors discussed the limitations.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Strong performance compared to recent work.\n2. The presentation is easy to follow and well structured. Related works are introduced to give good contexts.\n3. Extensive comparison between baselines, detailed abalation study, and has extra experiments on distillation across CNNs and transformers in appendix.", "weaknesses": "One of the motivation for the WKD-L is the cross-category comparison, however, it is not clear to me why the \"cross-category\" comparison is attrubted as the source of improvement. Firstly, there is implicit cross-category comparision for KL based methods (line 117), so it is not a differentiator. Secondly, without the story of cross-category comparison, the WDK-F also shows improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 5GY4,\n\nThank you for reviewing our paper and for providing constructive and insightful comments. Particularly, we appreciate your acknowledgement that our paper has excellent contribution, strong performance and good presentation. We hope our detailed responses can address your concerns, and could you please consider increasing your rating accordingly.\n\n\n> ### Could you explain the statement at line 120 \"this implicit effect is insignificant\"? it is not clear to me why the \"cross-category\" comparison is attributed as the source of improvement.\n\nThanks for your concerns.\n\nFirst, please allow us to explain what \"implicit effect\" exactly means in classical KD that is a logit-based distillation method. While comparing WKD-L (Figure 1b (left)) to classical KD (Figure 2), we can see that KL-Div only performs category-to-category comparison, **but lack of mechanism to *explicitly* exploit cross-category comparison via pair-wise IRs in Eq. (1),** which contain rich knowledge among categories. Nevertheless, for KL-Div based KD, through backpropagation the probability of one category affects probabilities of the other categories due to softmax function; in this sense, we say that the cross-category effect in KL-Div is implicit. \n\nBy stating \"this implicit effect is insignificant\", we mean that **gains brought by explicit cross-category comparison in WD is more significant than that brought by the implicit influence in KL-Div, as experimentally shown in Table 2(a).** Specifically, WD vs. KL-Div in Top-1 Acc (%) is 72.04 vs. 71.03 without (w/o) target separation and is 72.49 vs. 71.96 with (w/) target separation. As such, we attribute the improvement to cross-category comparison inherent in WD. \n\nAdditionally, kindly note that WKD-F belongs to the type of feature-based distillation methods, quite different from the type of logit-based ones, such as WKD-L and classical KD & its variants. Therefore, the effectiveness of WKD-F cannot suggest that the improvement in WKD-F is not attributed to cross-category comparison. \n\nWe are sorry for the possible ambiguity and we’ll make further clarification in the modified paper.\n\n> ### What does the \"small size\" of features of an image refers to (Line 38)?\n\nThanks for the concern. \n\n**Here \"small size\" indicates that for an input image the number of convolutional features output by a CNN is small.** For example, for ResNet50 with a standard 224x224 input image, the commonly used feature maps of stage 5 (Conv\\_5x) contain only 49 (7$\\times$7) features, which however are of high-dimension (i.e. 2048). \n\nThe high-dimensional features of small size not only makes non-parametric density estimation (e.g., histogram) infeasible, but also leads to non-overlapping discrete distributions, both bringing  difficulty for KL-Div. One may turn to parametric distributions (e.g., Gaussian); however, KL-Div is limited as it is not a true metric and is unaware of geometry of the underlying manifold. **This downside of KL-Div motivates our 2nd contribution, i.e., WKD-F that uses Wasserstein distance as dis-similarity between Gaussians.** \n\n> ### Are the student networks randomly initialized or initialized from a trained weights for baseline methods (line 239)?\n\nThanks for the concern. \n\nFor fair comparison with previous arts, we use the framework of Faster-RCNN for object detection. **For the student models, the backbones are initialized with pre-trained weights on ImageNet, while Region Proposal Network (RPN), Feature Pyramid Network (FPN), and detection head consisting of a classification branch and a localization branch are all initialized randomly.**  Note that this is a standard practice widely adopted by knowledge distillation methods for object detection, as well as by all general object detection methods. \n\n> ### There is no grid setting for detection in the main text and can mislead readers to think grid is not used at all.\n\nThanks for your suggestion.\n\nWe are sorry for the possible ambiguity. As described in Section E.1 in Appendix, for object detection, we use a 4$\\times$4 spatial grid for computing Gaussian. We’ll make this clear in the main text.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GrwJt9kUdB", "reviewer": "Reviewer_F2zQ", "summary": "This paper proposes a new algorithm for knowledge distillation by replacing KL-divergence loss with Wasserstein distance loss.\nThe proposed algorithm contains two parts:       \n(1) Logits distillation with Wasserstein distance loss, implementing with an entropy regularized linear programming;        \n(2) Assuming the features obey Gaussian distribution, the Wasserstein distance loss can be solved with parametric tricks.\n\nExperiments on ImageNet and CIFAR100 show obvious improvements compared to baselines like KD, DKD, and NKD.", "questions": "(1) Ablation for $\\lambda$ in Eq. (3).     \n(2) Explanation and analysis of sensitivity of all hyper-parameters.         \n(3) How to calculate $\\mu^T$ and $\\mu^S$ in your implementation for Eq. (9)?                  \n(4) Does the proposed method work well in the setting of self-KD where the student and teacher share the same architecture?", "limitations": "Limitations are discussed in the paper", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The paper is written clearly and is easy to follow.       \n(2) Sound ablations for combinations of the proposed method and previous ones.", "weaknesses": "(1) Eq. (3) and Eq. (5) both have hyperparameter $\\lambda$. Does the two lambda have the same value in implementation?\n      The ablation for $\\lambda$ in Eq. (3) is missing.\n\n(2) There are too many hyper-parameters to be tuned.                       \n       lambda in Eq. (3),                 \n       lambda in Eq. (5),                   \n       k for calculating IR,                                  \n       gamma for a trade-off between D_mean and D_cov,                              \n       weight between WKD-L and WKD-F,                                \n       temperature.                                                                                     \n       Do all experiments in the paper adopt consistent hyper-parameters?\n         \n(3) What's the layer features are used for WKD-F? \n      Feature dimensions are usually higher than logits. Moreover, multiple-layer features are often used for feature-based methods.\n      Thus, it seems to be wired that WKD-F (207ms) is even faster than the original KD (215ms).\n\n(4) The paper claims that Wasserstein distance rivals Kullback-Leibler divergence for knowledge distillation as indicated by their title.\nHowever, as shown in Table 2(a) and Table 2(b), the KL-div method achieves 71.96 in the setting of separating target and non-target.\nThe WD-based method just achieves comparable results with the Polynomial kernel, class centroid, or classifier weight for calculating IR.\n\nIt seems that the IR is more important than the form of WD formulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer F2zQ,\n\nThank you for reviewing our paper and for providing constructive and insightful comments. We appreciate your acknowledgement of good soundness, presentation \\& contribution of our paper. We hope our detailed responses can address your concerns, and could you please consider increasing your rating accordingly.\n\n> ### Eq. (3) and Eq. (5) both have hyperparameter $\\lambda$. Does the two lambda have the same value in implementation? The ablation for the hyperparameter in Eq. (3) is missing.\n\nWe apologize for our typo that leads to confusion. \n\nThe **hyper-parameter (HP) in Eq. (3) is a typo and will be denoted by $\\eta$**, indicating the regularizing constant of entropic term, while  $\\lambda$ in Eq. (5) indicates the weight of WD loss; they have different values. We ablate $\\eta$ in Setting (a) and results (Acc, \\%) are shown below along with Figure R1 in PDF for Global Rebuttal. We see across a decently large range the performance changes smoothly, particularly for Top-5 Acc. Notably, **$\\eta$ in Eq. (3) is always set to 0.05 across the paper (Line 231)** in either classification on ImageNet \\& CIFAR100 or object detection on MS-COCO.\n\nMethod|$\\mid$|$\\quad \\eta$  |$\\mid$|Top-1|$\\mid$|Top-5|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|KD|$\\mid$|NA|$\\mid$|71.03|$\\mid$|90.05|\n||\n|WKD-L|$\\mid$|1.0e-2|$\\mid$|71.65|$\\mid$|90.21|\n|--|$\\mid$|2.0e-2|$\\mid$|72.15|$\\mid$|90.70|\n|--|$\\mid$|5.0e-2|$\\mid$|**72.49**|$\\mid$|**90.75**|\n|--|$\\mid$|1.0e-1|$\\mid$|72.25|$\\mid$|90.74|\n|--|$\\mid$|2.0e-1|$\\mid$|72.09|$\\mid$|90.53|\n||\n\n> ### Explanation and analysis of sensitivity of all hyper-parameters.\n\nAbove all, kindly note **as logit-based WKD-L and feature-based WKD-F are two distinct methods, their hyper-parameters (HPs) should be analyzed separately.**\n\nWKD-L is to address the downside of lack of cross-category comparison in classical KL-Div. **We ablate its HPs containing temperature, sharpening parameter and weight in Section B.1 (Lines 636--649 and Figure 5),** where we see that they are insensitive in decently large ranges. $\\eta$ in Eq. (3) is always fixed across the paper.  \n\nWKD-F aims to address the problem of KL-Div that cannot exploit geometry of data manifold. **We ablate its HPs (i.e., mean-cov ratio and the weight) in Section B.1 (Lines 650--659 and Figure 6),** and observe that they are invulnerable to large variation of values.  \n\nFurther, combining WKD-L and WKD-F leads to further improvement; in this case, we simply add their losses **where all HPs are unchanged.** \n\nFinally, for classification on ImageNet a summary of HPs is given in Section B.2; on CIFAR100, for a fair comparison, we follow DKD [3], CAT [51], ReviewKD [29], FCFD [8] and WTTM [5] *for KD within CNNs,* and follow OFA [62] *for KD across CNNs and Transformers,*  tuning HPs separately for different settings (Section C). For object detection, the framework of Faster-RCNN is quite different from that of classification, and how HPs are turned are given in Section E.  \n\n> ### How to calculate $\\mu^S$ and $\\mu^T$ in your implementation for Eq. (9)?\n\nOn Lines 602--610 we explain concretely how to compute the means. For teacher, we calculate directly $\\mu^T$ using features output by **single stage of Conv5_x with 1x1 spatial grid**; for student we use the corresponding features after the projection layer  for calculating $\\mu^S$.\n  \n> ### Does the proposed method work well in the setting of self-KD where the student and teacher share the same architecture?\n\nThanks for the suggestion. \n\nWe implement self-KD in the framework of Born-again network (BAN) (Furlanello et al., ICML’18). Specifically, we first train a network model $S_{0}$ with ground truths. With $S_{0}$ as the teacher, we use WKD-L and distill a new model $S_{1}$ with same architecture. For simplicity's sake, we do not perform the 2nd-generation distillation, i.e., distilling a new model $S_{2}$ with $S_{1}$ as the teacher, etc.\n\nWe experiment with ResNet18 and the HPs are same as those in Setting (a). For logit-distillation, state-of-the-art USKD [4] is slightly better than BAN (KL-Div), both improves the regular training without self-KD; our BAN (WKD-L) performs much better than USKD, suggesting superiority of WKD-L in self-KD setting. \n\n|$\\ \\ $Method|\\||Self-KD |\\||Dis-similarity|\\||Top-1|\n|:-:|-|:-:|-|:-:|-|:-:|\n|Regular|\\||$\\times$|\\||NA|\\||69.75|\n||\n|USKD|\\||$\\checkmark$|\\||KL-Div|\\||70.75|\n|BAN|\\||$\\checkmark$|\\||KL-Div|\\||70.50|\n|BAN (WKD-L)|\\||$\\checkmark$|\\||WD|\\||**71.35**|\n||\n\n\n\n> ### What's the layer features are used for WKD-F? It seems to be wired that WKD-F (207ms) is even faster than the original KD (215ms).\n\nFor WKD-F, we use the **features of single stage of Conv5\\_x,** i.e., 49 features of 512-dimension that is lower than logits (i.e., 1000). Our ablation in Table 3(c) shows that single stage with 1x1 grid is the best option. Kindly note that for WKD-F, we only need to compute the mean and variances for Gaussian (Diag) that are very efficient. To resolve your concern, we also measure latencies (ms) in Setting (b), where we have 49 features of 2048-dimension for distribution matching. Compared to Setting (a), the latencies of both methods increase while that of WKD-F is slightly higher. \n\n|Method|$\\mid$|Strategy|$\\mid$|Dimension|$\\mid$|Latency|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Setting (a)\n|KD|$\\mid$|Logit|$\\mid$|1000|$\\mid$|215|\n|WKD-F|$\\mid$|Feature|$\\mid$|512|$\\mid$|207|\n||\n|Setting (b)\n|KD|$\\mid$|Logit|$\\mid$|1000|$\\mid$|268|\n|WKD-F|$\\mid$|Feature|$\\mid$|2048|$\\mid$|276|\n||\n\n \n\n\n> ### It seems that the IR is more important than the form of WD formulation.\n\nThanks for the concern.\n\nKindly note **IR is a key ingredient of our WD formulation,** without which IRs cannot play the roles, e.g., IRs cannot be used in KL-Div based knowledge distillation. \n\nIt is worth noting that the results in Table 2(b) show that WD formulation with each IR performs better than strongest KL-Div variant [4] (71.96), suggesting it is superior to KL-Div formulation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, Area Chairs and Program Chairs,

We thank all reviewers again for their thoughtful and constructive comments, which are helpful in improving the quality of our paper. After carefully reading all comments and questions, we conducted additional experiments and discussions to address the reviewers' concerns. In the following, we provide a summary of the reviewers' comments and our responses.

**Firstly**, we are pleased to observe that majority of reviewers have highly recognized the innovativeness, clarity of presentation, experiments and performance of our work, initially recommending **1 accept** and **3 borderline accept**. Specifically, we are grateful for the following positive comments about our work:

* The paper provides **a fresh perspective** of WD in distillation (Reviewers ScyW and Zit9) and **a new algorithm** for knowledge distillation (Reviewer F2zQ); the use of parametric methods for feature distribution modeling, specifically Gaussian distributions, **is innovative.** (Reviewer Zit9).
* The proposed methods exhibit **strong performance** compared to recent work (Reviewer 5GY4), and show **obvious improvements** compared to baselines (Reviewer F2zQ). The experiments conduct **extensive comparison** (Reviewer 5GY4), **are decently presented** (Reviewer ScyW)  and provide **sound ablations** (Reviewer F2zQ). The paper provides **extra experiments on distillation *across CNNs and transformers*** in appendix (Reviewer 5GY4).
* The paper is **easy to understand** (Reviewer ScyW), **easy to follow, written clearly** (Reviewer F2zQ) and **well structured** (Reviewer 5GY4). Additionally, the related works are introduced to **give good contexts** (Reviewer 5GY4).

---

**Secondly**, we provide point-by-point responses to all four reviewers, striving to address their concerns and questions. We will carefully revise our manuscript according to these discussions. Here, we list major experiments and discussions that have been added in responses.

> ### 1. Compared to other parametric methods for feature distribution modeling.

**Following the suggestion of Reviewer Zit9**, we compare to additional parametric distributions including Laplace and exponential distributions. **With KL-Div**, Gaussian distribution is a better option in capturing the feature distribution and knowledge transferring. **By leveraging the Wasserstein distance (WD)**, Gaussian distribution achieved further improvements in performance. For detailed results and discussions, please refer to our response to Reviewer Zit9 and Table R1 in the attached PDF. It is worth mentioning that **we have already compared to parametric methods including 1st- or 2nd-moments (spatial and channel)** in the paper (Table 3 (a) and Lines 267--276).

> ### 2. Additional experiments on self-KD settings.

**Following the suggestion of Reviewer F2zQ**, we implement self-KD in the framework of Born-again network (BAN) (Furlanello et al., ICML’18). Our **WKD-L performs much better than classical KD and the state-of-the-art method USKD**, suggesting the effectiveness of WKD in self-KD settings. The detailed results can be found in our response to Reviewer F2zQ and Table R2 in the attached PDF.

> ### 3. Ablation study on the regularizing constant of entropic term in Eq. (3).

**Following the suggestion of reviewer F2zQ**, we have conducted an ablation study on the regularizing constant of the entropic term $\eta$ in Eq. (3). Across a decently large range, the performance changes smoothly, indicating that our method is not sensitive to $\eta$. **Note that $\eta$ is always fixed across the paper.** The detailed results are provided in our response to Reviewer F2zQ and Figure R1 in the attached PDF.

> ### 4. Evaluation of training latency for Setting (b).

We have measured training latencies (ms) for Setting(a) in Table 5, where WKD-F is slightly faster than classical KD. **To resolve the concern of reviewer F2zQ**, we further measure the latencies for Setting (b), where the latencies of both methods increased while that of WKD-F is slightly higher. The detailed results are shown in response to Reviewer F2zQ and Table R3 in the attached PDF.

---

**Finally**, based on the constructive comments and suggestions from all reviewers, we will carefully revise our manuscript. We believe we can effectively address these concerns, and we kindly ask you to consider increasing your scores. We are looking forward to your positive feedback on our rebuttals, and please feel free to reach out if you have any additional questions.

Meta-review: The paper provides an interesting perspective of WD in distillation and a new algorithm for knowledge distillation. After initial rating, the authors provide a nice rebuttal which addressed all the issues mentioned by the reviewers. Now, all the reviewers though there is no further issues with this paper and the AC also learns toward an acceptance for this submission.
Decision: Accept (poster)