Knowledge Diffusion for Distillation
Authors: Tao Huang
Yuan Zhang
Mingkai Zheng
Shan You
Fei Wang
Chen Qian
Chang Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge distillation
diffusion models
TL;DR: This paper proposes a method that aligns teacher and student features by denoising student feature with a diffusion model.

Abstract: The representation gap between teacher and student is an emerging topic in knowledge distillation (KD). To reduce the gap and improve the performance, current methods often resort to complicated training schemes, loss functions, and feature alignments, which are task-specific and feature-specific. In this paper, we state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature, and propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models. Our approach is based on the observation that student features typically contain more noises than teacher features due to the smaller capacity of student model. To address this, we propose to denoise student features using a diffusion model trained by teacher features. This allows us to perform better distillation between the refined clean feature and teacher feature. Additionally, we introduce a light-weight diffusion model with a linear autoencoder to reduce the computation cost and an adaptive noise matching module to improve the denoising performance. Extensive experiments demonstrate that DiffKD is effective across various types of features and achieves state-of-the-art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at https://github.com/hunto/DiffKD.

Knowledge Diffusion for Distillation
Tao Huang 1.2
Yuan Zhang² Mingkai Zheng' Chang Xu'
Shan You²*
Fei Wang*
Chen Qian
'Scoool of Computer Science, Faculty of Engineering, The University of Sydney
Gbense'T'mee Research $Peking University 4 UUniversity of Science and Technology of China
Abstract
The representation gap between teacher and student an emerging topic in knowl- edge distillation (KD). To reduce the gap and improve the performance, current methods often resort to complicated training schemes, loss functions, and feature alignments, which are task-specific and feature-specific. In this paper, we state that the essence of these methods is to discard the noisy information and distill the valu- able information in the feature, and propose novel KD method dubbed DifIKD, to explicitly denoise and match features using diffusion models. Our approach is based on the observation that student features typically contain more noises than teacher features due to the smaller capacity of student model. To address this, we propose to denoise student features using a diffusion model trained by teacher fea- tures. This allows us to perform better distillation between the refined clean feature and teacher feature. Additionally, we introduce a light-weight diffusion model with a linear autoencoder to reduce the computation cost and an adaptive noise matching module to improve the denoising performance. Extensive experiments demonstrate that DiffKD is effective across various types of features and achieves state-of-the- art performance consistently on image classification, object detection, and semantic segmentation tasks. Code is available at https//githbb.com/uttt//iDi
Introduction
The success of deep neural networks generally accomplished with the requirements of large computation and memory, which restricts their applications on resource-limited devices. One widely- used solution is knowledge distillation (KD) [14], which aims to boost the performance of efficient model (student) by transferring the knowledge of a larger model (teacher).
The key to knowledge distillation lies how to transfer the knowledge from teacher to student by matching the output features (e.g., representations and logits). Recently, some studies [17, 29] have shown that the discrepancy between student feature and teacher feature can be significantly large due to the capacity gap between the two models. Directly aligning those mismatched features would disturb the optimization of student and weaken the performance. As a result, the essence of most state-of-the-art KD methods is shrink this discrepancy and only select the valuable information for distillation. For example, TAKD [29] introduces multiple middle-sized teach assistant models to bridge the gap; SFTN [30] learns mudeet--rieedly teacher by regularizing the teacher training with student; DIST [17] relaxes the exact matching of teacher and student features of Kullback- Leibler (KL) divergence loss by proposing a correlation-bant
loss; MasKD [18] distills the valuable information in the features and ignores the noisy regions by learning to identify receptive regions that contribute to the task precision. However, these methods need to resort to either complicated training schemes or task-specific priors, making them challenging to apply to various tasks and feature types.
"Correspondence to: Shan You <youshannSensettine.. com
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Image
Student feature
Denoised feature
KD Loss
Gaussian noise"
Student
Teacher
T
2t
21-1
Zo
Feature
Feature
Teacher feature
Figure 1: Diffusion model in DiffKD. The diffusion model is trained with teacher feature in diffusion process q (red-dashed arrow), while we feed the student feature to the reverse denoising process Pe (blue arrows) to obtain a denoised feature for distillation. We find that due to capacity limitation, student feature contains more noises and its semantic information is not salient the teacher's s. Therefore, we treat student feature as a noisy version of teacher feature, and propose to denoise student feature using diffusion model trained with teacher feature.
In this paper, we proceed from a different perspective and argue that the devil of knowledge distillation is in the noise within the distillation features. Intuitively, we regard the student noisy version of the teacher due to its limited capacity or training recipe to learn truly valuable and decent features. However, distilling knowledge with this noise can be detrimental for the student, and may even lead to undesired degradation. Therefore, we propose to eliminate the noisy information within student and distill only the valuable information accordingly. Concretely, inspired by the success of generative tasks, we leverage diffusion models [15, 40], a class of probabilistic generative models that can gradually remove the noise from an image or feature, to perform the denoising module. An overview of our DiffKD is illustrated in Fig. 1. We empirically show that this simple denoising process can generate a denoised student feature that is very similar to the corresponding teacher feature, ensuring that our distillation can be performed in more consistent manner.
Nevertheless, directly leveraging diffusion models in knowledge distillation has two major issues. (1) Expensive computation cost. The conventional diffusion models use a UNet-based architecture to predict the noise, and take large amount of computations generate high-quality images". In DiffKD, lighter diffusion model would suffice since we only need to denoise the student feature. We therefore propose light-weight diffusion model consisting of two bottleneck blocks ResNet [1]]. Besides, inspired by Latent Diffusion [35], we also adopt linear autoencoder to compress the teacher feature, which further reduces the computation cost. (2) Inexact noisy level of student feature. The reverse denoising process in diffusion requires to start from a certain initial timestep, but in DiffKD, the student feature is used as the initial noisy feature and we cannot directly get its corresponding noisy level (timestep); therefore, the inexact noisy level would weaken the denoising performance. To solve this problem, we propose an adaptive noise matching module, which measures the noisy level of each student feature adaptively and specifies corresponding Gaussian noise to the feature to match the correct noisy level in initialization. With these two improvements, our resulting method DiffKD is efficient and effective, and can be easily implemented on various tasks.
It is worth noting that one of the merits of our method DiftKD is feature- agnoosic, and the knowledge diffusion can be applied to different types of features including intermediate feature, classification output, and regression output. Extensive experimental results show our DiffKD surpasses current state- of-the-art art methods consistently on standard model settings of image classification, object detection, and semantic segmentation tasks. For instance, DiffKD obtains 73.62% accuracy with MobileNetV1 student and ResNet-50 teacher on ImageNet, surpassing DKD [53] by 1.57%; while on semantic segmentation, DiffKD outperforms MasKD [18] by 1% with PSPNet- R8 student on Cityscapes test set. Moreover, to demonstrate our efficacy in eliminating discrepancy between teacher and student features, we also implement DiffKD on stronger teacher settings that have much more advanced teacher models. and our method significantly outperforms existing methods. For example, with Swin-T student and Swin-L teacher, our DiffKD achieves remarkable 82.5% accuracy on ImageNet, improving KD bascline with large margin of 1%.
For example, the diffusion model in Stable Diffusion [35] takes 104 GFLOPs on an input resolution of 256 x 256 [33].

---

Image
Student
Denoised stadent
Teacher
Image
Student
Denoised student
Teacher
Figure 2: Visualization of student features, denoised student features and teacher features on COCO dataset. Details and more visualizations can be found in Appendix F.
2 Preliminaries
2.1 Knowledge distillation
Conventional knowledge distillation methods transfer the knowledge of pretrained and fixed teacher model to a student model by minimizing the discrepancies between teacher and student outputs. Typically, the outputs features of model. Given the student outputs F(*) and teacher outputs F(), the knowledge distillation loss is defined as
be the predictions (e.g., logits in classification task) and intermediate
Lka = d(F(s) F(
(I)
where d denotes distance function that measures the discrepancy of two outputs. For example, we can use Kullback-Leibler (KL) divergence for probabilistic outputs, and mean square error (MSE) for intermediate features and regression outputs.
2.2 Diffusion models
Diffusion models class of probabilistic generative models that progressively add noise the sample data, and then learn to reverse this process by predicting and removing the noise. Formally, given the sample data zo CxCxxx the forward noise process iteratively adds Gaussian noise to it, i.e.
<<z:|20)= =- (Zzz (1 az)
T". a. == 11 - 11 B.) where is the transformed noisy data timestep E (0.
notation for directly sampling 2; at arbitrary timestep with noise variance schedule [15]. Therefore, can express 2, as linear combination of and noise variable e,:
2,
V0 + vI a.Et:
where E, E WN I). During training, a neural network (= f) is trained to predict the noise in z w.r.t 2g by minimizing the L2 loss between them, i.e.,
Last ||e, 4 t)
(
During inference, with the initial noise 2,, the data sample zo is reconstructed with an iterative denoising process using the trained network Øg:
P0 = (1 1 (),z,)t),of)) o)),
(5)
where of denotes the transition variance DDIM [40], which accelerates denoising process by sampling with small number of score function evaluations (NFEs), i.e., #T 2T-A 20. where the sampling interval.
In this paper, we leverage diffusion model to eliminate the noises in student feature in our knowledge distillation method DiffKD, which will be introduced in the next section.
3 Method
In this section, we formulate our proposed knowledge distillation method DiffKD. We first convert the feature alignment task in KD to the denoising procedure diffusion models, this enables us to use diffusion models to match student and teacher outputs for more accurate and effective distillation.

---

Noie Magter
Dillasoo Madel
Studan
Aclapte
timeddep
Anhensadir
aylmeu
Butkenck
Tine embholaing
Trait
Avg Ruo
2
Berlewok
labest feature
Biitwds Model
DV Locs
PC
denoleed stadent featurre
Ciey [+]
Pac
nolsed Teaturea!
Tetter predictum
Figure 3: Architecture of DiffKD. Bottleneck denotes the Bottneck block in ResNet [1]].
To further improve the computation efficiency, we introduce feature autoencoder to reduce the dimensions of feature maps, thereby streamlining the diffusion process. Additionally, we propose an adaptive noise matching module to enhance the denoising performance of student feature. The architecture of DiffKD is illustrated Fig. 3.
3.1 Knowledge diffusion for distillation
Generally, models with different capacities and architectures produce varying preferences on feature representations, even when trained on the same dataset. This discrepancy between teacher and student is crucial to the success of knowledge distillation. Previous studies [19, 21J have investigated the differences between teacher and student features. Kundu al. [19] observes that the predicted probabilistic distribution of teacher is more sharp and confident than the student's. s. Similarly, ATS [21] discovers that the variance of wrong class probabilities of teacher smaller than that of student, indicating that the teacher output is cleaner and more salient. In summary, both studies find that the student has larger values and variances on wrong classes than the teacher, suggesting that the predictions of student contains more noises than the teacher's.
In this paper, we demonstrate that the same trend holds for intermediate features. We visualize the first feature map of FPN in RetinaNet [23] on COCO dataset [24] in Fig. 2 and find that the semantic information in teacher feature much more salient than the student feature. Therefore, we can conclude that both predictions and intermediate features of student model contain more noises than the teacher's, s, and these noises are difficult to eliminate through simple imitation of the teacher model in KD due to the capacity gap [17]. As a result, a proper solution is to disregard the noises and only imitate the valuable information from both teacher and student. Inspired by the success of eliminating noises in diffusion models [15, 35, 40], we propose to treat the student feature as a noisy version of teacher feature, and train a diffusion model with teacher feature, then use it to denoise the student feature. With the denoised feature that contains only valuable information as teacher feature, can perform noiseless distillation on them.
Formally, with teacher feature F(tea) and student feature F(sta) used in distillation, we use ((eea) in the forward noise process q( (FF (((ee)) (Eq. (2)) to train the diffusion model with Laif (Eq. (4)). Then the student feature is fed into the iterative denoising process of the learned diffusion model, i.e., po (o(((tt)) 1 denoising process, obtain a denoised student feature F(stu), which is used to compute the KD loss with the original teacher feature F in Eq. (1).
Ee(att) in Eq. (5), where F(stu) is the initial noisy feature of the process. After this
3.2 Efficient diffuusion model with linear autoencoder
However, we find that the denoising process in DiffKD can be computationally expensive due the large dimensions of the teacher feature. During training. DifIKD needs to forward the noise prediction network 4 for T times (we 7 5 in our method) for denoising the student feature and I time for training the noise prediction network with teacher feature. This T + times of forwarding can result in high computation cost when the dimension of teacher feature is large. To reduce the computation cost of diffusion model, we propose light diffusion model which is stacked with two bottleneck block in ResNet [11]. Then we follow Latent Diffusion Models [35] and propose to compress the number of channels using a linear autoencoder module. The compressed latent feature is used as the input of diffusion model. shown in Fig. 3, our linear autoencoder module is composed of two convolutions only, one is the encoder for reducing the number of channels, another one is the decoder

---

Table 1: Training strategies image classification tasks. BS: batch size; LR: learning rate; WD: weight decay; LS: label smoothing; EMA: model exponential moving average; RA: RandAugment [7]; RE: random erasing; CJ: color jitter.
Total Initial
Straloyy
Dataset
Epochs
BS
LR
Opümizer
WD
LS EMA LR scheduler
Data augmentation
CIFAR-100
240
SGD
x 10
x 00.1 150,180,210 opochs crop + flip
B1
ImageNet
100
256
0.1
SGD
1 10
x every 30 epochs
crop flip
ImageNet
RMSProp
x0.97 every 2.4 epochs AdamW 5 10 0.1 cosine
x 10
D.1 0.9999
(BI) +
B3 ImugeNet
300
1024
5e-4
(82) + CJ + Mixup + CuiMix
for reconstruct the teacher features. The output feature of encoder is used to train the diffusion models (Eq. (2)) and supervise the student.
The autoencoder is trained with reconstruction loss only, which is the mean square error between the original teacher teacher (toa)) and reconstructed teacher feature ((tea) .iee
Lae
[(tea)
Note that the latent teacher feature Z (tea) used to train diffusion model is detached and has gradient backward from the diffusion model.
We also use a convolution layer to project the student feature to the same dimension teacher latent feature Z denoted as (sstu) It is then passed to the diffusion model to perform a reverse denoising process (Eq. (5)) and generate the denoised student feature (stu), Now we have the latent representation (teea) of teacher and the denoised representation of student (stu), they are then used to compute the KD loss and supervise the student, i.e.,
Laitnd == d( 2 (atu) z (tea)).
Note that our DiffKD is generic and applics to various tasks (e.g., classification, object detection, and semantic segmentaion) and feature types (e.g.. intermediate feature, classification output, and regression output). We use simple MSE loss and KL divergence loss as the distance function d to compute the discrepancy of denoised student feature and teacher feature baseline implementation of DiffKD, while it is possible to achieve better performance with more advanced distillation losses.
3.3 Adaptive noise matching
As previously discussed, we treat student feature as noisy version of teacher feature. However, the noisy level, which represents the gap between the teacher and student features, is unknown and may vary depending different training samples. Therefore, we cannot directly determine which initial timestep we should start the diffusion process. address this issue, we introduce an adaptive noise matching module to match the noise level of student feature pre-defined noise level.
As shown in Fig. 3, we construct simple convolutional module to learn a weight y that fuses student output and Gaussian noise, which helps match the student output to the same noisy level of noisy feature initial time step T. Therefore, the initial noisy feature the denoising process becomes
Z (stu)
(stu) + 7)E-
(8)
This noise adaptation can be naturally optimized with the KD loss Led. since the optimal denoised student feature that has minimal discrepancy to the teacher feature is obtained when the student feature matches the appropriate noise level the denoising process.
Overall loss function. The overall loss function of DiffKD is composed of the original task loss, diffusion loss that optimize the diffusion model, a reconstruction loss to learn the autoencoder, and KD loss for distillation on teacher features and denoised student features, i.e.,
Liaai
Lrack + X Caitt + XaLao + >gLaiffld
where 71, λ2, and Ag are loss weights balance the losses. We simply set = >a in all experiments.
5

---

Table 2: Evaluation results of baseline settings on ImageNet. We use ResNet-34 and ResNet-50 released by Torchvision [28] as our teacher networks, and follow the standard training strategy (B1).
MSE: we implement our baseline for comparisons. f: we replace KL divergence loss with more advanced DIST loss in DiffKD.
Student (teacher)
Tea.
Stu.
KD [14]
Review [6]
DKD [53]
DIST [17]
MSE
DifKKD
DifiKD
Top-1
73.31
69.76
70.66
71.61
71.70
72.07
70.58
72.22
72.49
R18 (R33)
Top-5
91.42
89.08
89.88
90.51
90.41
90.42
89.95
90.64
90.71
Top-1
76.16
70.13
70.68
72.56
72.05
73.24
72.39
73.62
73.78
MBVI (R50)
Top-5
92.86 89.49
90.30
91.00
91.05
91.12
90.74
91.34
91.48
Table 3: Performance of students trained with strong strategies on ImageNet. The Swin-T trained with strategy B3 in Table others are trained with B2. The ResNet-50 is trained by TIMM [44], and Swin-L is pretrained on ImageNet-22K.
Top-1 ACC (%)
Teacher
Student
Tea.
Stu.
KD [14]
RKD [31]
SRRL [47]
DIST [17]
DiffKD
ResNet-34
76.8
77.2
76.6
76.7
77.8
78.1
ResNet-50
MobileNetV2
80.1
73.6
71.7
73.1
69.2
74.4
74.9
EfficientNet-BB
78.0
77.4
77.5
77.3
78.6
78.8
ResNet-50
78.5
80.0
78.9
78.6
80.2
80.5
Swin-I.
86.3
Swin-T
81.3
81.5
81.2
81.5
82.3
82.5
4
Experiments
In this paper, to sufficiently validate the generalization of our DifIKD, we conduct extensive experi- ments image classification, object detection, and semantic segmentation tasks.
4.1 ImageNet classification
Settings. Following DIST [17], we conduct experiments on bascline settings and stronger teacher settings. The training strategies for CIFAR-100 and ImageNet datasets summarized in Tab. 1. On baseline settings, we use ResNet-18 [11] and MobileNet V1 [16] as student models, and ResNet-34 and ResNet-50 as teachers, respectively; and the training strategy (B1) is the most common in previous methods [6, 17, 53]. While for the stronger teacher settings, we train students with much stronger teacher models (ResNet-50 and Swin-L (26)) and strategies (B2 and B3).
implement our DiffKD on the output feature of backbone before average pooling, and the output logits of classification head, and the distance functions are MSE and KL divergence (with temperature factor of 1), respectively. We set A X2 = λ3 1.
Results on baseline settings. We summarized the results on baseline settings in Tab. 2. Our methods outperforms existing KD methods with a large margin, especially on the MobileNet and ResNet- 50 setting, DiffKD significantly surpasses previous state-of-tt of-the-art DIST [17] by 0.38% on top-1 accuracy. For comparisons with our baseline one feature distillation, we also report the MSE results with the same distillation location as DiffKD. We can see that, by only adding our diffusion model for feature alignment, DiffKD obviously improves on MobileNet V1. Moreover, we replace the KI. divergence loss in the output logits of DiffKD with the advanced loss function DIST. which achieves further improvements. For instance, DiffKD with DIST loss improves DIST by 0.42% ResNet-18. This indicates that feature alignment in DifIKD is generic to different KD losses and can be further improved by changing the losses.
MSE results by 1. 1.74% on ResNet- and 1. 1.23%
Results on stronger teacher settings. To fully investigate the efficacy of DiffKD on reducing the distillation gap between teacher feature and student feature, we further conduct experiments on much stronger teachers and training strategies following DIST. From the results summarized in Tab. 3, we can see that DifTKD outperforms DIST on model settings, especially MobileNetV2 in the table, DiffKD surpasses DIST by 0.5%. It is worth to remind that our DiffKD only uses the simple KI. divergence loss and MSE loss on the stronger settings, the performance could be further improved if we use more advanced loss functions such DKD [53] and DIST [17].
the most Hight-weight

---

Table 4: Object detection performance with baseline settings on COCO val set. teacher. S: student. i: we replace MSE with an attention- based MSE loss.
Table 5: Object detection performance with stronger teachers on COCO val set. CM RCNN: Cascade Mask RCNN. t: we replace MSE with an attention traentioo-based MSE loss.
Method
AP 50 AP7s APs AP AP,.
Method
AP AP AP7s APs APM APr
Two-stage detectors
Two-stage detectors
T: Faster RCNN-RI01
39.8 60.1 43.3 22,5
43.6
52.8
T: CM RCNN-XI01
45.6 64.1 49.7 26.2 49.6 60.0
S: Faster RCNN-R50
38.4 59.0 42.0
21.5
42.1
50.3
S: Faster RCNN-R50
38.4 59.0 42.0
21.5
42.1
50.3
Fitnet [36]
38.9 59.5
42,4
21.9
42.2
51.6
COFD [13]
38.9 60.1
42.6
21.8
42.7
50.7
FRS [9]
39.5 60.1
43.3
22.3
43.6
51.7
FKD [50]
41.5 62.2
45.1
23.5
45.0
55.3
FGD [48]
40.4
22.8
44.5
53.5
FGD [48]
42.0
23.7
46.4
55.5
DifIKD
40.6 60.9 43,9 23.0
44.5
54.0
DifIKD
42.2 62.8 46.0 24.2
46.6
55.3
DiffKD
40.7 61.0 44.3 22.6
44.6
53.7
DiffKD
42.4 62.9 46.4 24.0
46.7
55.2
One-stage detectors
One-stage detectors
T: RetinaNet- R10iaaNet-R1013333
38.9 58.0 41.5 21.0 42.8 52.4
T: RetinaNet-XIO0
41.2 62.1 45.1 24.0 45.5
53.5
S: RetinaNet-R50
37.4 56.7 39.6 20.0
40.7 49.7
S: RetinaNet-R50
37.4 56.7 39.6
20.0
40.7
49.7
Fitnet [36]
37.4 57.1 40.0
20.8
40.8
50.9
COFD [13]
37.8 58.3
41.1
21.6
41.2
48.3
FRS [9]
39.3 58.8
42.0
21.5
43.3
52.6
FKD [50]
39.6 58.8
42.1
22.7
43.3
52.5
FGD [48]
39.6
22.9
43.7
53.6
FGD [48]
40.4
23.4
44.7
54.1
DifKK
39.7 58.6 42.1 21.6
43.8
53.3
DiffKD
40.7 60.0 43.2
22.2
45.0
55.2
DifKDD'
39.8 58.7 42.5 21.5
43.6
53.2
DifKKD
41.4 60.7 44.0 23.0
45.4
55.8
Anchor-free detectors
Anchor-free detectors
FCOS-RI01
40.8 60.0 44.0 24.2 44.3 52.4
ReePoinns-XXO0
44.2 65.5 47.8 26.2 48.4 58.5
FCOS-R50
38.5 57.7 41.0 21.9
42.8 48.6
RepPoints-RS.
38.6 59.6 41.6 22.5
42.2 50.4
FRS [9]
40.9 60.3
43.6
25.7
45.2
51.2
FKD [50]
40.6 61.7
43.8
23.4
44.6 53.0
FGD [48]
42.1
27.0
46.0
54.6
FGD [48]
41.3
24.5
45.2
54.0
DiffKD
42.4 61.0
45.8
26.6
45.9
54.8
DiffKD
41.7 62.6
44.9
23.6
45.4
55.9
DiffKD
42.5 61.1 45.6 25.2
46.8 55.1
DiffKD
41.9 62.8 45.0
24.4
45.7
55.3
Results for CIFAR-100 dataset are summarized in Appendix C.
4.2
Object detection
Settings. Following FGD [48], conduct experiments baseline settings and stronger teacher settings. On baseline settings, we adopt various network architectures, including two-stage detector Faster-RCNN [34], one-stage detector RetinaNet [23], and anchor-free detector FCOS [42], and use ResNet-50 [11] as student models, and 101 as teachers, respectively. The training strategy the most common one in previous methods [9, 18, 48]. While for the stronger teacher settings, we train students with much stronger teacher models, including two-stage detector Cascade Mask RCNN [2], stage detector RetinaNet [23], and anchor-free detector RepPoints [49], and stronger backbone ResNeXt- 101 (X101) [45].
We conduct feature distillation on the predicted feature maps, and train the student with DiffKD loss Ldiffidd regression KD loss, and task loss. Note that we do not use linear autoencoder in DiffKD since the number of channels in FPN is only 256. We set X X2 1. Besides using MSE loss in feature distillation, we also adopted a simple attention- based MSE loss (marked with in Tab. 4) inspired by FGD [48] to balance the foreground and background distillations, which is acknowledged important for detection KD [10, 18, 48]. Details can be found in Appendix B.
Results on baseline settings. Our results compared with previous methods are summarized in Table Our DiffKD can significantly improve the performance of student models over their teachers on various network architectures. For instance, DiffKD improve FCOS-R50 by 4.0 AP and surpasses FGD [48] by 0.4 AP. Besides, the attention based MSE loss affords consistent improvements on vanilla DiffKD by 0.1 AP.
Results on stronger teacher settings. We further investigate our efficacy on stronger teachers whose backbones are replaced by stronger ResNeXts [45]. As in Table 5, student detectors achieve
7

---

Table 6: Semantic segmentation results on Cityscapes dataset. t: trained from scratch. Other models are pretrained on ImageNet. FLOPs measured based on an input size of 1024 2048.
Params
FLOPs
mloU (%)
Params
FL.OPs
mlaU (%)
Method
Method
(M)
(G)
Val
Test
(M)
(G)
Val
Test
T: Deepl.bbV3.R10
61.1
2371.7
78.07
77.46
T: Deept.ahV3.R10
61.1
2371.7
78.07
77.46
S: 1epllabV3-R18
13.6
572.0
74.21
73.45
S: PSPNet-R18
12.9
507.4
72.55
72.29
CWD [38]
13.6
572.0
75.55
74.07
CWD [38]
12.9
507.4
74.36
73.57
CIRKD [46]
13.6
572.0
76.38
75.05
CIRKD [46]
12.9
507.4
74.73
74.05
MasKD [18]
13.6
572.0
77.00
75.59
MasKD [18]
12.9
507.4
75.34
74.61
DiffKD
13.6
572.0
77.78
76.24
DiffKD
12.9
507.4
75.83
75.61
S: DespLabV3-R18
13.6
572.0
65.17
65.47
DepLabV3-MBV2
3.2
128.9
73.12
72.36
CWD [38]
13.6
572.0
67.74
67.35
CWD [38]
3.2
128.9
74.66
73.25
CIRKD [46]
13.6
572.0
68.18
68.22
CIRKD [46]
128.9
75.42
74.03
MasKD [18]
13.6
572.0
73.95
73.74
MasKD [18]
3.2
128.9
75,26
74.23
DiffKD
13.6
572.0
74.45
74.52
DiffKD
3.2
128.9
75.71
74.96
more enhancements with our DiffKD, especially when with a Retina-XI01 teacher, DiffKD gains substantial improvement of 4.0 AP over the Retina-R50, Additionally, methods outperforms existing KD methods with a large margin, and significantly surpasses FGD [48] when distilling on RetinaNet and RepPoints, by 1.0 and 0.6 AP, respectively. Moreover, comparing Table 4 with
5, we can infer that when the teacher is stronger, the benefit DiffKD is more significant, as the
discrepancy between the student and stronger teacher is larger.
4.3 Semantic segmentation
Settings. Following CIRKD [46], we use DeepLabV3 [5] framework with ResNet-101 (R101) [11] backbone as the teacher network. While for the students, we use various frameworks (DeepLabV3 and PSPNet [54]) and backbones (ResNet-18 [11] and MobileNetV2 [37]) to valid our efficacy.
We conduct feature distillation on the predicted segmentation maps, and train the student with our DiffKD loss and task loss, as formulated in Eq. (9). Note that do
linear autoencoder in DiffKD since the number of channels in segmentation map is only 19. Detailed training strategies are summarized in Appendix B.
Results. The experimental results are summarized Tab, 6. Our DiffKD significantly outperforms the state-of-the-art MasKD all settings. For example, on DeepLabV3-R18 student, DiffKD improves MasKD by 0.78% on val set and 0.65 on test set.
4.4 Ablation study
Effects of adopting DiffKD on different types of features. On image classification, we distill both intermediate feature (the output feature of back- bone before average pooling) and output logits with DiffKD. Here we conduct experiments to compare the effects distillations on different features of MobileNetV1 student and ResNet-50 teacher in Tab. 7. can see that, by adopting DifIKD to align the student features, both feature-level Dif- fKD and logits-level DiffKD can obtain obvious improvements without changing the loss functions. Meanwhile, combining feature and logits distilla- tions together in final method achieves the optimal 73.62% top-1 accuracy.
Table 7: Comparisons of different distilla- tion features in DiffKD.
Method
Top-1 (%)
w/o KD
70.13
MSE
72.39
KL div.
70.68
DifIKD (feature)
73.16
DifiKD (logits)
72.89
DiffKD (feature logits)
73.62
Effects of linear autoencoder. We compare different dimensions linear autoencoder in Tab. 8. We that, the linear autoencoder significantly reduces the FLOPs of diffusion model, but if we

---

Table 8: Comparisons of different dimensions of linear autoencoder in DiffKD. We report the top-1 accuracy and FLOPs of diffusion models with MobileNetVI student and ResNet-50 teacher on ImageNet.
w/o AE
256
512
1024
2048
Top-1 1op-1cp (%)
73.47
72.84
73.22
73.53
73.62
73.58
FLOPs (G)
1.41
0.04
0.09
0.21
0.58
1.82
conduct high compression ratios like 128 and 256 dimensions, the distillation performance will be severely weakened. Interestingly, AE with 512, 1024, and 2048 dimensions can outperform the one without AE, a possible reason that the AE encodes common and valuable information in the original feature for reconstruction and would have better feature representations than the original feature. As result, we use AE with 1024 dimension in our experiments for better performance-efficiengg trade-off.
Effects of adaptive noise matching. We propose adaptive noise matching (ANM) to match the noisy level of student feature to the correct initial level in the denoising process. Here we conduct experiments validate its efficacy. We train MobileNetVI student with the only removal of ANM in our final method, and the DiffKD without ANM obtains 73.34% top-1 accuracy, which has decrease of 0.28 on our DiffKD with ANM. This indicates that ANM can improve the performance by generating better denoised student feature.
Ablation numbers of score function evalua- tions (NFEs). In diffusion models, the number of score function evaluations (NFEs) is important factor for controlling the generation quality and ef- ficiency. The early method such DDPM [15] requires to run a complete timesteps in the re- verse denoising process as training, which leads to heavy computational budget. Recently, some
73.7
73:82
73.65
73.6
ACC (%)
73.5
works [27, 40] have been proposed to accelerate the denoising process by sampling small number of timesteps. this paper, we use DDIM [40] for
73.44
73.4
73.36
73.3
speedup. Here, we conduct experiments to show the influence of different NFEs. As shown Fig. 4, compared to the 72.39% accuracy of MSE bascline without denoising on student feature, only one-step denoising also achieves significant improvement. However, its performance is weaker than that of those larger NFEs due to the limitation of denoising quality. A NFEs would suffice in our KD setting to achieve promising performance, and we use it in all experiments for better efficiency-accuran trade-off.
1
3
5
10
20
NFEs
Figure 4: Results of DiffKD with different NFEs. We use MobileNetV1 student and ResNet-50 teacher on ImageNet.
5 Conclusion
In this paper, we investigate the discrepancy between teacher and student in knowledge distillation. To reduce the discrepancy and improve the distillation performance, we proceed from new perspective and propose explicitly eliminate the noises in student feature with a diffusion model. Based on this idea, we further introduce a light-weight diffusion model with a linear autoencoder to reduce the computation cost of our method, and an adaptive noise matching module align the student feature with the correct noisy level, thus improving the denoising performance. Extensive experiments on image classification, object detection, and semantic segmentation tasks validate efficacy and generalization.
Acknowledgements
This work was supported in part by the Australian Research Council under Projects DP210101859 and FT230100549.

---

References
[1] S. Ahn, S. X. Hu, A. Damianou, N. D. Lawrence, and Z. Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163-9171, 2019. 15
[2] Z. Cai and N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6154-6162, 2018.
[3] G. Chen, W. Choi, X. Yu, T. Han, and M. Chandraker. Learning efficient object detection models with knowledge distillation. Advances in neural information processing systems, 30, 2017. 13
[4] K. Chen, J. Wang, J. Pang, Y. Cao, Y. Xiong, X. Li, S. Sun, W. Feng, Z. Liu, J. Xu, et al. Mmdetection:
Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 09050090000,5, 2019, 14
[5] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801 118,, 2018.
[6] P. Chen, S. Liu, H. Zhao, and J. Jia. Distilling knowledge via knowledge review. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 5008-5017, 2021. 6
[7] E. D. Cubuk, B. Zoph, J. Shlens, and Q V. Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020. 5
[8] C. Du, Y. Li, Z. Qiu, and C. Xu. Stable diffusion is untable. arXiv preprint arXiv: 230.02330. 2023,
[9] Z. Du, R. Zhang, M. Chang, X. Zhang, S. Liu, I. Chen, and Y. Chen. Distilling object detectors with feature richness. In 35th Conference on Neural Information Processing Systems, 2021. 7
[10] J. Guo, K. Han, Y. Wang, H. Wu, X. Chen, C. Xu, and C. Xu. Distilling object detectors via decoupled features. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages
2154-2164, 2021. 7, 13
[11] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016. 4, 6,7,8
[12] T. He, C. Shen, 7. Tian, D. Gong, C. Sun, and Y. Yan. Knowledge adaptation for efficient semantic segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 578-587, 2019. 13
[13] B. Hco, J. Kim, S. Yun, H. Park, N. Kwak, and J. Y. Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1921-1930, 2019. 7
[14] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: Nki.:1503.023// 2015. 1, 6, 15, 17
[15] Ho, A. Jain, and P. Abheel. Denoising diffusion prohabilistic models. Advances in Neural Information Processing Systems, 13::840-685// 2020. 2,
[16] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint
arXiv: 2017. 6
[17] T. Huang. S. You, Wang, C. Qian, and C. Xu. Knowledge distillation from a stronger teacher. A. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022. 1, 4.4, 6, 13, 14, 15, 17
[18] T. Huang, Y. Zhang. S. You, F. Wang. C. Qian, J. Cao, and C. Xu. Masked distillation with receptive tokens. In The Eleventh International Conference on Learning Representations, 2023. 1, 7, 8, 13, 14
[19] S. Kundu, Sun. Y. Fu, M. Pedram, and P. Beerel. Analyzing the confidentiality of undistillable teachers in knowledge distillation. Advances Neural Information Processing Systems, 34:9181-991 19919999222222 2021, 4
[20] Q. Li, S. Jin, and J. Yan. Mimicking very efficient network for object detection. In Proceedings of the leee conference on computer vision and pattern recognition, pages 6356-6364, 2017. 13
[21] X.-C. Li, W.S.. Fan, S. Song, Y. Li, S. Yunfeng, D.-C. 7han, et al. Asymmetric temperature scaling makes larger networks teach well again. Advances in Neural Information Processing Systems, 35:3830-3842, 2022. 4
[22] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongic. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2117-2125, 2017. 13
[23] T.-Y. Lin. P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980 2988, 2017. 4,7
10

---

[24] T.-Y. Lin. M. Maire, S. Belongie. J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects context. In European conference on computer vision, pages 740-755. Springer, 2014. 4
[25] Y. Liu, K. Chen, C. Liu, Z. Qin, Z. Luo, and J. Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2604-2613, 2019. 13
[26] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 2021. 6
[27] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, DPM-solver: A fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[28] S. Marcel and Y. Rodriguez. Torchvision the machine-vision package of torch. In Proceedings of the 18th ACM international conference on Multimedia, pages 1485-1488, 2010.
[29] S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, and H. Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 5191-5198, 2020. 1, 13
[30] D. Y. Park, M.-H. Cha, D. Kim, B. Han, et al. Learning student-friendly teacher networks for knowledge distillation. Advances in Neural Information Processing Systems, 34:13292-13303 2021. 1, 13
[31] W. Park. D. Kim, Y. Lu, and M. Cho. Relational knowledge distillation. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3967-3976, 2019. 6, 15
[32] N. Passalis, M. Tzelepi, and A. Tefas. Probabilistic knowledge transfer for lightweight deep representation learning. IEEE Transactions on Neural Networks and Learning Systems, 32(5):2030 2020. 15
[33] W. Peebles and S. Xie. Scalable diffusion models with transformers. arXiv preprint arX9:2221..09748 2022. 2
[34] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-can: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[35] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022. 2, 4
[36] A. Romero, N. Ballas. S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio. Fitnets: Hints for thin deep nets. arXiv preprint 141.::11..6655. 2014. 7, 15
[37] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenccks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018. 8
[38] C. Shu, Y. Liu, J. Gao, Z. Yan, and C. Shen. Channel-wise knowledge distillation for dense prediction. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 531 5311-5320, 2021. 8, 13
[39] W. Son, J. Na, J. Choi, and W. Hwang. Densely guided knowledge distillation using multiple teacher assistants. In Proceedings of the IEFF//VV International Conference on Computer Vision, pages 9395- 9404, 2021. 13
[40] J. Song, C. Meng. and S. Ermon. Denoising diffusion implicit models. In International Conference on Learning Reprexentations, 2021. 2, 3, 14
[41] Y. Tian, D. Krishnan, and P. Isola. Contrastive representation distillation. In International Conference on Learning Representations, 2019. 15, 16
[42] Z. Tian, C. Shen, H. Chen, and T. He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the JEEF/CVF international conference on computer vision, pages 9627-9636, 2019. 7
[43] Y. Wang, W. Zhou, T. Jiang, X. Bai, and Y. Xu. Intra-class feature variation distillation for semantic segmentation. In European Conference Computer Vision, pages 346-362. Springer. 2020. 13
[44] R. Wightman, H. Touvron, and H. Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint 2rXiv:21 2021. 6
[45] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017. 7
[46] C. Yang. H. Zhou, Z. An, X. Jiang, Y. Xu, and Q. Zhang. Cross-image relational knowledge distillation for semantic segmentation. arXiv preprint arXiv:2204.06996 2022. 8, 13, 14
11

---

[47] J. Yang, B. Martinez, A. Bulat, and G. Tzimiropoulos. Knowledge distillation via softmax regression
representation learning. In International Conference Learning Reprexentations, 2020.
[48] Z. Yang. Z. Li, X. Jiang. Y. Gong, Z. Yuan, D. Zhao, and C. Yuan. Focal and global knowledge distillation for detectors. arXiv preprint arXiv: 11.1 1887, 2021. 7, 8, 13, 17
[49] Z. Yang. Liu, Hu, L. Wang, and S. Lin. Reppoints: Point set representation object detection. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 9657-9666, 2019. 7
[50] L. Zhang and K. Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. International Conference Learning Representations, 2021.
[51] Y. Zhang, W. Chen, Y. Lu, T. Huang, X. Sun, and J. Cao, Avatar knowledge distillation: Self-ensemble teacher paradigm with uncertainty. arXiv preprint vrXiv:2305.02722 2023. 13
[52] Y. Zhang, T. Huang, Liu, Jiang, K. Cheng, and S. Zhang. Freekd: Knowledge distillation via semantic frequency prompt, 2023. 13
[53] Zhao, Q Cui, R. Song, Y. Qiu, and Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 1953-11962, 2022. 2,6
[54] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In Proceedings of the IEFF conference computer vision and pattern recognition, pages 2881-28999 2017. 8
12

---

A Related Work
A.1 Representation gap in knowledge distillation
An emerging topic in knowledge distillation is the representation gap between teacher and student. Recently, the models have been designed larger and more complicated, with remarkable performance improvements compared to the traditional models. As result, an intuitive idea to improve efficient models is to distill them from stronger teacher model. However, recent studies [29, 39] find that KD performance with a stronger teacher is poor and even worse than the normal teacher. TAKD [29] states that the student can only learn the knowledge effectively from teacher model up fixed capacity, and proposes training multiple teaching assistants that have moderate capacities compared to the narrow student and a huge teacher. The teach assistants are trained sequentially with their former larger teach assistants, and the final smallest teach assistant is used to train the student. DAKD [39] further improves TAKD by densely connecting all models (teacher, assistant teachers, and student) together and letting the student choose the optimal teacher for each sample. SFTN [30] proposes to learn the teacher model with the supervision of student model, which obtains a student-friendly teacher with smaller representation gap to the student. However, such methods suffer from complex distillation algorithms and heavy computational cost for training the teacher model, therefore, they are not applicable in practice. More recently, DIST [17] proposes an efficient and simple approach, which aims at relaxing the exact matching in previous KL divergence loss with correlation- based loss, which performs better when the discrepancy between teacher and student is large. However, such heuristic loss in DIST only adapts to classification outputs, and thus for dense prediction tasks such as object detection, which requires detailed semantic information in intermediate features, the efficacy limited. Moreover, the Pearson correlation in DIST only has shift and scale invariances, whereas for the complex situation of dicrepancy in features, it cannot ignore all noises for a clean distillation.
As a result, we present DiffKD, which adapts well to multiple types of features and tasks, and handles better in eliminating the discrepancy in distillation features.
A.2 KD in dense prediction tasks
Different from the image classification task that only needs to recognize the overall classification of the image, object detection and semantic segmentation are referred as dense prediction tasks that have to predict the bounding boxes and classes for all the objects inside the image, or the pixel-level segments of image. As a result, effectively distilling the knowledge of the teacher in these tasks is more challenging than classification.
Various methods [3, 10, 20, 38, 48, 51] are proposed to improve KD performance in object detection. Chen et al. [3] first proposes performing KD on the classification logits and regressions of the Rol head. Mimicking [20] states that the feature maps in detection model contains richer semantic information than the responses, and proposes to distill the FPN [22] features of teacher. However, distilling from the features suffers from severe imbalance of the foreground and background pixels in object detection. To address this issue, recent methods focus on selecting valuable features and propose various loss functions based on this [10, 18, 38, 48, 52].
In semantic segmentation, knowledge distillation techniques typically prioritize maintaining the structural semantic connections between the teacher and student models. To address the inconsistency between teacher and student features, He et al. [12] employ a pretrained autoencoder to optimize feature similarity in transferred latent space. They also transfer non-local pairwise affinity maps to minimize feature discrepancies. SKD [25] conducts pairwise distillation among pixels to preserve pixel relationships and adversarial distillation
score maps to distill holistic knowledge. IFVD [43] transfers intra-class feature variation from teacher to student to establish more robust relations with class-wise prototypes. CWD [38] proposes channel-wise distillation to better mimic spatial scores along the channel dimension. CIRKD [46] proposes intra-image and cross- pross-image relational distillations to learn better semantic relations from the teacher.
However, existing state-of-the-ee KD approaches in dense prediction tasks are designed specifically for the tasks, which are difficult to be applied to various tasks, resulting in large experiment cost in evaluating and adapting those methods. As a result, an interesting direction in KD to generalize and unify KD methods in different tasks.
13

---

B
Implementation Details
B.1
Diffusion model
In all experiments, we use DDIM [40] as the noise scheduler in reverse denoising process. The total range of timesteps for training 1000, and the initial time step for denoising is 500.
B.2 Image classification
On ImageNet classification task, we simply set \ 22 A3 in all experiments. (1) Feature distillation. For experiments that use ResNet-50 or ResNet-34 as the teacher model, we set the number of latent channels in our autoencoder to 1024; while for Swin-I. teacher, its feature for distillation has 1536 channels, so we compress to 768 with autoencoder. (2) Logits distillation. We also add a DiffKD loss on the predicted classification logits. Different from the Bottleneck block used feature distillation, we use MLP (two linear layers associated with an activation function) as alternate of Bottleneck since the logits have only two dimensions (no spatial dimensions). Besides, the autoencoder is not used logits distillation. DiffKD on the output feature before average pooling layer and classification logits, but remove the autoencoder since the computation cost on CIFAR model is small.
CIFAR- 100 dataset, we also implement
B.3
Object detection
For object detection task, we conduct feature distillation the predicted feature maps, and train the student with our DiffKD loss Lafffkd. regression KD loss, and task loss. Note that we do not use linear autoencoder in DiffKD since the number of channels in FPN is only 256. We set λ Àg 1. We adopt ImageNet pre-trained backbones during training following previous KD works [18]. All the models trained with the official strategies (SGD, weight decay of 1e-4) of 2x schedule in MMDetection [4]. We run all the models on 8 V100 GPUs
Loss weights: We set DifKKD loss weight to 5 and regression loss weight to on Faster RCNN students. For other detection frameworks, we simply adjust the loss weight of DiffKD to keep a similar amount of loss value Faster RCNN. Concretely, the loss weights of LDIRKD on RetinaNet, FCOS, and RepPoints are 5, 5, and 15, respectively.
B.4 Semantic segmentation
Following CIRKD [46] and MasKD [18], train the models with standard data augmentations including random flipping, random scaling in the range of [0.5, 2). and crop with size 512 x 1024. An SGD optimizer with momentum 0.9 is adopted, and the learning rate is annealed using polynomial scheduler with an initial value of 0.02. For DiffKD, we distill the knowledge of the segmentation probabilistic distributions,
DIST [17] the distance function d in KD loss. We do not involve autoencoder since the segmentation prediction has only 19 channels Cityscapes dataset, and set A >g
predictions of teacher. Since these predictions
C More Experiments
C.1 Results on CIFAR-100 dataset
We summarize the CIFAR-100 results in Tab. 9. Our DiffKD surpasses previous methods in most cases. Moreover, comparing homogeneous architecture settings and heterogeneous architecture settings, DiffKD gains more significant improvements on heterogeneous settings compared to the standalone training results, which indicates that our method can deal better with the discrepancy between teacher and student models.
C.2 Effect of efficient diffusion model
We train RetinaNet R50 student with R101 teacher with xx schedule in COCO, and compare our Efffeicnt DM with original UNet in DDPM using DiffKD. shown Tab. 10, with feature shape (256, 80, 124), the original UNet has much larger parameters and GFLOPs, and thus leads to 3x
14

---

Table 9: Evaluation results on CIFAR-100 dataset. The upper and lower models denote teacher and student, respectively.
Homogeneous architecture style
Heterogeneous architecture style
Method
WRN-40-2
ResNet-56 ResNet-20
ResNet-32x4
ResNet-50
ResNet-32x4
ResNet-32x4
WRN-40-1
ResNet-8x4
MobileNetV2
ShuffleNetVi ShuffleNetV2
Teacher
75.61
72.34
79.42
79.34
79.42
79.42
Student
71.98
69.06
72.50
64.6
70.5
71.82
FitNet [36]
72.24:10.24
69.2110.33
73.50=0.28
63.16:10.47
73.5910.15
73.54:10.22
VID [1]
73.30+00.13
70.38=0.14
73.09+0.21
67.57=0.28
73.38
73.40=0.17
RKD [31]
72.22++0.20
69.61+0.006
71.90+0.11
64.43++0:42
72.28+0.39
73.21+0.28
PKT [32]
73.45:10.19
70.34.10.04
73.6410.18
66.52.10.33
74.10.10.25
74.69.10.4
CRD [41]
74.14=0.22
71.16:0.17
75.51
69.11
75.11
75.65
KD [14]
73.54=0.20
70.66+0.24
73.33=0.25
67.35++0.22
74.07=0.19
74.45++0.27
DIST [17]
74.73+00.24 71.75+0.30
76.31+0.119
68.66++0.23
76.34:+0.18
77.35+0.25
DifKKD
74.09110.09 71.92.10.14 76.72.L0.15
69.21
76.57.10.13
77.52.10.1
Table 10: Comparisons of different dimensions of linear autoencoder in DiffKD. We report the top-1 accuracy and FLOPs of diffusion models with MobileNetVI student and ResNet-50 teacher on ImageNet.
w/o AE
128
256
512
1024
2048
Top-1 (%)
73.47
72.84
73.22
73.53
73.62
73.58
FLOPs (G)
1.41
0.04
0.09
0.21
0.58
1.82
training time. The original UNet only achieves similar performance our Efficient DM, as the generation of features easier than images, and a small model would suffice.
C.3 Effect of adaptive noise matching (ANM)
To validate the efficacy of ANM more thoroughly. conducted further experiments in more model settings and training strategies, as summarized Tab. 11. We can see that, when with stronger strategy and teacher, the improvement of ANM is more significant (1.2% improvement compared to 0.3% and 0.5% improvements on MobileNetVI and ResNet-18). One possible reason is that, when the augmentations and teacher become stronger, the noisy gaps between predicted features of teacher and student become more various, and therefore ANM is more effective in matching the noisy levels.
MobileNetV2
C.4 Statistics of the learned noise weight
To analyze the effectiveness of ANM, we first show the distribution of noise weight y in Fig. 5 (a) of the rebuttal PDF. Revealing that the student feature is noised with Z 7Z (stu) + -)r), the larger denotes smaller additional noise added.
We can that large amount of values is in the range of > 0.9, indicating that the student feature itself contains nonnegligible noises and only requires small noises to match the initial noise level, while there also exist some cleaner samples that require large noises.
also plot the curves of average 7 in each epoch during training. Fig. (b) indicates that, at beginning of training, the student feature contains more noises, only small weights of noises should be added. When the model gets converged, the noise in student feature becomes smaller and ? goes smaller to match the noise level accordingly.
15

---

Table 11: Effects of adaptive noise matching (ANM) on various model settings.
Student
Teacher
Strategy
w/ ANM
ANM
MobileNetV1
ResNet-50
B1
73.6
73.3
ResNet-118 18
ResNet-34
B1
72.2
71.7
MobileNetV2
ResNet-50-SB
B2
74.9
73.7
20
1.00
0.98
Percentie (S)
Average
0.94
10
0.92
5-
0.90
0.7
0.8
1.0
10
60
100
Epoch
(a)
(b)
Figure 5: Statistics of learned noise weight y on MobileNetVI student. (a) Histogram of learned ? during training. (b) curves of average y in each epoch during training.
C.5 Effects of different feature types of distillations on object detection
We conduct experiments on COCO dataset to compare the effects of performing distillations on FPN features, classification outputs, and regression outputs. As results shown in Tab. 12, compared with the original KD without feature denoise in DiffKD, our method obtains significant improvements on all the feature rypes. Besides, comparing these features, distillation on FPN feature obtains relative high performance, which demonstrates that the semantic information in intermediate features is more valuable than the responses. We also conduct experiment to combine all the feature distillations in DiffKD together, while it achieves similar performance of FPN feature, this infers that the distillation on FPN feature is sufficient to obtain a good performance.
Table 12: Comparison of original diffusion model (DM) and our efficient DM on COCO dataset.
DM
Params
GFLOPs
Training time
AP
AP
AP
UNet (DDPM)
13.62
650.77
1.80x8 GPU Days
39.1
58.0
41.9
Efficient DM (ours)
0.21
132.76
0.59x8 GPU Days
39.2
42.0
C.6 Efficiency analysis
In DiffKD, we use light-weight diffusion model to denoise the student feature. Here we analyze the computation efficiency with comparisons other methods. Compared the vanilla KD, DifIKD has additional computations to denoise the intermediate feature and output logits on ImageNet. Specifically, for ResNet-18 student and ResNet-34 teacher, the additional FLOPs 800M. However, compared to other feature-based KD methods, the cost is acceptable. For example, CRD [41] uses extra 260M FLOPs, ReviewKD has an addtional computation cost of 1900M.
D Discussion
D.I Limitation
In this paper, we only implement our diffusion model with simple convolutions and DDIM inference algorithm, while there exists recent advances of diffusion models that propose better transformer- based models and more efficient inference algorithm. Besides, we only use the traditional mean square error and KL divergence
KD loss functions, while many novel losses could be leveraged
16

---

to further improve the distillation performance. The computational cost is larger than the simple logits distillation methods such as KD [14] and DIST [17], but the cost is comparable to existing feature distillation methods and does not affect the computation cost in inference.
D.2 Societal impacts
Investigating the efficacy of the proposed method would consume considerable computing resources. These efforts can contribute to increased carbon emissions, which could raise environmental concerns. However, the proposed knowledge distillation method can improve the performance of light-weight compact models, where replacing the heavy models with light models in production could save more energy consumption, and it is necessary validate the efficacy of DiffKD adequately.
weight
E
Visualization of Predicted Classification Scores
We visualize the predicted classification scores in Fig. 6. The original predictions of student, have different sharpness compared to the teacher's s, while the denoised predictions align better to the predictions of teacher.
Student
1444
Studest
04
Teacher
Texcher
B3
rdde
Figure 6: Visualization of predicted scores of MobileNetVi student ImageNet validation set. DifIKD has an effect of matching the sharpness of the probabilistic distributions of teacher and student.
F
Visualization of Features
F.1
Visualization details
We visualize the features of student and teacher models in the first output (stride 4) of FPN. The models used to extract the feature are RetinaNet-RSD (student) and RetinaNet-X101 (teacher) trained on COCO dataset. Following FGD [48], we average the feature map along channel axis and perform softmax on the spatial axis measure the saliency of each pixel. Formally, with a given feature map HW X E x0xHW, we first average the channels and get X' E FWW, where X = nw
(X..).
Then generate the attention map for visualization
V = Н. W softmax(X*) X'/T),
(10)
where is the temperature factor for controlling the softness of distribution, and set 0.5.
F.2 More visualizations
We visualize the student features, denoised student features, and teacher features in Fig. First, by comparing the original student and teacher features, we can conclude that the discrepancy between student and teacher features is fairly large, and the student feature contains more noises and not as salient the teacher feature. While for the denoised feature generated by our DiffKD, it is very similar the teacher feature, this infers that distillation on the denoised student feature and teacher feature get rid of the noises that disturb the optimization.
17

---

Image
Student
Denoised student
Teacher
Figure 7: Visualizations of student features, denoised student features and teacher features COCO dataset.
18


Reviews:
[{"review_id": "tbCvybtsTa", "reviewer": "Reviewer_NwaZ", "summary": "This work proposes the use of the diffusion model for denoising the noisy feature of student. \nIt tackles the issues of employing the diffusion model for knowledge distillation which are heavy computation and inexact noisy level of student feature.\nTo tackle the issues, it proposes a light-weight diffusion model consisting of two bottleneck blocks in ResNet and adopt a linear auto encoder to compress the teacher feature. It also proposes an adaptive noise matching module which measures the noisy level of each student feature adaptively and specifies a corresponding Gaussian noise to the feature to match the correct noisy level in initialization.", "questions": "Please refer to the Weaknesses.", "limitations": "Please refer to the Weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed idea is simple.\n\nShowed what is the problem of directly applying the diffusion model to knowledge distillation. (Expensive Computation Cost, Inexact noisy level of student feature)\n\nIt has conducted various experiments on various vision tasks to show that the proposed method can be widely applicable. (classification, object detection, semantic segmentation).\n\nIt shows good performance gain on several benchmarks.", "weaknesses": "Applying diffusion model to denoise the student feature seems somewhat obvious approach, in other words, not novel.\nThe fact that the feature of student is noisier than that of the teacher is already well studied by other papers as mentioned in line 109-117.\nIn my opinion, the contribution of this paper comes from showing what are the practical problems of applying diffusion process to KD.\nIt showed two problems (Expensive Computation Cost, Inexact noisy level of student feature).\nHowever, I am not sure if this paper has done enough experiments to show that it solved the above problems. \nThere are not enough analysis to prove the proposed method is efficient. (Table 8 and Figure 4 seems to be the only ones).\n\nHow is the diffusion process applied on the logit-level?\n\nThe effect of ANM is not really convincing, there is only a marginal performance gain. It would be more convincing how the ANM is helping to denoise the student feature better by analyzing the student feature with and without the noise adapter. Visualization of student feature with/without ANM could help to better understand the effect of ANM.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts in reviewing our paper. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> 1. Applying diffusion model to denoise the student feature is not novel.\n\nWe summarize our novelties as follows. (1) The existing methods solve the discrepancy between teacher and student by improving the loss functions or training strategies, while we are the first work that uses a new perspective of distilling valuable knowledge from the teacher by eliminating the noise in the student feature. (2) There is no practice on how to involve diffusion model (DM) in KD before. We assume the student feature is a noisy version of teacher feature so that the DM is naturally involved with a teacher-training and student-denoising manner. DM enjoys better generation performance than the other transformation modules in previous methods. (3) The original diffusion model follows a heavy UNet architecture which is computationally-expensive and heavily slows down the training speed of KD, we introduce efficient DM and autoencoder that effectively reduce the computation cost. (4) We design an adaptive noise matching module to solve the issue of unknown noise levels in student features.\n\n---\n\n> 2. Experiments to show the efficacies of efficient model and ANM.\n\n**(1) Efficacy of efficient diffusion model.**\n\nWe train RetinaNet R50 student with R101 teacher with 1x schedule in COCO, and compare our Effieicnt DM with original UNet in DDPM using DiffKD. As shown in the following table, with feature shape (256, 80, 124), the original UNet has much larger parameters and GFLOPs, and thus leads to ~3x training time. The original UNet only achieves similar performance as our Efficient DM, as the generation of features is easier than images, and a small model would suffice.\n\n|DM|Params|GFLOPs|Training time|AP|AP50|AP75|\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n|UNet (DDPM)|13.62 M|650.77|1.80*8 GPU Days|39.1|58.0|41.9|\n|Efficient DM (Ours)|0.21 M|132.76|0.59*8 GPU Days|39.2|58.1|42.0|\n\n**(2) Efficacy of ANM.**\n\nActually, for the simple training strategy B1, the 0.28 performance gain of ANM on MobileNetV1 is not marginal since the 73.6% accuracy is high w.r.t to its capacity and it is challenging to obtain significant increment on the simple 100-epoch training strategy. To validate the efficacy of ANM more sufficiently, we further conducted experiments on more model settings and training strategies, as summarized in the following table.\n\n|Student|Teacher|Strategy|w/ ANM|w/o ANM|\n|:--:|:--:|:--:|:--:|:--:|\n|MobileNetV1|ResNet-50|B1|73.6|73.3|\n|ResNet-18|ResNet-34|B1|72.2|71.7|\n|MobileNetV2|ResNet-50-SB|B2|74.9|73.7|\n\nWe can see that, when with stronger strategy and teacher, the improvement of ANM is more significant (1.2% improvement on MobileNetV2 compared to 0.3% and 0.5% improvements on MobileNetV1 and ResNet-18). One possible reason is that, when the augmentations and teacher become stronger, the noisy gaps between predicted features of teacher and student become more various, and therefore ANM is more effective in matching the noisy levels.\n\n---\n\n> 3. How is the diffusion process applied on the logit-level?\n\nWe have elaborated the implementation details and comparison of DiffKD on feature level and logits level in lines 481~488 of our supplementary material. The predicted logits can also be regarded as a feature that has only one dimension compared to the intermediate feature of 3 dimensions (height, width, and channels), and we can also use diffusion model (DM) to denoise the student logits feature. To achieve this, we replace the convolutional networks of our DM with MLP network, and the goal of DM is to predict the 1-D noise of the logits.\n\n---\n\n> 4. The performance gain of ANM is marginal.\n\nSee response 2 (2).\n\n---\n\n> 5. Analyzing the student feature with or without ANM.\n\n**(1) Statistics of learned noise weight $\\gamma$.**\n\nTo analyze the effectiveness of ANM, we first show the distribution of noise weight $\\gamma$ in Fig. 1 (a) of the rebuttal PDF. Revealing that the student feature is noised with $\\boldsymbol{Z}^{(stu)}_T = \\gamma\\boldsymbol{Z}^{(stu)} + (1 - \\gamma)\\boldsymbol{\\epsilon}_T,$ the larger $\\gamma$ denotes smaller additional noise added.\n\nWe can see that a large amount of values is in the range of $\\gamma > 0.9$, indicating that the student feature itself contains non-negligible noises and only requires small noises to match the initial noise level, while there also exist some cleaner samples that require large noises.\n\nWe also plot the curves of average $\\gamma$ in each epoch during training. The Fig. 1 (b) indicates that, at the beginning of training, the student feature contains more noises, so only small weights of noises should be added. When the model gets converged, the noise in student feature becomes smaller and $\\gamma$ goes smaller to match the noise level accordingly.\n\n**(2) Comparison of features with or without ANM.**\n\nTo validate how much ANM can improve the denoised student features, we measure the mean distance between teacher (ResNet-50) features and denoised student (MobileNetV1) features with or without ANM on ImageNet validation set, as summarized in the following table.\n\n|Type|Metric|Distance w/ ANM|Distance w/o ANM|\n|:--:|:--:|:--:|:--:|\n|Intermediate feature|MSE|2.87|3.29|\n|Logits|KL div.|0.21|0.35|\n\nWe can infer that ANM effectively reduces the discrepancy between teacher and student features, and therefore leads to better distillation performance."}, {"review_id": "fV19cVivTT", "reviewer": "Reviewer_tFex", "summary": "The authors propose DiffKD, a knowledge distillation technique based on the hypothesis that the student's feature is a noisy version of the teacher's feature. Based on this assumption, they use a diffusion model to iteratively denoise the student's features before matching with the teacher. In addition, they propose to use a linear autoencoder to make diffusion more efficient to compute, and they propose a noise adapter module to predict the student's noise level.", "questions": "1. Could you please share some statistics about the predicted noise level from the noise adapter?", "limitations": "Limitations are discussed in the Appendix.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Treating the student's feature as a noisy version of the teacher's feature is an interesting idea.\n2. The authors conducted experiments on three tasks and showed improvements for DiffKD.", "weaknesses": "1. The authors did not show strong evidence for the claim that the student's features are the noisy version of the teacher's features. While Figure 2 provides a visualization, it would be best if this can be analyzed quantitatively, as it's the central hypothesis that DiffKD is based on.\n2. Figure 4 shows that denoising may not be very important, since the performance is not much better even if more denoising steps are taken. This suggests that it's possible that the noise in the student is very weak, which is further supported by the experiments showing very little improvement over existing methods (less than 1 point), despite the heavy machinery of the diffusion model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts in reviewing our paper and positive evaluation. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> 1. Evidence for the claim that the student feature is the noisy version of the teacher feature.\n\nIn knowledge distillation, given the student's objective of imitating the teacher's outputs, i.e., $\\mathcal{L}_\\mathrm{kd} := d(\\boldsymbol{F}^{(s)}, \\boldsymbol{F}^{(t)})$, we can infer that the student feature is an approximation of the teacher feature. However, in practice, the student's approximation cannot be identical to the teacher feature. This indicates that the equivalence $\\boldsymbol{F}^{(s)} - \\boldsymbol{F}^{(t)} = \\delta$ with $\\delta$ being an arbitrary non-zero noise always holds.\n\nFor empirical analysis, we take mean square error (MSE) and peak signal-to-noise ratio (PSNR) as the metrics to evaluate the noise ratio of original student features and denoised student features w.r.t. teacher features. As shown in the following table, the original student feature has a small PSNR value, while the denoised student feature has higher PSNR comared to the original feature, indicating that the original student contains noise, and our DiffKD can effectively reduce the noise ratio.\n\n|Feat|MSE|PSNR|\n|:--:|:--:|:--:|\n|Original student|4.62|24.32|\n|Denoised student|2.87|33.99|\n\n---\n\n> 2. Denoising may not be very important, since the performance is not much better even if more denoising steps are taken.\n\nWe find that just a few denoising steps (e.g., 5) can achieve good KD performance, since generating the cleaned student feature is not as difficult as generating an image in DM.\nHowever, this does not mean that the denoising is unimportant. We want to clarify that our performance gain does not solely come from the denoising process of diffusion model (DM), but also because of the training strategy of DM. The training target of DM is to predict the noise of different levels (zero to full gaussian noise) of noisy features, so the DM would be more robust and accurate to denoise and align the student feature to the teacher feature, compared to directly using student feature to predict the teacher feature. As shown in the following table, we compare DiffKD with simple transformation (we use the same architecture of DM to transform the student feature and compute distillation loss on transformed feature, instead of using the denoising process in DM). The results show that DiffKD significantly outperforms transformation even when they have the same FLOPs (NFEs=1).\n\n|DiffKD (NFEs=5)|DiffKD (NFEs=1)|Transformation|\n|:--:|:--:|:--:|\n|73.62|73.36|72.33|\n\n---\n\n> 3. Statistics about the predicted noise level from the noise adapter.\n\nWe first show the distribution of noise weight $\\gamma$ in Fig. 1 (a) of the rebuttal PDF. Revealing that the student feature is noised with $\\boldsymbol{Z}^{(stu)}_T = \\gamma\\boldsymbol{Z}^{(stu)} + (1 - \\gamma)\\boldsymbol{\\epsilon}_T,$ the larger $\\gamma$ denotes smaller additional noise added.\n\nWe can see that a large amount of values is in the range of $\\gamma > 0.9$, indicating that the student feature itself contains non-negligible noises and only requires small noises to match the initial noise level, while there also exist some cleaner samples that require large noises.\n\nWe also plot the curves of average $\\gamma$ in each epoch during training. The Fig. 1 (b) indicates that, at the beginning of training, the student feature contains more noises, so only small weights of noises should be added. When the model gets converged, the noise in student feature becomes smaller and $\\gamma$ goes smaller to match the noise level accordingly."}, {"review_id": "R1bO7PllJY", "reviewer": "Reviewer_pbtu", "summary": "This paper presents a novel knowledge distillation (KD) approach. The difference from the existing methods lies in the computation of discrepancy between the teacher and student signals. This paper formulates it using a diffusion model and uses a denoising procedure to reconstruct the teacher features from the student features. Experiments are performed on image classification, object detection, and semantic segmentation datasets.", "questions": "Please address the concerns above.", "limitations": "Overall, I think the idea of this paper is interesting. However, there are insufficient validations on either the performance (experiments are not strong enough) or the principle. I am looking forward to further results and/or explanations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The idea of measuring the discrepancy using diffusion models is novel.\n+ The paper is well-written.", "weaknesses": "- Although using diffusion model to model the discrepancy is a reasonable idea, this paper lacks sufficient analysis on the essential benefit of using diffusion models for this purpose. Does this paper mean that the diffusion procedure finds the shortest path in a distorted feature space (rather than the plain Euclidean space) which is better in measuring the discrepancy? If yes, are there any validations (metrics, visualizations, etc.) for the statement?\n\n- The proposed method mixes a large number of loss terms (Eqn 9). In the ablation part, the contribution of each term is not thoroughly ablated.\n\n- The results for image classification, object detection and semantic segmentation are mostly conducted on weak student models. It is questionable whether the method works well on strong student models (because improvement becomes more difficult).\n\n- The improvement beyond some competitive results is not strong enough (e.g. 0.1-0.3% gain on object detection, given that the student model is relatively weak).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts in reviewing our paper and the positive evaluation. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> 1. Does this paper mean that the diffusion procedure finds the shortest path in a distorted feature space (rather than the plain Euclidean space) which is better in measuring the discrepancy? If yes, are there any validations (metrics, visualizations, etc.) for the statement?\n\nYes, the denoising yields a distorted feature space that is better for optimizing the discrepancy. Compared to simple linear transformation or other generative models such as GAN, diffusion model (DM) offers desirable properties such as distribution coverage, a stationary training objective, and easy scalability.\n\nTo validate the statement, we measure the discrepancies of original student feature and denoised student feature to the teacher feature, respectively. The discrepancy metrics are mean square error (MSE) and peak signal-to-noise ratio (PSNR, higher is better). As shown in the following table, the denoised student feature has smaller discrepancy comared to the original feature, this indicates that the distillation with denoised feature has shorter path in minimizing the discrepancy.\n\n|Feat|MSE|PSNR|\n|:--:|:--:|:--:|\n|Original student|4.62|24.32|\n|Denoised student|2.87|33.99|\n\n---\n\n> 2. The contribution of each loss term.\n\nIn Eq. (9), our loss is composed of task loss, diffusion loss $L_\\mathrm{diff}$, autoencoder loss $L_\\mathrm{ae}$, and kd loss $L_\\mathrm{diffkd}$. We added experiments to validate the contribution of loss terms by removing each loss in the equation, as shown in the following table.\n\n|w/o KD|MSE baseline|Ours (Eq. (9))|Ours w/o $\\mathcal{L}_\\mathrm{diff}$|Ours w/o $\\mathcal{L}_\\mathrm{ae}$|\n|:--:|:--:|:--:|:--:|:--:|\n|70.13|72.39|73.62|72.37|72.68|\n\nWe can infer that, both the diffusion loss $L_\\mathrm{diff}$ and autoencoder loss $L_\\mathrm{ae}$ are important to performance.\n\n---\n\n> 3. Performance on strong student models.\n\nWe have conducted experiments for stronger student models (ResNet-50, Swin-T) on ImageNet classification in Table 3. We can see that DiffKD can improve the standalone training of ResNet-50 and Swin-T by 2.0% and 1.2%, respectively, which are significant.\n\n---\n\n> 4. The improvement beyond some competitive results is not strong enough (e.g., COCO dataset).\n\nOur method achieves consistent improvements on all the tasks, which is sufficient to demonstrate our superiority. Besides, our method uses simple distillation losses such as KL divergence and MSE, and our improvements over these losses are fairly significant. For COCO dataset, unlike the compared methods such as FGD, which are specifically-designed for object detection task and uses the ground-truth bounding boxes to assist the distillation, our method only uses features as input and MSE loss, yet achieves consistent improvements on all models with simple MSE loss."}, {"review_id": "UsGMtZtgxj", "reviewer": "Reviewer_hqhp", "summary": "The paper introduces a new knowledge distillation (KD) method named DiffKD, which aims to bridge the representations between teacher and student features via a diffusion model. The motivation is based on the finding that the student feature is noisier than the teacher feature, and therefore diffusion models can be leveraged to denoise the student feature. Additionally, an efficient diffusion model and a noise matching module are proposed for better efficiency and accuracy. The authors conduct KD on image classification, object detection, and semantic segmentation tasks to validate the method.", "questions": "- In Table 8, why is the performance of DiffKD without AE worse than DiffKD with AE of 512, 1024, 2048 channels?", "limitations": "Applicable.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of reducing the gap between teacher and student features is an emerging and important topic in KD. This paper discusses the discrepancy between teacher and student, then proposes using diffusion models to reduce the discrepancy, which is interesting and straightforward.\n\n- It is good to see that the method can generalize to various feature types and tasks. Unlike existing methods that often focus on specific tasks and design complex loss functions, this method can be used in various tasks and achieve advanced performance with simple loss functions.\n\n- The technical contribution is evident. Instead of directly adopting classical diffusion models, this paper introduces a lightweight architecture with an autoencoder to speed up the model and a noise-matching module to improve performance.\n\n- The improvements are significant in image classification, detection, and segmentation tasks.\n", "weaknesses": "- The method of denoising the student feature with a teacher-trained diffusion model is a bit strange. According to my understanding, the diffusion model is a generative model. Why don't the authors directly train a diffusion model to take the student model as input and generate a feature that is similar to the teacher for distillation?\n\n- In Table 3, compared to DIST, the improvement of DiffKD on Swin-T is not as strong as that on smaller models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts in reviewing our paper and the positive evaluation. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> 1. Train diffusion model with teacher feature vs. Student feature.\n\nTraining the diffusion model requires the features to be stable and consistent, while the student feature changes iteratively through the training, it is inappropriate to use student feature as the generative target. We added experiments to validate our intuition.\n\n|DiffKD|Student target|\n|:--:|:--:|\n|73.62|72.78|\n\nWe can see that, training diffusion models with student target causes a significant performance drop of 0.84%, which demonstrates our intuition.\n\n---\n\n> 2. Improvement on Swin-T is not as strong as that on smaller models.\n\nThe Swin-T is a strong model that can obtain high ImageNet accuracy with standalone training, so gaining improvement on it is more difficult compared to the small models with much lower accuracies. Besides, the discrepancy between Swin-T and Swin-L is also not as large as the settings with small students. Therefore, the denoising effect is also limited.\n\n---\n\n> 3. Why is the performance without AE worse than DiffKD with AE 512, 1024, 2048 channels?\n\nWe think the AE may have an effect of purifying the features, and it focuses more on the factors that help reconstruct the input features. Therefore, distillation on AE-encoded features could guide the student to learn more from valuable features."}, {"review_id": "iQDeH7nSnc", "reviewer": "Reviewer_78h4", "summary": "This paper proposes a novel method of knowledge distillation. It uses a diffusion model to denoise the student model features, reducing the gap between  the teacher and student model. An auto-encoder is also designed to reduce the computational efforts and an adaptive noise module to improve the denoising effect. The method is extensively evaluated and achieves SOTA results in image classification, object detection, and semantic segmentation.", "questions": "1. Is the training of the autoencoder and diffusion model done at the same time as the distillation training (according to Equation 9)? Does this cause inadequate training of teacher features and denoising effects at the beginning of training to interfere with student model learning?\n\n2. Why are the methods of comparison in Tables 2 and 3 different?", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper is innovative in applying a diffusion model to knowledge distillation to reduce the gap between the teacher and the student model.\nThe method proposed in this paper is highly applicable and can be used for different types of features. It works well for a variety of tasks such as image classification, target detection, and semantic segmentation.", "weaknesses": "Some of the hyperparameters are not described clearly. \nLine 177 says λ1=λ2=1, does this indicate that λ3 is set differently? line 220 says λ1=λ3=1.\n\nDistillation methods are usually very sensitive to different backbones/downstream tasks? More evaluations and clarifications of this can be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the efforts in reviewing our paper and the positive evaluation. Our responses according to the reviewer's comments are summarized as follows.\n\n---\n\n> 1. Hyperparameters of $\\lambda_1$, $\\lambda_2$, and $\\lambda_3$.\n\nThere is a typo in line 220, and $\\lambda_3$ should be $\\lambda_2$. We clarify our hyperparameters as follows. All our experiments use $\\lambda_1=\\lambda_2=1$. For image classification and semantic segmentation tasks, the $\\lambda_3$ is also set to 1; while for object detection task, we set $\\lambda_3=1$ on Faster RCNN students, while for other detection frameworks, $lambda_3$ is adjusted according to their loss values compared to Faster RCNN (details are summarized in lines 498~501 of the appendix).\n\n---\n\n> 2. Are distillation methods sensitive to different backbones or downstream tasks?\n\nMost recent advanced distillation methods are designed specifically for one task. For example, DKD [1] and DIST [2] are designed for distilling logits on classification task, while feature-level distillation on downstream tasks such as object detection is more effective (DIST obtains 40.4 mAP on Faster RCNN-R50, while the state-of-the-art FGD [3] has 42.0 mAP). Nevertheless, FGD is sophisticatedly-designed for object detection task, and it requires the ground-truth bounding boxes to compute the loss, which cannot be directly applied to other tasks. One recent work, MGD [4], proposes a feature-level distillation loss and conducted experiments on classification, detection, and segmentation tasks, but its performance on ImageNet classification is not as competitive as that on downstream tasks (e.g., MGD has 71.58% on ResNet-18, while DIST achieves 72.07% accuracy).\n\nTo this end, our work does not focus on customizing the loss function, but aims to design a general method for all the feature types and tasks and gain consistent improvements with the most common distillation losses.\n\n---\n\n> 3. Is the training of autoencoder and diffusion model done at the same time as distillation training? Does this cause inadequate training of teacher features and denoising effect at the beginning of training?\n\nYes, the autoencoder and diffusion model are trained simultaneously with distillation training, as we found that they converge very quickly and may have minor impact on training stability. We added experiments on ResNet-50 teacher and MobileNetV1 student to prove this. As the table shows below, we train DiffKD with pretrained and fixed AE, DM, and AE & DM, respectively, and all settings have similar accuracies.\n\n|Ours|Pretrained AE|Pretrained DM|Pretrained AE & DM|\n|:--:|:--:|:--:|:--:|\n|73.62|73.68|73.51|73.55|\n\n---\n\n> 4. Why are the methods of comparison in Table 2 and 3 different?\n\nThe settings in Table 2 are the most common benchmark for ImageNet distillation, we select the recent state-of-the-art methods and directly report the results in their papers for basic comparisons. While the settings in Table 3 are with stronger models and strategies proposed by DIST [2], we regretfully have not enough computation resources to re-implement all methods in Table 2 on these settings.\n\n---\n\n**References**\n\n[1] Zhao, Borui, et al. \"Decoupled knowledge distillation.\" Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2022.\n\n[2] Huang, Tao, et al. \"Knowledge distillation from a stronger teacher.\" Advances in Neural Information Processing Systems 35 (2022): 33716-33727.\n\n[3] Yang, Zhendong, et al. \"Focal and global knowledge distillation for detectors.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[4] Yang, Zhendong, et al. \"Masked generative distillation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022."}, {"review_id": "TeLt7RPanK", "reviewer": "Reviewer_AKTb", "summary": "This authors propose to explicitly eliminate the noises in student feature with a diffusion model to reduce the dicrepancy between student and teacher model for better knowledge distillation. Specifically, they build a lightweight diffusion model to reduce computation cost and introduce an adaptive noise matching module to align the student feature with the approximate noisy level of intermediate diffusion.", "questions": "See Section Weakness", "limitations": "The paper is incrementally novel .", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.The paper adopts the diffusion model to model the teacher feature and denoise the student feature for reducing the dicrepancy between student and teacher model. \n2.Quantitative experiments on multiple high-level tasks are provided to evaluate the effectiveness of the proposed method. \n3.The paper is well organized and presented.", "weaknesses": "1.For methods,\n(1)The technique contribution of this work is not significant.\n(2)It is intuitive to regard the student as a noisy version of the teacher, which lacks the theoretical analysis. And the NFEs of diffusion model also depend on the capacity of student models for different noise levels. \n(3)The design principle of adaptive noise matching is not explained clearly. For different student models, the final learned γ should be analyzed.\n(4)It is still confused about how to determine the initial timesteps for the student model to reverse diffusion. The reverse diffusion is started from pure Gaussian noise, how to perform the inference for the feature output from the noise adapter. Would the γ near 0？ \n2.For experiments,\n(1)For Table 2 and Table 3, it seems that the proposed method achieves marginal performance improvement for students trained with strong strategies. The reason should be further discussed and analyzed. Besides, it only obtains comparable performance on object detection task, which did not show obvious advantages than other methods. \n(2)More visualized results should be provided, such as the affinity matrix.\n(3)Whether other generative models, e.g., GANs, FLOW, can be adopted to model the dicrepancy between student and teacher model or not? It should be discussed and compared to demonstrate the effectiveness of diffusion model.\n(4)In Table 6, Params and FLOPs are not shown completely.\n(5)Since different model capacity determines the dicrepancy between student and teacher model, how about the difference of NFEs for different student models?\n(6)Some important details are missing, such as the total timesteps for training the diffusion model, the initial timesteps for the student model to reverse diffusion.\n3.Some writing typos, e.g., \nLine 13:  adpative -> adaptive \nLine 136:  We -> we", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> 1.1 The technical contribution of this work is not significant.\n\nWe respectfully believe our contribution is sufficient. Directly applying diffusion models to KD is difficult and not straight-forward, and we provide effective solution on it with the following adaptations. (1) We assume the student feature is a noisy version of teacher feature so that the DM is naturally involved with a teacher-training and student-denoising manner. (2) The original diffusion model is computationally-expensive, we introduce efficient DM and autoencoder that effectively reduce the cost. (3) We design an adaptive noise matching module to solve the issue of unknown noise levels in student features.\n\n> 1.2.1 Theoretical analysis of regarding student as a noisy version of teacher.\n\nIn KD, the student's objective is approximating the teacher outputs, i.e., $L_{kd} := d(F^{(s)}, F^{(t)})$. However, in practice, this approximation cannot be identical to the teacher feature ($L_{kd}=0$). This indicates that the equivalence $F^{(s)} - F^{(t)} = \\delta$, with $\\delta$ being an arbitrary non-zero tensor always holds. We regard the student as an additive noise model [1] where its output is equal to the addition of teacher output and noise $\\delta$.\n\nFor a shorter and more appropriate optimization of the approximation in KD, we want to discard the bias $\\delta$ with a function $f$ so that it's possible to achieve the optima $F^{(t)} = f(F^{(s)})$, where\n$f(F^{(s)}) = F^{(s)} - \\delta$.\n\nThe transformation function $f$ needs to eliminate the additive noise $\\delta$, which is similar to the task of image denoising. Therefore, we adopt diffusion models, a popular variant of generative models that can effectively generate clean images or features from the noisy ones.\n\n**References**\n\n[1] Hoyer, Patrik, et al. \"Nonlinear causal discovery with additive noise models.\" NIPS 2008.\n\n\n> 1.2.2 NFEs of diffusion model depend on the capacity of student models for different noisy levels.\n\nActually, in conventional diffusion models, all the numbers of NFEs are valid for generating images, but larger NFEs usually result in higher generation quality. Similarly, we can also use the same NFEs for different settings in DiffKD, and we empirically find that NFEs of 5 can obtain promising results for all the settings (see response 2.5).\n\n> 1.3 Analysis of the final learned $\\gamma$ of different student models.\n\nWe measured the average final learned $\\gamma$ of different student and teacher models, as shown in the Table 1 of rebuttal PDF.\n\nWe can see that, the $\\gamma$ tends to be larger when the accuracies or architectures of teacher and student are more varied, as the noise in student feature is large and it needs smaller additional noise.\n\n> 1.4 Design principle of ANM & how to determine the initial timesteps for the student model.\n\nThe student features would have different noise levels for different samples, which may not always match the fixed initial noise level. Therefore, we propose adaptive noise matching (ANM) to match the noise level to our defined level. The ANM is optimized with distillation loss, since the denoising effect becomes optimal when the noise level is exactly matched.\n\n> 2.1.1 Marginal performance improvements for students trained with strong strategies.\n\nThe students trained with strong strategies have very competitive accuracies, so it is difficult to obtain significant improvements over them. But we still achieve 0.3%, 0.5%, and 0.3% gains on ResNet-34, MobileNetV2, and ResNet-50 compared to previous SOTA DIST, which we believe is sufficient to show our superiority.\n\n> 2.1.2 DiffKD only obtains comparable performance on object detection task.\n\nDiffKD is a general KD method that applies to various tasks. Unlike the compared methods such as FGD, which are specifically-designed for object detection task and have to use the ground-truth bounding boxes to assist the distillation, our method only uses features as input and MSE loss, yet achieves consistent improvements on all models with simple MSE loss. We think this is sufficient to show our superiority.\n\n> 2.2 More visualizations.\n\nSee rebuttal PDF.\n\n> 2.3 Can other generative models be adopted to model the discrepancy?\n\nApplying diffusion model is more natural since we treat the student feature as a noisy teacher feature. Besides, we can also use other generative models to predict the denoised student feature, but they may face some issues. For example, GANs are known to suffer from a variety of issues such as non-convergence and training instability, and the autoencoder and flow-based models may have inferior performance. In contrast, DM offers desirable properties such as distribution coverage, a stationary training objective, and easy scalability, which is more friendly to our task.\n\n> 2.4 In Table 6, Params and FLOPs are not shown completely.\n\nSorry for the confusion. Actually, the values of Params and FLOPs inside the table apply to all compared methods since the models are the same. We will refine this table.\n\n> 2.5 Different NFEs for different student models?\n\nTo explore whether we should adopt different NFEs for different model settings or not, we added experiments to show the performance of various models under multiple NFEs, as summarized in Table 2 of rebuttal PDF. Note that due to the limitation of our computational resource, we only trained the EfficientNet-B0 model for 100 epochs with B1 strategy.\n\nWe can see that all the three settings have relatively high performance at 5 NFEs, and their preferences on NFEs are similar. However, we do acknowledge that different NFEs for different settings or even samples could achieve better performance-efficiency trade-off, and it is a valuable direction for future increments.\n\n> 2.6 Some important implementation details are missing.\n\nThe total range of timesteps for training is 1000, and the initial time step for denoising is 500. We will report more details and release our code in the publication."}]

Author rebuttal:
Dear Reviewers,


We thank all the reviewers for their valuable comments and efforts in reviewing our paper. 

We are delighted to see that Reviewer tFex, pbtu, hqhp, and 78h4 stated that our method is interesting and novel; Reviewer hqhp and 78h4 acknowledged that our method is widely applicable and has evident technical contribution.

We have also responded all the reviewers' concerns such as the effectivenesses of proposed modules, evidence of our statements, and significance of improvements with additional experiments, visualizations, and explanations.

The rebuttal PDF containing new figures and tables is attached in this comment for your reading.

Regards,

Authors

Meta-review: This work presents a method to denoise features in a student-teacher model using diffusion process.

The work is found novel with good results.

All reviewers recommended acceptance.
Decision: Accept (poster)