Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolution
Authors: Simiao Li
Yun Zhang
Wei Li
Hanting Chen
Wenjia Wang
Bingyi Jing
Shaohui Lin
Jie Hu
Venue: ICLR 2025 Spotlight (2025)
Keywords: Image Super-Resolution
Knowledge Distillation
Model Compression
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Knowledge distillation (KD) is a promising yet challenging model compression approach that transmits rich learning representations from robust but resource-demanding teacher models to efficient student models. Previous methods for image super-resolution (SR) are often tailored to specific teacher-student architectures, limiting their potential for improvement and hindering broader applications. This work presents a novel KD framework for SR models, the multi-granularity Mixture of Priors Knowledge Distillation (MiPKD), which can be universally applied to a wide range of architectures at both feature and block levels. The teacher’s knowledge is effectively integrated with the student's feature via the Feature Prior Mixer, and the reconstructed feature propagates dynamically in the training phase with the Block Prior Mixer. Extensive experiments illustrate the significance of the proposed MiPKD technique.

Published as a conference paper at ICLR 2025
KNOWLEDGE DISTILLATION WITH MULTI- GRANULARITY MIXTURE OF PRIORS FOR IMAGE
SUPER-RESOLUTION
Simiao Li'
Yun Zhang"
Huawei Noah's Ark Lab
The Hong Kong University of Science and Technology (Guangzhou)
Wei Li Hanting Chen Huawei Noah's Ark Lab
Wenjia Wang
The Hong Kong University of Science and Technology (Guangzhou)
Bingyi Jing Southern University Science and Technology
Shaohui Lin
East China Normal University
Jie Hu
Huawei Noah's Ark Lab
ABSTRACT
Knowledge distillation (KD) promising yet challenging model compression approach that transmits rich learning representations from robust but resource- demanding teacher models to efficient student models. Previous methods for image super-resolution (SR) are often tailored to specific teacher-student architectures, limiting their potential for improvement and hindering broader applications. This work presents novel KD framework for SR models, the quuti-granularity Mixture Priors Knowledge Distillation (MiPKD), which can be universally applied to wide range of architectures at both feature and block levels. The teacher's knowledge is effectively integrated with the student's feature via the Feature Prior Mixer, and the reconstructed feature propagates dynamically in the training phase with the Block Prior Mixer. Extensive experiments illustrate the significance of the proposed MiPKD technique.
INTRODUCTION
Super-resolution (SR) poses key challenge in computer vision (CV) DDng et al.
2015 Liang
al. 2021 Chen et al. 202I), reconstructing high-resolution (HR) images from their low-resolution
(LR) versions. In the past decade, the convolutional neural network (CNN) (Dong et al. 2014 Kim
al. 2016 Lim al. 2017) and the Transformer (Wang et al. [022c] Zamir al.) 2022
Qiao et al.
2024b a Tu et al. 2024; Wang et al. 2023) Zhang et al. (024) have demonstrated exceptional success SR. However, deploying these models directly on eesurce-constraine devices impractical
due to their substantial computational overhead (Zhang et al.) 2021b). Therefore, there is increasing attention on model compression techniques for super-resolution (SR) models to enhance their practical deployment.
Knowledge distillation, emerging as an effective model compression method, can significantly reduce computation overload, facilitating the student by transmitting prior knowledge from the competent but resource- inteessive teacher model to the compact student model (Zhang et al.) |021 [uo et al.)
2021 Hui et al. 2019 Lee et al. 2020 Liu et al.
2023) In contrast to alternative model compression strategies like pruning (Wang et al.] 2021a la bb. quantization
([i et al. 2020
Hong et al. 2022).
*Co-first author.
I Project leader.
1 "Correppnniigg Author. shaohuilin007e @gmail.ommmmmmmmmmmmmmmm hujic23 huawei.com

---

Published as a conference paper at ICLR 2025
MiPKD
MiPKD
MiPKD
FAKD
CSD
FAKD
Logits-KD>
Lo
Scratch
RKD
Serttch
Logits
Seratch
RKD
AT
FAKD
PitNet
Applicable
FAKD
26.29
26.19
26.49
26.99
26:01
36.11
26:31
25.5
25.8
(a) Depth compression
(b) Widlh compression
(c) Compoundod compresssion
Figure 1:
The PSNR values of student models on 10 test set under different compression settings. In the depth compression (a), there are barely KD methods outperforming vanilla logits-KD. For width compression (b), CSD performs well but only satisfies this setting. For compounded compression, almost all KD underperforms training without KD.
compact block design (Ahn et al., 2018; Song et al. 2021| Nie al. 2021; Wang et al. (022a), and neural architecture search (NAS) (Zoph & Le 2016; Wan et al. 2020 Ren et al. 2021). KD is widely recognized method that can be incorporated with these approaches to further enhance the compactness of the student model. KD for SR has also attracted wide attention recently and has
gained remarkable progress (Li et al. 2020 Lee et al.) 2020 Zhang et al., 2021a He et al. 2020 Wang et al. 2021b). These methods can be broadly classified into response- eessonse-based based KD and feature- based KD The former supervises the student model using the teacher model's output, while the latter focuses aligning the latent space representations between the teacher and student models. (Gou
ct al. 2021 Wang et al. 2021b; He et al. 2020).
Although previous KD methods show promising results in SR, several issues hinder their wide applications. First, existing KD techniques for SR are tailored to specific teacher-student architectures. They support network depth (Figure a)) or network width (Figure (b)) compression (He et al.) 2020). and deteriorate the student dramatically when they are adopted into another setting. For instance, FAKD (He al. 2020) boosts the student model in depth compression but deteriorates the student when applied to width compression circumstance. CSD (Wang et al., 2021b) improves the student model significantly (Figure but is not compatible with depth compression in Figure It's necessary to propose a more flexible KD framework that is closer to real-world application. While few methods have discussed compounded compression on both depth and width dimensions, which is much more general but challenging scenario. Existing KD methods for SR, including feature-based methods adapted from high-level CV tasks, such as RKD (Park et al.) 2019). AT (Zagoruyko & Komodakis 2016). and FitNet (Romero et al. 2014), offer limited benefit to the student model. Figure shows that the previous depth and channel distillation methods can just obtain a marginal performance gain
deteriorate the student in most cases. To alleviate these issues, we introduce a novel knowledge distillation technique for SR models, the multi multi-granuulaity Mixture of Priors knowledge for Knowledge Distillation student frameworks at feature and block levels. Specifically, the feature prior mixer dynamically combines prior knowledge from the teacher and student models' intermediate feature maps. Then its output-enhanced feature map is supervised by the teacher model's feature map. The block prior mixer adopts a coarser- grasseed prior mixture at the network block level that dynamically and stochastically switches the normal forward propagation path to the teacher or the student. The output SR image of this ensembled sub-network is supervised by the teacher's output. The primary contributions of this paper are outlined as follows:
that universally applicable various teacher-
We present MiPKD, a KD framework for efficient SR, transferring the teacher model's dark knowledge from both network width and depth levels. It's flexible and applicable to various teacher-student frameworks.
We propose the feature and block prior mixers to mitigate the impact of model capacity disparity on the effectiveness of KD to achieve better alignment. The former integrates the feature maps in a unified latent space, while the latter assembles a dynamic combination of network blocks from teacher and student models.
Extensive experiments on various benchmarks show that the proposed MiPKD framework signifi- cantly outperforms the previous arts.
2

---

Published as a conference paper at ICLR 2025
Touder Festar
Sidde
B
foato
F
p
Pirur Mixer
Form Plnce Misir
JL..
MIPDD Medale
MINKO Medade
X
Studked Netwvrkk
datillation
Compunze
Forward propagation pulls the -i-th distilatio, position
Dieillaia Puaition
(d) Overall framework MiPKD
(b) Feature Prior Mixer
Figure 2: Framework of the MiPKD method. MiPKD utilizes the multi granularity dark knowledge
mixture to constrain the KD process. At the k and (A: + i)-th distillation position, the Feature Prior Mixer stochastically integrates teacher-providee features with the student model, and the Block Prior Mixer employs coarser-grained prior fusion at the network block level.
RELATED WORK
Efficient SISR. To improve the model efficiency, there have been various approaches to make the SR
model less redundant, such neural architecture search (NAS) (Chu al.
2021: [Song et al. 22200]
pruning (Wang et al., 202 la b), low-bit quantization (Ma et al.
2019 9 Li et al.
2020
Hong et al.
2022), and compact net block design (Ahn al. 2018 (ong al. 2021 Nie al.
2021 Wang et
2022a e Zamir et al. 2022). The strength of NAS manifests in searching the optimal architecture
but time-consuming and computationally expensive due the massive search space. Afterwards, compact SR model designs have attracted rising attention and achieved remarkable progress (Zhang) al. 2022; Huu al., 20199 Ahn et al. 2018 Dong et al., 2016). ELAN,, proposed by [Zhang et al. (2022). incorporates the GMSA module that effectively exploits long-range image dependencies and achieves superior performance compared to transformer-
super-resolution models while being much less complex. Pruning (Wang et al. 202 202laj b) and quantization (Ma al. 2019 Li et al., Hong et al.) 2022) are other two types of methods to remove model redundancy by sparsity and low-bit quantization mappings. Despite the considerable progress made by these lightweight networks, significant computational resources are still in demand.
Feature-based Knowledge Distillation:
Knowledge distillation is widely recognized as an effective neural network compression technique that is able to significantly reduce the computation overload and improve student's capability by transferring "dark knowledge" as prior information from the large teacher model to the lightweight student model (Gou al., [222]] Yim al. 2017 Hinton al. 2015). Feature-based KD methods extend beyond simple output alignment by focusing on matching intermediate representations. FitNet Romero et al. (2015) aligns feature maps directly, while FAKD He et al. (2020) distills
correlation infommatonn from the affinity matrix. KD-SRRL Yang et al. (2021) 1) uses Softmax regression loss by feeding student features into the teacher's 's classification head. Qiu et al. (2022) introduces a dynamic knowledge mechanism that injects teacher features into the student with hyper- parameters. While MiPKD integrates the teacher's s dark knowledge and the student's representations at two levels: feature level and block level. This is achieved through latent space encoding and two stochastic mixing mechanisms, ensuring comprehensive alignment of the learned representations. PEFD Chen al. (2022) employs projector ensemble to extract task-relevant discriminative features and avoid overlitting the teacher's 's feature space. DMAE Bai et al, (2022) aligns intermediate features using a feature-based KD loss and reconstructs the original image via the student's 's decoder, similar
PEFD
3

---

Published as a conference paper at ICLR 2025
to FitNet but focused on the direct alignment of intermediate features. ViTKD Yang et al. (2024) combines the FitNet loss for shallow layers with a generative loss for deeper ones.
Knowledge Distillation for SISR. Recently, several attempts have also been made for image super- resolution knowledge distillation. Lee et al. (020) employ a trainable encoder-decoder network to perform information extraction, and use the statistics computed from the scale maps of the decoder to distill student models. He et al. (2020) proposed FAKD to distill the correlation information from the affinity matrix of feature maps. Wang al. Ib) proposed CSD that incorporates self-distillation and contrastive learning by introducing extra simply upsampled LR images as negative samples. MTKDSR (Yao al., 2022) employed two teacher models with different SR objectives (PSNR, perceptual) to guide the student model simultancously. CrossKD (Fang et al. (202)) divides the teacher and student networks into two segments that are interchanged and connected to perform forward propagation. RDEN (Ren et al. 2024) introduces an efficient SR network design by leveraging re- parameterization and progressive training strategy. Especially, the FitNet loss is used for distillation in the second training stage. Existing SRKD techniques for SR are tailored to specific teacher-student architectures, focusing on either network depth (Wang et al. 202 or channel compression (He et al. 2020), which is infeasible for practical compounded compression applications.
3
METHODOLOGG
3.1 PRELIMINARIES AND NOTATIONS
Given a low-resolution input image IRR, the deep SR model F()) aims to reconstruct the high- resolution image ISR F(ILR; Q) with fine details and consistent content with corresponding high-resolution image RH where Ô denotes the model parameters, The boset KD method compels the student model Fs to produce the same output as the teacher model
Liogits | Islll
(1)
where 1R FI( 05) and I5 FT O"') denote the SR images produced by the student and teacher networks. The feature-based KD methods seek mimic the rich implicit hidden representations between the teacher and the student, which also can be represented by the feature distillation loss
Leea = (TT((P)) Te(F
where F and F represent the feature maps of the student and teacher model at the k-thh distillation position, respectively, Tr and T, are the transformations applied on raw feature maps.
3.2 MIXTURE OF PRIOR KNOWLEDGE DISTILLATION
Inspired by MAE (He et al. 2022) that reconstructs the missing pixels from the masked input patches, proposed the dark inowledge synthesis approach for KD on SR tasks in both feature and block levels. The prior knowledge mixers are applied to the raw feature maps of the student and teacher models in order encode them into unified latent space, in which the models' dark knowledge is mixed. Subsequently, the mixed latent feature map is decoded to its original space, enabling the reconstruction of the enhanced feature map and the performance of distillation. While the purpose of the MAE is to reconstruct the masked pixels, the encoder-decoder in the feature prior mixer reconstructs the portion of the teacher model feature map that is replaced by the student's. This allows the student model's intermediate representations to have similar distribution to the teacher model's. The block prior mixer optimizes the network's 's capacity process and represent information. This is achieved utilizing a dynamic combination of blocks, whereby the resulting fusion information transferred from the feature prior mixer to the enhanced network. The two granularity of prior mixtures follow the common idea of prior mixing and propagation, which effectively mitigates the adverse effects caused by the capacity disparity between the teacher and student.
Feature Prior Mixer. Figure 2 illustrates the hybrid dark knowledge framework at the feature level. At the k-th feature distillation position, initially, the feature maps of both the student model FÊ and teacher model F are processed through their respective encoders to extract the latent representations 5 2 E xxx*xxw in a unified latent space, where C, H, W represent the dimension of the feature maps. Subsequently, the encoded student and teacher feature maps are fused in accordance with
4

---

Published as a conference paper at ICLR 2025
pair of randomly generated complementary masks. And the decoder reverts the fused feature map Z E to the enhanced feature map representation F in the same space as raw feature maps
F = Decoder(Z,e) = Decoder(Zī o (1 (1- (1)-1a) + Z O )),
(3)
where IM E {0, 1] c x represents a random three-dimensional mask and + denotes the element- wise product between matrices. The student's ' feature map is combined with the teacher's knowledge with the above mixing mechanism to reduce the discrepancy between them at the feature level. FX is utilized input the subsequent block level prior mixer module. The feature distillation loss f featy of Feature Prior Mixer is computed between F and F as
C | - - 1
Additionally, in order to enhance the reconstruction capability of the decoder and ensure the stability of training. the beginning of training, the auxiliary enhanced feature map F is obtained by directly passing the teacher's 's feature map to the teacher's encoder and decoder without applying the above masking and mixing strategy. The auxiliary "auto-encoder" loss L is computed
Cx 1 rf
(5)
It requires the encoder and decoder to serve as an auto-encoder structure, ensuring the decoded enhanced feature map is comparable with F The enhancement of the decoder contributes to the overall effectiveness of the feature prior mixer module.
Block Prior Mixer. Existing feature-based distillation methods on SR tasks mostly align the feature maps in the original representation space with Mcan Absolute Error or Mean Square Error (MSE). The semantic information among the teacher and student networks are differently distributed. Solely aligning features at the present distillation node with the same magnitudes of distance can lead the student model to learn entirely different information. To tackle this issue, we propose to align the networks' ability to process and represent information by assembling dynamic combination of blocks and transmitting the fusion information from the Feature Prior Mixer to the enhanced network.
To construct enhanced network (FMock) the distillation position k, according to the Block Prior Mixing Option Rş randomly sampled from f0, 1). the output of Feature Prior Mixer F. is forwarding propagated to the student network (Rx = 1) or teacher network (R, = 0), as the propagation path exemplified in Figure? The H HSu, and HT,, represent the block from student and teacher models after the current position respectively. How, represents the mixed block at the current position based Rg, which can be computed as
Hom R + Rx)HHT
Based on this process, denote the output of such concatenated network SR
Ex
1t = Fbloock Qo) Hom (FF))
(7)
The feature knowledge distillation loss based on Block Prior Mixer is derived through the combined network's final output with the teacher model's output:
c feats
(8)
In addition, C. = 0 if the k-th feature distillation position is dropped out. It is anticipated that there will be an attainment of interchangeability between the corresponding teacher and student network blocks, facilitating the student in learning and replicating the abilities of the teacher model.
The Whole Pipeline. Compared to conventional feature-based KD methods, MiPKD uses enhanced feature maps and networks to impose more constraints on the student model. In general, for each feature distillation position k. based on the pair of F! and F as the input of Feature Prior Mixer, the random masked feature maps are fused in a unified latent space. And the c.( featt is computed to align the enhanced feature map F with the initial teacher feature map in the same representation space. Subsequently, the randomly sampled R2 determines the propagation option of FV the networks' blocks randomly exchanged and the knowledge is transmitted from the teacher to student model, as shown in Figure Besides logits-KD loss Liogits: reconstruction loss Lrees the feature losses in block and feature levels are accumulated:
Liotal Atagiigits + Xreehree
Σ (Afeat Seatr + Abtock (eat
RSK
5

---

Published as a conference paper at ICLR 2025
Table 1: The specifications for the SR model under x4 experimental settings, including #Params, FLOPs and FPS are calculated using an input image with dimensions 256x256x3, And Frames per second is evaluated on NVIDIA V100 GPU.
Network
Model
Role
FLOPs (G)
#Params (M) FPS
Channel Block Group
Teacher
256
3293.35
43.09
3.2
EDSR
Student
64
32
207.28
2.70
33,958
Student 2
64
16
129.97 (25.3x)
1.52 (28.3x)
53.3
Teacher
64
20
10
1044.03
15.59
6.3
RCAN
Student
64
6
10
366.98
5.17
12.3
Teacher
180
6
861.27
11.90
0.459
SwinlR
Student
121.48
1.24
0.874
where Akd. Arecs X feats Abook represent the weights for logits-kd loss, reconstruction loss, feature prior mixer and block prior mixer respectively. The teacher's knowledge is effectively transferred through this multi-level distillation process.
EXPERIMENTAL RESULTS
4.1 EXPERIMENT SETUPS
Backbones and Evaluation. EDSR (Lim et al., 2017). RCAN (Zhang et al. 2018). and SwinIR (Liang et al., (02T) as backbone architectures are utilized to assess the significance of MiPKD and contrast it against previous KD) techniques on x2, x3, and x4 super-resolving scales. The SR network specifications and some statistics are presented in Table [ including the num- ber of channels, residual blocks and residual groups (RCAN), number of parameters (#Params), FLOPs(Floating Point Operations per Second), and inference speed (frame per second, FPS).
We compare MiPKD with the baselines: train from scratch, Logits KD (Hinton al. 2015) RKD (Park et al., 2019), AT (Zagoruyko & Komodakis 2016). FitNet (Romero et al. 2014). FAKD (He al. 2020). CrossKD (Fang et al. [223)) and CSD (Wang al. (2221b) Since the CSD is self-distillation method in the channel-splitting manner, it's not applicable to the RCAN experiments of network depth distillation. The results for x4 EDSR trained with CSD are obtained by lesting the provided checkpoint, and the x2 and x3 ones are reproduced by us since the checkpoints unavailable. The peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) are computed to evaluate the quality of the SR model's output. We utilize DIV2K (imoftte et al. 2017 dataset for training, and evaluate models on various standard test sets: Set14 (Zeyde al. 2012). Set5 (Bevilacqua et al. 2012), BSD100 (Martin et al. 22001) and Urban100 (Huang et al. 2015).
Training Details. The Adam optimizer (Kingma Ba 2014) is employed for training models, utilizing parameters 31 = 0.9, Ar = 0.99 and 6 = le 8 with 2.5e5 iterations. The learning and reduced by factor of 10 each 1e5 iteration. We set the loss weights Aka Afet and Abtock to 1, and 0. respectively. The proposed MiPKD is implemented by the BasicSR (Wang et al. 2022b) and PyTorch (Paszke et al., 2019) framework and train them using 4 NVIDIA V100 GPUs For both training and evaluation, the LR images produced by applying bicubic samn-sampling to the HR images. Augmentation through random cropping, flips, and rotations are applied during training
rate initialized at 1e
4.2 RESULTS AND COMPARISON
Comparison with Baseline Methods. Quantitative results for training EDSR (Lim al. (017).
RCAN (Zhang et al. 2018). and SwinIR (Liang et al., (021) of three SR scales are presented in Table Table and Table 1I from which we can draw the following conclusions:
(1) Existing KD methods for SR have limited effects, some may even deteriorate the student model. The KD methods originally designed for high-level CV tasks (RKD, AT, FitNet), though applicable,

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparison of distilling EDSR (Lim et al. 2017) the benchmark datasets. In
these experiments, the EDSR student model of c64b32 is distilled ed by the teacher model of c256b32.
SetS
Set14
BSDI00
Urban100
Scale
Method
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
Teacher
38.20/0.
34.02/0.9204
32.37/0. 9018
33.10/0.9363
Scratch
38.00/0.9605
33.57/0.9171
32.17/0.899
31.96/0.9268
KD
38.04/0.
33.58/0.9172
32.19/0. $2.19/0.8998
31,98/0.9288
RKD
38.03/0.9600
33.57/0.9173
32.18/0.8999
31.96/0.9277
X2
AT
37.96/0.9600
33.48/0.9167
32.12/0. 8990
31.71/0. 9241
FitNet
99599/09559
33.09/0.9136
31,79/0.8 8953
30,46/0.9111
FAKD
37.99/0 9666
33.60/0.9173
19/19/0.88
32.04/0.9277
CSD
38.06/0. 9607
33.65/0.9179
32.22/0. 9004
32.26/0.9300
MipKD
38.18/0.9611
33.82/0.9197
32.30/0.9011
32.56/0.9323
Teacher
34.76/0.929
30.66/0.8481
29.32/0.8 & 104
29.02/0.8683
Scratch
34.39/0.927
30.32/0.8417
29.08/0.800 8046
97/99/88448
KD
34.43/0.9273
30.34/0.847
29. 29.10/0.80 1050
28. 8491
RKD
34.43/0.9274
30.33/0.844 8423
29.09/88001
27. 8499
x3
AT
34.29/0.9262
30.26/0.8406
29.03/0.8033
27.76/0.8 8443
FitNet
33.35/0.9178
29.71/0.8323
28. 18.66200.744 7949
26,61/0.810
FAKI
34.39/0.9272
30.34/0.8457 8426
29. 10,10/0.800.
28.07/0.855 851
CSD
34.45/0.9275
30.32/0. 8430
29. 19.11/0.806
28.21/0.855 8549
MipKD
34.60/0.2288
30.50/0.8454
29.21/0.8079
28.52/0.8592
Teacher
32.65/0.9005
28. 28.95/0. 7903
27. 17..1/
26,87/0.888 8086
Scratch
32,29/0.8965
28. 28.68/0.7840
27.64 27.64/0.7380
26.21/0.788 7893
KD
32.30/0.8965
78.70/0.7842
27.64/0.7382
26.21/0.7897
RKD
32.30/0.8965
28.69/0.7842
27.64/0.7383
26.20/0.7899
32.22/0.8952
28.60/0.7825
27.59/0.7365
25.97/0.7825
FitNet
31.65/0.8873
28.33/0.7768
27.38/0.7309
25.40/0.7633
FAKD
32.27/0.880
78.65/0.7836
27.62/0.7379
26. 7895
CSD
32.34/0.8974
28. 18.72/0.7866
27.68/0.7390
26.34/0.7948
MipKD
32.45/0.8980
28.79/0.7865
27.71/0.7400
26.46/0.7968
hardly improve the SR models over training from scratch. For instance, AT and FitNet underperform the vanilla student models trained without KD among all settings.
(2) The presented MiPKD outperforms existing KD methods baselines for model compression. For example, MiPKD outperforms the vanilla student in the most challenging dataset Urban100 in x2, x3 and x4 settings by 0.6 dB, 0.53 dB, 0.25 dB in terms PSNR, respectively as Table shown. Compared with training from seratch, 0.35 dB, 0.30 dB, 0.30 dB in terms of PSNR improved, respectively, on Urban 100 dataset in RCAN x2, x3, and x4 settings as Table 3 shown.
(3) The MiPKD is applicable to the transformer network and able boost the model's performance. Conventional feature-base KD methods are not directly applicable to the Transformer-type networks, so we compare MiPKD with training from scratch and the response-based KD (Hinton al. 2015) in the experiments. The results in Table II indicate that the MiPKD could improve the transformer SR model by a large margin, further emphasizing its superior performance.
Visual Comparison. Figure 3 compares the output of x4 EDSR models from the Urban 100 dataset with various KD methods. details than all baseline works. FAKD are prone to artifacts in the left-bottom of the building and the vanilla student, Logits- KD, FAKD, and FitNet are over-blurred. In contrast, MiPKD alleviates blurring distortions and reconstructs much more fine-gra grained structural features. Similar observations are observed across other examples, e.g. the characters and anisotropic textures in img_077 The visual analysis aligns with the quantitative findings, highlighting the advantages of MiPKD. Further visual comparisons are supplied in the supplementary section.
instance, for img_ 047, MiPKD can reconstruct much better fine
Comparison of training costs: As shown in Table 4 MiPKD significantly outperforms Logits-KL by 0.12dB PNSR, while with an increase only 0.38s training time per step. It indicates that our MiPKD achieves the best trade-off between performance and training time.
7

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative comparison RCAN Zhang al., 2018) architecture the benchmark datasets. In these experiments, the RCAN student model of c64b6 is distilled by the teacher model of c64b20.
Set5
Set14
BSD100
Urban 100
Scale
Method
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
Teacher
38.27/0.9614
34.13/0.9216
32.41/0.9027
33.34/0.9384
Scratch
13/0.999661
33.78/0.9194
32.26/0.900)
32.63/0.9327
KD
18.197199611
33.83/0.9197
32.29/0.9010
32.67/0.9329
RKD
18.18/0.9612
33.78/0.9191
32.29/0
92.70/0.9333
x2
AT
38.13/0.9610
33.70/0.9187
32.25/0.908 9005
32.48/0.9313
FitNet
9602
33.57/0.9174
32.19/0.899 8999
32.06/0.9277
FAKD
18/199999
33.83/0.9199
32.29/0.901
32.65/0.9330
CrossKD
38. 18/09119661
33.82/0.9195
32.29/0.9011 9012
32.69/0.933 9331
MiPKD
38.26/0.9614
34.02/0.9210
32.35/0.9017
32.98/0.9357
Teacher
34.74/0.9299
10.65/0.8482
29.32/0.8111
29.09/0.8702
Scratch
34.61/0.9288
30,45/0.844 8444
29.18/0.807
28.59/0.8610
KD
34.61/0.9291
30,47/0.8447
29.2 99.8100.8080
28 62/0.8611
RKD
34.67/0.9292
30,48/0.8451
29.21/0.
89,00 8080
28 6610
AT
34.55/0.9287
30.43/0.8438
29.17/0. 8070
28.43/0.8577 8577
FilNet
34,21/0.9248
30.20/0.8399
29.05/0.
80.45 8044
27.89/0.844
FAKD
34.63/0.9290
30.51/0.8453
29,21/0.88
8079 8079
28 28.62/0.861
CrossKD
34.66/0.9291
30.50/0 8448
29,22/0 8082
28, 28.64/0.861
MiPKD
34.76/0.9299
30.61/0.8467
29.28/0.8090
28.89/0.8658
Teacher
32,63/0.9002
28.87/0.7888
27.77/0.7453
26.82/0.8087
Scratch
32.38/0.8971
28.69/0.
28.69/0.7842 7842
27.63/0.7379
26.36/0.7944 7947
KD
32.45/0.89% 8980
28.76/0.786
27. 27.67/0.7400
26.49/0.7982
RKD
32.39/0.8977 8974
28.74/0. 7856
27.67/0. .399
26.47/0.7998
x4
AT
32.31/0.8967
28.69/0.78..
27.64/0.7385
26.29/0.7927
FitNet
1..99/0.8899
28. 78.
27.55/0.7355
25 25.90/0.7791
FAKD
32.46/0.8980
78.77/0.7860
27.68/0.7400
26.50/0.7980
CrossKD
32.45/0.899 8984
28.81 78.81/077866
27.69/0.7400
26.53/0.7992
MiPKD
32.58/0.8998
28.84/0.7875
27.75/0.7418
26.66/0.8029
Table 4: Training expenses of KD methods for distilling x2 EDSR model.
KD methods
Logits-KD) KD)
FitNet
FAKD
CSD
MiPKI)
Time (s/step)
0.49
0.56
0.56
1.18
0.87
Urban PSNR
31.98
30.46
32.04
32.26
32.56
ABLATION STUDY
To illustrate the significance of the proposed MiPKD method, we conduct detailed ablation studies on RCAN and EDSR networks.
Ablation on the feature and block prior mixers for MipKD. There are two fine-grained prior mixer modules in MiPKD, namely, the feature and block prior mixers. Their individual effects ablated in Table 5 The result shows that employing the feature prior mixers leads to significant performance improvement and the block prior mixer based it could further boost the student model.
Ablation on the MiPKD feature prior mixer module. In the feature prior mixer module of MiPKD, the teacher and student models' feature maps are mapped the latent space through corresponding encoders, then randomly mixed and stitched. We present analysis on the encoders in Table [ comparing MiPKD with 1) removing the encoders, aligning and utilizing the teacher's feature map directly and 2) sharing the encoder between the teacher and student model. Removing the encoders would substantially deteriorate the student model's performance. Due to the different distribution of teacher and student models' feature maps, shared encoder cannot effectively map them to the same latent space, leading to noisy mixtures. Assigning separate encoders to the teacher and student models yields the best results, indicating that mixing feature priors in the same latent space is necessary.
8

---

Published as a conference paper at ICLR 2025
STIPSNR)
NOKO
SITNET OURS 969
GT(PSN))
NOKD 560
(20.5.)
FAKO (20
FINEET (20.60)
OURS (2) 50)
GTIPSNRI
NOKD
(12.55)
FAND
112
OURS (13.55)
GTISSNR)
KOKD (20 220
(20.36)
FAKI (20.03)
FITNET (19.95)
DURS (21.25)
GTIPSNN)
NOKO (19 141
(19.25)
FAKO
FITNET (18.93)
OURS (20
(PANNI
NOND (23
FAKO TITNE QURN
GTUPNRI
NOKD (16.45)
KD (16.911
FAKD (16-54)
PINNET (16.05)
OURS (18 221
Figure 3: Visual comparison (x4) with existing SRKD methods from Urban100. The numbers in the bracket denote the PSNR of the presented patches.
Table 5: Ablation on the two prior mixers. The RCAN student model of c32b5g5 is distilled by the teacher model of c32b6g 10.
Table 6: Ablation on the encoder type in MiPKD feature mixer module without block prior mixer module.
Feature Prior Mixer
Block Prior Mixer
Urban100
Urban100
Encoder Type
PSNR SSIM
PSNR
SSIM
25.60 0.7700
No Encoder
24.51
0.7149
25.63 0.77
Shared Encoder
25.61
0.7704
X
25.65 0.7717
Separate Encoder
25.63
0.7711
25.69 0.7728
Table 7 compares the encoder and decoder of different network architectures with similar sizes. The convolutional neural network can better project the representations to the unified latent space, the result shows that CNN exhibits better performance than the MLP encoder/decoder.
Ablation the "auto-encoder" loss L We compared the MiPKD with and without Lg in Table & The results indicate that the auxiliary "auto-encoder" loss makes the mapping between the feature maps" space and the latent space more accurate, leading better student model.

---

Published as a conference paper at ICLR 2025
Table 7: Comparison of different encoder and de- coder network settings.
Table 8: Ablation study the auto-encoder
loss Lx
Urban 100
Urban 100
Encoder/Decoder Type
Auto-encoder Loss
PSNR/SSIM
PSNR/SSIM
MLP
26.42/0. 7964
26.42/0.7977 26.66/0.8029
Conv
26.66/0.8029
Besides, the mask generation strategies are compared in Table Compared with 1) masking according to the Cosine or CKA similarity between teacher and student models' feature maps or 2) generating the complementary pairs of feature map by fixed grid pattern, the random 3D mask exhibits the best performance and least calculation consumption. A more flexible, generalizable strategy applied in the prior mixer module.
Table 9: Ablation analysis the masking strategy for feature prior mixture.
Urban100 PSNR/SSIM
masking strategy
Cosine
25.62/0.7711
Grid mask
25.61/0.7669
CKA
25.63/0.7713
Random
25.69/0.7728
Ablation on the Loss weights setting of feature and block mixers. The impact of various weights of feature mixers loss and block mixer loss is evaluated as Table I0 shown, where Arees Akd. À feats Abtock represent the weights for reconstruction loss, logits-kd loss, feature prior mixer and block prior mixer respectively. Considering the initial fluctuation caused by mixing the block from networks, Abtock is applied since 0.1 presented the best student performance the Table & shown. In addition, the reconstruction loss of the auto-encoder in the feature prior mixer is introduced in the initial stage of training. As the reconstruction ability of the decoder improves, it's beneficial for the prior mixer fuse dark knowledge and restore the enhanced feature map efficiently.
Table 10: Ablation analysis on the weights of different losses
Set5
Set14
BSD100
Urban 100
Arec
Aued
> fent
ABtock
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
I
02.46/0.8972
28.7510.7851
77/88/07399
26.53/0.7976
1
0.1
32.34/0.8977 8970
28.73/0.7849
27 77.67/0.7394
26.47/0.7966 7960
0.1
0.1
32.42/0.80 8980
28.75/0.7857
27.68/0.7399
26.51/0.7988
32.58/0.899)
28. 74.847778875
27.75/0.7418
26.66/0.800 8029
CONCLUSION
In this paper, proposed the dark mixing mechanism for KD SR in feature and block levels. The teacher's knowledge is effectively integrated with the student's ' feature via the Feature Prior Mixer, and the reconstructed feature propagates stochastically by the Block Prior Mixer. The masked feature maps are fused in unified latent space, and the mixed prior narrows the optimization space. The Block Prior Mixer propagates the reconstructed feature and re-ensembles the networks constrain the student model. The two granularity of the prior mixtures follow the common idea of prior mixing and propagation, which alleviates the impact of capacity variation on the performance knowledge transfer. In-depth evaluations prove the significance of the proposed MiPKD strategy.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super- resolution with cascading residual network. In ECCV, pp. 252-2668 2018.
Yutong Bai, Zeyu Wang, Junfei Xiao, Chen Wei, Huiyu Wang, Alan Yuille, Yuyin Zhou, and Cihang Xie. Masked autoencoders enable efficient knowledge distillers. 2022.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based nonnegative neighhor embedding. BMVC, 2012.
Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma. Chunjing Xu, Chao Xu, and Wen Gao. Pre-trained image processing transformer. In CVPR, pp. 12299-12310, 2021.
Yudong Chen, Sen Wang. Jiajun Liu, Xuwei Xu, Frank de Hoog, and Zi Huang. Improved feature distillation via projector ensemble. 2022.
Xiangxiang Chu, Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. Fast, accurate and lightweight super-resolution with neural architecture search. In ICPR, pp. 69.64. IEEE, 2021.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In ECCV, pp. 184-199. Springer, 2014.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 38(2):295-307, 2015.
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In ECCV, pp. 391-407. Springer, 2016.
Hangxiang Fang, Yongwen Long, Xinyi Hu, Yangtao Ou, Yuanjia Huang, and Haoji Hu. Dual cross knowledge distillation for image super-resolution. Journal of Visual Communication and Image Representation, 95: 103858, 2023.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. IJCV, 129: 188-11819, 2021.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Zibin He, Tao Dai. Jian Lu, Yong Jiang, and Shu-Tao Xia. Fakd: Feature-affinity based knowledge distillation for efficient image super-resolution. In ICIP, pp. 518-522. IEEE, 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.00 02531, 2(7), 2015.
Cheeun Hong, Sungyong Baik, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Cadyq: Content- aware dynamic quantization for image super-resolution. In ECCV, 2022.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, pp. 5197-5206, 2015.
Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACMMM, pp. 2024-2032, 2019.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, pp. 1646-1654, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. 14X1v:1412.6980. 2014.
Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged informa- tion for efficient image super-resolution. In ECCV, pp. 465-482, Springer, 2020.
11

---

Published as a conference paper at ICLR 2025
Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Baochang Zhang, Fan Yang. and Rongrong Ji. Pams: Quantized super-resolution via parameterized max scale. In ECCV, pp. 564-580, 2020.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir:
Image restoration using swin transformer. In ICCV, pp. 1833-1844, 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, pp. 136-144, 2017.
Dongyang Liu, Meina Kan, Shiguang Shan, and Xilin Chen. Function-consisten feature distillation. arXiv preprint arXiv:2304.11833 2023.
Xiaotong Luo, Qiuyuan Liang. Ding Liu, and Yanyun Qu. Boosting lightweight single image super-resolution via ooittdiisillattin. In ACMMM, pp. 1535-1543, 2021.
Yinglan Ma, Hongyu Xiong, Zhe Hu, and Lizhuang Ma. Efficient super resolution using binarized neural network. In CVPR Workshops, pp. 0-0, 2019.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, volume 2, pp. 416-423. IEEE, 2001.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, volume 34, pp. 5191-5198, 2020.
Ying Nie, Kai Han, Zhenhua Liu, An Xiao, Yiping Deng, Chunjing Xu, and Yunhe Wang. Ghostsr: Learning ghost features for efficient image super-resolution. arXiv preprint arXiv:2101.08525, 2021.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, pp. 3967-3976, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurlPS, 32, 2019.
Junbo Qiao. Wei Li, Haizhen Xie, Hanting Chen, Yunshuai Zhou, Zhijun Tu, Jie Hu, and Shaohui Lin. Lipt: Latency-aware image processing transformer. arXiv preprint 2oXiv:240400617 2024a.
Junbo Qiao, Jincheng Liao, Wei Li, Yulun Zhang, Yong Guo, Yi Wen, Zhangxizi Qiu, Jiao Xie, Jie Hu, and Shaohui Lin. Hi Hi-mamba: Hicrarchical mamba for efficient image super-resolution. arXiv preprint arXiv:2410.100 10140, 2024b.
Zengyu Qiu, Xinzhu Ma, Kunlin Yang. Chunya Liu, Jun Hou, Shuai Yi, and Wanli Ouyang. Better teacher better student: Dynamic prior knowledge for knowledge distillation. 2022.
Bin Ren, Yawei Li, Nancy Mehta, Radu Timofte, Hongyuan Yu, and Cheng Wan. The ninth ntire 2024 efficient sepertresolution challenge report. In CVPRW2024. 2024.
Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yac Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang. A comprehensive survey of neural architecture search: Challenges and solutions. ACM Computing Surveys (CSUR), 54(4):1-34, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. CoRR, abs/ 111.//502. 2014.
Adriana Romero et al. Fitnets: Hints for thin deep nets. In ICLR, 2015.
Dehua Song, Chang Xu, Xu Jia, Yiyi Chen, Chunjing Xu, and Yunhe Wang. Efficient residual dense block search for image super-resolution. In AAAI, volume 34, pp. 12007-12014, 2020.
Dehua Song, Yunhe Wang, Hanting Chen, Chang Xu, Chunjing Xu, and DaCheng Tao. Addersr: Towards energy efficient image super-resolution. In CVPR, pp. 35648-15657 2021.
12

---

Published as a conference paper at ICLR 2025
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming- Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution Methods and results. In CVPRW, pp. 114-125, 2017.
Zhijun Tu, Kunpeng Du, Hanting Chen, Hailing Wang, Wei Li, Jie Hu, and Yunhe Wang. 1pt-v2: Effi- cient image processing transformer using hierarchical attentions. arXiv preprint arXiv:2404.0088 00633, 2024.
Alvin Wan, Xiaoliang Dai, Peizhao Zhang, Zijian He, Yuandong Tian, Saining Xie, Bichen Wu, Matthew Yu, Tao Xu, Kan Chen, et al. Fbnetv2: Differentiable neural architecture search for spatial and channel dimensions. In CVPR, pp. 12965-12974, 2020.
Hailing Wang, Wei Li, Yuanyuan Xi, Jie Hu, Hanting Chen, Longyu Li, and Yunhe Wang. Ift: Image fusion transformer for ghost-free high dynamic range imaging. arXiv preprint arXiv:2309.1500 2023.
Li Wang, Dong Li, Lu Tian, and Yi Shan. Efficient image super-resolution with collapsible linear blocks. In CVPR, pp. 817-823, 2022a.
Longguang Wang, Xiaoyu Dong, Yingqian Wang, Xinyi Ying, Zaiping Lin, Wei An, and Yulan Guo. Exploring sparsity in image super-resolution for efficient inference. In CVPR, pp. 4917-4926, 2021a.
Xintao Wang, Liangbin Xie, Ke Yu, Kelvin C.K. Chan, Chen Change Loy, and Chao Dong. BasicSR: Open source image and video restoration toolbox. https://github. BasicSR 2022b.
Yanbo Wang, Shaohui Lin, Yanyun Qu, Haiyan Wu, Zhizhong Zhang, Yuan Xie, and Angela Yao. Towards compact single image super-resolution via contrastive self-distillation. In ICCAI, 2021b.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: general u-shaped transformer for image restoration. In CVPR, pp. 17683-17693, 2022c.
Jing Yang, Brais Martínez, Adrian Bulat, and Georgios Tzimiropoulos. Knowledge distillation via softmax regression representation learning. In International Conference on Learning Representa- tions, 2021. URL URL htttt:tt////ee h
www.nteetttt.mm ins.aannsscchharr org/ Corpus ID: 235613564
Zhendong Yang, Zhe Li, Ailing Zeng, Zexian Li, Chun Yuan, and Yu Li. Vitkd: Feature-hased knowledge distillation for vision transformers. In 2024 IEEE/CV Conference Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1379-1388. IEEE, 2024. ISBN 9798350365474,
Gengqi Yao, Zhan Li, Bir Bhanu, Zhiqing Kang. Ziyi Zhong, and Qingfeng Zhang. Mtkdsr: Multi- teacher knowledge distillation for super resolution image reconstruction. 2022 26th International Conference Pattern Recognition (ICPR), pp. 352-358. IEEE, 2022.
Junho Yim, Donggyu Joo, Jihoon Bac, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, pp. 4133-4141, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 1612.03928, 2016.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pp. 2022.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Curves and Surfaces: 7th International Conference, Avignon, France, June 24-30, 2010, Revised Selected Papers 7, pp. 711-730. Springer, 2012.
Quan Zhang, Xiaoyu Liu, Wei Li. Hanting Chen, Junchao Liu, Jie Hu, Zhiwei Xiong, Chun Yuan, and Yunhe Wang. Distilling semantic priors from sam to efficient image restoration models. arXiv:2403. 16368, 2024.
13

---

Published as a conference paper at ICLR 2025
Xindong Zhang, Hui Zeng, Shi Guo, and Lei Zhang. Efficient long-range attention network for image super-resolttion. In ECCV, pp. 649-667. Springer, 2022.
Yiman Zhang, Hanting Chen, Xinghao Chen, Yiping Deng. Chunjing Xu, and Yunhe Wang, Data-free knowledge distillation for image super-resolution. In CVPR, pp. 7852-7861, 2021a.
Yulun Zhang, Kumpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, pp. 286-301, 2018.
Yulun Zhang, Huan Wang. Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. NeurIPS, 34:2695-2706, 2021b.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprini /rXiv:/61.001577 2016.
14

---

Published as a conference paper at ICLR 2025
A SUPPLYMENTARY EXPERIMENT RESULTS
A.1 EXPERIMENT RESULTS ON SWINIR MODEL
We compare MiPKD with other applicable KD methods on distilling transformer-basee SR model, SwinIR. The result shows the superiority and universality of MiPKD.
Table 11: Quantitative comparison of distilling SwinIR (Liang al.
2021
the benchmark datasets.
Set5
Set14
BSDI00
Urban
Scale
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Teacher
38.36
0,9620
34.14
0.9227
32.45
0.9030
33.40
0,9394
Scratch
38.00
0.9607
33.56
0.9178
32.19
0.9000
32.05
0.9279
2
KD
38.04
0.9608
33.61
0.9184
32.22
0.9003
32.09
0.9282
MipKD
38.14
0.9611
33.76
0.9194
32.29
0.9011
32.46
0.9313
Teacher
34.89
0.9312
30.77
0.8503
29.37
0.8124
29.29
0.8744
Scratch
34.41
0.9273
30.43
0.8437
29.12
0.8062
28.20
0.8537
KD
34.44
0.9275
30.45
0.8443
29.14
0.8066
28.23
0.8545
MipKD
34.53
0.9283
30.52
0.8456
29.19
0.8079
28.47
0.8591
Teacher
32.72
0.9021
28.94
0.7914
27.83
0.7459
27.07
0.8164
Scratch
32.31
0.8955
28.67
0.7833
27.61
0.7379
26.15
0.7884
4
KD 32.27
0.8954
28.67
0.7833
27.62
0.7380
26.15
0.7887
FitNet
32.08
0.8925
28.51
0.7800
27.53
0.7354
25.80
0.7779
FAKD
32.06
0.8926
28.52
0.7800
27.53
0.7354
25.81
0.7780
MipKD
32.39
0.8971
28.76
0.7854
27.68
0.7403
26.37
0.7956
A.2 EVALUATE MIPKD ON HIGH COMPRESSION RATE SETTING.
Table 12 shows the results of training x4 scale EDSR model with both network width and depth compression. The number of parameters is reduced to about 28 of teacher model's. Directly distilling the student model by the teacher model yields negative effects its performance. Two-stage KD with an intermediate maching-assistan (TA) model Mirzadeh et al. (2020) are preferred in such case. To make use of the CSD method, compared different TA options, (I) Seacher=>Studeet1>>Stueeee the whole pipeline is in width-then--detth compression order. We adopt CSD to train the TA and FAKD for student model (2) Depth-ther -henntthheeeeee width: then-width: we first perform depth compression and then width compression. We adopt FAKD+CSD and MiPKD+MiPKD for the two-stage distillations. The result indicates that distill TA and student model with MiPKD yields the best performance.
Table 12: Quantitative comparison on training x4 EDSR models with higher compression rate on the benchmark datasets. In these experiments, the EDSR student model of c64b16 is distilled by the teacher model of c256b32.
SetS
Set14
BSD100
Urban
Method
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
Teacher NOKD)
32.65/0.9003
28.95/0.7903
27.81/0.7448 7440
26.87/0.
32.01/0.8924
28.46/0.7782
17.47/0.7324
25.61/0.770M
KD
31.99/0.80 8921
28.46/0.7784
27.47/0.7327
25.60/0.7700
FitNet
31.92/0.8912
28.42/0.7776
27.4A/0.7317
25.52/0.7672
FAKD
31.65/0.8877
28.32/0.7760
27.37/0.7303
25.38/0.7629
FAKD+CSD
32.00/0.8930
28.47/0.7800
27.51/0.7340
25.79/0.7790
CSD+FAKD
31.86/0 8907
.988//0.907
28.42/0.7786
27.46/0.7332
25.58/0.7709
MiPKD+MiPKD
32.17/0.8947
28.57/0.7812
27.57/0.7354
25.89/0.7794
A.3 ABLATION ON THE FREQUENCY OF BLOCK PRIOR MIXER
In the MiPKD, the block prior mixer each distillation position randomly determines if the enhanced feature map F. is forward propagated to the student or teacher network. In Table 13 we analyzed the impact of frequency of this random propagation. It's also the number of each input image
15

---

Published as a conference paper at ICLR 2025
passing through the enhanced mixed SR networks. It shows that performing single random forward propagation yields the best results, which in practice is the most efficient as well.
Table 13: Ablation for the frequency of random propagation of the Block Prior Mixer. The RCAN student model of c64b5g10 distilled by the teacher model of c64b6g10 10.
Set5
Set14
BSI1000
Urban 100
#propagation per sample
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
31.99
0,8919
28.43
0.7777
27.46
0.732
25.69
0.7728
2
3.887
0,8884
28.38
0.7632
27.42
0.7311
25.62
0.7699
31.86
0.8867
28.37
0.7619
27.41
0.7306
25.57
0.7697
B MORE VISUAL RESULTS
In the experiment section of main text, we provided some visual results of EDSR model. We present more visual comparisons of MiPKD with other KD methods over the SwinlR network in Figure MiPKD reconstructs more structural details and alleviates the blurring artifacts, which is consistent with the observations on EDSR network.
GT PSNRR
NOKD (21.26)
KD. (20.91)
OURS (23.44)
Ing
GT(PSNR)
NOKD (16.12)
KD (17.02)
OURS (20.36)
GUPSNRI
NOKD [22 89)
KD (22.64)
QUBS (24 39)
ing. 090 from Urbun 100
(17.82)
(18.09)
(18.13)
Figure 4: The x4 super resolution results of SwinIR models on img024, and img098 from Urban 00. PSNRs of the eropped regions are annotated below each image.
C FEATURE MAP ANALYSIS FOR THE MASK STRATEGY
The comparison of feature maps at deep layer (the convolution layer following all residual groups of RCAN) across FitNet, RKD, and MiPKD could highlight the effectiveness of MiPKD's random masking strategy, as shown in Figure 5 The MiPKD excels in capturing fine-grained texture details, producing feature maps (far right) with sharper contrasts between high- figh-freuuncyy regions with complex patterns and low-freque frequeenyycy background regions. This enhanced representation serves higher-qualinn input for the network's 's reconstruction layers, ultimately leading to superior visual quality in the output SR images.
16

---

Published as a conference paper at ICLR 2025
The Feature Prior Mixer ensures that the student learns to reconstruct the teacher's fine- tne-grained features while maintaining contextual consistency, effectively narrowing the gap between teacher and student feature distributions. Simultaneously, the Block Prior Mixer introduces stochastic routing of feature representations, enabling the student to inherit complex transformation patterns from the teacher. This granularity prior mixing mechanism enhances the fidelity of high-frequency textures while preserving structural integrity, reducing artifacts and producing visually pleasing super-resolution results with sharper edges and more realistic textures.
100
300
120
Figure 5: The feature maps at the deep layer ('conv afterbody") distillation point for FitNet, RKD, and MiPKD methods on the x4 super resolution RCAN model
D TRAINING STABILITY OF MIPKD
The MiPKD adopts random masking strategy the feature and block prior mixer, which introduces diverse mixing patterns. As shown in Figure 6, the random strategy is as stable stable other feature- based KD methods during training.
Training loss curve comparison
0.8
Lagits-KD
FAKD
Training loss
0.6
MiPKD
0.4
0.0
50000
100000
150000
200000
250000
Step
Figure 6: Loss curve of distilling x4 RCAN with different KD methods.
17


Reviews:
[{"review_id": "KlB8fp45Vr", "reviewer": "Reviewer_pGYs", "summary": "This paper introduces a multi-granularity prior knowledge distillation (MiPKD) framework for image super-resolution (SR) tasks. By incorporating both a feature prior mixer and a block prior mixer, MiPKD effectively transfers knowledge from a larger teacher model to a compact student model, enhancing model compression while preserving SR performance. Unlike conventional KD methods tailored to specific teacher-student architectures, MiPKD flexibly accommodates different network depths and widths.", "questions": "1. Does the block-level mixing lead to instability in training for certain models? Were there hyper-parameter adaptations needed for different architectures during training?\n\n2. Have the authors considered extending MiPKD to other vision tasks, such as object detection or semantic segmentation, to verify the broader applicability of the proposed framework?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Originality: MiPKD’s innovation lies in its multi-granularity knowledge mixing mechanism. Through coordinated use of feature and block prior mixers, MiPKD enables adaptable knowledge transfer across different teacher-student architectures, a notable improvement in KD design.\n\n2. Quality: Experimental design is thorough and covers a wide array of SR network architectures and compression configurations. Results indicate MiPKD’s superior performance on multiple datasets, confirming its generalizability and efficacy across tasks.\n\n3. Clarity: The paper’s organization is clear, with helpful visuals and an accessible writing style that conveys the method’s complexity effectively.", "weaknesses": "1.Explanability. Although MiPKD seems good in experiments, I still don't know why it is effective. The method introduces random masks I and R_k in the feature prior mixture and block prior mixture. but it needs to clarify why these masks are effective in distilling SR networks. I will be appreciated if the author could convince me through feature map analysis or theoretical deductions. \n\n2. Experiment Details: While MiPKD demonstrates good results, further details regarding the impact of different mask generation strategies in the feature mixer could clarify the specific role and contribution of these strategies.\n\n3. Generality: The paper primarily discusses MiPKD’s performance in SR tasks, but it seems that MiPKD is not specially designed for the SR task. Thus, the authors are encouraged to show the applicability to other CV tasks. Additional experiments on a broader range of tasks could enhance the method’s perceived generalizability.\n\n4. Experiments on current SOTA SR networks. Experiments are carried out on EDSR, RCAN and SwinIR networks. If possible, the authors could do experiments on more recent SOTA networks, such as DRCT-L and HMANet etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CzUv3H0Z7Q", "reviewer": "Reviewer_B9sn", "summary": "The authors propose MiPKD, a multi-granularity mixture of prior knowledge distillation framework designed for image super-resolution tasks. MiPKD facilitates the transfer of “dark knowledge” from teacher models to student models across diverse network architectures. The framework employs feature and block prior mixers to reduce the capacity disparity between teacher and student models for effective knowledge alignment and transfer. Extensive experiments are conducted on three SR models and four datasets, demonstrating that MiPKD significantly surpasses existing KD methods.", "questions": "1. What is the rationale for randomly sampling the forward propagation path in the block prior mixer? If the output of Feature Prior Mixer is passed to both teacher and student models and compute two losses, will the knowledge distillation be more effective?\n2. What are the specifications of the encoder and decoder networks (e.g., number of layers, hidden dimensions)? Would a larger auto-encoder result in a better student model?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The integration of feature fusion within a unified latent space  and stochastic network block fusion are innovative for SR model knowledge distillation.\n2. The paper provides clear explanations and discussion, supported by well-organized charts and ablation studies, which effectively highlight the framework’s contributions and innovations.\n3. The paper is well organized and the motivation of MiPKD is clear", "weaknesses": "1. The writing is sometimes unclear, with inconsistent notations and undefined terms:\n\t- The feature maps $F$ in Fig. 2, Eq. 2, and Eq. 3 are not consistently bolded.\n\t- The three feature maps in Eq. 3 are inconsistently formatted.\n\t- The loss weights $\\lambda_1$ and $\\lambda_2$, mentioned in lines 358–359, are not defined or referenced elsewhere in the paper.\n2. While the paper provides comparisons for EDSR and RCAN, the evaluation on SwinIR is not as comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BwOGOMiSpT", "reviewer": "Reviewer_qbB2", "summary": "This work proposes a new knowledge distillation method which considers multi-granularity mixturw of priors for image super resolution. Motivated by the success of masked autor encoder in reconstructing missing pixels from masked input, this work explores this machinism into feature prior mixer and block prior mixer to distill teacher's feature. Experiments are done on various datasets to show it effectiveness but the improvement is not significant,", "questions": "The main concern with the proposed method is the lack of comparison with closely related feature-based knowledge distillation approaches, such as ViTKD [1], Improved Feature Distillation via Projector Ensemble [2], and Knowledge Distillation via Softmax Regression Representation Learning [3]. These methods, while not specifically targeting super-resolution, focus on feature projection before distillation, making them relevant benchmarks. Including these comparisons would provide a clearer understanding of the proposed method’s strengths and positioning relative to state-of-the-art techniques. Additionally, a deeper discussion on the differences between the proposed method and Masked Autoencoders [4] would clarify its unique contributions.   [5] serves as a benchmark for super-resolution where knowledge distillation is applied to enhance performance. Including a discussion of this benchmark and similar methods would help clarify how the proposed method contributes to this area.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is well motivated and grounded.  This work tackles the challenge of aligning representations between models of different sizes, which is important for improving the performance of lightweight models.\n2. Experiments are done on various datasets (Set5, Set14, BSD100, and Urban100) to show its superiority over competitors in super-resolution task.\n3. The authors perform ablation studies to evaluate the contribution of each component (feature prior mixer, block prior mixer, encoder type) and discuss different loss weighting strategies.", "weaknesses": "1. This method lacks comparison with SOTA feature-based knowledge distillation methods, such as, [1][2][3][4] which all contributed to projecting features before distillation to improve student's performance albeit not specifically in the super-resolution domain.\n[1]ViTKD: Feature-based Knowledge Distillation for Vision Transformers\n[2]Improved feature distillation via projector ensemble\n[3]Knowledge distillation via softmax regression representation learning\n[4]Masked Autoencoders Enable Efficient Knowledge Distillers\n2. As a closely related work [4] also proposes masked auto encoder for feature distillation.  It would be beneficial to discuss this work in both the introduction and experimental sections, highlighting the differences and demonstrating where the proposed method offers improvements.\nBesides, [5] has included knowledge distillation used for super resolution.  Method such as RDEN in [5] should be included in related work.\n[5]The Ninth NTIRE2024 Efficient Super-Resolution Challenge Report\n3. Three key factors influence Knowledge distillation performance: knowledge, position, loss. Current ablations studies have discusses knowledge and loss but lacks discussion of where to put.  Understanding the optimal positions for applying the feature prior mixer and block prior mixer would provide valuable insights into the flexibility and effectiveness of MiPKD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)