AugKD: Ingenious Augmentations Empower Knowledge Distillation for Image Super-Resolution
Authors: Yun Zhang
Wei Li
Simiao Li
Hanting Chen
Zhijun Tu
Bingyi Jing
Shaohui Lin
Jie Hu
Wenjia Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Image Super-Resolution
Knowledge Distillation
Model Compression
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Knowledge distillation (KD) compresses deep neural networks by transferring task-related knowledge from cumbersome pre-trained teacher models to more compact student models. However, vanilla KD for image super-resolution (SR) networks yields only limited improvements due to the inherent nature of SR tasks, where the outputs of teacher models are noisy approximations of high-quality label images. In this work, we show that the potential of vanilla KD has been underestimated and demonstrate that the ingenious application of data augmentation methods can close the gap between it and more complex, well-designed methods. Unlike conventional training processes typically applying image augmentations simultaneously to both low-quality inputs and high-quality labels, we propose AugKD utilizing unpaired data augmentations to 1) generate auxiliary distillation samples and 2) impose label consistency regularization. Comprehensive experiments show that the AugKD significantly outperforms existing state-of-the-art KD methods across a range of SR tasks.

Published as a conference paper at ICLR 2025
AUGKD: INGENIOUS AUGMENTATIONS EMPOWER KNOWLEDGE DISTILLATION FOR IMAGE SUPER- RESOLUTION
Yun Zhang'
Wei Li', Simiao Li, Hanting Chen, Zhijun Tu
The Hong Kong University of Science and Technology
Huawei Noah's 's Ark Lab
Bingyi Jing
Shaohui Lin
Southern University Science and Technology
East China Normal University
Jie Hu
Wenjia Wang
Huawei Noah's Ark Lab
The Hong Kong University of Science and Technology (Guangzhou)
ABSTRACT
Knowledge distillation (KD) compresses deep neural networks by transferring task- related knowledge from cumbersome pre-trained teacher models to more compact student models. However, vanilla KD for image super-resolution (SR) networks yields only limited improvements due to the inherent nature of SR tasks, where the outputs of teacher models noisy approximations of high-quality label images. In this work, we show that the potential of vanilla KD has been underestimated and demonstrate that the ingenious application of data augmentation methods can close the gap between it and more complex, well-designed methods. Unlike conventional training processes typically applying image augmentations simultaneously to both low-quality inputs and high-qualit labels, we propose AugKD utilizing unpaired data augmentations to 1) generate auxiliary distillation samples and 2) impose label consistency regularization. Comprehensive experiments show that the AugKD significantly outperforms existing state-of-the-ar KD methods across a range SR tasks.
INTRODUCTION
is an essential yet challenging task computer vision (CV) that focuses on reconstructing high- resolution (HR) image from its low-resolution (LR) counterpart (Lim et al. 2017 Zhang et al. 201 8). In recent years, convolutional neural networks (CNNs) and Transformers (Lingg al. 2021 Yang
al. 2020 Wang et al. 2022b Zamir et al. 2022) have achieved significant success SR. However, despite the impressive performance of deep learning-
SR models, their practical deployment is often constrained by the high computational and memory requirements (Zhang et al. 2021c). As result, there has been an increasing focus on developing SR model compression techniques to enable their use in real-world applications, especially for resource-constrained devices.
KD is an effective technique for reducing computational costs and memory requirements during model deployment, while also enhancing the performance of student models. works by transferring the "dark knowledge" from a well well-performing but computationally heavy teacher model to a more lightweight student model (Gao al., [0199] Hui et al. 2019 Zhang et al. (021b). Compared to other model compression techniques, such as quantization (Gupta et al. 2015: Hubara et al. 2016
Ignatov al., 2021] Wu al., (016). pruning (Anwar et al. 2017 Wang et al. 2021b Liu al.
2019), and neural architecture search (NAS) (Wu et al. 2019 Howard et 2019 Guo et al. 2020
KD has garnered significant attention due to its outstanding performance and wide applicability.
"Co-first author.
'Coresponding author. hujie23 huawei.com, huuttnggo@huuutttg..uuu....m

---

Published as a conference paper at ICLR 2025
8
Out
Teacher Model
T(a; 0*)
LiKe
ER
Arg ritl
Aug
Saudent Model 0(x;
(1
Zoent In
HRR
Auxiliary Distillation Samples
Label Consistency Regularization
Figure 1: Framework of the AugKD method. Facilitates the transfer of knowledge through the auxiliary distillation samples and label consistency regularization.
The effectiveness of KD has been well-established in natural language processing (NLP) (Gou et al.
2021 (anh et al., 2019) and conventional high-level CV tasks, such as classification, detection, and
segmentation (Hinton et al. 2015 Park et al. 2019) Tung & Mori 2019). However, its application to
SR tasks remains relatively underexplored (He et al. 2020 Wang et al. 2021c| Zhang et al. 20211b
Lee al. 2020). The use of standard response- based KD methods (Hinton et al.][2015 or those
optimized for high-level CV tasks (Romero et al. 2014 Yim et al.) 2017 Zagoruyk & Komodakis 2016) typically results in only marginal improvements and may even have negative effects when applied to distilling SR networks, observed by He al. (2020) and confirmed by our experiments in section 4 Previous KD methods specifically designed for SR are mostly feature-based, where the student model is forced to mimic the intermediate features of the teacher model directly (He let al. 2020) or through pre-trained perceptual feature extractor like VGG (Yao et al., [022a] Wang et al. 2021c). However, these Peature-based approaches have limited applicability. In practice, the architecture of teacher models is often inaccessible due to commercial, privacy, or safety restrictions, making feature-hased methods impractical for some real-world applications.
For the knowledge distillation of super-resolution models, most of previous explanations for the mechanism of KD no longer hold due to the unique task characteristics. Since the teacher model's output, as noisy approximation to the ground-truth (GT) high-quality image, contains barely extra information exceeding GT, the "dark knowledge" of teacher are hardly transferred to student model through KD. Distinct to exist feature-based methods, we propose AugKD, an alternative approach to enhance the knowledge distillation via the data augmentations. developing various knowledge types (Gou et al.) (02T) to more task-adapted training data mining and construction with effective utilization of pre-trained teacher. Specifically, the AugKD consists of two major modules: auxiliary distillation sample generation and label consistency regularization. The auxiliary training examples are built from (LR, HR) pairs by zoom-in and zoom-out augmentations. Then teacher model is able to guide the student model with these image samples. It frees the teacher model from merely echoing the GT labels inaccurately. Moreover, we realize the label consistency regularization into the KD for SR by defining several invertible data augmentation operations. The student model is forced to yield the same output as the teacher model, given the augmented inputs. The regularization makes the student model exposed to a diverse range of inputs, substantially improving performance (Oliver et al., 2018 Jeong et al. 2019 Englesson & Azizpour 2021). The AugKD is logits-basee based and independent of network architectures. It shows great universality among a diverse array of SR model families and SR tasks. In summary, our main contributions are three-fold:
shifts the paradigm from
We analyze the mechanisms of KD for SR, and propose AugKD adapted to the unique task properties. It facilitates the student model's learning by auxiliary training samples.
We leverage the label consistency regularization into KD for SR by specifying several invertible data augmentations. It improves the model's generalizability.
The proposed AugKD, applies broadly to multiple teacher-student configurations, promising cutting-edge KD approach for SR.
2

---

Published as a conference paper at ICLR 2025
DI22K Training Set
Urban100
45
(QUR(S.T
POIRITAT
15
38
44-
13.00
34
40.51
33,00
34,52
27.0
26.87
31.0
30.5
30.20
30:45
30:17
30:41
30:50
30.57
26.5
25.83
26.21
26-21
30.0
25.5
CatafresKD Scratch
AugKD
DataFreooD Scretch
FARD
C5D
Aug
Figure 2: Similarity between the student and teacher x4 EDSR models under different training approaches (X-axis). PSNR (S, T) represents the average PSNR between the student and teacher outputs, with higher values indicating greater similarity, PSNR (S, GT) shows the average PSNR between the student output and ground-truth HR image, with higher values indicating better fitting (left: training set) or generalization (right: testing set).
RELATED WORKS
2.1 IMAGE
Deep neural networks (DNNs) have achieved impressive success in image SR. (Rnn et al. (2014) introduced a CNN SR model with only three convolutional layers first. Subsequently, residual learning was incorporated into the VDSR model (Kim et al. 2016), which expanded the network to 20 convolutional layers. Lim et al. (201)) proposed EDSR, which utilized simplified residual blocks (He al., 2016), and Zhang et al. (20T8) introduced the even deeper RCAN network. These methods, among others, have set state-of state-of-the-art performance benchmarks by increasing network depth and width. More recently, Transformers have gained significant attention in the field of image restoration. The SwinIR model (Liang et al.) 2021) applies the Swin Transformer architecture for deep feature extraction. The Restormer model (Zamir et al., (0222) proposed hierarchical multi-scale structure, introducing a more efficient Transformer blocks that alters the attention mechanisms and the feed-forward network. While CNNs and Transformers models demonstrate extraordinary effect in SR, they are often associated with high memory and computational costs.
2.2 KNOWLEDGE DISTILLATION
KD for high-level CV. Knowledge distillation is widely recognized as an effective model com- pression method that can significantly reduce the computation overload and improve student's capability (Hinton et al., 2015 Yim et al. 2017 Gou et al. 202T). The response-based KD methods simple yet effective where the student models directly imitate the predictions or logits of the teacher model (Hinton et al. 2015) Zhao et al. 2022] Chen et al., 2017). The proposed AugKD method falls into this category since only the final outputs of models are aligned. Besides the output of the networks, the intermediate features can also be used to improve the student model, by matching feature maps directly, after dimension standardization (Zagoruyko & Komodakis 2016) or extra
modules (Kim et al. 20188 Passban et al. 2021: Guo el al.) 2021 }). The relations between layers or samples can be used for KD, such correlation (Yim et al. 2017 You al., 2017), mutual information (Passalis et al. 2020), or pairwise or triple-wise geometric relations (Park et al. 2019).
KD for super resolution. Lately, there has been an increasing number of efforts made on the KD for super-resolution networks. He et al. (2020) proposed the FAKD to align the dimensions of models' feature maps by spatial affinity matrix to train the student model. [ee et al. (2020) employed an encoder to extract the compact features from HR images to initialize the generator network and thereby perform feature distillation. Wang et al. (202lc) proposed a channel-sharing self-distillation method with perceptual contrastive losses. To train SR network under the privacy and data transmission limitations, Zhang et al. (202 employed a generator to support data-free KD. MTKDSR Yao et al.) 2022b) employed two teacher models with different SR objectives (PSNR, perceptual) to guide the student model simultancously. CrossKD (Fang et al. 2023) divides the teacher and student networks into two segments that are interchanged and connected perform
3

---

Published as a conference paper at ICLR 2025
Traking Sd
Trannng Set
Aeillyy waa
Di UOS GUI
Sampting (f
e Md
Teuskwr
Stake
Tacker
(5122)) - TUgSI
1
(AT 1V2
based Knowledge Destilationn
Dena-Foee Knowledge Dvetilation
A tnbballe :wiigg Darm S zo00
8 penix
Law
Data Augmentation empowered KD fir
Figure 3: Comparison between the logits-KD,, Data Free KD, and AugKD. The first two fail to enable the function of teacher model.
forward propagation. The common limitation of these methods is that they are only applicable to CNN-based models and have certain requirements on the teacher-student structure.
3 METHODOLOGY
3.1 NOTATIONS AND PRELIMINARIES
Let (x;; (*) and S(z; 8*) be a teacher and student SR model with parameters O and 0 for the super-resolution of input x, respectively. Given an input LR image In RHxWXx x3, the output
SR images of the two networks are denoted by SR S(T LRi 0*) E HR x3, where x W is the input size and 8, Z+ the scaling factor. The Ll-norm reconstruction loss is computed as:
((t) T LRi 0*) E BR5.Hxs and S(()
Lree
(1)
where Hn is the ground-truth HR label. And the vanilla response-based KD loss is given by
Ld
S(t)
T())
11
(2)
SR
SR
which is computed directly by the output of teacher and student models.
3.2 MOTIVATION
Since the introduction of knowledge distillation by Hinton et al. (2015), numerous studies have analyzed and discussed mechanisms through which teacher supervision enhances the performance of student models (Tang al. 2020: Stanton et al.) 2021 Wang et al. 202la| Zhang et al. 2022 Harutyunyan et al. 2023). It is widely accepted that in response- based KD, the "dark knowledge from the teacher model encompasses the inter-class and inter-example relational information found in the output logits, which is not present in the ground-truth labels.
However, there is barely such benefit in SR tasks that reconstruct image pixels. Since the outputs of the SR network (T(0) are noisy and inaccurate approximations of the ground-truth distribution of high-resolution image flt) as shown in Figure (a). Directly aligning the model outputs hardly transfers knowledge and may even mislead the student model. The guide capability of the teacher model is shaded by [() resulting limited KD effects. To verify this hypothesis, we train an EDSR network of scale x4 using different training methods (data-free KD, supervised training without distillation, Logits-KD KD., FAKD, CSD and the proposed Aug To make the models comparable, the data-free KD uses the LR of the training set and discards HR, assuming that there is an oracle image generator G. Then we compute the PSNR metries between the outputs of teacher and student models, on the training and testing sets, respectively. It reflects the similarities between networks, and the extent to which the student model is impacted by the teacher model. The results shown in Figure indicate that the existing KD approaches make the student model performs more like teacher only to small extent, since the the improvements of P PSNR (S, T) over training without KD are limited. Therefore, the PSNR referring GT, PSNR (S, GT), are also low on both training and testing sets.
4

---

Published as a conference paper at ICLR 2025
Oupees TT3)
Frozen Teacher Model (x; 0*)
SR
Original
Model
Original
Caugl
Student Model S(x; 8*) A
Depurity
9B
Augmemedd
Angrerad
(t) consistency regularization
(b) consistency regularization SR
Figure 4: Comparison of the label consistency regularization in high-level CV and KD for SR. The augmentations should be invertible make the models' output comparable.
This issue cannot be addressed by simple data augmentation that reuses the training image pairs to produce augmented I. and HR with pairwise rotations or flips, i.e. (JLn, TRn) = (Tr.g (RR) The "recycled" data inadequate enable the function of teacher model. The data-free knowledge distillation methods (Zhang et al. (2021b) stay out of this problem due to the discard of HR references from the training data. The supervision signals solely come from the teacher model, as illustrated in Figure 3 (b). Although teacher model's knowledge are transferred to student model (it's only supervised by the teacher model's output), it's impractical to discard the labels from training data especially when they are available. Besides, the teacher model may yield noisier output on the generated training images.
Above findings motivate us to build more task-adapted KD) framework by mining information from the training data. Specifically, we construct auxiliary training inputs through data augmentations to function the KD. These data closely related with the training set to prevent distributional bias. And we introduce the label consistency regularization through invertible data augmentations.
3.3 AUXILIARY DISTILLATIONN SAMPLES
Since the teacher model's knowledge is shaded by the ground-truth HR labels, we perform knowledge distillation by extra LR images rather than the raw training data. To make the generation process efficient and the generated images distributed closely with the original, the auxiliary training samples are obtained from original LR, HR pairs, as demonstrated in Figure []] Two image zooming oper- ations are employed: The zoom-in f0 operation is facilitated by randomly cropping patches from HR The cropped patches have the same size as the LR image L for the convenience of batch processing. Conversely, the zoom-out e operation is carried out by wwwn--smpling the LR image in the same manner of degradation as (()) (i)
(0)
Lt-- The two obtained auxiliary LR images are denoted as I.R.. E #///s.s//sxxx and I.R... HRHWxx
For pair of original training examples (7/)- 1 the output of the zoom-out operation is unique, but the zoom-in operation on I(() could result in various outcomes according to the strategy of patch selection. Beyond random cropping, regions can also be selected based on their reconstruction difficulty or texture complexity. It's observed our experiments that adapted selection would incur higher computational cost with marginal performance gains.
After generating auxiliary distillation samples, the teacher model provides corresponding SR images for the zoom-in and zoom-out LR inputs to supervise the student model. Since there is only the teacher model's 's supervision for these training samples, teacher's distribution information are unshaded from GT and able to impact the student model. AugKD provides more refined and data-centric approach to KD, reflecting its benefits in effective data utilization and superior performance in SR tasks. The overall loss is constructed by adding extra loss term that is computed on the auxiliary distillation samples to the reconstruction loss (Equation (T) and conventional KD loss (Equation 2
Laughs = S - |l1 + ( - 1T - 111:
||
where 15 = -( 8r), T(() T(ri)) 8' and the other terms are computed similarly, zoom-out performed, we compute the reconstruction loss between gitte and [ also. To up.
L Lree + deaLed + daugrdhangkd;
5

---

Published as a conference paper at ICLR 2025
Table 1:
SR model specifications and statisties (x4 scale). The FLOPs and frames per second (FPS) are computed with 3x256x256 input image on single V100 GPU of 64GB VRAM. The block denotes the number of residual blocks for EDSR and RCAN (in each residual group) or Swin transformer blocks for SwinIR models.
Network
Model
Role
FLOPs (G)
#Params
FPS
Channel
Block
Group
Teacher Student
256
32
3293.35
43.09 M
3.233
EDSR
64
32
207.28
2.70 M
33.958
RCAN
Teacher
Student
64
20
10
10
1044.03
366.98
15.59 M
5.17 M
12.337
6.162
SwinIR
Teacher
180
6
861.27
11.90 M
0.459
Student
60
4
121.48
1.24 M
0.874
where Akd and Aaugkd are the loss weights.
3.4 LABEL CONSISTENCY REGULARIZATION
Consistency regularization is commonly used in semi-supervised and self-supervised learning. As illustrated in Figure 4 (a), it encourages the prediction of the network be consistent over perturbed training examples, leading to robustness against corrupted data in test time (Oliver et al.) [2088] Englesson & Azizpour [20T]] Jeong et al., 2019). The model is trained
identify the crucial semantic information related to specific tasks from the input images, despite the possible noise and perturbations. The regularization is based on various image augmentation techniques, like rotation, shearing, cutout, and translation.
KD encourages the student model to produce the same predictions as the teacher model. Such characteristic should hold even their inputs are differently augmented, as the task-related semantic information remains unchanged and the input perturbations should not significantly distinguish between the outputs of the teacher and student models. To realize label consistency regularization, we apply data augmentations on the input of student model while keeping the teacher model's input unperturbed. This approach allows the student to learn invariant representations across diverse transformations. Meanwhile, the student is guided by a more powerful teacher model, whose supervision are from non-perturbed inputs that inherently provide superior quality compared those from augmented inputs. Thereby the auxiliary distillation samples and the teacher model are fully leveraged. Taking the zoom-in LR sample an example, the consistency regularization can be represented as:
L = [IH - T(HHn 11)
where FF)) denotes the perturbation function.
However, as super resolution is a pixel-level image-to-image CV task that is weakly relevant to semantic information of image subject, any tweak the input can alter the model's output. For KD, the student model's output would consequently be incomparable with the teacher model's. Therefore, need to perform inverse augmentation, namely F-1((), on the output of the student model. The label consistency regularization becomes:
|| || ((HR,)
The selected augmentations should be invertible and relevant to the SR task for maintaining the crucial pixel-level details after augmentation. It requires that for any input image I, F- (F(()) I. Hence, a number of popular image augmentations, such as blurring, cutout, brightness adjustment, and cropping, are not applicable as they do not meet this prerequisite. Instead, we employ two geometric transformations, horizontal/verticall flip and 90%/180'/70"' rotations, along with the color inversion transformation that subtracts each pixel intensity value of the input image from 255 (or 1 if normalized): F(I) = 255 I. The color inversion is invertible and maintains the relative magnitude among pixel values. It also prompts the student models to be more sensitive to essential structural features such as lines and edges. Right bottom of Figure Tillustrates the three types of invertible data augmentations employed to realize label consistency.

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparison (average PSNR/SSIM) between AugKD and other distillation methods for EDSR of three SR scales. The best and second-best performances are highlighted in bold and underlined, respectively. An asterisk indicates that the results in row are from reproduction.
Sct5
Set14
BSD100
Urb
Scale
Method
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
Scratch
38.00/0.966
33.57/0.9171
32.17/0. 8996
31.96/0.9200 9268
KD
15.04/0.9600
33.58/0.9172
22.19/0.8998
31.98/0.9269
FitNet
37.59/0 9589
33.09/0.9136
31.79/0.8955
30.46/0.9111
AT
37.96/0.
33.48/0.9167
32.12/0.89 8990
31.71/0.9241
x2
RKD
38.03/0. 9606
33.57/0.9173
32.18/0.898 8998
31.96/0.9277
FAKD
9606
33.60/0.9173
12.8999.8998 89998
32.04/0.9275
CSD*
38.06/0 9607
33.65/0.9179
32.22/0. 9004
32.26/0.9300
AugKD
38.15/0.9610
33.80/0.9195
32.27/0.9007
32.53/0.9320
Scratch
34.39/0.9270
30.32/0.8417
29.08/0.88 29.08/0.8046
27.99/0.8489
KD
34.43/0.9273
30.34/0.8422
29.10/0.8055
28.00/0.CC 28.00/0.8491
FitNet
33.35/0.9178
29.7110.8323
28.62/0. 7949
26.61/0.8166
AT
34.29/0.9262
30.26/0.8406
29.03/0.803 8035
27.76/0.8443
x3
RKD
34.43/0.9274
30.33/0.8423
29.09/0.805T
27.96/0.849 8493
FAKD
34.39/0.9272
30.34/0.8426
29. 29.10/0.8052 8052
28.07/0.8511
CSD*
34.45/0.9275
30.32/0.8430
29.11/0.8061
28.21/0.8544
AugKD
34.59/0.9287
30.47/0.8448
29.20/0.8073
28.44/0.8878
Scratch
32.29/0.8965
28.68/0.7840
27.64/0.7380
26.21/0.7893
KD
32.30/0.896 8965
28. 15.7000.742
27.64/0.7382
78.997
FitNet
31.65/0.8877
28.33/0.7768
27.38/0.7309
25.40/0.763
AT
32.22/0.8955
08.63/0.7825
27.59/0.7365
25.97/0. 7825
x4
RKD
02.30/0.8966
28.69/0.784 7842
27.64/0.7383
26.200007799
FAKD*
01.77/0.890
28 28.65/0. 7836
27.62/0.7377
26. 7895
CSD
32.34/0 8974
28.72/0.7856
27.68/0.7368
26.34/0.7948 7948
AugKD
32.47/0.8981
28.80/0.7866
27.71/0.7403
26.45/0.7963
4 EXPERIMENTS
4.1 EXPERIMENT SETUPS
Backbones and Baselines. We use EDSR (Lim et al. 2017). RCAN Zhang et al. 2018), and SwinIR (Liang et al., 2021) as backbone models to evaluate AugKD and compare with existing KD methods. The specifications of the teacher and student networks, along with statistics such as FLOPs, number parameters, and inference speed (FPS), are shown in Table T We compare AugKD with the following baseline methods: training from scratch, response- based KD (Hinton Komodakis 2016 RKD (Park et al. 2019 FAKD (He et 2020)) CSD (Wang et al. 2021c), and CrossKD (Fang al.) 2023). While FitNet, AT, and RKD were originally designed for high-level CV tasks, they are also applicable to SR models. However, CSD. being self-distillation method, is not suitable for distilling RCAN (depth compression) and SwinIR (transformer- based) models. Performance is evaluated using peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) the Y channel of the YCbCr color space. Detailed training settings are provided in Appendix A.1
al..._00 2015)), FitNet (Romero al. 2014), AT (Zagoruyko
4.2 RESULTS AND COMPARISON
Comparison with Baseline Methods. The quantitative results (PSNR / SSIM) for training EDSR, RCAN, and SwinIR networks are presented in Table 2 and [T] respectively. for x2, x3, and x4 scales. The following conclusions can be drawn from these results: (1) Existing KD methods have limited benefits and some even result in student models worse than those trained without KD. For instance, EDSR models trained with FAKD sometimes underperform the ones trained from scratch. It showcases that the "dark knowledge" camnot be directly simply transferred to the student SR model. (2) The KD methods initially designed for high-level CV tasks (FitNet, AT, RKD), while applicable, hardly improve the SR models over training from scratch. It's caused by the intrinsic difference
7

---

Published as a conference paper at ICLR 2025
Table 3:
Quantitative comparison between AugKD and other distillation methods for RCAN. The best and second- performances are highlighted in bold and underlined, respectively.
Set5
Set14
BSD100
Urban100
Scale
Method
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
Scratch
38.13/0.9610
33.78/0.9194
32.26/0.900
32.63/0.9327
KD
38.18/0.9611
33.83/0.9197
32.29/0.9010
32.67/0.9329
FitNet
37.97/0.9602
33.57/0.9174
32.19/0.899 9999
32.06/0.9279
x2
AT
38.13/0.9610
33.70/0.9187
32.25/0.9000 9005
32.48/0.9313
RKD
38.18/0.9622
33.78/0.9191
32.29/0.9011
32.70/0.9330
FAKD"
38.17/0.9612
33.83/0.9199
32.29/0.900
32.65/0.9330
CrossKD
38.18/0.9612
33.81/0.9194
32.30/0. 9011
32.66/0.9332
AugKD
38.23/0.9614
33.90/0.9201
32.33/0.9016
32.87/0.9349
Scratch
34.61/0.9288
30.45/0.8444
29.18/0. 18/0.8074
28.59/0.80 3910
KD
34.61/0.9291
30.47/0.8447
29.21/0.8808
28.62/0.8811
FitNel
34.21/0.9248
20,2000.8399
29.05/0. 8044
27.89/0.88 8472
x3
AT
34.55/0.9287
30.43/0.8438
28 43/0.87777
RKD
34.67/0.9292
30.48/0.8451
29.21/0. 8080
28.60/0.8610
FAKD"
34.63/0.9290
30.51/0.8453
29.21/0.880 8079
28. 28.62/0.8612
CrossKD
34.65/0.9299 9299
50/50/0.8499
29.21/0.8077
28. 60/0.8610
AugKD
34.74/0.9296
30.54/0.8458
29.25/0.800 80.88
28.79/0.8646
Scratch
32.31/0.8966
28. 13.6990.7884 7842
27.64/0.7338
26.37/0. 7949
KD
22.45/0.8980
28. 76/0. 7860
27.67/0.
26.49/0.
FitNet
31.99/0.8899
28. 7789
27.55/0. 7353
25.90/0.779 7791
x4
AT
32.31/0.8967
28.69/0. 7839
27.64/0. 7385
26.29/0. 7927
RKD
32.39/0.8974
28. 74/0 7856
27.67/0. 7399
26.47/0.7988
FAKD"
32.46/0.8980
08.77/0.7860
27.68/0.7408
36.50/0.7980
CrossKD
32.46/0.8984
28. 18.799007866
27.69/0. 7405
26.52/0.7999
AugKD
32.56/0.8990
28.83/0.7870
27.72/0.7410
26.62/0.8020
between SR and other CV tasks. (3) The proposed AugKD method consistently outperforms the existing KD methods in all experimental settings by a large margin. For example, compared with the response-based KD method, the average PSNR improvements for the three types of networks on the Urban
100 100 test set over three SR scales are 0.43 dB, 0.31 dB, 0.31 dB, respectively. Most existed KD methods are inapplicable to the transformer architecture network, but AugKD, as a response- based KD method, is able compress the SwinIR model while exhibiting great performance.
AugKD facilitates the student model to mimic the teacher model. In Figure 2] we compare the effect of different KD methods by comparing the similarity of students' output and teacher's 's on the training and Urban 100 testing sets, to evaluate how well the student learns to mimic the teacher model. It shows that AugKD makes the student not only effectively fit the teacher model the training set but also imitate it on the test sets so that it can generalize better.
Table 4: The results of heterogeneous distillation using AugKD on the x4 scale RCAN model.
Table 5: NIQE scores on several real-world SR test- ing datasets. The lower, the better. Visual compar- isons are provided in the appendix.
BSD100
Urban
Method
#Params
RealSR
DRealSR
OST300
Teacher
PSNR/SSIM
PSNR/SSIM
Scratch
11.9M
4.771
4.847
2.932
(Scratch)
27.64/0.7384
26.37/0.7949
Scratch KD
5.810
5.757
3.788
EDSR
27.71/0.7406
26.59/0.8014
1.24M
5,425
5.408
3.652
SwinIR
27.72/0.7408
26.59/0.8007
AugKD
5.398
5.378
3.494
Experiment Results on Heterogeneous Settings. We extend the experiments to heterogeneous settings where the teacher and student models have different network architectures, presented in T Conventional feature- based KD or self-distillation methods are inapplicable to the cross- architecture setting, while AugKD can still effectively improve the student models. For instance,
8

---

Published as a conference paper at ICLR 2025
(I:A)
HURUPSNNA
Scrmch
KLD (25A8)
FARD LSS
CSP (TTORA
FAKD 125.39
CSD 12062
NarKD 26,61
SCOURILIN
Figure 5: The x4 SR examples of EDSR models on img004, img019, img089 and img096 from Urban100. PSNRs (dB) of the cropped regions are annotated below each image.
compared to the RCAN model trained from scratch, utilizing AugKD with an EDSR or SwinIR teacher model yields an increase in PSNR by 0.22dB at x4 scale on Urban100 test set.
Visual Comparison. In Figure 5 we compare the visual quality of output images of the x4 EDSR model trained by AugKD and other methods. To underscore the differences in detailed pattern and texture reconstruction, we took relatively small cropped portions and computed local PSNR metrics. Generally, higher PSNR aligns with superior visual effect. For the reconstruction of textures (e.g. lines, edges, and complex patterns), the model trained with AugKD yields outputs that are both sharper and more similar HR, indicating the superiority of AugKD.
Experiment Results on Real-world SR task.
To evaluate the performance of Aug in real-world SR, we continue training the PSNR-oriented student SwinIR models at x4 scale using the BSRGAN degradation model (Zhang et al.) 2021a Liang
al.
202T> on the DF2K dataset. The models are tested on three benchmark datasets: RealSR (Cai
et al. 2019), DRealSR (Wei et al.) 22020), and OST300 (Wang et al.) 2018). The non-reference image quality assessment (NIQE) scores (Mittal al. 2012 are presented in Table The model trained with AugKD achieves lower NIQE scores and produces output images with more visually pleasing results, as shown in the supplementary material.
4.3 ABLATION ANALYSIS
Impact of auxiliary distillation samples and label consistency regularization. Table 6 shows the effect of the presented two modules, using EDSR baseline model (#Channel=64, #Block=16) distilled by our student model (#Channel=64, #Block=32). Further, Table 7 ablates the zoom-in e and zoom- out e operations. The result shows that adopting auxiliary distillation samples and label consistency regularization could lead significant performance improvement upon logits-KI -DD, whether used separately together. For example, simply introduce the auxiliary images by zoom-in draws ddB PSNR increment on Urban 100 test set, and adding zoom-out and label consistency regularization yields additional 0. 16dB improvement.
Integrate AugKD into other model compression methods. We integrate AugKD with a SOTA SR network quantization method, Distribution-Aaan Quantization (DAQ) (Hong et al. 2022). and use the full- full-precision model to supervise the quantized ones. Figure 6 shows the PSNR of quantized x4 scale EDSR baseline models trained with and without KD, and the full results provided in supplementary. It shows that vanilla Logits- KD has barely effects on the quantization, while AugKD could improve the quantized model by large margin. We also integrate AugKD with the FAKD

---

Published as a conference paper at ICLR 2025
Table 6: Ablation study of auxiliary distillation samples and label consistency regularization.
Table Ablation study of the zoom in and zoom out operations.
Auxiliary samples
Label
Urban100
Zoom
Zoom
Urban
consistency
PSNR/SSIM
8
9
PSNR SSIM
X
24.87 0.7433
X
X
24.87 0.7431
V
x
25.18 / 0.7551
25.20 0.7558
X
25.18 0.7552
/
25.34 / 0.7609
/
/
25.20 0.7558
method in Table 8 The resulting models outperform the ones trained by FAKD greatly. The results indicate that A can be effectively aggregated with other model compression techniques.
Table 8: Experiment results of combin- ing AugKD and FAKD.
quantieed baseline
Urban100
24.0-
Model
Method
PSNR SSIM
24.
34.75
2432 24.72
Logits KD
26.21 0.7899
24.61
FAKD
26.18 7.7895
EDSR
FAKD+AugKD
26.30 7979
A262
AugKD
26.45 0,7966
Figure 6: PSNR of quantized bascline EDSR
model trained with and without KD.
Table 9: Comparison with data expansion. DF2K denotes DIV2K=Flickr2K.
Training set
BSD100
Urban100
#Images
Training steps
Method
PSNR/SSIM
PSNR/SSIM
Scratch 27.57/0.7356 25. 25.94/0.7809
DIV2K
800
2.5 x 10
AugKD 27.68/0.7390
26.32/0.7927
Scratch 27.62/0.7372 26. 15/077077872
DF2K
3450
5 105
KD 27.67/0.7390 26.31/0.799
Comparison with data expansion The proposed AugKD generates auxiliary distillation samples by simple data augmentations. Comparing with using a parametric generator or introducing additional training image data sources, it's more efficient and able keep the training data have similar inputs. Table compares AugKD with training or distilling with expanded data. We train the x4 scale EDSR models on a much larger dataset (DF2K: DIV2K (Timofte et al. 2017) 7 with 3450 images). The number of iterations is doubled for the larger training set since the previous configuration (2.5 x 109) is insufficient for the models to converge. Except that the x4 SR networks are not initialized with the x2 ones in this comparison, the other settings of the training recipe are the same. AugKD is superior to training with more input data terms of both efficiency and performance.
5 CONCLUSION
In this work, we investigated the issues existing in KD for SR networks. Motivated by the findings, we present AugKD, simple yet significant KD framework for SR, which outperforms existing methods and is applicable to a wide array of network architecture and SR tasks. Central to our approach is the auxiliary distillation samples generated by zooming augmentations, which facilitates the knowledge transfer from teacher to student model. Besides, we realize label consistency regularization in KD for SR, which further bolsters the student model's generalization capabilities. Extensive experiments conducted across various SR tasks, benchmark datasets and diverse network backbones, consistently showing the out-performance of AugKD and endorsing its robust and effective.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies Computing Systems (JETC), 13(3):1-18, 2017.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based nonnegative neighbor embedding. BMVC, 2012.
Jianrui Cai, Hui Zeng, Hongwei Yong. Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolttion: A new benchmark and a new model. In ICCV, pp. 3086-3095, 2019.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. NeurIPS, 30, 2017.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional network for image super-resolution. In ECCV, pp. 184-199. Springer, 2014.
Erik Englesson and Hossein Azizpour. Consistency regularization improve robustness to label noise. arXiv:21100012422 2021.
Hangxiang Fang, Yongwen Long, Xinyi Hu, Yangtao Ou, Yuanjia Huang, and Haoji Hu. Dual cross knowledge distillation for image super-resolution. Journal of Visual Communication and Image Representation, 95: 9038088 2023.
Qinquan Gao, Yan Zhao, Gen Li, and Tong Tong. Image super-resolution using knowledge distillation. In ACCV, pp. 527-541. Springer, 2019.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 129, 2021.
Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In CVPR, 2021.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In ECCV, pp. 544-560. Springer, 2020.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. ICML., pp. 1737-1746. PMLR, 2015.
Hrayr Harutyunyan, Ankit Singh Rawat, Aditya Krishna Menon, Seungyeon Kim, and Sanjiv Kumar. Supervision complexity and its role knowledge distillation. 12X1v:2301.12245, 2023.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, pp. 770-778. 2016.
Zibin He, Tao Dai, Jian Lu, Yong Jiang, and Shu-Tao Xia. Fakd: Feature-affinity based knowledge distillation for efficient image super-resolution. In ICIP, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. 15X3:05503.253/ 2015.
Cheeun Hong, Heewon Kim, Sungyong Baik, Junghun Oh, and Kyoung Mu Lee. Daq: Channel-wise distribution-aware quantization for deep image super-resolution networks. In ICCV, pp. 2675-2684, 2022.
Andrew Howard, Mark Sandler, Grace Chu, Liang- Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Searching for mobilenetv3. In ICCV, pp. 1314-1324, 2019.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, pp. 5197-5206. 2015.
11

---

Published as a conference paper at ICLR 2025
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. NewrIPS, 29, 2016.
Zheng Hui, Xinbo Gao, Yunchu Yang. and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACMMM, 2019.
Andrey Ignatov, Radu Timofte, Maurizio Denna, and Abdel Younes. Real-time quantized image
super-resolution on mobile npus, mobile ai 2021 challenge: Report. In CVPR, pp. 2525-2534, 2021.
Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak. Consistency-based semi-supervised learning for object detection. NeurlPS, 32, 2019.
Jangho Kim, SeongUk Park, and Nojun Kwak. Paraphrasing complex network: Network compression via factor transfer. NewrIPS, 31, 2018.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, pp. 1646-1654, 2016.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv:1412.6900 2014.
Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged informa- tion for efficient image super-resolution. In ECCV, 2020.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCV, 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPR, 2017.
Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang. Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In ICCV, pp. 3296-3305, 2019.
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, volume 2. IEEE, 2001.
Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Making "completely blind" image quality analyzer. IEFE Signal processing letters, 20(3):209-11 2012.
Avital Oliver, Augustus Odena, Colin A Raffel, Ekin Dogus Cubuk, and Ian Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. NeurIPS, 31, 2018.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
Nikolaos Passalis, Maria Tzelepi, and Anastasios Tefas. Heterogeneous knowledge distillation using information flow modeling. In CVPR, pp. 2339-2348, 2020.
Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh, and Qun Liu. Alp-kd: Attention-based layer projection for knowledge distillation. In AAAI, volume 35, pp. 13657-13665, 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and
Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv:1412.633 2014.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv:1910.01108 2019.
Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? NeurIPS, 34:6906-6919, 2021.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation. arXiv:2002.03553 2020.
12

---

Published as a conference paper at ICLR 2025
Radu Timofte, Eirikur Agustsson, Lue Van Gool, Ming- Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPRW, pp. 114-125, 2017.
Frederick Tung and Greg Mori. 2019.
knowledge distillation. In InCCV, pp. 1365-1374,
Jiyue Wang, Pei Zhang, Qianhua Hc, Yanxiong Li, and Yongjian Hu. Revisiting label smoothing regularization with knowledge distillation. Applied Sciences, 11(10), 2021a.
Longguang Wang, Xiaoyu Dong, Yingqian Wang, Xinyi Ying, Zaiping Lin, Wei An, and Yulan Guo. Exploring sparsity in image super-resolution for efficient inference. In CVPR, pp. 4917-4926, 2021b.
Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Recovering realistic texture in image super-resolution by deep spatial feature transform. In CVPR, pp. 606 615, 2018.
Xintao Wang, Liangbin Xie, Ke Yu, Kelvin C.K. Chan, Chen Change Loy, and Chao Dong. BasicSR: Open source image and video restoration toolbox. httpps://ithhb com FwtomXXPiiee wwt.m//xxxiirru/////// BasicsR 2022a.
Yanbo Wang, Shaohui Lin, Yanyun Qu, Haiyan Wu, Zhizhong Zhang, Yuan Xie, and Angela Yao. Towards compact single image super-resolution via contrastive self-distillation. arXiv:2105.1168
2021c.
Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: general u-shaped transformer for image restoration. In CVPR, pp. 17683-17693, 2022b.
Pengxu Wei, Ziwei Xie, Hannan Lu, Zongyuan Zhan, Qixiang Ye, Wangmeng Zuo, and Liang Lin. Component divide-and-cn and-cuuu for real-world image super-resolution. In ECCV, pp. 101-117. Springer, 2020.
Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-awan efficient convnet design via differentiable neural architecture search. In CVPR, pp. 10734-10742, 2019.
Jiaxiang Wu, Cong Leng. Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In CVPR, pp. 4820-4828, 2016.
Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo. Learning texture transformer network for image super-resolution. In CVPR, pp. 5791-5800, 2020.
Gengqi Yao, Zhan Li, Bir Bhanu, Zhiqing Kang. Ziyi Zhong, and Qingfeng Zhang. Mtkdsr: Multi- teacher knowledge distillation for super resolution image reconstruction. 2022 26th ICPR, pp. 352-358, 2022a.
Gengqi Yao, Zhan Li, Bir Bhanu, Zhiqing Kang, Ziyi Zhong, and Qingfeng Zhang. Mtkdsr; Multi- teacher knowledge distillation for super resolution image reconstruction. In ICPR, pp. 352-358. IEEE, 2022b.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017.
Shan You, Chang Xu, Chao Xu, and Dacheng Tao. Learning from multiple teacher networks. In SIGKDD, pp. 1285-1294, 2017.
Sergey Zagoruyko and Nikos Komodakis. Paying attention attention: Improving the performance of convolutional neural networks via attention transfer. arXiv:1612.03998, 2016.
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming- Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR, pp.
5728-5739, 2022.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. In Curves and Surfaces: 7th International Conference, 2012.
13

---

Published as a conference paper at ICLR 2025
Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In ICCV, pp. 4791-4800, 2021a.
Quanshi Zhang, Xu Cheng, Yilan Chen, and Zhefan Rao. Quantifying the knowledge in dnn to explain knowledge distillation for classification. PAMI, 45(4), 2022.
Yiman Zhang, Hanting Chen, Xinghao Chen, Yiping Deng. Chunjing Xu, and Yunhe Wang. Data-free knowledge distillation for image super-resolution. In CVPR, pp. 7852-7861, 2021b.
Yulun Zhang, Kunpeng Li. Kai Li, Lichen Wang. Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV, pp. 286-301, 2018.
Yulun Zhang, Huan Wang, Can Qin, and Yun Fu, Aligned structured sparsity learning for efficient image super-resolution. NeurIPS, 34:2695-2706, 2021c
Borui Zhao, Quan Cui, Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In CVPR, pp. 11953-11 11962, 2022.
14

---

Published as a conference paper at ICLR 2025
A SUPPLYMENTARY EXPERIMENT RESULTS
A.1 1 TRAINING DETAILS
The SR models are trained using 800 images from the DIV2K dataset (Timofte et al. 2017) and evaluated on four benchmark datasets: Set5 (Bevilacqua et al., 2012). Set14 (Zeyde et al.) 2012) BSDI00 (Martin et al. 2001), and 1r0 (Huang et al. 2015). The low-resolution (LR) images used for training are generated by down-sampling the high- resolution (HR) images using bicubic degradation. The x4 scale SR models are initialized from the corresponding x2 scale models. During training, the input LR images randomly cropped into 48 48 patches and augmented with random horizontal and vertical flips and rotations. For the FAKD and CSD methods, we follow the hyperparameter settings specified in the original papers and train the models ourselves checkpoints are not provided, as noted in the results table. The zoom-in e operation for AugKD is performed by randomly cropping for efficiency. The zoom-out f operation is skipped during training for SwinIR, as the I.... would be too small to serve as valid input to the model. The models are trained using the Adam optimizer (Kingma& & Ba 2014) with 31 = 0.9, B2 = 0.99, and e = 10-8, with a batch size of 16 and a total of 2.5 x 102 updates. The initial learning rate is set to 10- and decays by factor of 10 every 10t iterations. The proposed KD method is implemented using the BasicSR framework (Wang ct al. 2022a) and PyTorch 1.10, with training performed 4 NVIDIA V100 GPUs.
A.2 EXPERIMENT RESULTS OF SWINIR NETWORK
We compare AugKD with other applicable KD methods on distilling transformer-basse SR model, SwinIR. The result shows the superiority and universality of AugKD.
Table 10: Quantitative comparison (average PSNR/SSIM) between AugKD and other applicable distillation methods for SwinIR of three SR scales. Best performance is highlighted in bold.
Set5
Set14
BSD100
Urban 100
Scale
Method
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
Scratch
18.0110.9607
33.57/0.9178
32.19/0.900 9000
32.05/0.9279
x2
KD
1..04/0.96008
33.61/0.9184
32.22/0. 90033
32.09/0.9282
AugKD
38.13/0.9610
33.78/0.9194
32.26/0.9007
32.63/0.9327
Scratch
34.41/0.9273
30.43/0.8437
29.12/0.88 8062
28.20/0.8537
x3
KD
34.44/0.9277 9255
30.45/0.8443
29.14/0. 8066
28.23/0.855 8545
AugKD
34.55/0.9285
30.53/0.8456
29.20/0.8080
28.53/0.8604
Scratch
32.31/0.8955
28.67/0.7833
27.61/0.7379
26.15/0.7884
x4
KD
32.27/0.895.
28.67/0.7833
27.62/0.
75/0
AugKD
32.41/0.8973
28.79/0.7860
27.69/0.7405
26.43/0.7972
A.3 COMPARISON OF TRAINING COSTS
As shown in Table II AugKD outperforms Logits-KD by 0.55dB PSNR with an increase of only 0.21s training time per step. Considering the significant performance gains from AugKD, the extra cost on training time is mild and acceptable.
Table 11: Training expenses of KD methods for x2 SR on EDSR.
KD methods
KD
FitNet
FAKD
CSD
AugKD
Time (s/step)
0.49
0.56
0.56
1.18
0.70
Urban 100 PSNR
31.98
30.46
32.04
32.26
32.53
A.4 TEACHER MODELS' RESULTS
In this work, EDSR, RCAN, and SwinIR backbone models for the experiments. Their specifications and statistics are provided in Table We use the public checkpoints of these teacher models for distilling the student models, with quantitative results summarized in Table T2
15

---

Published as a conference paper at ICLR 2025
Table 12: Quantitative results of teacher models
Sct5
Sct14
BSD100
Urban 100
Scale
Model
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
PSNR/SSIM
EDSR
38.20/0. 9610
34.02/0.9204
32.37/0.9018
23.10/0.9363
x2
RCAN
38.27/0.9614
34.13/0.9216
32.39/0.9024
33.18/0.9377
SwinIR
98.36/0.9620
34.14/0.9227
32.45/0.9030
33.40/0.9394
EDSR
34.76/0.9299
30.66/0.844 8481
29.32/0.8 19.32/0.8104
29.02/0.C 8685
x3
RCAN
$4.74/0.9299
30,65/0.845 8852
29.32/0.8111
29.09/0.87 8700
SwinIR
34.89/0,9312
30,77/0.8503
29.37/0.8124
29. 19.22/0.8744.
EDSR
32.65/0.
28.95/0. 7903
27.81/0. 7440
26.87/0.808 8086
x4
RCAN
32.64/0.9000
28.85/0.7890
37.74/0.7430
36.75/0.800 8070
SwinIR
32.72/0.9021
98/9.4077914
27 27.83/0.7459
27 27.07/0.8 164
B MORE VISUAL RESULTS
In Figure 7 we present additional visual comparisons of AugKD with other KD methods on Urban AugKI) restores more structural details and reduces blurring artifacts. The AugKD evaluated on the real-world SR task in Table where it outperformed the baselines across several testing datasets. Figure we show visual comparisons for real-world SR, further highlighting its superior performance.
HRUSNKI
Sernchh (17
KDT117.51)
HREESNII
Scratch 222.46)
KDr22.0 471
1745
AugED
InguU33
AugED (23.64)
(17.84)
Imglas
CSD
ArgRD (TR.40)
(2064)
Aug (244))
Figure 7: The x4 super resolution results of EDSR models on image 033, 078, 058 and 024 from Urban PSNRs (dB) of the cropped regions are annotated below each image. Zoom in e for the best view.
16

---

Published as a conference paper at ICLR 2025
Imput
Teaeher
Sonatch
Inpur
Teacher
Scrarch
112
K11
AugKD
AugKD
Input
Teacher
Scrttch
Input
Tencher
Seratch
5
4
KD
AugKD
AugKD
11 12
10
Figure 8: Visual comparisons of representative real-world SR images at x4 SR scale. AugKI> outperforms other methods in artifact removal and detail restoration, producing outputs more similar to the teacher model. Zoom in 2 for optimal viewing.
17


Reviews:
[{"review_id": "fwtrEUKSnS", "reviewer": "Reviewer_bsJu", "summary": "This paper proposes a new method for knowledge distillation for image super-resolution. The authors propose using auxiliary training samples by zoom in and zoom out operations on the training images, and apply label consistency regularization by data augmentation and inverse augmentation.", "questions": "For the zoom out operation, we have the ground truth for I_LR_zo, and given the analysis in Section 3.2, the teacher model output for I_LR_zo would be shaded by the ground truth. So there seems to be additional complexity for this path to go through the teacher model.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The motivation makes sense that for image super-resolution knowledge distillation, the guidance of the teacher model is shaded by the ground truth. So the authors propose a specific knowledge distillation training paradigm for image super-resolution.\nExperiments show that the proposed method surpasses scratch training and 7 baseline knowledge distillation methods.\nThe ablation studies verifies that the proposed auxiliary distillation samples and label consistency regularization improve student model performance.", "weaknesses": "The question answered by the paper is not a major one, as it is a knowledge distillation method specifically for the image super-resolution task. Does it also apply to other low-level tasks?\nThe image super-resolution models used for experiments are not state-of-the-art. EDSR is from 2017 and RCAN is from 2018. SwinIR is newer from 2021 but only \"Scratch\" and \"KD\" is compared with the proposed method for SwinIR. As the proposed method is claimed to be model-agnostic, it is supposed to be applied to more advanced models to demonstrate the effectiveness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DrLKeLU5gy", "reviewer": "Reviewer_PKbV", "summary": "The paper proposes AugKD, a new knowledge distillation method for image super-resolution. AugKD contains two special designs, auxiliary\ndistillation sample generation, and label consistency regularization. By comparing with other distillation counterparts on four benchmark datasets, AugKD shows its superiority.", "questions": "See Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper has a deep insight into super-resolution tasks and its method is simple and effective.\n2. The code is also provided for reproduction.", "weaknesses": "1. **The paper writing could be further improved.** For instance, the authors claim the motivation - *'the teacher's output contains barely extra information exceeding GT, thus the “dark knowledge” of the teacher being hardly transferred to the student model through KD'* in Line #76-78 in the Introduction part. However, in Section 3.2 (Motivation) and Figure 2, the authors show the motivation by measuring the PSNR of outputs between the teacher and the student. It seems that the two statements are a little bit contradictory, as the former one indicates that the teacher's outputs can not be good learning materials but the second one leverages the the teacher's outputs as the reference for evaluating whether KD method is good or not. Such circumstances make it hard to understand the central idea of the paper.\n\n2. **The paper lacks further deep analysis of where the performance gains are from.** From the results in Figure 2, it seems that the gains are from improving the fidelity between the teacher and the student. A further question is *Why AugKD can improve the fidelity?* And in Lines #521-529, the authors compare AugKD with data expansion. Thus, a question arises *Does the improvement of fidelity from the expansion of the training set by augmentation?* From another perspective, the question is *how does the augmentation strength affect the fidelity and the final distillation results?* By answering such a series of questions, the paper can help the readers understand the intrinsic mechanism of AugKD.\n\n3. **A minor question about the design of the method.** Although I think the design of the inverse augmentation is clear and plausible, I'm still curious about what would happen if we dropped the inverse augmentation and added the augmentation at the end of the teacher's model in the training stage and still utilized the same architecture as the method in the inference stage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SMru8ijZs3", "reviewer": "Reviewer_oSVc", "summary": "The paper explores an improved augmentation strategy for knowledge distillation (KD) specifically in image super-resolution (SR). It introduces AugKD, which uses unpaired data augmentations to create auxiliary distillation samples and enforce label consistency. This approach addresses limitations in traditional KD methods by enhancing the student model’s learning process through diverse training samples, aiming to improve efficiency and effectiveness in SR tasks.", "questions": "- Please see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- **Motivation**: The paper provides a clear motivation for improving data augmentation strategies in knowledge distillation (KD) for super-resolution (SR), highlighting the unique challenges in SR tasks.\n- **Comprehensive Ablations**: Extensive ablation studies test various experimental setups for KD in SR, demonstrating a thorough examination of the method's effectiveness under different conditions.", "weaknesses": "- **Modest Improvements**: Results in Figure 2 and Tables 2–4 show only slight gains, questioning the practical value of AugKD over existing methods.\n- **Limited Insight on Augmentation Impact**: Ablation studies explain augmentation effects but don’t clarify *why* this strategy improves KD. Further detail on what specific features AugKD captures would be helpful.\n- **Augmentation Benefits Unclear**: It’s unclear how augmentations help representations learned through KD or why prior methods failed to capture these.\n- **Potential Architecture Constraints**: While AugKD is claimed to be generalizable, performance with some architectures (like SwinIR) suggests possible limitations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "inFi6Vclrr", "reviewer": "Reviewer_HXWn", "summary": "This paper introduces AugKD, a novel method aimed at improving image super-resolution (SR) by leveraging data augmentations to generate auxiliary distillation samples and enforce consistency regularization. This work analyzes the mechanisms of KD for SR and propose AugKD adapted to the unique task with label consistency regularization. Extensive experiments on various SR tasks are presented across multiple datasets to validate the proposed approach.", "questions": "1. How does AugKD affect training efficiency in comparison to other KD techniques?\n2. What's the rationale behind the specific choice of zoom-in and zoom-out augmentations for AugKD?\n3. Given that some other image augmentations, such as translation, are also invertible, would it be beneficial to include them in the consistency regularization module?\n4. What is the motivation of using label consistency regularization for SR? Is it also also suitable for other low-level tasks?\n5. Does the proposed method also adapt to other backbones, like Mamba?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper thoroughly analyzes the mechanics and distinct challenges of knowledge distillation (KD) in the context of SR, proposing the use of data augmentations to enhance distillation.\n2. The AugKD strategy is adaptable to different SR models and tasks, yielding substantial performance improvements across several networks and settings.\n3. The well-organized structure and clearly described method facilitates reproducibility.", "weaknesses": "1. The visualization in Fig. 2 is unclear. Replacing it with a bar plot may improve readability and convey the idea more effectively.\n2. Although lines 241-244 highlight that the adaptive selection of zoom-in samples is ineffective, it lacks sufficient experiments to support this claim. \n3. The motivation by using label consistency regularization is unclear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "i232S0A82D", "reviewer": "Reviewer_3UKt", "summary": "This paper proposes AugKD, an innovative knowledge distillation (KD) technique specifically designed for image super-resolution (SR). AugKD incorporates zooming augmentations and label consistency regularization to enhance the training process. In this approach, both randomly cropped high-resolution (HR) patches and their down-sampled low-resolution (LR) counterparts are fed into a pre-trainedd teacher model to generate target labels. These labels are then used to guide the training of the student model. To further improve the robustness and generalization of the student model, consistency regularization is applied through a series of invertible data augmentations. Extensive experiments have been conducted across multiple public image super-resolution datasets, demonstrating the effectiveness and versatility of the proposed method.", "questions": "1. Referred to Tab.9 in the paper, could you explain the reason why the performance of the combination of FAKD and AugKD is lower than AugKD?\n2. The performance of AugKD on the X4 scale RCAN model is presented in Tab.3, why is it better than the result of heterogeneous distillation in Tab.5?\n3. Does $L_{dukd}$ in Fig. 1 indicate the same with the $L_{augkd}$ computed in Fig.4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. A novel KD method is proposed in this paper. AugKD improves the training process by using zooming augmentations and label consistency regularization. To make the student model more robust and versatile, consistency regularization is applied using a series of invertible data augmentations. Extensive quantitative experiments and qualitative analysis are provided to demonstrate the validity of the methodology\n2. Compared with previous methods, the performance is improved on models with multiple scales. For instance, compared with training from scratch, the performance of RCAN on x4 scale is improved by 0.25dB on Urban dataset.\n3. AugKD is general and effective, easy to follow, and convenient for reproducing the method.\n4. Paper is well written and organized.", "weaknesses": "1. This paper proposed multiple effective improvements, while I'm curious that, besides zooming augmentations, could other data augmentation methods improve performance?\n2. The ablation of the label consistency is not sufficient. Have the authors tried other non-invertible ways of regularization?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)