QuantSR: Accurate Low-bit Quantization for Efficient Image Super-Resolution
Authors: Haotong Qin
Yulun Zhang
Yifu Ding
Yifan liu
Xianglong Liu
Martin Danelljan
Fisher Yu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Super Resolution
Model Quantization
Deep Learning
TL;DR: We propose a novel quantized image SR network, called QuantSR, which achieves accurate and efficient SR processing under low-bit quantization.

Abstract: Low-bit quantization in image super-resolution (SR) has attracted copious attention in recent research due to its ability to reduce parameters and operations significantly. However, many quantized SR models suffer from accuracy degradation compared to their full-precision counterparts, especially at ultra-low bit widths (2-4 bits), limiting their practical applications. To address this issue, we propose a novel quantized image SR network, called QuantSR, which achieves accurate and efficient SR processing under low-bit quantization. To overcome the representation homogeneity caused by quantization in the network, we introduce the Redistribution-driven Learnable Quantizer (RLQ). This is accomplished through an inference-agnostic efficient redistribution design, which adds additional information in both forward and backward passes to improve the representation ability of quantized networks. Furthermore, to achieve flexible inference and break the upper limit of accuracy, we propose the Depth-dynamic Quantized Architecture (DQA). Our DQA allows for the trade-off between efficiency and accuracy during inference through weight sharing. Our comprehensive experiments show that QuantSR outperforms existing state-of-the-art quantized SR networks in terms of accuracy while also providing more competitive computational efficiency. In addition, we demonstrate the scheme's satisfactory architecture generality by providing QuantSR-C and QuantSR-T for both convolution and Transformer versions, respectively. Our code and models are released at https://github.com/htqin/QuantSR .

QuantSR: Accurate Low-bit Quantization for Efficient Image Super-Resolution
Haotong Qin Yulun Zhang Yifu Ding', Yifan Liu?,
Xianglong Liu Martin Danelljan", Fisher Yu²
Beihang University 2ETH Zürich
Abstract
Low-bit quantization in image super-resolution (SR) has attracted copious attention in recent research due its ability to reduce parameters and operations signifi- cantly. However, many quantized SR models suffer from accuracy degradation compared to their full-precision counterparts, especially ultra-low bit widths (2-4 bits), limiting their practical applications. To address this issue, we propose novel quantized image SR network, called QuantSR, which achieves accurate and efficient SR processing under low-bit quantization. To overcome the repre- sentation homogeneity caused by quantization in the network, we introduce the Redistribution-driin Learnable Quantizer (RLQ). This is accomplished through an inference-agnosee efficient redistribution design, which adds additional informa- tion in both forward and backward passes to improve the representation ability of quantized networks. Furthermore, to achieve flexible inference and break the upper propose the Depth- dynamicc Quantized Architecture (DQA). Our DQA allows for the trade-off between efficiency and accuracy during inference through weight sharing. Our comprehensive experiments show that QuantSR out- performs existing state-of-the-art quantized SR networks in terms of accuracy while also providing more competitive computational efficiency. In addition, we demon- strate the scheme's ' satisfactory architecture generality by providing QuantSR-C and QuantSR-T for both convolution and Transformer versions, respectively. Our code and models are released at https://github..e. httpggiiub.o///gggnuuuuattteeeee
limit of accuracy,
Introduction
Single image super-resolution (SR) is the task of obtaining high-resolution (HR) version of low-resolution (LR) image by retrieving high-frequency details. It is an ill-posed problem since there are multiple HR candidates for the same LR input. To address this issue, researchers have explored the use of deep neural networks, including convolutional neural networks (CNNs) and Transformers [18, 26, 38, 41, 25, 3, 39, 43), to achieve high-quality reconstruction. However, existing SR models rely on expensive computational resources, which significantly limits the real-world SR applications on resource-constrainee
edge devices. Therefore, there is an urgent requirement to develop model compression techniques for SR models reduce the computational overhead.
Model quantization [5, 14, 24, 8] has emerged as powerful compression technique that compresses weights and activations computing units (such convolutional and linear layers) into low-bit representations. This results in heavy floating-poin operations being converted into efficient integer ones, making quantization highly desirable for edge devices and friendly for memristor-basee hardware. Meanwhile, the quantization process primarily focuses on parameter compression within operations. is considered architecture- and task-agnostic, making it popular among the deep learning
Equal contribution
e Corresponding authors: Yulun Zhang and Xianglong Liu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

HR/ Bit-width
DoReFa/ /-orai/4-iit 4-bit
PAMS/ /4-bit
CADyQ/ 4-bit
QuanSSR-C/HH /4-bit
Urban100: img_017 (x:4)
SRRReNeet/ 32-bit
DoReFa/ 2-bit
PAMS CADyQ/2-bi QuamSR-C C-2bb
Figure I: Visual comparison (x4) with quantized lightweight SR models in terms of 4-bit and 2-bit. We use SRResNet [20] as the full-precision SR backbone and quantize it with low bit width. We compare our QuantSR-C with recent quantization methods (i.e., DoReFa [44], PAMS [22], and CADyQ [10]). Our QuantSR-C performs obviously better than others in both 4-bit and 2-bit cases.
community. Additionally, the quantization function, known as the quantizer, can be customized for each bit width (usually 2-8 bits) achieve flexible balance between accuracy and efficiency.
Although quantization is known to enhance the efficiency of image SR models, it can also lead to significant performance degradation, particularly when using ultra-low bit width, e.g., 2-4 bits. It should be noted that another extreme I-bit setting (also known as binarization) [7, 28] is not considered here. This is mainly because 1-bit quantization suffers from much larger performance gap and has a different hardware implementation in practice when compared with low-bit quantization scttings. Despite attempts to minimize the loss compared to their full-pre preepiiciiionn counterparts, considerable performance gap still exists in existing low-bit quantized SR models.
We identify two main reasons for this performance degradation. Firstly, the quantizer aims to compress the parameters by discretizing them. results in the homogenization of the parameter representation in the SR model and causes the loss of gradient information during the backward pass. Secondly, the current quantized SR models are specific bit-width mappings of the original full-precision counterparts, and the accuracy of the latter determines its upper performance bound. And once trained, the quantized SR model loses the ability to balance accuracy and efficiency when deployed, limiting its flexibility in practice usage. In this work, set out to address these problems, to achieve efficient, yet accurate, flexible quantized image SR models.
We propose accurate Quantized model for Resoluteso (QuantSR). We first propose method called Redistribution--rit driven Learnable Quantizer (RLQ) that mitigates the representation homogenization caused by discretization. We achieve this by designing redistributable and learnable quantizer that improves the forward and backward information. Our RLQ results in significant diversification of the representation without incurring additional inference burden. We further propose a Depth-dynamic Quantized Architecture (DQA) that advances the performance of quantized SR and provides flexibility in inference resource utilization. We construct weight-sharing deep- dynamic models from an SR architecture base with higher accuracy upper bound, enabling us to achieve superior performance and resource adaptation during inference.
Our comprehensive experiments show that QuantSR outperforms existing quantized SR models across various bit widths (see Fig. 1) by a substantial margin. Notably, our QuantSR with 4 bits achieves performance comparable existing methods using bits. We also demonstrate the effectiveness of our QuantSR on CNN- and Transformer-based SR networks, highlighting its versatility across different architectures. Our main contributions are summarized as follows:
We propose QuantSR, novel accurate quantization scheme for efficient image SR. provides the potential to deploy the SR application on edge devices and further narrow the performance gap between the quantized model and its full-precision counterpart.
We propose a Redistribution-die driven Learnable Quantizer (RLQ). Specifically, our RLQ diversifies quantized representation and gradient information by redistribution in quantizers. RLQ greatly enhances representations of quantized SR models with little inference burden.
We propose a Depth-dynamic Quantized Architecture (DQA) to achieve superior perfor- mance with the same network depth. Such dynamic strategy further allows for multiple models with different depths, which is resource adaptation during inference.
We employ our QuantSR to compress CNN- and Transformer- based SR networks to lower bit-width, resulting in the corresponding quantized baselines, QuantSR-C and QuantSR-T. Our QuantSR achieves superior performance over SOTA quantized SR methods.

---

2 Related Work
2.1 Efficient Image Super-Resolution (SR)
Recently, there has been a growing interest in efficient image super-resolution (SR) models due to their resource-friendly characteristics. Typically, researchers aim to develop lightweight networks through various methods, such as architecture design, neural architecture search (NAS), knowledge distillation (KD), and network pruning. For instance, Ahn et al. proposed a cascading method using a residual network [1]. While Hui et al. designed an information multi distiliitttl network [15]. Additionally, model compression techniques have also been applied to lightweight SR, such as NAS in FALSR [6], KD employed in training lighter SR student networks [9, 21]. and network pruning in ASSL [42]. These techniques have obtained promising results, but they either overlook the fine- grained parameter redundancy or require dramatically additional computations, leaving significant room for compression from bit-width perspective (i.e., low-bit quantization).
2.2 Low-bit Quantization for SR Networks
There exist two primary methods in the quantization field: Post- Trsinirann Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ has gained popularity due to its capability to quantize models without necessitating retraining. But, the absence of training and the fixed parameters in the pre-trained model constrain its potential for achieving extreme low-bit quantization [5, 17, 14, 24, 8]. Fortunately, QAT provides us with an opportunity to leverage the entire training process to achieve aggressive low-bit quantization and has shown promising performance [22, 10, 44, 4]. This method enables more comprehensive model optimization and allows for optimal training of the quantized model. QAT thus considered powerful to achieve extremely low-bit quantization, e.g. 2-4 bits.
The practical utility of existing SR networks on essource-lmiites devices is hindered by their extensive memory demands and computational burden. One of the significant obstacles to their usage is the extensive usage of Hlotting-poin storage and operations in these networks. As a result, there is ample opportunity to compress these networks by adopting low-bit quantization approach, which can drastically reduce memory and computational requirements. This approach provides a compelling rationale for investigating the development of low-bit quantized SR models [22, 10, 12, 33, 16, 27]. Li el al. presented PAMS,, which utilizes a trainable truncated parameter to dynamically determine the upper limit of the quantization range for SR models [22]. CADyQ is proposed as a technique designed for SR networks and optimizes the bit allocation for local regions and layers in the input distribution-aww
quantization scheme for SR models [12]. Despite these efforts, quantized SR models continue to exhibit a noticeable performance gap in comparison their full-precision counterparts.
image [10]. Hong et al. proposed DAQ, a channel-
3 Method
This section first provides an overview of low-bit quantization for image super-resolution (SR) and points out the limitations of existing low-bit SR networks. We then present our QuantSR, an accurate Quantization approach for efficient image Super-Resolutee consists of two novel techniques: Redissribution-driven Learnable Quantizer (RLQ) and Depth- dynamic Quantized Architecture (DQA). RLQ diversifies the information of forward and backward representations. While DQA pushes the accuracy upper bound and achieves dynamic-resoure inference. Finally, discuss how to utilize QuantSR for image SR and optimize the quantized SR network. The implementation details are also provided.
(see Fig. 2). QuantSR mainly
3.1 Preliminaries
SR Network Architecture. We first outline the basic architecture of quantized SR networks. These networks are designed to take a low-resolution (LR) image denoted as IL and produce a corresponding super-resolved image Isg. The process can be expressed follows IsR M([IR),
(I) where M(-) denotes the quantized super-resolution (SR) model. Specifically, the image SR network M()) can be generally divided into three parts: low-level feature extractor (-).). high-level feature extractor En(-), and reconstruction R.).). The common practice to apply the network quantization to the high-level feature extractor [22]. which costs most of the computational resources among the
3

---

Redistribution-diien Learnable Quantizer for Representation Recovery
Qu
Redistribution
Loarnalle Quantiorr
Soet Gradent
((x)
forword
L
backward
x [w. a))
Depth-dynamic Quantized Architecture for Flexible and Accurate Inference
head
XSR
Stage
Xo
Learaale shorlcut
Forward propagation
XSR
Skippedlnon-skippeee layer
Unselected shortcut
Stage
2: 72
22: 712 Suipping threshold
Figure 2: Overview our QuantSR for image super-resolution network. The upper part is the Redistribution-drivv Learnable Quantizer (RLQ), where the representation information of quan- tizers enhanced in forward and backward propagation of the SR network. The lower one the Depth-dynamic Quantized Architecture (DQA) that breaks the accuracy limitation of quantized SR architecture and allows for the flexible accuracy-efficiency trade-off in inference.
whole SR network. Thus, the image SR process in Eq. (1) can be further expressed as follows IsR ((IR) R EH (LIR),
(2)
where En denotes the high-level feature extractor and denotes the connection among network parts.
Quantization Framework. For the quantized SR network, the weight ur and activation a of computing units (such convolution, linear, and matmul layers) compressed low bit-widths by the quantizers Qu (w) and Qa (a), respectively. As the common practice [35], these parameters are compressed lower bit-widths by following quantizer Q with symmetric mode:
(x(x) = round (clip(æ))
Cb.
Ubb
where denotes the weight w or activation a, and Q*(-) denotes their quantizer. The function clip(.) = max maxx(min(2,, (z, a), aa) is to limit the range of the inputs, where represents the maximum of the absolute value of æ. And vs is the map function that scales the higher precision inputs their 2--1-1 Since the direct differentiation of the discrete quantizer causes
lower bit reflections as all-zero gradients and hinders the backward propagation, straight- thrhughh estimation (STE) is used to approximate the gradient of parameters:
(Q*(x)
[1
if æ (-u,a)
δx 0 otherwise
After quantizing the SR networks, the storage size and computation load can be significantly reduced due to the low bit-width and efficient integer operations.
3.2 Redistribatiin-drivven Learnable Quantizer
3.2.1
Quantization-indueed Representation Degradation
While quantization promises less storage and faster inference, it also causes significant performance degradation of compressed SR models. The intuitive reason is that the discretization brought by the quantizer hinders the information representations in both forward and backward propagations. In forward propagation, the information of quantized parameters is significantly reduced due to the limitation of low bit-width, especially in ultra-low bit quantization. For example, in 2-bit quantization, the number of candidates for single element drops from 232 in floating point form to only 4. Furthermore, quantizers such as Eq. (3) also lead homogeneous quantization mappings throughout the SR network. In backward propagation, since the gradient of quantizers cannot be used directly, the gradient obtained by estimators cannot completely reflect effect quantizers. This phenomenon results in an information mismatch always between backward and forward propagation, with the former consistently lagging behind. Therefore, the quantizer should be improved to achieve more accurate forward and backward processes, enhancing the representations of the quantized SR network.

---

3.2.2 Redistribution-driven Learnable Quantizer for Representation Recovery In this work, we proposed novel Redistribution- driven Learnable Quantizer (RLQ) for our QuantSR (Fig. 3). RLQ is driven by parameter redistribution designs, which introduce learnable parameters into the quantizer to diversify the forward quantization mapping in the whole quantized SR model. The transformation function is added in each quantiza- tion interval to enrich the gradient information in the backward. Our RLQ can be expressed as
forward
backward
C Da)
Q
L
Ohuy(: Dg.
Qa
O
Figure 3: Forward and backward propagation of
RLQ. Blue notations are learnable parameters.
(ciipi clip(1 + +)
tanh (2(x
[
1) +
Q,.((1 (z. 0b. f) round
68.
((x))
a + (5)
thb
tanh 1
where the in and / denote the learnable interval and mean-shifting parameters, respectively, which initialized as --ंー and O. The function <(.) is embedded within each quantization interval as transformation function. This function does not alter the rounded value but rather reduces the gradient of elements that are distant from the interval center. The aim of this function is to incorporate the information reflecting the quantizer's actual behavior while maintaining optimization stability.
The derivative w.r.f. the input and learnable parameters used in the backward pass are
(1. 06. f)
i E (-a,a)
aQiio f).+) =1+
80(a + f)
dx
otherwise
Of
07
(6)
aa úb, f)
round m F5
+26
if a E (-a,a)
8%
Dos
or
otherwise
Our proposed QRLO (ar, 0s, +) quantizer is embedded in the entire quantized SR network for quantization- aware training to compress weights and activations to low bit-width.
After applying RIQ. both the forward and backward representations of QuantSR are greatly enhanced. (I) For the forward propagation, since introducing the learnable mean-shifting and quantization scale parameters to RLQ. the quantizers are gradually diversified during training. The learnable parameters of the quantizers in the entire network diversify continuously throughout the training process, which significantly reduces the impact of parameter representation degradation caused by parameter discretization. (2) For the backward propagation, the embedded transformation function o(-) can introduce information in the gradient to help accurate updates. Compared with using STE directly, RLQ provides additional gradient-guide the farther away from the center of the region, the smaller the gradient. The transformation function embedded in the round function and can merged inference time without additional burden.
information, that is, in each quantization interval,
3.3 Depth-dynamic Quantized Architecture
3.3.1 Latent Architecture Bounds for Quantized SR Network
Quantization is a highly ef- fective compression tech- nique that employs low- bit parameters and efficient bitwise operations, making it ideal for fast inference on resource-constrained de- vices. One of the key hene- fits quantization is that it can be applied to SR mod- els without affecting their underlying architecture. However, in investigation, we observed that the quantized SR net- work's 's accuracy is typically limited by that of its full-precision counterpart and that quantization inevitably leads to some loss in accuracy. Furthermore, since SR models come in various sizes to meet different accuracy requirements, striking a balance between accuracy and efficiency becomes critical in practical applications. To address this challenge, propose dynamic depth quantization architecture that allows for immediate and adaptive accuracy-efficiency trade-offs in SR applications. This architecture also generates quantization models with greater accuracy at the same depth.
Insirable shorlats
layer
skipped layar
feewnd propagation
Xg
X
Stage
shuricals
Stage 2. Joinyy buin SUD- wo--ntworko
Figure 4: The architecture and training pipeline of DQA, which uses two-stage strategy to train weight-shared variants with different depths.
5

---

3.3.2 Depth-dynamic Quantized Architecture for Flexible and Accurate Inference
We propose Depth-dynamic Quantized Architecture (DQA) for the quantized high-level feature extractor & (Fig. 4), which selects a more lightweight model with fewer layers at runtime, directly reducing the computational consumption. We first build the main quantized architecture, which is computationally expensive with learnable short connections. The main architecture &H containing 2N blocks (N E (2", n E }), where the initialized i-th block ADoA expressed as
$POA (ar)) = (a((); ) + o
(7)
where x; denotes the input feature of i-th block, (p() denotes the quantized feature extractor consisting of convolution and activation layers, and denotes the learnable scaling of the skip connection.
The training strategy of our architecture is divided into two stages. The first stage is warm-up. which takes 1/5 of the training iterations. In this stage, the entire structure is optimized according to the original loss, and the skip connection is updated during the training process. In the next stage, the architecture is derived into dynamic-depth version. Different derivatives are jointly trained. Specifically, the dynamic-depth architecture is allowed to be derived into different variants with the 100%, 50%, and 25% number of blocks. The variants of DQA can be expressed as
& (H,, var)
II d 4 (z;, var) 2=1
II (b: (var))));; + o,æ:;), A=1
(8)
where II denotes composition, var E (0. 100%) and be (var)) E {0. 1). The coefficient by (var) of i-th block is determined based on the value of its scaling a, of each block. The b b[var] is determined based on the value of its short connection a each block.
For specific variant var, we select the corresponding proportion of blocks with the smallest short connection values (meaning that the block is less likely be skipped) after warming up and set their coefficient b[var] to 1. The coefficients corresponding to other blocks are set to 0 and their feature extraction processes are skipped. The reserved short connections can also be merged into the previous blocks to achieve acceleration during inference.
3.4 Training Pipeline of QuantSR
To optimize the SR model quantized by the proposed QuantSR, we adopt the joint training strategy to update multiple DQA variants, where the RLQ is applied to quantized weight and activation. On the K
given training dataset D= {LiR- [iR i=1 with K low-resolution inputs IR and corresponding high- resolution ground-truth /HR, for specific variant var of quantized SR model M, the conventional pixel-wise CPIX loss can be expressed as
PIX var
Σ
|A
MM var)
1=1
And the losses of different variants of DQA are accumulated and used to jointly update the network:
CPIX
Cis
var (100%, 50%, 25%}.
(10)
var
varEvar
where |- le, denotes to normalization. With the well-designed quantizer, architecture, and training pipeline, our QuantSR enjoys accuracy and efficiency with CNN- and Transformer-bas based backbones.
4
Experimental Results
4.1 Settings
Dataset. We adhere to the standard procedure in image SR, training on DIV2K [32] and evaluating on Set5 [2], Set14 [37], 1100 [29], 1r0 [13], and Mangal09 [30].
Evaluation. We report the reconstruction performance measured by PSNR and SSIM [34] on the channel of the YCbCr space. And we also evaluate the amounts of parameters and computation for the quantized SR model and the full full-precision one.
Proposed Quantization Baselines. We quantize image SR models with our proposed method, including the CNN- and Transformer- based architectures. For CNN-based SR models, we follow PAMS [22] and CADyQ [10], and use SRResNet as the backbone. For Transformer-bass SR models,

---

Method
#Bit
#BIk
Set5
Set14
B100
Urban
Mungu109
(wnla)
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SRReeNet
32/32
38,00
0.9605
33.59
0.9171
32.19
0.8997
32.11
0,9282
38.56
0,9770
DoReFa
4/4
37,60
0.9589
33.10
0.9133
31,87
0.8954
30.87
0.9151
37,65
0,9746
RLQ
4/4
16
37.72
0.9594
33.25
0.9147
31.98
0.8971
31.15
0.9187
37.99
0.9755
32
37.77
0.9596
33.25
0,9150
32.00
0.8973
31.30
0.9200
38.12
0.9758
DAQ
4/4
37.69
0,9593
33.15
0.9140
31,92
0.8963
30.65
0.9163
37.80
0.9751
37.51
0.9584
32.99
0,9122
31.78
0.8943
30.50
0.9111
37.38
0.9736
32
37.95
0.9603
33,59
0.9177
32.17
0.8996
31.98
0,9274
38.62
0,9771
QuantRR
4/4
16
37.88
0.9600
33.45
0,9166
32.10
0.8988
31.72
0.9249
38.37
0.9766
37.74
0.9595
33.25
0.9150
31,99
0.8973
31.27
0.9199
37.98
0.9757
Table I: Ablation study (x2 scale) about our proposed Redistribution-drivan Learnable Quantizer for
Representation Recovery (RLQ) and Depth-dynamic Quantized Architecture (DQA) for flexible and
accurate inference. 'wla' denotes the weight/activation bits. '#BIk' means residual block number.
quantize lightweight SwinIR [25]. The implementations of comparison methods follow the official codes [11, 23] and are trained with the same settings as ours. We quantize both the weight and activation of the body part in all the models with low bit-width (e.g.. 2, 4, or 8). We denote w-bit weight and a-bit activation as w/a. To ensure a fair comparison, QuantSR reports the performance of the same-depth variant other networks (e.g.. 16 blocks for SRResNet) in the comparison. The complete efficiency performance (32, 16, and 8 blocks) is demonstrated in Sec. 4.3.
Training Strategy. In our training process, we follow the practices of previous studies [26, 40, 36, 25] by conducting data augmentation. which involves random rotations 90", 180", flipping. The models are trained for 300K iterations, with each training batch consisting of 32 image patches. The input size of each patch is 64 64. To optimize our model, we utilize the Adam optimizer [19]. The learning rate is initially set to 2 10 -4 and is then halved at the 250K-th iteration. All experiments are conducted on NVIDIA RTX A6000 GPUs with PyTorch [31].
270", and horizontal
4.2 Ablation Study
To showcase the efficacy of the techniques employed in our QuantSR, we conduct ablation studies on RLQ and DQA. We employ SRResNet [20] as the image SR backbone and trained it for 200K iterations with 4-bit setting. To establish a quantization baseline, we used the vanilla quantization method, DoReFa [44]. Subsequently, we incorporate RLQ and/or DQA into SRResNet and binarized it. The results of our experiments are presented in Tab. 1, where we report the PSNR/SSIM values on five test benchmarks. Additionally, we provide detailed analyses of all variants (e.g., 32, 16, and 8 blocks) for the depth-dynamic DQA and QuantSR.
Redistribution-driven Learnable Quantizer. The vanilla version of the quantized SR model, DoReFa [44], has demonstrated basic SR performance. However, we have improved upon this by incorporating RLQ for activations, which allows us to learn the shift and numerical scale. Our proposed RLQ has significantly enhanced the performance of the quantized network while reducing the performance drop. In Tab. 1, it is clear that our RLQ achieves remarkable improvements in PSNR and SSIM with little additional computation overhead, particularly with the 4-bit setting, where we achieve around O. 12-0.22 dB and 0.0005-00.00% improvements, respectively. Our enhanced quantizer significantly improves the representation of information during both forward and backward propagation, leading to improved performance in quantized SR networks. This enhancement results in enhanced representation capabilities and more precise optimization.
Depth-dynamis Quantized Architecture. The proposed dynamic quantization algorithm (DQA) can adaptively operate on different versions of a model, based on the energy budget of real-world application scenarios. As shown in Tab. 1, we can use the proposed optimization scheme to jointly train models with varying degrees of quantization, where the variable Eq. (10) can take values of 100%, 50%, and 25% The 16-block DQA has achieved an improvement of around 0.05 dB and 0.0010 over the baseline, surpassing the upper limit of accuracy achievable by full-precision models. Notably, the DQA can effectively reduce computational complexity and memory usage by selectively ranking and skipping blocks based on a percentage threshold. Consequently, the 8-block model with the 4/4 setting has the smallest computational and storage footprint. The 32-block variant exhibits a noteworthy accuracy improvement at the cost of increased computational overhead, e.g., the improvement in Set5 is 0.17 dB and 0.0005. The accuracy results of QuantSR also demonstrate that employing both RLQ and DQA techniques can enhance the performance jointly and even elevate it to full precision levels, accomplished by utilizing an efficient bit width.
7

---

Method
Scale
#Bit
Set5
Serl4
BTO0
Urban 1OO
Manga 109
(wla)
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
Bicubic
x2
-/-
33.66
0,9299
30.24
0.8688
29.56
0.8431
26,88
0.8403
30.80
0.9339
SRReeNett [20]
x2
32/32
38.00
0.9605
33.59
0.9171
32.19
0,8997
32.11
0.9282
38.56
0.9770
SwinIR S [25]
x2
32/32
38.14
0.9611
33.86
0.9206
32.31
0,9012
32.76
0.9340
39.12
0.9783
DokeFa [64]
X2
8/8
37.32
0,9520
32,90
0.8680
31.69
0.8504
30.32
0,8800
37.01
0:9450
CADyQ [10]
x2
8/8
37.79
0.9590
33.37
0.9150
32.02
0,8980
31.53
0.9230
38.06
0.9760
DoReFa [44]
x2
414
37.31
0.9510
32.48
0.9091
0.8901
30.18
0.8780
36,95
0.9440
PAMS [22]
x2
4/4
37.67
0.9588
33.19
0.9146
31.90
0.8966
31.10
0.9194
37.62
0.9400
CADy [10]
x2
4/4
37.58
0.9580
33.14
0.9140
31.87
0.8960
30.94
0.9170
37.31
0.9740
QuanSR-C (ours)
X2
4/4
37.80
0.9597
33,35
0.9158
32.04
0.8979
31.46
0.9221
38,25
0.9762
QuantSR-T (ours)
X2
4/4
38.10
0,9604
33.65
0.9186
32.21
0.8998
32.20
0.9295
38.85
0.9774
DoRcFa [44]
x2
272
36.91
0,9470
32.55
0.9071
31.41
0.8868
29.60
0.8740
36,132
0.9410
PAMS [22]
X2
2/2
34.04
0,8270
30.91
0.8751
30.11
0.8592
27,57
0.8400
31.79
0.9110
CADyQ [10]
x2
2/2
18.51
0.4810
19.70
0.4760
17.97
0.4550
17.346
0.5830
QuaniSR C (ours)
x2
2/2
37.57
0.9589
33.09
0.9136
31.84
0.8954
30.77
0.9149
37.60
0.9745
QuanSR-T (ours)
x2
2/2
37.55
0,9587
33.12
0.9145
31.89
0.8958
30,96
0.9172
37.61
0.9745
Bicubic
x4
-/-
28.42
0.8104
26.00
0.7027
25.96
0.6675
23.14
0.6577
24.89
0.7866
SRResNet [20]
x4
32/32
32.16
28, 60
0.7822
27.58
0.7364
26.11
0,7870
30.46
0.9089
SwinIR S [25]
x4
32/32
32.44
0.8976
28.77
0.7858
27.69
0,7406
26.47
0.7980
30.92
0.9151
DoReFa [44]
x4
4/4
29.57
0.7352
26.47
0.6971
23.75
0.6898
27.89
0.8634
PAMS [22]
x4
4/4
31.59
0,8851
28.
0.7725
27.32
0.7220
25.32
0.7624
28.86
0.8805
CADyQ [10)
x4
4/4
31.48
0,8830
28.05
0.7690
27.21
0.7240
25.09
0.7520
28.82
0.8840
QuaniSR-C C tours)
x4
4/4
32.00
0.8924
28.50
0.7799
27.52
0,7342
25.88
0.7807
30.15
0.9040
QuantSR-1 (ours)
x4
4/4
32.18
0.8941
28.63
0.7822
27.59
0.7367
26.11
0.7871
30.49
0.9087
DoRara [44]
x4
272
30.54
0.8610
27.30
0.7538
26.90
0.709%
24.44
0.7242
27.31
0.8502
PAMS [22]
x4
2/2
29.20
0.8239
26.61
0.7273
26.36
0.6934
23,58
0.6812
25.59
0.8012
CADyQ [10]
x4
2/2
19:67
19.30
0.4740
19.80
0.4620
17.97
0.4360
0.5640
QuaniSR C (ours)
x4
2/2
31.30
0,8819
28.08
0.7694
27.23
0,7246
25.13
0.7537
28.81
0.8844
Quan (ours)
x4
2/2
31.53
0,8845
28.16
0.7715
27.28
0.7274
25.26
0.7609
29.06
0.8898
Table 2: Quantitative results. SRResNet and SwinlR-S are used full-precision backbones, 'wla"
denotes the weight/activation bits. The best and second best results are colored with red and cyan.
4.3 Image Super-Resolutte (SR)
We select SRResNet [20] (i.e., 1,367K (x2) and 1,515K (x4) Params) and SwinlR_ S [25] (i.e., lightweight SwinIR with 910K (x2) and 930K (x4) Params) as CNN and Transformer backbones, respectively. The corresponding QuantSR variants are QuantSR-C and QuantSR-T, respectively. And we then compare with recent quantization methods, e.g., DoReFa [44], PAMS [22], and CADyQ [10].
Quantitative Results. In Tab. 2. we provide PSNR and SSIM. Our 4-bit QuantSR-C achieves comparable or superior PSNR/SSIM scores than 8- DoReFa and CADyQ with scale x2. In the 4- bit case, our QuantSR-C achieves 0.52 dB/0.0051 (x2) and 0.79 dB/0 /8/800087 (x4) higher PSNR/SSIM values than CADyQ [10] on Urban100) 100. The case of the Transformer is more challenging, as we observe a larger performance gap between the quantized Q and the full-precision model compared to the quantized SRResNet. However, our QuantSR effectively narrows this gap, with a constant improvement compared to previous techniques, particularly at lower bit widths. For instance, with a 2-bit setting on SRResNet, our Qua outperforms PAMS by significant 3.20 dB/0.0049 (x2) and 1.55 dBB0000225 (x4) on Urban 100. It also is significantly higher than the performance obtained with the SOTA CADYQ and approaches that of the full-precision model. Similarly, on Urban100, our QuantSR-T T improves 1.26 dB/0.0125 (x2) over CADyQ in the 4-bit setting.
Compression Ratio. By utilizing combination of quantization and dynamic, lightweight architecture, compression ratio of image SR models is significantly increased. In Tab. 3, we present the compression ratio and speedup in terms of Params and Ops, respec- tively. By quantizing the full-precision SRResNet to 2 and bit-widths, we can reduce both the model size (i.e., Params) and operations (i.e., Ops) considerably. Following PAMS [22], we only quantize the weights and activations in the high-level feature extractor part, but we calculate the compression ratio and speedup for the entire model. With a 4-bit setting, our QuantSR-C achieves approximately 77.8% and 77.5% compression ratio for parameters and operations, respectively, through quantization alone. By integrating DQA for the 8-block variant, we were able to push the compression ratios to 83. 33.1% 1% and 82.9%, respectively. Furthermore, with a 2-bit setting, the operation compression ratios of 16- and 8-block variants can even reach 87.9% and 88.3%, respectively. Regarding the variant with 32
#Bit
Params (K)
Ops (G) (L Ratio)
Urban 100
Method
(w)o)
#Bik
(1 Ratio)
PSNR
SSIM
SRResNet
32/32
16
1,307 (0%)
90.1 (0%)
32.16
0.8951
32
451 (J 67.0%)
29.9 (1 66.9%)
32.17
0.8943
QuantRR-C
4/4
16
303 (1 77.8%)
20.2 (L 77.5%)
32.00
0.8924
8
230 () 83.1%)
15.4 (L 82.9%)
31.75
0.8894
32
170 ($ 87.6%)
11.5 (L 87.2%)
31.48
0.8849
QuantSR-C
2/2
16
161 (1 88.2%)
10.9 (L 87.9%)
31.30
0.8819
156 (1 88.6%)
10.6 (L 88.3%)
31.04
0.8771
Table 3:
Compression ratio of 2-b and bit SRResNet (x2), and their input sizes are 3 x 25622556 for calculating Ops.
8

---

HR / Bitth
Bicubic +
SRReeNet [20] /32-bit
DoReFa [44] /4-bit
Unbam 100: img 098 (x44
PAMS [22] /4 bit
CADyQ /10//4.bi
QuaniSR C (ours) /44 bit
QuantSR T (ours) / 4. bii
HR / Bit-width
Bicubic /-
SRReeNeet [20] /32-bit
DoReFa [44] 2-bit
100: (47 (xc4)
PAMS 1221 2-bbi
CADyQ t1ol/ 2-bit
QuaniSR-C (ours) / 2-bit QuantSR-T (ours) 2-bit
Figure 5: Visual comparison (x4) with lightweight SR in terms of 4-bit and 2-bit.
(a) 6g weight quantizer.
(b) activation quantizer. (c) Gradient effect of Φ(.)
Figure The statistics of RLQ demonstrate its representation improvement forward and backwart
blocks, 4-bit QuantSR-C. model also achieves notable parameter and operation savings of 67.0% and 66.9%, respectively, while obtaining comparable accuracy with the 16-block full-precision model.
4.4 Visualization
Visual Results. In Fig. 5, provide visual results of representative methods with scale x4 in terms of 2-bit and 4-bit cases. For each case, we compare with several quantization methods, like DoReFa [44], PAMS [22], and CADyQ [10]. Our QuantSR-CC and QuantSR-T recover more structural details and alleviate more blurring artifacts than other quantization methods. Consequently, the visual difference between QuantSR and the full-precision one is small. These visual comparisons further demonstrate the effectiveness our QuantSR, which is consistent with the observations in Tab. 2.
Training Statistics. In Fig. 6, we present statistics of the learnable parameters and function effects in our RLQ for every 100 iterations of training. Specifically, we report the statistics Op. which represents the activations, f, which represents the weights, and the gradient effects of 6(-). Figures 6 (a) and (b) illustrate that the learnable parameters are initially initialized in close proximity to each other. However, as training progresses, the optimized parameters diverge in various directions. This indicates that the quantizers containing these parameters in QuantSR are becoming more diverse, thus recovering the lost representation information due to discretization. In Fig. (c), depict the gradient change induced by specific embedded transformation function Φ(-) during backpropagation. While the forward results and training stability remain unaffected, the gradient guidance information provided by this function is significant. Consequently, promotes the optimization of QuantSR, leading to overall improvement in the results.
Conclusion
In this paper, we propose QuantSR, novel quantized image super-resolution (SR) network that overcomes the limitations of existing low-bit quantized SR models. The proposed network employs a Redistribution-driver Learnable Quantizer (RLQ) improve the representation ability of quantized networks and a Depth-dynamic Quantized Architecture (DQA) to achieve flexible inference and break the upper limit of accuracy. The authors demonstrate that QuantSR outperforms existing state-of-the quantized SR networks in both accuracy and computational efficiency. Additionally, provide QuantSR-C and QuantSR-T for both convolution and Transformer versions, respectively, to demonstrate the scheme's 's satisfactory architecture generality.
Acknowledgements. This work was supported by the National Natural Science Foundation of China (No. 62022009), the State Key Laboratory of Software Development Environment (SKL.DDE- 2022ZX-23), and the Huawei Technologies Oy (Finland) Project

---

References
[1] Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In ECCV, 2018. 3
[2] Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alheri-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012. 6
[3] Zheng Chen, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Cross aggregation transformer for image restoration. In NeurIPS, 2022.
[4] Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks, arXiv preprint arxiv: w 2018.
[5] Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCVW, 2019. 1,3
[6] Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. Fast, accurate and lightweight super-resolution with neural architecture search. arXiv preprint arXiv:1901.0226. 2019. 3
[7] Matthicu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1, arXiv preprint /602 02830, 2016. 2
[8] Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai, Junjie Liu, Xiaolin Wei, and Xianglong Liu. Towards accurate post-training quantization for vision transformer. ACM MM, 2022. 1,3
[9] Zibin He, Tao Dai, Jian Lu, Yong Jiang, and Shu-Tao Xia. Fakd: Feature-affinity based knowledge distillation for efficient image super-resolution. In ICIP, 2020. 3
[10] Cheeun Hong, Sungyong Baik, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Cadyq: Content-aware dynamic quantization for image super-resolution. In ECCV, 2022. 2, 9
[11] Cheeun Hong, Sungyong Baik, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Cadyq, 05 2023. 7
[12] Checun Hong, Hcewon Kim, Sungyong Baik, Junghun Oh, and Kyoung Mu Loc. Daq: Channel-wisc distribution-avee quantization for deep image super-resolution networks. In WACV, 2022. 3
[13] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015. 6
[14] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training
quantization with small calibration sets. In ICML, 2021. 1,3
[15] Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACM MM, 2019. 3
[16] Andrey Ignatov, Radu Timofte, Maurizio Denna, Abdel Younes, Ganzorig Gankhuyag, Jingang Huh, Myeong Kyun Kim. Kihwan Yoon, Hyeon- flyeon-Cheol Moon, Seungho Lee, et al. Efficient and accurate quantized image super-resolution mobile npus, mobile & aim 2022 challenge: report. In ECCVW. 2022. 3
[17] Divyansh Jhunjhunwala, Advait Gadhikar, Gauri Joshi, and Yonina C Eldar. Adaptive quantization of model updates for communication-effiiceen federated learning. In ICASSP, 2021. 3
[18] Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recunsive convolutional network for image super-resolution. In CVPR, 2016.
[19] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICI.R, 2015. 7
[20] Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, 2017. 2, 8,9
[21] Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged information for efficient image super-resolution. In ECCV, 2020. 3
[22] Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Baochang Zhang, Fan Yang, and Rongrong Ji. Pams: Quantized super-resolution via parameterized max scale. In ECCV, 2020. 2, 3, 8.9
10

---

[23] Huixia Li, Chenqian Yan, Shaohui Lin. Xiawu Zheng, Baochang Zhang, Fan Yang, and Rongrong Ji. Pams, 05 2023. 7
[24] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang. and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. In ICLR, 2021. 1,3
[25] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCVW, 2021. 1, 7,7,8
[26] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017. 1,7
[27] Jingyu Liu, Qiong Wang, Dunbo Zhang, and Shen. Super-resolution model quantized in multi-precision. Electronics, 10(17)2117,, 2021. 3
[28] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang- Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cans with improved representational capability and advanced training algorithm. In ECCV, 2018. 2
[29] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001. 6
[30] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga 109 dataset. Multimedia Tools and Applications. 2017.
[31] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito, Zeming Lin, Alhan Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurlPS, 2017. 7
[32] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Hiung- Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, al. Ntire 2017 challenge single image super-resolution: Methods and results. In CVPRW, 2017.
[33] Hu Wang, Peng Chen, Bohan Zhuang, and Chunhua Shen. Fully quantized image super-resolution networks. In ACM MM, 2021.
[34] Zhou Wang. Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004. 6
[35] Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint 2rX1v:2004009602, 2020. 4
[36] Jingwei Xin, Nannan Wang, Xinrui Jiang, Jie Li, Heng Huang, and Xinbo Gao. Binarized neural network for single image super resolution. In ECCV, 2020. 7
[37] Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representations. In Proc. 7th Int. Conf. Curves Surf., 2010. 6
[38] He Zhang and Vishal Patel. Densely connected pyramid dehazing network. CVPR, 2018.
[39] Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. In ICLR, 2023. I
[40] Kai Zhang, Wangmeng Zuo, and Ici Zhang. Learning single convolutional super-resolution network for multiple degradations. In CVPR, 2018. 7
[41] Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, 2018.
[42] Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. In NewrtPS, 2021. 3
[43] Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Jiangshe Zhang, and Pengfei Li. Didfuse: Deep image decomposition for infrared and visible image fusion. In IJCAI, 2020.
[44] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606..6111 2016. 2, 3. 7, 8,9
11


Reviews:
[{"review_id": "QFnqrSNP06", "reviewer": "Reviewer_VLLV", "summary": "This paper propose an low-bit quantization method for efficient image super-resolution by introduce a Redistribution-driven Learnable Quantizer (RLQ). Besides, the proposed Depth-dynamic Quantized Architecture (DQA) allows for the trade-off between efficiency and accuracy during inference through weight sharing. Experiments show that the proposed method outperforms existing state-of-the-art quantized SR networks in terms of accuracy while also providing more competitive computational efficiency.", "questions": "Please refer to comments in the **weakness** part.", "limitations": "Not applicable", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- **Originality:** This paper propose a novel accurate quantization scheme for efficient image SR by introducing Redistribution-driven Learnable Quantizer (RLQ) and Depth-dynamic Quantized Architecture (DQA). The proposed scheme can be applied into both CNN- and Transformer- based SR networks.\n\n- **Quality:** This paper is well organized and easy to follow.", "weaknesses": "- Motivation is unclear: Although the proposed method outperforms existing state-of-the-art quantized SR networks in terms of trade-off between accuracy and complexity, the contribution is still limited. If we only look at the method section, it seems that the proposed module can be applied to Low-bit Quantization of model compression for various tasks. Are there any special designs for SR tasks? If so, it is suggested to provide more discussion to highlight the contributions.\n\n- As claimed in Introduction section, \"existing SR models rely on expensive computational resources, which significantly limits the real-world SR applications on resource-constrained edge devices. Therefore, there is an urgent requirement to develop model compression techniques for SR models to reduce the computational overhead\". As far as I know, the low complexity SR model is indeed an important research task, but the model compression is only one of them. In fact, many lightweight SR algorithms have been proposed recently, such as look-up table based method, *e.g.,* SR-LUT (CVPR2021), MuLUT(ECCV2022). It is suggested to discuss the superior of the proposed method.\n\n- The proposed method is evaluated only on two classical SR networks, SRResNet (CVPR'17) for CNN-based and SwinIR_S (ICCVW'21) for Transform-based backbones, respectively. Can the proposed method be extended to the recently published SR models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: … Are there any special designs for SR tasks? If so, it is suggested to provide more discussion to highlight the contributions.\n\n**A1**: We clarify that in this work, the techniques in QuantSR focus on the SR task since it possesses certain attributes not found in other high-level tasks, such as the retention of pixel-level representations, making the quantizing SR models challenging. However, despite our techniques being designed with SR in mind, they also have the potential to be applicable to other tasks. We intend to investigate this in future work.\n\nSpecifically, the Redistribution-driven Learnable Quantizer (RLQ) targets the preservation of fine-grained pixel information essential for SR tasks. The information recovery of RLQ for quantized SR networks is accomplished through forward and backward propagation by utilizing learnable quantizer parameters and the embedded soft function, respectively. And the Depth-dynamic Quantized Architecture (DQA) leverages the blockwise characteristics of the quantized SR model, effectively pushing the boundaries of quantization performance. Benefiting from screening from a stronger initial SR model with more stacked quantized blocks, the upper limit of the model accuracy is shattered.\n\nBoth RLQ and DQA operate independently but synergistically, collectively enhancing the performance of quantized SR models. In addition, we keep the general nature of the proposed techniques, which are applicable to a variety of SR architectures, including CNNs and transformers.\n\n> **Q2**: … many lightweight SR algorithms have been proposed recently, such as look-up table based method, e.g., SR-LUT (CVPR2021), MuLUT(ECCV2022). It is suggested to discuss the superior of the proposed method.\n\n**A2**: We will follow the reviewer's suggestions to discuss the mentioned works. Currently, numerous researchers are devoted to achieving lightweight quantization algorithms, such as SR-LUT and MuLUT, which innovatively utilize lookup table-based methods to achieve fast inference and good reasoning speed. However, they still exhibit a significant gap in accuracy compared to advanced SR models based on deep networks. For instance,the PSNR indicator for the 4x Set5 dataset: MuLUT-SDY-X2 30.60 vs. 4-bit QuantSR-T 32.18. Our proposed QuantSR approach aims to bridge the gap between quantized SR models and their full-precision versions, bringing its performance close to that of their full-precision counterparts and surpassing existing methods, including both quantized and lookup-table-based SR models. Additionally, since the quantizer in QuantSR works at a general operator level, it has the potential to further accelerate convolutional units in lookup-table-based SR models, maintaining performance while pushing their efficiency limits further.\n\n> **Q3**: … Can the proposed method be extended to the recently published SR models?\n\n**A3**: Yes, thanks to the general nature of our QuantSR, it can be flexibly applied to various types of SR models. In this study, we utilize our QuantSR to quantize the CAT model under 4-bit and present the relevant results in Table A3 (part of *Table 1 in the attached PDF*). The findings demonstrate that QuantSR significantly enhances the model’s performance compared to PAMS, the currently existing state-of-the-art quantized SR model. This outcome illustrates the robust improvements our proposed method brings to the quantization of various SR architectures.\n\nTable A3: Quantitative results. CAT-R [1] is used as full-precision backbones.\n|                             |                         | \\#Bit   | Set5 |  | Set14 | | B100 | | Urban100 | | Manga109 | |\n|-----------------------------|-------------------------|-----------|-------|--------|-------|--------|-------|--------|-------|--------|-------|-------------|\n| Method    | Scale | ($w$/$a$) | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM |\n| Bicubic                     | $\t\\times$4                | -/-       | 28.42 | 0.8104 | 26.00 | 0.7027 | 25.96 | 0.6675 | 23.14 | 0.6577 | 24.89 | 0.7866      |\n| CAT\\_R | $\\times$4               | 32/32     | 32.89 | 0.9044 | 29.13 | 0.7955 | 27.95 | 0.7500 | 27.62 | 0.8292 | 32.16 | 0.9269      |\n| PAMS                        | $\\times$4               | 4/4       | 31.77 | 0.8885 | 28.33 | 0.7752 | 27.40 | 0.7295 | 25.55 | 0.7672 | 29.62 | 0.8966      |\n| QuantSR-T                   | $\\times$4               | 4/4       |  32.31 | 0.8955 | 28.74 | 0.7836 | 27.61 | 0.7389 | 26.31 | 0.7896 | 30.69 | 0.9118     |\n\n[1] Chen et al. Cross Aggregation Transformer for Image Restoration. NeurIPS, 2022.\n"}, {"review_id": "ns0Lr0FdIG", "reviewer": "Reviewer_G4U2", "summary": "This paper propose a new quantization method for single image super resolution, including Redistribution-driven Learnable Quantizer (RLQ) and Depth-dynamic Quantized Architecture (DQA). The previous one diversifies the representation and gradient information of quantized values by redistribution in quantizers, the last one improve the performance of quantized SR and achieves resource adaptation in inference. Extensive experiments demonstate the superiority of QuantSR.", "questions": "1. what is the theory of mapping method in Equation~5, or which function that inspires you?\n\n2. The true reason that cause the degradation is the high-dynamic activation [1][2], Can QuantSR solve this problem as well?\n\n3. Depth-dynamic Quantized Architecture is similar with dynamic network, could show more comparison with previous methods.\n\n\n[1] Tu Z, Hu J, Chen H, et al. Toward Accurate Post-Training Quantization for Image Super Resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 5856-5865.\n\n[2] Zhong Y, Lin M, Li X, et al. Dynamic dual trainable bounds for ultra-low precision super-resolution networks[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 1-18.", "limitations": "Th authors addressed the limitations in this paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The analysis of quantization for performance degradation is quite motivating and the proposed method is well-designed based on the analysis. The experiments show that QuantSR is better than previous methods.", "weaknesses": "1. The two main reasons for this performance degradation is much common, and not specific to super resolution task. Thus the method proposed in this paper is not SR-optimal.\n\n2. Depth-dynamic Quantized Architecture seems not much related with quantization, which is unneccessary.\n\n3. The designing of mapping method in Equation~5 is unclear, and is much similar with previous quantization method for image classifacation.\n\n4. Learnable quantizer and soft gradient is not new for quantization.\n\n5. Loss the latest baseline, DDTB[3].\n\n[1] Zhou S, Wu Y, Ni Z, et al. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients[J]. arXiv preprint arXiv:1606.06160, 2016.\n\n[2] Yamamoto K. Learnable companding quantization for accurate low-bit neural networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 5029-5038.\n\n[3] Zhong Y, Lin M, Li X, et al. Dynamic dual trainable bounds for ultra-low precision super-resolution networks[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 1-18.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: The two main reasons for this performance degradation is much common, ... Thus the method proposed in this paper is not SR-optimal.\n\n**A1**: We clarify that in this work, the techniques in QuantSR focus on the SR task since it possesses certain attributes not found in other high-level tasks, such as the retention of pixel-level representations, making the quantizing SR models challenging. However, despite our techniques being designed with SR in mind, they also have the potential to be applicable to other tasks. We intend to investigate this in future work.\n\n> **Q2**: DQA seems not much related with quantization, which is unneccessary.\n\n**A2**: We clarify that the DQA in QuantSR focuses on improving the accuracy of the quantized SR network by breaking the accuracy limit of their full-precision counterparts.\n\nAs we discussed in Sec 3.1, in previous methods, the full-precision counterpart is commonly assumed to represent the performance upper limit of the quantized model. Our DQA breaks through this limitation through dynamic architecture and training strategies. By applying selective quantized blocks from the 2x-size architecture, the performance upper limit of QuantSR far surpasses that of directly training the same architecture. The weight sharing among variants also enables a versatile trade-off between accuracy and efficiency during inference.\n\n> **Q3**: The designing of mapping method in Equation~5 is unclear, and is much similar with previous quantization method for image classifacation.\n\n**A3**: As mentioned in Sec 3.2, the purpose of $\\phi$ is to incorporate information reflecting the quantizer's actual behavior while ensuring optimization stability. In Eq. (5), our $\\phi$ function is obtained by applying the $tanh(x)$ transformation and embedding it within each quantization interval. Specifically, in the interval $x \\in [0, 1]$, the function is $\\phi(x) = \\frac{tanh(2x−1)}{tanh(1)}+0.5$, and the shape is repeated in other intervals.\n\nCompared to existing quantization methods that use soft functions, such as the tanh-based DSQ [1] and quadratic function [2], the $\\phi$ function in RLQ has a stable impact in once backward while the former two are more intense. This property allows the quantized SR model to update stably without collapsing and ultimately enhances the accuracy by introducing forward vectorized information into the gradients. The results in *Table 2 in the attached PDF* demonstrate that QuantSR significantly outperforms other SR model quantized soft-function-based methods.\n\n[1] Gong R, et al. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. ICCV, 2019.\n\n[2] Liu Z, et al. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. ECCV, 2018.\n\n> **Q4**: Learnable quantizer and soft gradient is not new for quantization.\n\n**A4**: Please note that the techniques in QuantSR are specifically designed to address the performance degradation of the quantized SR model and are different from existing quantization methods, as described next.\n\nFor the learnable quantizer: Since the SR task focuses on reconstructing images at the pixel level, making it crucial to preserve as much pixel-level information as possible during quantization. Previous quantized SR models using statistical quantizers lead to inaccurate information and significantly degraded precision. The forward mapping of our RLQ allows for flexible adjustments and diversification across the network, achieved by utilizing learnable intervals and zeros in quantizers.\n\nFor soft gradient: As we respond in A3, the purpose of $\\phi$ is to incorporate information reflecting the actual behavior of quantizers while ensuring optimization stability, and it makes QuantSR significantly outperform existing quantization methods using soft gradient approximation.\n\n> **Q5**: Loss the latest baseline, DDTB[3].\n\n**A5**: We will compare DDTB in the paper as the reviewer suggests. As we demonstrate in  *Table 2 of the attached PDF*, QuantSR consistently outperforms the DDTB method on various datasets.\n\n> **Q6**: what is the theory of mapping method in Equation~5, or which function that inspires you?\n\n**A6**: Regarding the mapping method of Eq. (5) in RLQ, it offers valuable gradient-guided information. Specifically, within each quantization interval, the gradient becomes smaller as the distance from the center of the region increases (L181 in our paper). This characteristic allows RLQ to incorporate information that accurately reflects the behavior of quantizers while maintaining optimization stability. Furthermore, in our response A3 above, we conduct a comparative analysis between RLQ and other existing soft quantization methods.\n\n> **Q7**: The true reason that cause the degradation is the high-dynamic activation [1][2], Can QuantSR solve this problem as well?\n\n**A7**: In our QuantSR, the learnable parameters in RLQ effectively tackle the quantization problem of high dynamic activations in SR networks. Relevant discussions on this approach will be included in the paper.\n\nSince the interval and zero-point of RLQ are learnable, it can gradually adapt to diverse and asymmetric input distributions. This implies that it can handle high dynamic activations better than statistics-based quantizers. As shown in Fig. 6(b), with increasing training, the offset of the activation quantizer in the network becomes more diverse, confirming the activation in the SR network is highly dynamic and demonstrating the ability of RLQ to adapt and cope well with such cases during training.\n\n> **Q8**: DQA is similar with dynamic network, could show more comparison with previous methods.\n\n**A8**: Our DQA is devoted to utilizing a dynamic architecture to break through the accuracy limitation of quantized models and also achieves dynamic trade-offs during inference. We will incorporate more related work about dynamic networks in the final version."}, {"review_id": "fhwxRJ9cGC", "reviewer": "Reviewer_E4FQ", "summary": "The paper proposes a new quantization scheme for super-resolution (SR) networks. The paper starts with a claim that weight quantization results in the homogeneity of parameters, leading to the loss of gradient information during backward pass. To resolve the claimed issue, the paper introduces Redistribution-driven Learnable Quantizer (RLQ) to reduce homogenization that is caused by discretization. RLQ is implemented by redistributable and learnable quantizer that improves the quality of information during both forward and backward passes, resulting in diverse representation, without additional inference overhead. The paper further introduces a Depth-dynamic Quantized Architecture (DQA) to better handle the performance-efficiency trade-off through learnable short-cut connections and weight sharing from different networks.", "questions": "Refer to weakness section.", "limitations": "The authors have addressed the limitations in the supplementary section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to read and well-written\n- The paper introduces a new method RLQ to handle weight homogeneity issues, without additional inference overhead.\n- The paper introduces DQA to better control the trade-off between performance and efficiency.", "weaknesses": "- Missing discussions and quantitative comparisons against related works. What is the major difference between DQA and DropConnect [A] and AIG [B]? Also, why limiting to variants 100%, 50%, and 25% in comparison to these two related works?\n\n---\n\n- The paper claims that there exists weight homogeneity due to quantization, which the proposed method mitigates. However, the paper does not present the demonstration (neither statistics nor visualization) of the extent of weight homogeneity before and after the proposed quantization scheme.\n\n---\n\n- Can the authors provide intuitive explanation as to how the proposed method mitigates the weight homogeneity issues? Isn't the proposed method still susceptible to weight homogeneity, due to the limited possible number of values each weight can take? I'm guessing the learnable mean-shifting and quantization scale parameters are optimized to diversify weight values in the given possible values each weight can take. But, I can't see how the objective function and the current design can guarantee such diversification. More intuitive and detailed explanations could be helpful.\n\n---\n\n- Can the authors provide intuitive explanation as to how the proposed method mitigates the weight homogeneity issues? Isn't the proposed method still susceptible to weight homogeneity, due to the limited possible number of values each weight can take?\n\n---\n\n- Last but not least, there is no latency/bit operation efficiency comparisons against related works, such as PAMS.\n\n\n---\n\n[A] Regularization of Neural Networks using DropConnect \\\n[B] Convolutional Networks with Adaptive Inference Graphs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: Missing discussions and quantitative comparisons against related works. What is the major difference between DQA and DropConnect [A] and AIG [B]? Also, why limiting to variants 100%, 50%, and 25% in comparison to these two related works?\n\n**A1**: We will incorporate the related work suggested by the reviewer.\n\nCompared to these dynamic networks, DQA employs low-bit parameters and undergoes quantization-aware training. It is more significant that DQA is mainly devoted to utilizing a dynamic architecture to break through the precision limitation of quantized models. Benefiting from screening from a stronger initial SR model with more stacked quantized blocks, the upper boundary of the model performance with the original block size is shattered, resulting in enhanced accuracy. This means that the architecture not only allows quantized SR networks to achieve dynamic trade-offs during inference but also significantly improves the accuracy at the same model size.\n\nDue to the joint training of various variants, we just use three variants for QuantSR to control the training costs. For example, for 16-block SRResNet, QuantSR simultaneously trains weight-shared quantized variants with 32, 16, and 8 blocks, where the first and last one strike a tilt in accuracy and efficiency, respectively. It is worth noting that our method, based on the stacking of blocks, thus also allows for a more fine-grained selection of variants with varying the number of blocks based on specific requirements.\n\n> **Q2**: The paper claims that there exists weight homogeneity due to quantization, which the proposed method mitigates. However, the paper does not present the demonstration (neither statistics nor visualization) of the extent of weight homogeneity before and after the proposed quantization scheme.\n\n**A2**: Our RLQ achieves diversification of quantizers throughout the network during training by utilizing learnable intervals and zero points. Fig. 6(a) in the manuscript demonstrates that the learnable parameters of the weight quantizers gradually diversify with increasing training epochs. This indicates that the discretized mapping of weights in different layers starts to vary, resulting in diversified quantized weights across the entire network.\n\nFollowing your suggestion, we further present a comparison between visualizations of quantized weights with and without trainable parameters (see *Fig. 1 in the attached PDF*). As observed, the presence of learnable parameters in QuantSR leads to a progressive diversification of quantizers, consequently resulting in gradually diversified quantized weights across the entire network. In the quantized SR network without learnable parameters, the weights are more homogenized during training, and their distribution remains nearly unchanged. This illustrates the significant role of RLQ in addressing weight homogeneity.\n\nOverall, these findings demonstrate the effectiveness of RLQ in promoting diversity among quantizers and mitigating weight homogenization in the quantization process.\n\n> **Q3**: Can the authors provide an intuitive explanation as to how the proposed method mitigates the weight homogeneity issues? …\n\n**A3**: Our RLQ achieves the diversification of the quantizer by utilizing learnable intervals and zero points, guiding the diversified expression of quantized weights throughout QuantSR. While the quantization interval for each quantizer remains unchanged (e.g., 2^4 for 4-bit), the quantized weights exhibit diverse trends due to different quantizers learning distinct mappings during the training process. Detailed discussions and visualizations are presented in A2.\n\nFurthermore, it is worth noting that our RLQ does not require additional constraints and can be directly optimized using the original loss function. Since discretization leads to the degradation of pixel-level information in the SR network, introducing learnable parameters into the quantizer naturally promotes the diversification of both weights and quantizers without the need for extra objectives. This point is demonstrated by the visualizations in Fig. 6 of our original paper and *Fig. 1 in the attached PDF*.\n\n> **Q4**: Last but not least, there is no latency/bit operation efficiency comparisons against related works, such as PAMS.\n\n**A4**: We will compare the efficiency with other methods in the paper as you suggested. Since our techniques do not introduce any additional computation on inference, QuantSR has the same efficiency as other quantized SR models with significantly improved accuracy under the architecture with the same number of blocks. We present the comparison in Table A4.\n\nTable A4. Comparison of the number of parameters and operations.\n|  Method    | \\#Block     | \\#Bit($w$/$a$) | \\#Params (M) | Ops (G) |\n|-------------|-----------|----------------|--------------|-----------|\n|  SRResNet  | 16  | 32/32          | 1.37        | 90.1      |\n|  DoReFa    | 16 | 4/4            | 0.30         | 20.2      |\n|  DSQ       | 16 | 4/4            | 0.30         |      20.2     |\n|  Bi-Real   | 16 | 4/4            | 0.30         | 20.2      |\n|  PAMS      | 16 | 4/4            | 0.30         | 20.2      |\n|  CADyQ     | 16 | 4/4            | 0.30         | 20.2      |\n|  QuantSR-C | 16 | 4/4            | 0.30      |     20.2      |"}, {"review_id": "Xr0a7GVMdk", "reviewer": "Reviewer_j1Cc", "summary": "This paper proposes a novel quantization method for SR models, including two new technologies to improve the unit representation and architectural potential of quantized SR models and push low-bit SR models to full-precision performance. The results on CNN and Transformer models on SR tasks show that QuantSR achieves SOTA performance and exceeds existing SR quantization methods. The most significant point of this paper is that the dynamic quantized architecture of QuantSR pushes up the performance ceiling of quantized SR from a block-stacking perspective and allows flexible reasoning according to resources in actual deployment, leading to a new promising way to lossless quantized SR.", "questions": "See weaknesses.", "limitations": "The author proposes an effective quantification method for SR tasks in this paper, and I suggest the author discuss the potential of QuantSR in more low-level vision tasks in more detail, which will make the contributions of the proposed method more significant and wide.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "In this paper, the authors propose novel RLQ and DQA technologies to jointly improve quantized SR models from two orthogonal perspectives of the unit representation and architectural potential:\n\na) RLQ is an effective technique. By introducing learnable parameters and redistribution functions in the forward and backward propagation of quantized computing units, the amount of information in the quantization training propagation process is significantly improved and more accurate, and the representation ability of the quantization unit is improved.\n\nb) DQA is more attractive to me. In addition to being effective, this is a novel attempt to improve the quantized SR model from an architectural perspective. QuantSR's dynamic quantization architecture pushes up the performance ceiling of quantized SR from a block-stacking perspective, thus providing a new promising way towards lossless quantized SR models. In addition, it also allows the quantized SR model to perform flexible reasoning according to the resources in the actual deployment, which means that the practicality of the quantized SR model is greatly enhanced.\n\nThe experimental results are superior. QuantSR shows significant performance improvement under all bit widths and surpasses the existing quantized SR methods. This enables QuantSR to achieve a SOTA structure while being novel. The visualization results clearly show that the effect of the proposed technologies follows their motivation.\n", "weaknesses": "a) One notable issue is there are some places in the paper that should be revised and clarified, including but not limited to the following:\n- In the box of Redistribution in Fig.2, should the y-axis coordinate be PDF(a), that is, the probability density function of a\n- In the box of Learnable Quantizer in Fig.2, according to the formula Eq.5 in the paper, s should be corrected to vˆb\n- The gamma in Stage 2 in Fig.2 is undefined\n- Symbol a is defined repeatedly as activation or range in Eq.4\n- According to the implementation in the article, the options of var may be actually 200% (32 blocks), 100% (16 blocks), and 50% (8 blocks)\n\nb) In RLQ, the ϕ function seems to be used as an estimator to replace STE, so why it needs to be used in the forward pass and not just backward? In addition, the authors also need to explain why the ϕ function adopts this shape instead of the existing soft quantization functions such as DSQ [1] and Bireal [2].\n\nc) The author should compare more quantized transformer SR models, and existing experiments only show the QuantSR performance on this type of network, other quantized methods are also needed. And it is necessary to show the comparison of the parameters and calculation amount of SwinIR and SRResNet, including full precision and quantized versions, to make the comparison more fair and clear.\n\n[1] Gong R, Liu X, Jiang S, et al. Differentiable soft quantization: Bridging full-precision and low-bit neural networks, ICCV, 2019.\n[2] Liu Z, Wu B, Luo W, et al. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm, ECCV, 2018.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1**: One notable issue is there are some places in the paper that should be revised and clarified, including but not limited to the following: [...]\n\n**A1**: Thank you for pointing them out. We will carefully revise them in our final revision. \n\n> **Q2a**: In RLQ, …, so why it needs to be used in the forward pass and not just backward?\n\nA2a: We clarify that the $\\phi$ function can be accurately embedded in the quantization interval of RLQ without affecting the quantized result and introducing information into the gradient when deriving backwards. Therefore, the $\\phi$ function does not affect the inference speed of the quantized SR model, but only improves the optimization during training.\n\n> **Q2b**: In addition, … why the ϕ function adopts this shape instead of the existing soft quantization functions such as DSQ [1] and Bireal [2].\n\n**A2b**: Compared to existing quantization methods that use soft functions, such as the tanh-based DSQ function [1] and quadratic-based function [2], the $\\phi$ function in RLQ has a stable impact in once backward while the former two are more intense. This property allows the quantized SR model to update stably without collapsing and ultimately enhances the accuracy by introducing forward vectorized information into the gradients. The results in Table 4 demonstrate that QuantSR significantly outperforms other SR model quantized soft-function-based methods.\n\n> **Q3a**: The author should compare more quantized transformer SR models, and existing experiments only show the QuantSR performance on this type of network, other quantized methods are also needed. \n\n**A3a**: We follow the reviewer's suggestion to compare more quantized transformer-based SR models. Specifically, we demonstrate the results of 4-bit SwinIR and CAT quantized by DoReFa and PAMS in Table A3a (*Table 1 of the attached PDF*). Our QuantSR method is able to stably outperform existing methods on different transformer-based SR models, implying that the advantages of the proposed techniques of QuantSR are robust across different architectures.\n\nTable A3a: Quantitative results. SwinIR and CAT-R are used as full-precision backbones.\n|                             |                         | \\#Bit   | Set5 |  | Set14 | | B100 | | Urban100 | | Manga109 | |\n|-----------------------------|-------------------------|-----------|-------|--------|-------|--------|-------|--------|-------|--------|-------|-------------|\n| Method    | Scale | ($w$/$a$) | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM |\n| Bicubic   | $\\times$4  | -/-   | 28.42                  | 0.8104                  | 26.00                  | 0.7027                  | 25.96                  | 0.6675                  | 23.14                  | 0.6577                  | 24.89                  | 0.7866                  |\n| SwinIR\\_S | $\\times$4 | 32/32 | 32.44                  | 0.8976                  | 28.77                  | 0.7858                  | 27.69                  | 0.7406                  | 26.47                  | 0.7980                  | 30.92                  | 0.9151                  |\n| DoRaFa    | $\\times$4 | 4/4   | 29.65                  | 0.8416                  | 26.92                  | 0.7383                  | 26.53                  | 0.6993                  | 23.85                  | 0.6957                  | 26.15                  | 0.8218                  |\n| PAMS      | $\\times$4 | 4/4   | 31.52                  | 0.8865                  | 28.19                  | 0.7727                  | 27.33                  | 0.7279                  | 25.35                  | 0.7620                  | 29.24                  | 0.8924                  |\n| QuantSR-T | $\\times$4 | 4/4   | 32.18 | 0.8941 | 28.63 | 0.7822 | 27.59 | 0.7367 | 26.11 | 0.7871 | 30.49 | 0.9087 |\n| Bicubic                     | $\t\\times$4                | -/-       | 28.42 | 0.8104 | 26.00 | 0.7027 | 25.96 | 0.6675 | 23.14 | 0.6577 | 24.89 | 0.7866      |\n| CAT\\_R | $\\times$4               | 32/32     | 32.89 | 0.9044 | 29.13 | 0.7955 | 27.95 | 0.7500 | 27.62 | 0.8292 | 32.16 | 0.9269      |\n| PAMS                        | $\\times$4               | 4/4       | 31.77 | 0.8885 | 28.33 | 0.7752 | 27.40 | 0.7295 | 25.55 | 0.7672 | 29.62 | 0.8966      |\n| QuantSR-T                   | $\\times$4               | 4/4       |  32.31 | 0.8955 | 28.74 | 0.7836 | 27.61 | 0.7389 | 26.31 | 0.7896 | 30.69 | 0.9118     |\n\n> **Q3b**: And it is necessary to show the comparison of the parameters and calculation amount of SwinIR and SRResNet, including full precision and quantized versions, to make the comparison more fair and clear.\n\n**A3b**: We further present the FLOPs and storage of the full-precision and quantized transformer-based SwinIR models, and compare them with SRResNet and QuantSR-C. As shown in Table A3b, QuantSR can also bring significant speedup and compression to transformer-based models, which shows that QuantSR brings general efficiency improvements on different architectures.\n\nTable A3b. Comparison of the number of parameters and operations.\n|  Method    | \\#Block     | \\#Bit($w$/$a$) | \\#Params (M) | Ops (G) |\n|-------------|-----------|----------------|--------------|-----------|\n|  SRResNet  | 16  | 32/32          | 1.37        | 90.1      |\n|  QuantSR-C | 16 | 4/4            | 0.30      |     20.2      |\n|  SwinIR-S  | 4  | 32/32          | 0.88 | 195.6      |\n|  QuantSR-T | 4 | 4/4            | 0.20      |     43.4      |"}, {"review_id": "s9eugPJscu", "reviewer": "Reviewer_otEX", "summary": "This manuscript is devoted to pushing the super-resolution (SR) models to ultra-low bit-width (2-4 bits). It proposes two methods and pushes the bit-width to ultra-low 2/4 bits with little accuracy loss. Meanwhile, the proposed methods not only boost the accuracy performance but also reduces the parameters and computation. It is rather difficult since the fewer parameters bring limited representation capability. The methods are well-motivated and effective, clearly demonstrating their motivation.", "questions": "The authors should respond to the raised issues in the weakness part.\n\n(1) Here is a concern about the RLQ. As far as I know, the LSQ quantization method [1] has been proposed to learn quantization steps for better projecting the floating-point values to the integer space. So can you clarify the difference between the RLQ in QuantSR and the existing quantization method? \n[1] LEARNED STEP SIZE QUANTIZATION. ICLR, 2021.\n\n(2) I notice that there are more learnable parts introduced to QuantSR, including the learnable interval and mean-shifting parameters in RLQ, and the trainable shortcut scores in DQA. I want to know the training difficulty of the proposed methods—any training algorithms/tricks and detailed strategies that are used in the training framework.", "limitations": "See weaknesses and questions parts.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "There are several strengths here:\n(1) This work proposes two methods; the redistribution-driven learnable quantizer (RLQ) mainly helps to improve the accuracy performance with little computation overhead, while the Depth-dynamic quantized architecture (DQA) further reduces the computation and parameters by integrating structured pruning. It is an interesting attempt to combine learnable structured pruning with learnable low-bit quantization. \n\n(2) Results shown in tables and visualizations are impressive. Figure 1 compares several quantization methods, and QuantSR shows a dominant advantage that can be clearly recognized. Many recently proposed methods are listed and compared, while QuantSR achieves consistent performance under 4/2 bit-width. \n\n(3) First, the motivation is clear: SR models rely on expensive computational resources, and model quantization is an effective approach that can reduce the model parameters and computation. Second, the equations are correct, and notations are well demonstrated, and the figures are helpful in understanding. Moreover, the references are extensive and cited in the correct places. \n\n(4) Compared with existing quantized SR methods, QuantSR achieves higher accuracy performance under 2/4 bit-width, which is close to the full-precision methods. And the compression ratio shown in Table 3 also shows the potential of quantization for practical value. Both quantizing and block dropping save the parameters and operations while keeping the accuracy. It can promote the implementation of quantizing SR networks in real-world scenarios.", "weaknesses": "There are several weaknesses here:\n(1) The functions in RLQ should be further clarified. Firstly, during the forward propagation, RLQ includes the function \\phi, but it is not clear whether it plays a role in the forward. The author needs to clarify its function and discuss whether it would reduce the inference efficiency of the quantized model. Secondly, I am confused about the three equations in the backward propagation. As I understand, the function \\phi primarily affects x (the input to the quantization function) in the backward, as indicated in the first equation of Eq 6. I suggest the author further discuss the meaning of these Eqs.\n\n(2) The results among CNNs and transformers are strong, but the latter seems to lack comparison, I suggest authors implement SOTA quantized SR method to transformer-based SR networks and report their results, it would be significantly helpful to improve the experiment part. \n\n(3) The writing of the paper should be improved, there are grammatical errors in the existing manuscript, and some symbols are not clarified. The authors should correct them thoroughly in the next revision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Q1a**: The functions in RLQ should be further clarified. …, but it is not clear whether it plays a role in the forward. The author needs to clarify its function and discuss whether it would reduce the inference efficiency of the quantized model. \n\n**A1a**: We clarify that the $\\phi$ function can be accurately embedded in the quantization interval of RLQ without affecting the quantized result in forward, and introducing information into the gradient when deriving backwards. Therefore, the $\\phi$ function **does not affect the inference speed** of the quantized SR model, but only improves the optimization during training.\n\n> **Q1b**: Secondly, I am confused about the three equations in the backward propagation. …\n\n**A1b**: We clarify that Eq. (6) presents the derivative functions of the learnable parameters in the backward propagation. These include the derivative function for the input $\\mathbf x$ (weights $\\mathbf w$ or activation $\\mathbf a$) (first equation), and the derivative function for the zero point $\\tau$ (second equation), and the derivative function with respect to the quantization interval $v$ (third equation). In these derivative functions, the influence of the embedding function $\\phi$ is considered and thus introduces information into the gradient to improve training.\n\n> **Q2**: The results among CNNs and transformers are strong, but the latter seems to lack comparison, …\n\n**A2**: We follow the reviewer's suggestion to compare more quantized transformer-based SR models. Specifically, we demonstrate the results of 4-bit SwinIR and CAT quantized by DoReFa and PAMS in Table A2 (*Table 1 of the attached PDF*). Our QuantSR method is able to stably outperform existing methods on different transformer-based SR models, implying that the advantages of the proposed techniques of QuantSR are robust across different architectures. \n\nTable A3a: Quantitative results. SwinIR and CAT-R are used as full-precision backbones.\n|                             |                         | \\#Bit   | Set5 |  | Set14 | | B100 | | Urban100 | | Manga109 | |\n|-----------------------------|-------------------------|-----------|-------|--------|-------|--------|-------|--------|-------|--------|-------|-------------|\n| Method    | Scale | ($w$/$a$) | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM   | PSNR  | SSIM |\n| Bicubic   | $\\times$4  | -/-   | 28.42                  | 0.8104                  | 26.00                  | 0.7027                  | 25.96                  | 0.6675                  | 23.14                  | 0.6577                  | 24.89                  | 0.7866                  |\n| SwinIR\\_S | $\\times$4 | 32/32 | 32.44                  | 0.8976                  | 28.77                  | 0.7858                  | 27.69                  | 0.7406                  | 26.47                  | 0.7980                  | 30.92                  | 0.9151                  |\n| DoRaFa    | $\\times$4 | 4/4   | 29.65                  | 0.8416                  | 26.92                  | 0.7383                  | 26.53                  | 0.6993                  | 23.85                  | 0.6957                  | 26.15                  | 0.8218                  |\n| PAMS      | $\\times$4 | 4/4   | 31.52                  | 0.8865                  | 28.19                  | 0.7727                  | 27.33                  | 0.7279                  | 25.35                  | 0.7620                  | 29.24                  | 0.8924                  |\n| QuantSR-T | $\\times$4 | 4/4   | 32.18 | 0.8941 | 28.63 | 0.7822 | 27.59 | 0.7367 | 26.11 | 0.7871 | 30.49 | 0.9087 |\n| Bicubic                     | $\t\\times$4                | -/-       | 28.42 | 0.8104 | 26.00 | 0.7027 | 25.96 | 0.6675 | 23.14 | 0.6577 | 24.89 | 0.7866      |\n| CAT\\_R | $\\times$4               | 32/32     | 32.89 | 0.9044 | 29.13 | 0.7955 | 27.95 | 0.7500 | 27.62 | 0.8292 | 32.16 | 0.9269      |\n| PAMS                        | $\\times$4               | 4/4       | 31.77 | 0.8885 | 28.33 | 0.7752 | 27.40 | 0.7295 | 25.55 | 0.7672 | 29.62 | 0.8966      |\n| QuantSR-T                   | $\\times$4               | 4/4       |  32.31 | 0.8955 | 28.74 | 0.7836 | 27.61 | 0.7389 | 26.31 | 0.7896 | 30.69 | 0.9118     |\n\n> **Q3**: The writing of the paper should be improved, …\n\n**A3**: We will follow your suggestions to carefully refine our manuscript in the final revision. \n\n> **Q4**: … can you clarify the difference between the RLQ in QuantSR and the existing quantization method (LSQ [1])?\n\n**A4**: Compared with LSQ mentioned by the reviewer, our RLQ comprehensively improves the forward and backward of the quantizer to enhance the propagated information in the quantized SR network. \n\nSpecifically, for forward propagation, RLQ not only learns the quantization interval but also the zero-point of the quantizer, allowing for significant diversification of the quantized parameter in the whole network. For backward propagation, the soft embedded function is to incorporate information reflecting the quantizer's actual behavior while ensuring optimization stability. These improvements make RLQ significantly outperform the existing quantizer. \n\n> **Q5**: … I want to know the training difficulty of the proposed methods—any training algorithms/tricks and detailed strategies that are used in the training framework.\n\n**A5**: The training of QuantSR exhibits stability and robustness. We provide a comprehensive explanation of our training pipeline and experimental setup in Sec 3.4 and 4.1, respectively. The training of all learnable parameters is uniformly performed using the loss function defined by Eq. (10), without any additional constraints."}]

Author rebuttal:
We appreciate all reviewers for the constructive reviews and positive feedback to our QuantSR. Your expertise and insightful comments help us to further improve our paper. 

The attached PDF includes:

* Figure 1: Visual comparison of weights in QuantSR with and without learnable quantizer parameters.

* Table 1: Quantitative results on transformer-based SwinIR-S and CAT-R architectures (4-bit, 4$\times$ scale)

* Table 2: Quantitative results on CNN-based SRResNet architectures (4-bit, 4$\times$ scale)

For detailed instructions, please see our responses to each reviewer.

Meta-review: Before the discussion and rebuttal, reviewers highlighted the following strengths and weaknesses of Submission 422:

*Strengths:*
- The paper introduces a novel quantization scheme for efficient image super-resolution (SR) using Redistribution-driven Learnable Quantizer (RLQ) and Depth-dynamic Quantized Architecture (DQA). This scheme can be applied to both CNN and Transformer-based SR networks.
- The paper is well-organized, easy to follow, and well-written.
- The proposed RLQ addresses weight homogeneity issues without additional inference overhead, and DQA effectively manages the performance-efficiency trade-off.
- Experiments demonstrate that the proposed method, QuantSR, outperforms existing quantized SR networks in terms of accuracy, and the results show significant performance improvement across various bit widths.
- The paper offers a combination of learnable structured pruning with learnable low-bit quantization, which is an innovative attempt.

*Weaknesses:*
- The motivation for the proposed method is unclear, and the unique designs tailored for SR tasks are not highlighted sufficiently.
- The paper does not sufficiently discuss or compare its contributions against existing lightweight SR algorithms or other model compression techniques.
- There is limited evaluation on recently published SR models, raising concerns about the method's generalizability.
- Some reviewers found aspects of the proposed RLQ and DQA methods, such as the function \phi and its role in forward propagation, to be ambiguous or unclear.
- The paper lacks a demonstration of the extent of weight homogeneity before and after the proposed quantization scheme.

Post-rebuttal, reviewers appreciated the clear and convincing responses provided by the authors. They acknowledged that most concerns were well addressed, leading to raised scores. While one reviewer still had reservations about certain comparisons, they recognized the paper's value and agreed with its acceptance. The decision is to accept the paper.
Decision: Accept (spotlight)