Title: QuantSR: Accurate Low-bit Quantization for Efficient Image Super-Resolution
Venue: NeurIPS 2023
Accepted: yes
Keywords: Super Resolution

Model Quantization

Deep Learning
Abstract: Low-bit quantization in image super-resolution (SR) has attracted copious attention in recent research due to its ability to reduce parameters and operations significantly. However, many quantized SR models suffer from accuracy degradation compared to their full-precision counterparts, especially at ultra-low bit widths (2-4 bits), limiting their practical applications. To address this issue, we propose a novel quantized image SR network, called QuantSR, which achieves accurate and efficient SR processing under low-bit quantization. To overcome the representation homogeneity caused by quantization in the network, we introduce the Redistribution-driven Learnable Quantizer (RLQ). This is accomplished through an inference-agnostic efficient redistribution design, which adds additional information in both forward and backward passes to improve the representation ability of quantized networks. Furthermore, to achieve flexible inference and break the upper limit of accuracy, we propose the Depth-dynamic Quantized Architecture (DQA). Our DQA allows for the trade-off between efficiency and accuracy during inference through weight sharing. Our comprehensive experiments show that QuantSR outperforms existing state-of-the-art quantized SR networks in terms of accuracy while also providing more competitive computational efficiency. In addition, we demonstrate the scheme's satisfactory architecture generality by providing QuantSR-C and QuantSR-T for both convolution and Transformer versions, respectively. Our code and models are released at https://github.com/htqin/QuantSR .

QuantSR: Accurate Low-bit Quantization for Efficient Image Super-Resolution

Abstract

Low-bit quantization in image super-resolution (SR) has attracted copious attention in recent research due to its ability to reduce parameters and operations significantly. However, many quantized SR models suffer from accuracy degradation compared to their full-precision counterparts, especially at ultra-low bit widths (2-4 bits), limiting their practical applications. To address this issue, we propose a novel quantized image SR network, called QuantSR, which achieves accurate and efficient SR processing under low-bit quantization. To overcome the representation homogeneity caused by quantization in the network, we introduce the Redistribution-driven Learnable Quantizer (RLQ). This is accomplished through an inference-agnostic efficient redistribution design, which adds additional information in both forward and backward passes to improve the representation ability of quantized networks. Furthermore, to achieve flexible inference and break the upper limit of accuracy, we propose the Depth-dynamic Quantized Architecture (DQA). Our DQA allows for the trade-off between efficiency and accuracy during inference through weight sharing. Our comprehensive experiments show that QuantSR outperforms existing state-of-the-art quantized SR networks in terms of accuracy while also providing more competitive computational efficiency. In addition, we demonstrate the scheme's satisfactory architecture generality by providing QuantSR-C and QuantSR-T for both convolution and Transformer versions, respectively. Our code and models are released at https://github.com/htqin/QuantSR.

1 Introduction

Single image super-resolution (SR) is the task of obtaining a high-resolution (HR) version of a low-resolution (LR) image by retrieving high-frequency details. It is an ill-posed problem since there are multiple HR candidates for the same LR input. To address this issue, researchers have explored the use of deep neural networks, including convolutional neural networks (CNNs) and Transformers [18, 26, 38, 41, 25, 3, 39, 43], to achieve high-quality reconstruction. However, existing SR models rely on expensive computational resources, which significantly limits the real-world SR applications on resource-constrained edge devices. Therefore, there is an urgent requirement to develop model compression techniques for SR models to reduce the computational overhead.

Model quantization [5, 14, 24, 8] has emerged as a powerful compression technique that compresses weights and activations in computing units (such as convolutional and linear layers) into low-bit representations. This results in heavy floating-point operations being converted into efficient integer ones, making quantization highly desirable for edge devices and friendly for memristor-based hardware. Meanwhile, the quantization process primarily focuses on parameter compression within operations. It is considered architecture- and task-agnostic, making it popular among the deep learning

Urban100: img_017 (

×

4)

HR / Bit-width

DoReFa / 4-bit

PAMS / 4-bit

CADyQ / 4-bit

QuantSR-C / 4-bit

SRResNet / 32-bit

DoReFa / 2-bit

PAMS / 2-bit

CADyQ / 2-bit

QuantSR-C / 2-bit

community. Additionally, the quantization function, known as the quantizer, can be customized for each bit width (usually 2-8 bits) to achieve a flexible balance between accuracy and efficiency.

Although quantization is known to enhance the efficiency of image SR models, it can also lead to significant performance degradation, particularly when using ultra-low bit width, e.g. , 2-4 bits. It should be noted that another extreme 1-bit setting (also known as binarization) [7, 28] is not considered here. This is mainly because 1-bit quantization suffers from a much larger performance gap and has a different hardware implementation in practice when compared with low-bit quantization settings. Despite attempts to minimize the loss compared to their full-precision counterparts, a considerable performance gap still exists in existing low-bit quantized SR models.

We identify two main reasons for this performance degradation. Firstly , the quantizer aims to compress the parameters by discretizing them. It results in the homogenization of the parameter representation in the SR model and causes the loss of gradient information during the backward pass. Secondly , the current quantized SR models are specific bit-width mappings of the original full-precision counterparts, and the accuracy of the latter determines its upper performance bound. And once trained, the quantized SR model loses the ability to balance accuracy and efficiency when deployed, limiting its flexibility in practice usage. In this work, we set out to address these problems, to achieve efficient, yet accurate, flexible quantized image SR models.

We propose an accurate Quant ized model for S uperR esolution ( QuantSR ). We first propose a method called Redistribution-driven Learnable Quantizer (RLQ) that mitigates the representation homogenization caused by discretization. We achieve this by designing a redistributable and learnable quantizer that improves the forward and backward information. Our RLQ results in significant diversification of the representation without incurring additional inference burden. We further propose a Depth-dynamic Quantized Architecture (DQA) that advances the performance of quantized SR and provides flexibility in inference resource utilization. We construct weight-sharing deepdynamic models from an SR architecture base with higher accuracy upper bound, enabling us to achieve superior performance and resource adaptation during inference.

Our comprehensive experiments show that QuantSR outperforms existing quantized SR models across various bit widths (see Fig. 1) by a substantial margin. Notably, our QuantSR with 4 bits achieves performance comparable to existing methods using 8 bits. We also demonstrate the effectiveness of our QuantSR on CNN- and Transformer-based SR networks, highlighting its versatility across different architectures. Our main contributions are summarized as follows:

We propose QuantSR, a novel accurate quantization scheme for efficient image SR. It provides the potential to deploy the SR application on edge devices and further narrow the performance gap between the quantized model and its full-precision counterpart.

We propose a Redistribution-driven Learnable Quantizer (RLQ). Specifically, our RLQ diversifies quantized representation and gradient information by redistribution in quantizers. RLQ greatly enhances representations of quantized SR models with little inference burden.

We propose a Depth-dynamic Quantized Architecture (DQA) to achieve superior performance with the same network depth. Such a dynamic strategy further allows for multiple models with different depths, which is resource adaptation during inference.

We employ our QuantSR to compress CNN- and Transformer- based SR networks to lower bit-width, resulting in the corresponding quantized baselines, QuantSR-C and QuantSR-T. Our QuantSR achieves superior performance over SOTA quantized SR methods.

2 Related Work

2.1 Efficient Image Super-Resolution (SR)

Recently, there has been a growing interest in efficient image super-resolution (SR) models due to their resource-friendly characteristics. Typically, researchers aim to develop lightweight networks through various methods, such as architecture design, neural architecture search (NAS), knowledge distillation (KD), and network pruning. For instance, Ahn et al. proposed a cascading method using a residual network [1]. While Hui et al. designed an information multi-distillation network [15]. Additionally, model compression techniques have also been applied to lightweight SR, such as NAS in FALSR [6], KD employed in training lighter SR student networks [9, 21], and network pruning in ASSL [42]. These techniques have obtained promising results, but they either overlook the finegrained parameter redundancy or require dramatically additional computations, leaving significant room for compression from a bit-width perspective ( i.e. , low-bit quantization).

2.2 Low-bit Quantization for SR Networks

There exist two primary methods in the quantization field: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ has gained popularity due to its capability to quantize models without necessitating retraining. But, the absence of training and the fixed parameters in the pre-trained model constrain its potential for achieving extreme low-bit quantization [5, 17, 14, 24, 8]. Fortunately, QAT provides us with an opportunity to leverage the entire training process to achieve aggressive low-bit quantization and has shown promising performance [22, 10, 44, 4]. This method enables more comprehensive model optimization and allows for optimal training of the quantized model. QAT is thus considered powerful to achieve extremely low-bit quantization, e.g. , 2-4 bits.

The practical utility of existing SR networks on resource-limited devices is hindered by their extensive memory demands and computational burden. One of the significant obstacles to their usage is the extensive usage of floating-point storage and operations in these networks. As a result, there is ample opportunity to compress these networks by adopting a low-bit quantization approach, which can drastically reduce memory and computational requirements. This approach provides a compelling rationale for investigating the development of low-bit quantized SR models [22, 10, 12, 33, 16, 27]. Li et al. presented PAMS, which utilizes a trainable truncated parameter to dynamically determine the upper limit of the quantization range for SR models [22]. CADyQ is proposed as a technique designed for SR networks and optimizes the bit allocation for local regions and layers in the input image [10]. Hong et al. proposed DAQ, a channel-wise distribution-aware quantization scheme for SR models [12]. Despite these efforts, quantized SR models continue to exhibit a noticeable performance gap in comparison to their full-precision counterparts.

3 Method

This section first provides an overview of low-bit quantization for image super-resolution (SR) and points out the limitations of existing low-bit SR networks. We then present our QuantSR , an accurate Quant ization approach for efficient image S uperR esolution (see Fig. 2). QuantSR mainly consists of two novel techniques: Redistribution-driven Learnable Quantizer (RLQ) and Depthdynamic Quantized Architecture (DQA). RLQ diversifies the information of forward and backward representations. While DQA pushes the accuracy upper bound and achieves dynamic-resource inference. Finally, we discuss how to utilize QuantSR for image SR and optimize the quantized SR network. The implementation details are also provided.

3.1 Preliminaries

SR Network Architecture. We first outline the basic architecture of quantized SR networks. These networks are designed to take a low-resolution (LR) image denoted as I LR and produce a corresponding super-resolved image I SR. The process can be expressed as follows

where M ( · ) denotes the quantized super-resolution (SR) model. Specifically, the image SR network M ( · ) can be generally divided into three parts: low-level feature extractor E L ( · ) , high-level feature extractor E H ( · ) , and reconstruction R ( · ) . The common practice is to apply the network quantization to the high-level feature extractor [22], which costs most of the computational resources among the

Depth-dynamic Quantized Architecture for Flexible and Accurate Inference

> 𝛾

!

> 𝛾

!

𝑥

#$

head

tail

…

> 𝛾

"

𝑥

%

head

tail

…

> 𝛾

"

𝑥

#$

> 𝛾

"

> 𝛾

"

Skipping threshold

Skipped/non-skipped layer

Selected shortcut

tail

head

𝑥

#$

…

𝑥

%

Stage 1

Stage 2

Learnable shortcut

Unselected shortcut

Forward propagation

forward

backward

𝑥

𝑸

!

𝜏

𝑎

𝑸

"

Redistribution

𝑥

𝑸

!

𝑠

𝑥

𝑔(𝑥)

ℒ

Redistribution-driven Learnable Quantizer for Representation Recovery

(

*

𝑥 ∈ {𝑤, 𝑎}

)

Soft Gradient

where E H denotes the high-level feature extractor and ◦ denotes the connection among network parts.

Quantization Framework. For the quantized SR network, the weight w and activation a of computing units (such as convolution, linear, and matmul layers) are compressed to low bit-widths by the quantizers Q w ( w ) and Q a ( a ) , respectively. As the common practice [35], these parameters are compressed to lower b bit-widths by following quantizer Q with a symmetric mode:

where x denotes the weight w or activation a , and Q b ( · ) denotes their quantizer. The function clip( · ) = max(min( x , a ) , -a ) is to limit the range of the inputs, where a represents the maximum of the absolute value of x . And v b is the map function that scales the higher precision inputs to their lower bit reflections as v b = a 2 b -1 -1 . Since the direct differentiation of the discrete quantizer causes all-zero gradients and hinders the backward propagation, straight-through estimation (STE) is used to approximate the gradient of parameters:

After quantizing the SR networks, the storage size and computation load can be significantly reduced due to the low bit-width and efficient integer operations.

3.2 Redistribution-driven Learnable Quantizer

3.2.1 Quantization-induced Representation Degradation

While quantization promises less storage and faster inference, it also causes significant performance degradation of compressed SR models. The intuitive reason is that the discretization brought by the quantizer hinders the information of representations in both forward and backward propagations. In forward propagation, the information of quantized parameters is significantly reduced due to the limitation of low bit-width, especially in ultra-low bit quantization. For example, in 2-bit quantization, the number of candidates for a single element drops from 2 32 in floating point form to only 4. Furthermore, quantizers such as Eq. (3) also lead to homogeneous quantization mappings throughout the SR network. In backward propagation, since the gradient of quantizers cannot be used directly, the gradient obtained by estimators cannot completely reflect the effect of quantizers. This phenomenon results in an information mismatch always between backward and forward propagation, with the former consistently lagging behind. Therefore, the quantizer should be improved to achieve more accurate forward and backward processes, enhancing the representations of the quantized SR network.

3.2.2 Redistribution-driven Learnable Quantizer for Representation Recovery

In this work, we proposed a novel Redistributiondriven Learnable Quantizer (RLQ) for our QuantSR (Fig. 3). RLQ is driven by parameter redistribution designs, which introduce learnable parameters into the quantizer to diversify the forward quantization mapping in the whole quantized SR model. The transformation function is added in each quantization interval to enrich the gradient information in the backward. Our RLQ can be expressed as

𝒘

𝑸

𝒘

𝑄

!"#

$

" ;

$ 𝒗

𝒃

𝒂

𝑸

𝒂

𝑄

!"#

$

" ;

$ 𝒗

𝒃

,

' 𝝉

ℒ

𝒐

backward

forward

where the ˆ v b and ˆ τ denote the learnable interval and mean-shifting parameters, respectively, which are initialized as a 2 b -1 -1 and 0 . The function ϕ ( · ) is embedded within each quantization interval as a transformation function. This function does not alter the rounded value but rather reduces the gradient of elements that are distant from the interval center. The aim of this function is to incorporate the information reflecting the quantizer's actual behavior while maintaining optimization stability.

The derivative w.r.t. the input and learnable parameters used in the backward pass are

Our proposed Q b RLQ ( x , ˆ v b , ˆ τ ) quantizer is embedded in the entire quantized SR network for quantization-aware training to compress weights and activations to low bit-width.

After applying RLQ, both the forward and backward representations of QuantSR are greatly enhanced. (1) For the forward propagation, since introducing the learnable mean-shifting and quantization scale parameters to RLQ, the quantizers are gradually diversified during training. The learnable parameters of the quantizers in the entire network diversify continuously throughout the training process, which significantly reduces the impact of parameter representation degradation caused by parameter discretization. (2) For the backward propagation, the embedded transformation function ϕ ( · ) can introduce information in the gradient to help accurate updates. Compared with using STE directly, RLQ provides additional gradient-guided information, that is, in each quantization interval, the farther away from the center of the region, the smaller the gradient. The transformation function is embedded in the round function and can be merged at inference time without additional burden.

3.3 Depth-dynamic Quantized Architecture

3.3.1 Latent Architecture Bounds for Quantized SR Network

Quantization is a highly effective compression technique that employs lowbit parameters and efficient bitwise operations, making it ideal for fast inference on resource-constrained devices. One of the key benefits of quantization is that it can be applied to SR models without affecting their

𝑥

!

Original network

Stage 1: Train shortcuts

Stage 2: Jointly train sub-networks

𝑥

"#

𝑥

!

𝑥

"#

𝑥

!

𝑥

"#

𝑥

"#

𝑥

"#

trainable shortcuts

layer

skipped layer

forward propagation

underlying architecture. However, in our investigation, we observed that the quantized SR network's accuracy is typically limited by that of its full-precision counterpart and that quantization inevitably leads to some loss in accuracy. Furthermore, since SR models come in various sizes to meet different accuracy requirements, striking a balance between accuracy and efficiency becomes critical in practical applications. To address this challenge, we propose a dynamic depth quantization architecture that allows for immediate and adaptive accuracy-efficiency trade-offs in SR applications. This architecture also generates quantization models with greater accuracy at the same depth.

3.3.2 Depth-dynamic Quantized Architecture for Flexible and Accurate Inference

We propose a Depth-dynamic Quantized Architecture (DQA) for the quantized high-level feature extractor E ′ H (Fig. 4), which selects a more lightweight model with fewer layers at runtime, directly reducing the computational consumption. We first build the main quantized architecture, which is computationally expensive with learnable short connections. The main architecture E DQA H containing 2 N blocks ( N ∈ { 2 n , n ∈ Z + } ), where the initialized i -th block ˆ Φ DQA i is expressed as

where x i denotes the input feature of i -th block, φ ( · ) denotes the quantized feature extractor consisting of convolution and activation layers, and α denotes the learnable scaling of the skip connection.

The training strategy of our architecture is divided into two stages. The first stage is a warm-up, which takes 1/5 of the training iterations. In this stage, the entire structure is optimized according to the original loss, and the skip connection is updated during the training process. In the next stage , the architecture is derived into a dynamic-depth version. Different derivatives are jointly trained. Specifically, the dynamic-depth architecture is allowed to be derived into different variants with the 100%, 50%, and 25% number of blocks. The variants of DQA can be expressed as

where ∏ denotes composition, var ∈ (0 , 100%] and b i ( var ) ∈ { 0 , 1 } . The coefficient b i ( var ) of i -th block is determined based on the value of its scaling α i of each block. The b [ var ] is determined based on the value of its short connection α in each block.

For a specific variant var, we select the corresponding proportion of blocks with the smallest short connection values (meaning that the block is less likely to be skipped) after warming up and set their coefficient b [ var ] to 1. The coefficients corresponding to other blocks are set to 0 and their feature extraction processes are skipped. The reserved short connections can also be merged into the previous blocks to achieve acceleration during inference.

3.4 Training Pipeline of QuantSR

To optimize the SR model quantized by the proposed QuantSR, we adopt the joint training strategy to update multiple DQA variants, where the RLQ is applied to quantized weight and activation. On the given training dataset D = { I i LR , I i HR } K i =1 with K low-resolution inputs I LR and corresponding highresolution ground-truth I HR, for a specific variant var of quantized SR model M , the conventional pixel-wise L PIX var loss can be expressed as

And the losses of different variants of DQA are accumulated and used to jointly update the network:

where | · | ℓ 0 denotes ℓ 0 normalization. With the well-designed quantizer, architecture, and training pipeline, our QuantSR enjoys accuracy and efficiency with CNN- and Transformer-based backbones.

4 Experimental Results

4.1 Settings

Dataset. We adhere to the standard procedure in image SR, training on DIV2K [32] and evaluating on Set5 [2], Set14 [37], B100 [29], Urban100 [13], and Manga109 [30].

Evaluation. We report the reconstruction performance measured by PSNR and SSIM [34] on the Y channel of the YCbCr space. And we also evaluate the amounts of parameters and computation for the quantized SR model and the full-precision one.

Proposed Quantization Baselines. We quantize image SR models with our proposed method, including the CNN- and Transformer- based architectures. For CNN-based SR models, we follow PAMS [22] and CADyQ [10], and use SRResNet as the backbone. For Transformer-based SR models,

we quantize lightweight SwinIR [25]. The implementations of comparison methods follow the official codes [11, 23] and are trained with the same settings as ours. We quantize both the weight and activation of the body part in all the models with low bit-width ( e.g. , 2, 4, or 8). We denote w -bit weight and a -bit activation as w / a . To ensure a fair comparison, QuantSR reports the performance of the same-depth variant to other networks ( e.g. , 16 blocks for SRResNet) in the comparison. The complete efficiency performance (32, 16, and 8 blocks) is demonstrated in Sec. 4.3.

Training Strategy. In our training process, we follow the practices of previous studies [26, 40, 36, 25] by conducting data augmentation, which involves random rotations of 90 ◦ , 180 ◦ , 270 ◦ , and horizontal flipping. The models are trained for 300K iterations, with each training batch consisting of 32 image patches. The input size of each patch is 64 × 64. To optimize our model, we utilize the Adam optimizer [19]. The learning rate is initially set to 2 × 10 -4 and is then halved at the 250K-th iteration. All experiments are conducted on NVIDIA RTX A6000 GPUs with PyTorch [31].

4.2 Ablation Study

To showcase the efficacy of the techniques employed in our QuantSR, we conduct ablation studies on RLQ and DQA. We employ SRResNet [20] as the image SR backbone and trained it for 200K iterations with a 4-bit setting. To establish a quantization baseline, we used the vanilla quantization method, DoReFa [44]. Subsequently, we incorporate RLQ and/or DQA into SRResNet and binarized it. The results of our experiments are presented in Tab. 1, where we report the PSNR/SSIM values on five test benchmarks. Additionally, we provide detailed analyses of all variants ( e.g. , 32, 16, and 8 blocks) for the depth-dynamic DQA and QuantSR.

Redistribution-driven Learnable Quantizer . The vanilla version of the quantized SR model, DoReFa [44], has demonstrated basic SR performance. However, we have improved upon this by incorporating RLQ for activations, which allows us to learn the shift and numerical scale. Our proposed RLQ has significantly enhanced the performance of the quantized network while reducing the performance drop. In Tab. 1, it is clear that our RLQ achieves remarkable improvements in PSNR and SSIM with little additional computation overhead, particularly with the 4-bit setting, where we achieve around 0.12-0.28 dB and 0.0005-0.0036 improvements, respectively. Our enhanced quantizer significantly improves the representation of information during both forward and backward propagation, leading to improved performance in quantized SR networks. This enhancement results in enhanced representation capabilities and more precise optimization.

Depth-dynamic Quantized Architecture . The proposed dynamic quantization algorithm (DQA) can adaptively operate on different versions of a model, based on the energy budget of real-world application scenarios. As shown in Tab. 1, we can use the proposed optimization scheme to jointly train models with varying degrees of quantization, where the variable in Eq. (10) can take values of 100%, 50%, and 25%. The 16-block DQA has achieved an improvement of around 0.05 dB and 0.0010 over the baseline, surpassing the upper limit of accuracy achievable by full-precision models. Notably, the DQA can effectively reduce computational complexity and memory usage by selectively ranking and skipping blocks based on a percentage threshold. Consequently, the 8-block model with the 4/4 setting has the smallest computational and storage footprint. The 32-block variant exhibits a noteworthy accuracy improvement at the cost of increased computational overhead, e.g. , the improvement in Set5 is 0.17 dB and 0.0005. The accuracy results of QuantSR also demonstrate that employing both RLQ and DQA techniques can enhance the performance jointly and even elevate it to full precision levels, accomplished by utilizing an efficient 4-bit width.

4.3 Image Super-Resolution (SR)

We select SRResNet [20] ( i.e. , 1,367K ( × 2) and 1,515K ( × 4) Params) and SwinIR_S [25] ( i.e. , lightweight SwinIR with 910K ( × 2) and 930K ( × 4) Params) as CNN and Transformer backbones, respectively. The corresponding QuantSR variants are QuantSR-C and QuantSR-T, respectively. And we then compare with recent quantization methods, e.g. , DoReFa [44], PAMS [22], and CADyQ [10].

Quantitative Results. In Tab. 2, we provide PSNR and SSIM. Our 4-bit QuantSR-C achieves comparable or superior PSNR/SSIM scores than 8-bit DoReFa and CADyQ with scale × 2. In the 4bit case, our QuantSR-C achieves 0.52 dB/0.0051 ( × 2) and 0.79 dB/0.0287 ( × 4) higher PSNR/SSIM values than CADyQ [10] on Urban100. The case of the Transformer is more challenging, as we observe a larger performance gap between the quantized QuantSR-T and the full-precision model compared to the quantized SRResNet. However, our QuantSR effectively narrows this gap, with a constant improvement compared to previous techniques, particularly at lower bit widths. For instance, with a 2-bit setting on SRResNet, our QuantSR-C outperforms PAMS by a significant 3.20 dB/0.0749 ( × 2) and 1.55 dB/0.0725 ( × 4) on Urban100. It also is significantly higher than the performance obtained with the SOTA CADyQ and approaches that of the full-precision model. Similarly, on Urban100, our QuantSR-T improves 1.26 dB/0.0125 ( × 2) over CADyQ in the 4-bit setting.

Compression Ratio. By utilizing a combination of quantization and dynamic, lightweight architecture, compression ratio of image SR models is significantly increased. In Tab. 3, we present the compression ratio and speedup in terms of Params and Ops, respectively. By quantizing the

full-precision SRResNet to 2 and 4 bit-widths, we can reduce both the model size ( i.e. , Params) and operations ( i.e. , Ops) considerably. Following PAMS [22], we only quantize the weights and activations in the high-level feature extractor part, but we calculate the compression ratio and speedup for the entire model. With a 4-bit setting, our QuantSR-C achieves approximately 77.8% and 77.5% compression ratio for parameters and operations, respectively, through quantization alone. By integrating DQA for the 8-block variant, we were able to push the compression ratios to 83.1% and 82.9%, respectively. Furthermore, with a 2-bit setting, the operation compression ratios of 16and 8-block variants can even reach 87.9% and 88.3%, respectively. Regarding the variant with 32

1e-8

Urban100: img_098 (

×

4)

HR / Bit-width

Bicubic / -

SRResNet [20] / 32-bit

DoReFa [44] / 4-bit

PAMS [22] / 4-bit

CADyQ [10] / 4-bit

QuantSR-C (ours) / 4-bit

QuantSR-T (ours) / 4-bit

Urban100: img_047 (

×

4)

HR / Bit-width

Bicubic / -

SRResNet [20] / 32-bit

DoReFa [44] / 2-bit

PAMS [22] / 2-bit

CADyQ [10] / 2-bit

QuantSR-C (ours) / 2-bit

QuantSR-T (ours) / 2-bit

Figure 5: Visual comparison (

×

4) with lightweight SR in terms of 4-bit and 2-bit.

0.025

0.020

0.015

0.010

0.005

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

0.2

0.0

-0.2

-0.4

-0.6

-0.8

5

4

3

2

1

0

6

-2

-1

0

1

2

1e-8

1e5

(a)

ˆ

v

b

for weight quantizer.

0.025

0.020

0.015

0.010

0.005

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

0.2

0.0

-0.2

-0.4

-0.6

-0.8

5

4

3

2

1

0

6

-2

-1

0

1

2

1e5

(b)

ˆ

τ

for activation quantizer.

0.025

0.020

0.015

0.010

0.005

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

0.2

0.0

-0.2

-0.4

-0.6

-0.8

5

4

3

2

1

0

6

-2

-1

0

1

2

1e-8

1e5

(c) Gradient effect of

ϕ

(

·

)

(b) ̂ 𝜏 for activation (c) Changes in gradient (b) ̂ 𝜏 for activation (c) Changes in gradient (b) ̂ 𝜏 for activation (c) Changes in gradient blocks, 4-bit QuantSR-C model also achieves notable parameter and operation savings of 67.0% and 66.9%, respectively, while obtaining comparable accuracy with the 16-block full-precision model.

4.4 Visualization

Visual Results. In Fig. 5, we provide visual results of representative methods with scale × 4 in terms of 2-bit and 4-bit cases. For each case, we compare with several quantization methods, like DoReFa [44], PAMS [22], and CADyQ [10]. Our QuantSR-C and QuantSR-T recover more structural details and alleviate more blurring artifacts than other quantization methods. Consequently, the visual difference between our QuantSR and the full-precision one is small. These visual comparisons further demonstrate the effectiveness of our QuantSR, which is consistent with the observations in Tab. 2.

Training Statistics. In Fig. 6, we present statistics of the learnable parameters and function effects in our RLQ for every 100 iterations of training. Specifically, we report the statistics of ˆ v b , which represents the activations, ˆ τ , which represents the weights, and the gradient effects of ϕ ( · ) . Figures 6 (a) and (b) illustrate that the learnable parameters are initially initialized in close proximity to each other. However, as training progresses, the optimized parameters diverge in various directions. This indicates that the quantizers containing these parameters in QuantSR are becoming more diverse, thus recovering the lost representation information due to discretization. In Fig. 6 (c), we depict the gradient change induced by a specific embedded transformation function ϕ ( · ) during backpropagation. While the forward results and training stability remain unaffected, the gradient guidance information provided by this function is significant. Consequently, it promotes the optimization of QuantSR, leading to an overall improvement in the results.

5 Conclusion

In this paper, we propose QuantSR, a novel quantized image super-resolution (SR) network that overcomes the limitations of existing low-bit quantized SR models. The proposed network employs a Redistribution-driven Learnable Quantizer (RLQ) to improve the representation ability of quantized networks and a Depth-dynamic Quantized Architecture (DQA) to achieve flexible inference and break the upper limit of accuracy. The authors demonstrate that QuantSR outperforms existing state-of-the-art quantized SR networks in both accuracy and computational efficiency. Additionally, we provide QuantSR-C and QuantSR-T for both convolution and Transformer versions, respectively, to demonstrate the scheme's satisfactory architecture generality.

Acknowledgements. This work was supported by the National Natural Science Foundation of China (No. 62022009), the State Key Laboratory of Software Development Environment (SKLSDE2022ZX-23), and the Huawei Technologies Oy (Finland) Project.

References

Namhyuk Ahn, Byungkon Kang, and Kyung-Ah Sohn. Fast, accurate, and lightweight super-resolution with cascading residual network. In ECCV , 2018. 3

Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC , 2012. 6

Zheng Chen, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Cross aggregation transformer for image restoration. In NeurIPS , 2022. 1

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 , 2018. 3

Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCVW , 2019. 1, 3

Xiangxiang Chu, Bo Zhang, Hailong Ma, Ruijun Xu, and Qingyuan Li. Fast, accurate and lightweight super-resolution with neural architecture search. arXiv preprint arXiv:1901.07261 , 2019. 3

Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 , 2016. 2

Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai, Junjie Liu, Xiaolin Wei, and Xianglong Liu. Towards accurate post-training quantization for vision transformer. In ACM MM , 2022. 1, 3

Zibin He, Tao Dai, Jian Lu, Yong Jiang, and Shu-Tao Xia. Fakd: Feature-affinity based knowledge distillation for efficient image super-resolution. In ICIP , 2020. 3

Cheeun Hong, Sungyong Baik, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Cadyq: Content-aware dynamic quantization for image super-resolution. In ECCV , 2022. 2, 3, 6, 8, 9

Cheeun Hong, Sungyong Baik, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Cadyq, 05 2023. 7

Cheeun Hong, Heewon Kim, Sungyong Baik, Junghun Oh, and Kyoung Mu Lee. Daq: Channel-wise distribution-aware quantization for deep image super-resolution networks. In WACV , 2022. 3

Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR , 2015. 6

Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In ICML , 2021. 1, 3

Zheng Hui, Xinbo Gao, Yunchu Yang, and Xiumei Wang. Lightweight image super-resolution with information multi-distillation network. In ACM MM , 2019. 3

Andrey Ignatov, Radu Timofte, Maurizio Denna, Abdel Younes, Ganzorig Gankhuyag, Jingang Huh, Myeong Kyun Kim, Kihwan Yoon, Hyeon-Cheol Moon, Seungho Lee, et al. Efficient and accurate quantized image super-resolution on mobile npus, mobile ai & aim 2022 challenge: report. In ECCVW , 2022. 3

Divyansh Jhunjhunwala, Advait Gadhikar, Gauri Joshi, and Yonina C Eldar. Adaptive quantization of model updates for communication-efficient federated learning. In ICASSP , 2021. 3

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Deeply-recursive convolutional network for image super-resolution. In CVPR , 2016. 1

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015. 7

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR , 2017. 2, 7, 8, 9

Wonkyung Lee, Junghyup Lee, Dohyung Kim, and Bumsub Ham. Learning with privileged information for efficient image super-resolution. In ECCV , 2020. 3

Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Baochang Zhang, Fan Yang, and Rongrong Ji. Pams: Quantized super-resolution via parameterized max scale. In ECCV , 2020. 2, 3, 6, 8, 9

Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Baochang Zhang, Fan Yang, and Rongrong Ji. Pams, 05 2023. 7

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In ICLR , 2021. 1, 3

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCVW , 2021. 1, 7, 8

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW , 2017. 1, 7

Jingyu Liu, Qiong Wang, Dunbo Zhang, and Li Shen. Super-resolution model quantized in multi-precision. Electronics , 10(17):2176, 2021. 3

Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV , 2018. 2

David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV , 2001. 6

Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications , 2017. 6

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NeurIPS , 2017. 7

Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPRW , 2017. 6

Hu Wang, Peng Chen, Bohan Zhuang, and Chunhua Shen. Fully quantized image super-resolution networks. In ACM MM , 2021. 3

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP , 2004. 6

Hao Wu, Patrick Judd, Xiaojie Zhang, Mikhail Isaev, and Paulius Micikevicius. Integer quantization for deep learning inference: Principles and empirical evaluation. arXiv preprint arXiv:2004.09602 , 2020. 4

Jingwei Xin, Nannan Wang, Xinrui Jiang, Jie Li, Heng Huang, and Xinbo Gao. Binarized neural network for single image super resolution. In ECCV , 2020. 7

Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representations. In Proc. 7th Int. Conf. Curves Surf. , 2010. 6

He Zhang and Vishal M Patel. Densely connected pyramid dehazing network. In CVPR , 2018. 1

Jiale Zhang, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Accurate image restoration with attention retractable transformer. In ICLR , 2023. 1

Kai Zhang, Wangmeng Zuo, and Lei Zhang. Learning a single convolutional super-resolution network for multiple degradations. In CVPR , 2018. 7

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR , 2018. 1

Yulun Zhang, Huan Wang, Can Qin, and Yun Fu. Aligned structured sparsity learning for efficient image super-resolution. In NeurIPS , 2021. 3

Zixiang Zhao, Shuang Xu, Chunxia Zhang, Junmin Liu, Jiangshe Zhang, and Pengfei Li. Didfuse: Deep image decomposition for infrared and visible image fusion. In IJCAI , 2020. 1

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 , 2016. 2, 3, 7, 8, 9

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript is devoted to pushing the super-resolution (SR) models to ultra-low bit-width (2-4 bits). It proposes two methods and pushes the bit-width to ultra-low 2/4 bits with little accuracy loss. Meanwhile, the proposed methods not only boost the accuracy performance but also reduces the parameters and computation. It is rather difficult since the fewer parameters bring limited representation capability. The methods are well-motivated and effective, clearly demonstrating their motivation.\n\nSTRENGTHS:\nThere are several strengths here:\n(1) This work proposes two methods; the redistribution-driven learnable quantizer (RLQ) mainly helps to improve the accuracy performance with little computation overhead, while the Depth-dynamic quantized architecture (DQA) further reduces the computation and parameters by integrating structured pruning. It is an interesting attempt to combine learnable structured pruning with learnable low-bit quantization. \n\n(2) Results shown in tables and visualizations are impressive. Figure 1 compares several quantization methods, and QuantSR shows a dominant advantage that can be clearly recognized. Many recently proposed methods are listed and compared, while QuantSR achieves consistent performance under 4/2 bit-width. \n\n(3) First, the motivation is clear: SR models rely on expensive computational resources, and model quantization is an effective approach that can reduce the model parameters and computation. Second, the equations are correct, and notations are well demonstrated, and the figures are helpful in understanding. Moreover, the references are extensive and cited in the correct places. \n\n(4) Compared with existing quantized SR methods, QuantSR achieves higher accuracy performance under 2/4 bit-width, which is close to the full-precision methods. And the compression ratio shown in Table 3 also shows the potential of quantization for practical value. Both quantizing and block dropping save the parameters and operations while keeping the accuracy. It can promote the implementation of quantizing SR networks in real-world scenarios.\n\nWEAKNESSES:\nThere are several weaknesses here:\n(1) The functions in RLQ should be further clarified. Firstly, during the forward propagation, RLQ includes the function \\phi, but it is not clear whether it plays a role in the forward. The author needs to clarify its function and discuss whether it would reduce the inference efficiency of the quantized model. Secondly, I am confused about the three equations in the backward propagation. As I understand, the function \\phi primarily affects x (the input to the quantization function) in the backward, as indicated in the first equation of Eq 6. I suggest the author further discuss the meaning of these Eqs.\n\n(2) The results among CNNs and transformers are strong, but the latter seems to lack comparison, I suggest authors implement SOTA quantized SR method to transformer-based SR networks and report their results, it would be significantly helpful to improve the experiment part. \n\n(3) The writing of the paper should be improved, there are grammatical errors in the existing manuscript, and some symbols are not clarified. The authors should correct them thoroughly in the next revision.\n\nQUESTIONS:\nThe authors should respond to the raised issues in the weakness part.\n\n(1) Here is a concern about the RLQ. As far as I know, the LSQ quantization method [1] has been proposed to learn quantization steps for better projecting the floating-point values to the integer space. So can you clarify the difference between the RLQ in QuantSR and the existing quantization method? \n[1] LEARNED STEP SIZE QUANTIZATION. ICLR, 2021.\n\n(2) I notice that there are more learnable parts introduced to QuantSR, including the learnable interval and mean-shifting parameters in RLQ, and the trainable shortcut scores in DQA. I want to know the training difficulty of the proposed methods—any training algorithms/tricks and detailed strategies that are used in the training framework.\n\nLIMITATIONS:\nSee weaknesses and questions parts.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel quantization method for SR models, including two new technologies to improve the unit representation and architectural potential of quantized SR models and push low-bit SR models to full-precision performance. The results on CNN and Transformer models on SR tasks show that QuantSR achieves SOTA performance and exceeds existing SR quantization methods. The most significant point of this paper is that the dynamic quantized architecture of QuantSR pushes up the performance ceiling of quantized SR from a block-stacking perspective and allows flexible reasoning according to resources in actual deployment, leading to a new promising way to lossless quantized SR.\n\nSTRENGTHS:\nIn this paper, the authors propose novel RLQ and DQA technologies to jointly improve quantized SR models from two orthogonal perspectives of the unit representation and architectural potential:\n\na) RLQ is an effective technique. By introducing learnable parameters and redistribution functions in the forward and backward propagation of quantized computing units, the amount of information in the quantization training propagation process is significantly improved and more accurate, and the representation ability of the quantization unit is improved.\n\nb) DQA is more attractive to me. In addition to being effective, this is a novel attempt to improve the quantized SR model from an architectural perspective. QuantSR's dynamic quantization architecture pushes up the performance ceiling of quantized SR from a block-stacking perspective, thus providing a new promising way towards lossless quantized SR models. In addition, it also allows the quantized SR model to perform flexible reasoning according to the resources in the actual deployment, which means that the practicality of the quantized SR model is greatly enhanced.\n\nThe experimental results are superior. QuantSR shows significant performance improvement under all bit widths and surpasses the existing quantized SR methods. This enables QuantSR to achieve a SOTA structure while being novel. The visualization results clearly show that the effect of the proposed technologies follows their motivation.\n\nWEAKNESSES:\na) One notable issue is there are some places in the paper that should be revised and clarified, including but not limited to the following:\n- In the box of Redistribution in Fig.2, should the y-axis coordinate be PDF(a), that is, the probability density function of a\n- In the box of Learnable Quantizer in Fig.2, according to the formula Eq.5 in the paper, s should be corrected to vˆb\n- The gamma in Stage 2 in Fig.2 is undefined\n- Symbol a is defined repeatedly as activation or range in Eq.4\n- According to the implementation in the article, the options of var may be actually 200% (32 blocks), 100% (16 blocks), and 50% (8 blocks)\n\nb) In RLQ, the ϕ function seems to be used as an estimator to replace STE, so why it needs to be used in the forward pass and not just backward? In addition, the authors also need to explain why the ϕ function adopts this shape instead of the existing soft quantization functions such as DSQ [1] and Bireal [2].\n\nc) The author should compare more quantized transformer SR models, and existing experiments only show the QuantSR performance on this type of network, other quantized methods are also needed. And it is necessary to show the comparison of the parameters and calculation amount of SwinIR and SRResNet, including full precision and quantized versions, to make the comparison more fair and clear.\n\n[1] Gong R, Liu X, Jiang S, et al. Differentiable soft quantization: Bridging full-precision and low-bit neural networks, ICCV, 2019.\n[2] Liu Z, Wu B, Luo W, et al. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm, ECCV, 2018.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe author proposes an effective quantification method for SR tasks in this paper, and I suggest the author discuss the potential of QuantSR in more low-level vision tasks in more detail, which will make the contributions of the proposed method more significant and wide.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new quantization scheme for super-resolution (SR) networks. The paper starts with a claim that weight quantization results in the homogeneity of parameters, leading to the loss of gradient information during backward pass. To resolve the claimed issue, the paper introduces Redistribution-driven Learnable Quantizer (RLQ) to reduce homogenization that is caused by discretization. RLQ is implemented by redistributable and learnable quantizer that improves the quality of information during both forward and backward passes, resulting in diverse representation, without additional inference overhead. The paper further introduces a Depth-dynamic Quantized Architecture (DQA) to better handle the performance-efficiency trade-off through learnable short-cut connections and weight sharing from different networks.\n\nSTRENGTHS:\n- The paper is easy to read and well-written\n- The paper introduces a new method RLQ to handle weight homogeneity issues, without additional inference overhead.\n- The paper introduces DQA to better control the trade-off between performance and efficiency.\n\nWEAKNESSES:\n- Missing discussions and quantitative comparisons against related works. What is the major difference between DQA and DropConnect [A] and AIG [B]? Also, why limiting to variants 100%, 50%, and 25% in comparison to these two related works?\n\n---\n\n- The paper claims that there exists weight homogeneity due to quantization, which the proposed method mitigates. However, the paper does not present the demonstration (neither statistics nor visualization) of the extent of weight homogeneity before and after the proposed quantization scheme.\n\n---\n\n- Can the authors provide intuitive explanation as to how the proposed method mitigates the weight homogeneity issues? Isn't the proposed method still susceptible to weight homogeneity, due to the limited possible number of values each weight can take? I'm guessing the learnable mean-shifting and quantization scale parameters are optimized to diversify weight values in the given possible values each weight can take. But, I can't see how the objective function and the current design can guarantee such diversification. More intuitive and detailed explanations could be helpful.\n\n---\n\n- Can the authors provide intuitive explanation as to how the proposed method mitigates the weight homogeneity issues? Isn't the proposed method still susceptible to weight homogeneity, due to the limited possible number of values each weight can take?\n\n---\n\n- Last but not least, there is no latency/bit operation efficiency comparisons against related works, such as PAMS.\n\n\n---\n\n[A] Regularization of Neural Networks using DropConnect \\\n[B] Convolutional Networks with Adaptive Inference Graphs\n\nQUESTIONS:\nRefer to weakness section.\n\nLIMITATIONS:\nThe authors have addressed the limitations in the supplementary section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a new quantization method for single image super resolution, including Redistribution-driven Learnable Quantizer (RLQ) and Depth-dynamic Quantized Architecture (DQA). The previous one diversifies the representation and gradient information of quantized values by redistribution in quantizers, the last one improve the performance of quantized SR and achieves resource adaptation in inference. Extensive experiments demonstate the superiority of QuantSR.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. The analysis of quantization for performance degradation is quite motivating and the proposed method is well-designed based on the analysis. The experiments show that QuantSR is better than previous methods.\n\nWEAKNESSES:\n1. The two main reasons for this performance degradation is much common, and not specific to super resolution task. Thus the method proposed in this paper is not SR-optimal.\n\n2. Depth-dynamic Quantized Architecture seems not much related with quantization, which is unneccessary.\n\n3. The designing of mapping method in Equation~5 is unclear, and is much similar with previous quantization method for image classifacation.\n\n4. Learnable quantizer and soft gradient is not new for quantization.\n\n5. Loss the latest baseline, DDTB[3].\n\n[1] Zhou S, Wu Y, Ni Z, et al. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients[J]. arXiv preprint arXiv:1606.06160, 2016.\n\n[2] Yamamoto K. Learnable companding quantization for accurate low-bit neural networks[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021: 5029-5038.\n\n[3] Zhong Y, Lin M, Li X, et al. Dynamic dual trainable bounds for ultra-low precision super-resolution networks[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 1-18.\n\nQUESTIONS:\n1. what is the theory of mapping method in Equation~5, or which function that inspires you?\n\n2. The true reason that cause the degradation is the high-dynamic activation [1][2], Can QuantSR solve this problem as well?\n\n3. Depth-dynamic Quantized Architecture is similar with dynamic network, could show more comparison with previous methods.\n\n\n[1] Tu Z, Hu J, Chen H, et al. Toward Accurate Post-Training Quantization for Image Super Resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 5856-5865.\n\n[2] Zhong Y, Lin M, Li X, et al. Dynamic dual trainable bounds for ultra-low precision super-resolution networks[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 1-18.\n\nLIMITATIONS:\nTh authors addressed the limitations in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose an low-bit quantization method for efficient image super-resolution by introduce a Redistribution-driven Learnable Quantizer (RLQ). Besides, the proposed Depth-dynamic Quantized Architecture (DQA) allows for the trade-off between efficiency and accuracy during inference through weight sharing. Experiments show that the proposed method outperforms existing state-of-the-art quantized SR networks in terms of accuracy while also providing more competitive computational efficiency.\n\nSTRENGTHS:\n- **Originality:** This paper propose a novel accurate quantization scheme for efficient image SR by introducing Redistribution-driven Learnable Quantizer (RLQ) and Depth-dynamic Quantized Architecture (DQA). The proposed scheme can be applied into both CNN- and Transformer- based SR networks.\n\n- **Quality:** This paper is well organized and easy to follow.\n\nWEAKNESSES:\n- Motivation is unclear: Although the proposed method outperforms existing state-of-the-art quantized SR networks in terms of trade-off between accuracy and complexity, the contribution is still limited. If we only look at the method section, it seems that the proposed module can be applied to Low-bit Quantization of model compression for various tasks. Are there any special designs for SR tasks? If so, it is suggested to provide more discussion to highlight the contributions.\n\n- As claimed in Introduction section, \"existing SR models rely on expensive computational resources, which significantly limits the real-world SR applications on resource-constrained edge devices. Therefore, there is an urgent requirement to develop model compression techniques for SR models to reduce the computational overhead\". As far as I know, the low complexity SR model is indeed an important research task, but the model compression is only one of them. In fact, many lightweight SR algorithms have been proposed recently, such as look-up table based method, *e.g.,* SR-LUT (CVPR2021), MuLUT(ECCV2022). It is suggested to discuss the superior of the proposed method.\n\n- The proposed method is evaluated only on two classical SR networks, SRResNet (CVPR'17) for CNN-based and SwinIR_S (ICCVW'21) for Transform-based backbones, respectively. Can the proposed method be extended to the recently published SR models?\n\nQUESTIONS:\nPlease refer to comments in the **weakness** part.\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}