SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haotong Qin\*1,2, Yulun Zhang\*2, Yifu Ding1, Yifan Liu2, Xianglong Liu2, Martin Danelljan2, Fisher Yu2 1Beihang University ETH Zürich
Low-bit quantization in image super-resolution (SR) has attracted copious attention in recent research due to its ability to reduce parameters and operations significantly. However, many quantized SR models suffer from accuracy degradation compared to their full-precision counterparts, especially at ultra-low bit widths (2-4 bits), limiting their practical applications. To address this issue, we propose a novel quantized image SR network, called QuantSR, which achieves accurate and efficient SR processing under low-bit quantization. To overcome the representation homogeneity caused by quantization in the network, we introduce the Redistribution-driven Learnable Quantizer (RLQ). This is accomplished through an inference-agnostic efficient redistribution design, which adds additional information in both forward and backward passes to improve the representation ability of quantized networks. Furthermore, to achieve flexible inference and break the upper limit of accuracy, we propose the Depth-dynamic Quantized Architecture (DQA). Our DQA allows for the trade-off between efficiency and accuracy during inference through weight sharing. Our comprehensive experiments show that QuantSR outperforms existing state-of-the-art quantized SR networks in terms of accuracy while also providing more competitive computational efficiency. In addition, we demonstrate the scheme's satisfactory architecture generality by providing QuantSR-C and QuantSR-T for both convolution and Transformer versions, respectively. Our code and models are released at https://github.com/htqin/QuantSR.
Single image super-resolution (SR) is the task of obtaining a high-resolution (HR) version of a low-resolution (LR) image by retrieving high-frequency details. It is an ill-posed problem since there are multiple HR candidates for the same LR input. To address this issue, researchers have explored the use of deep neural networks, including convolutional neural networks (CNNs) and Transformers [18, 26, 38, 41, 25, 3, 39, 43], to achieve high-quality reconstruction. However, existing SR models rely on expensive computational resources, which significantly limits the real-world SR applications on resource-constrained edge devices. Therefore, there is an urgent requirement to develop model compression techniques for SR models to reduce the computational overhead.
Model quantization [5, 14, 24, 8] has emerged as a powerful compression technique that compresses weights and activations in computing units (such as convolutional and linear layers) into low-bit representations. This results in heavy floating-point operations being converted into efficient integer ones, making quantization highly desirable for edge devices and friendly for memristor-based hardware. Meanwhile, the quantization process primarily focuses on parameter compression within operations. It is considered architecture- and task-agnostic, making it popular among the deep learning
* Equal contribution Corresponding authors: Yulun Zhang and Xianglong Liu
Urban100: img_017  $(\times 4)$
Figure 1: Visual comparison  $(\times 4)$  with quantized lightweight SR models in terms of 4-bit and 2-bit. We use SRResNet [20] as the full-precision SR backbone and quantize it with low bit width. We compare our QuantSR-C with recent quantization methods (i.e., DoReFa [44], PAMS [22], and CADyQ [10]). Our QuantSR-C performs obviously better than others in both 4-bit and 2-bit cases.
HR / Bit-width
DoReFa / 4-bit
PAMS / 4-bit
CADyQ / 4-bit
QuantSR-C / 4-bit
community. Additionally, the quantization function, known as the quantizer, can be customized for each bit width (usually 2-8 bits) to achieve a flexible balance between accuracy and efficiency.
Although quantization is known to enhance the efficiency of image SR models, it can also lead to significant performance degradation, particularly when using ultra-low bit width, e.g., 2-4 bits. It should be noted that another extreme 1-bit setting (also known as binarization) [7, 28] is not considered here. This is mainly because 1-bit quantization suffers from a much larger performance gap and has a different hardware implementation in practice when compared with low-bit quantization settings. Despite attempts to minimize the loss compared to their full-precision counterparts, a considerable performance gap still exists in existing low-bit quantized SR models.
We identify two main reasons for this performance degradation. Firstly, the quantizer aims to compress the parameters by discretizing them. It results in the homogenization of the parameter representation in the SR model and causes the loss of gradient information during the backward pass. Secondly, the current quantized SR models are specific bit-width mappings of the original full-precision counterparts, and the accuracy of the latter determines its upper performance bound. And once trained, the quantized SR model loses the ability to balance accuracy and efficiency when deployed, limiting its flexibility in practice usage. In this work, we set out to address these problems, to achieve efficient, yet accurate, flexible quantized image SR models.
We propose an accurate Quantized model for Super-Resolution (QuantSR). We first propose a method called Redistribution-driven Learnable Quantizer (RLQ) that mitigates the representation homogenization caused by discretization. We achieve this by designing a redistributable and learnable quantizer that improves the forward and backward information. Our RLQ results in significant diversification of the representation without incurring additional inference burden. We further propose a Depth-dynamic Quantized Architecture (DQA) that advances the performance of quantized SR and provides flexibility in inference resource utilization. We construct weight-sharing deep-dynamic models from an SR architecture base with higher accuracy upper bound, enabling us to achieve superior performance and resource adaptation during inference.
Our comprehensive experiments show that QuantSR outperforms existing quantized SR models across various bit widths (see Fig. 1) by a substantial margin. Notably, our QuantSR with 4 bits achieves performance comparable to existing methods using 8 bits. We also demonstrate the effectiveness of our QuantSR on CNN- and Transformer-based SR networks, highlighting its versatility across different architectures. Our main contributions are summarized as follows:
- We propose QuantSR, a novel accurate quantization scheme for efficient image SR. It provides the potential to deploy the SR application on edge devices and further narrow the performance gap between the quantized model and its full-precision counterpart.
- We propose a Redistribution-driven Learnable Quantizer (RLQ). Specifically, our RLQ diversifies quantized representation and gradient information by redistribution in quantizers. RLQ greatly enhances representations of quantized SR models with little inference burden.
- We propose a Depth-dynamic Quantized Architecture (DQA) to achieve superior performance with the same network depth. Such a dynamic strategy further allows for multiple models with different depths, which is resource adaptation during inference.
- We employ our QuantSR to compress CNN- and Transformer- based SR networks to lower bit-width, resulting in the corresponding quantized baselines, QuantSR-C and QuantSR-T. Our QuantSR achieves superior performance over SOTA quantized SR methods.
2.1 Efficient Image Super-Resolution (SR)
Recently, there has been a growing interest in efficient image super-resolution (SR) models due to their resource-friendly characteristics. Typically, researchers aim to develop lightweight networks through various methods, such as architecture design, neural architecture search (NAS), knowledge distillation (KD), and network pruning. For instance, Ahn et al. proposed a cascading method using a residual network [1]. While Hui et al. designed an information multi-distillation network [15]. Additionally, model compression techniques have also been applied to lightweight SR, such as NAS in FALSR [6], KD employed in training lighter SR student networks [9, 21], and network pruning in ASSL [42]. These techniques have obtained promising results, but they either overlook the fine-grained parameter redundancy or require dramatically additional computations, leaving significant room for compression from a bit-width perspective (i.e., low-bit quantization).
2.2 Low-bit Quantization for SR Networks
There exist two primary methods in the quantization field: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT). PTQ has gained popularity due to its capability to quantize models without necessitating retraining. But, the absence of training and the fixed parameters in the pre-trained model constrain its potential for achieving extreme low-bit quantization [5, 17, 14, 24, 8]. Fortunately, QAT provides us with an opportunity to leverage the entire training process to achieve aggressive low-bit quantization and has shown promising performance [22, 10, 44, 4]. This method enables more comprehensive model optimization and allows for optimal training of the quantized model. QAT is thus considered powerful to achieve extremely low-bit quantization, e.g., 2-4 bits.
The practical utility of existing SR networks on resource-limited devices is hindered by their extensive memory demands and computational burden. One of the significant obstacles to their usage is the extensive usage of floating-point storage and operations in these networks. As a result, there is ample opportunity to compress these networks by adopting a low-bit quantization approach, which can drastically reduce memory and computational requirements. This approach provides a compelling rationale for investigating the development of low-bit quantized SR models [22, 10, 12, 33, 16, 27]. Li et al. presented PAMS, which utilizes a trainable truncated parameter to dynamically determine the upper limit of the quantization range for SR models [22]. CADyQ is proposed as a technique designed for SR networks and optimizes the bit allocation for local regions and layers in the input image [10]. Hong et al. proposed DAQ, a channel-wise distribution-aware quantization scheme for SR models [12]. Despite these efforts, quantized SR models continue to exhibit a noticeable performance gap in comparison to their full-precision counterparts.
This section first provides an overview of low-bit quantization for image super-resolution (SR) and points out the limitations of existing low-bit SR networks. We then present our QuantSR, an accurate Quantization approach for efficient image Super-Resolution (see Fig. 2). QuantSR mainly consists of two novel techniques: Redistribution-driven Learnable Quantizer (RLQ) and Depth-dynamic Quantized Architecture (DQA). RLQ diversifies the information of forward and backward representations. While DQA pushes the accuracy upper bound and achieves dynamic-resource inference. Finally, we discuss how to utilize QuantSR for image SR and optimize the quantized SR network. The implementation details are also provided.
SR Network Architecture. We first outline the basic architecture of quantized SR networks. These networks are designed to take a low-resolution (LR) image denoted as  $I_{\mathrm{LR}}$  and produce a corresponding super-resolved image  $I_{\mathrm{SR}}$ . The process can be expressed as follows
$$
I _ {\mathrm {S R}} = \mathcal {M} \left(I _ {\mathrm {L R}}\right), \tag {1}
$$
where  $\mathcal{M}(\cdot)$  denotes the quantized super-resolution (SR) model. Specifically, the image SR network  $\mathcal{M}(\cdot)$  can be generally divided into three parts: low-level feature extractor  $\mathcal{E}_{\mathrm{L}}(\cdot)$ , high-level feature extractor  $\mathcal{E}_{\mathrm{H}}(\cdot)$ , and reconstruction  $\mathcal{R}(\cdot)$ . The common practice is to apply the network quantization to the high-level feature extractor [22], which costs most of the computational resources among the
Redistribution-driven Learnable Quantizer for Representation Recovery
Depth-dynamic Quantized Architecture for Flexible and Accurate Inference
Figure 2: Overview of our QuantSR for image super-resolution network. The upper part is the Redistribution-driven Learnable Quantizer (RLQ), where the representation information of quantizers is enhanced in forward and backward propagation of the SR network. The lower one is the Depth-dynamic Quantized Architecture (DQA) that breaks the accuracy limitation of quantized SR architecture and allows for the flexible accuracy-efficiency trade-off in inference.
whole SR network. Thus, the image SR process in Eq. (1) can be further expressed as follows
$$
I _ {\mathrm {S R}} = \mathcal {M} \left(I _ {\mathrm {L R}}\right) = \mathcal {R} \circ \mathcal {E} _ {\mathrm {H}} \circ \mathcal {E} _ {\mathrm {L}} \left(I _ {\mathrm {L R}}\right), \tag {2}
$$
where  $\mathcal{E}_{\mathrm{H}}$  denotes the high-level feature extractor and  $\circ$  denotes the connection among network parts.
Quantization Framework. For the quantized SR network, the weight  $\mathbf{w}$  and activation  $\mathbf{a}$  of computing units (such as convolution, linear, and matmul layers) are compressed to low bit-widths by the quantizers  $Q_{w}(\mathbf{w})$  and  $Q_{a}(\mathbf{a})$ , respectively. As the common practice [35], these parameters are compressed to lower  $b$  bit-widths by following quantizer  $Q$  with a symmetric mode:
$$
Q ^ {b} (\boldsymbol {x}) = \operatorname {r o u n d} \left(\frac {\operatorname {c l i p} (\boldsymbol {x})}{v _ {b}}\right) v _ {b}, \tag {3}
$$
where  $\pmb{x}$  denotes the weight  $\pmb{w}$  or activation  $\pmb{a}$ , and  $Q^{b}(\cdot)$  denotes their quantizer. The function  $\mathrm{clip}(\cdot) = \max (\min (\pmb{x},a), - a)$  is to limit the range of the inputs, where  $a$  represents the maximum of the absolute value of  $\pmb{x}$ . And  $v_{b}$  is the map function that scales the higher precision inputs to their lower bit reflections as  $v_{b} = \frac{a}{2^{b - 1} - 1}$ . Since the direct differentiation of the discrete quantizer causes all-zero gradients and hinders the backward propagation, straight-through estimation (STE) is used to approximate the gradient of parameters:
$$
\frac {\partial Q ^ {b} (\boldsymbol {x})}{\partial \boldsymbol {x}} = \left\{ \begin{array}{l l} 1 & \text {i f} \boldsymbol {x} \in (- a, a) \\ 0 & \text {o t h e r w i s e} \end{array} . \right. \tag {4}
$$
After quantizing the SR networks, the storage size and computation load can be significantly reduced due to the low bit-width and efficient integer operations.
3.2 Redistribution-driven Learnable Quantizer
3.2.1 Quantization-induced Representation Degradation
While quantization promises less storage and faster inference, it also causes significant performance degradation of compressed SR models. The intuitive reason is that the discretization brought by the quantizer hinders the information of representations in both forward and backward propagations. In forward propagation, the information of quantized parameters is significantly reduced due to the limitation of low bit-width, especially in ultra-low bit quantization. For example, in 2-bit quantization, the number of candidates for a single element drops from  $2^{32}$  in floating point form to only 4. Furthermore, quantizers such as Eq. (3) also lead to homogeneous quantization mappings throughout the SR network. In backward propagation, since the gradient of quantizers cannot be used directly, the gradient obtained by estimators cannot completely reflect the effect of quantizers. This phenomenon results in an information mismatch always between backward and forward propagation, with the former consistently lagging behind. Therefore, the quantizer should be improved to achieve more accurate forward and backward processes, enhancing the representations of the quantized SR network.
3.2.2 Redistribution-driven Learnable Quantizer for Representation Recovery
In this work, we proposed a novel Redistribution-driven Learnable Quantizer (RLQ) for our QuantSR (Fig. 3). RLQ is driven by parameter redistribution designs, which introduce learnable parameters into the quantizer to diversify the forward quantization mapping in the whole quantized SR model. The transformation function is added in each quantization interval to enrich the gradient information in the backward. Our RLQ can be expressed as
Figure 3: Forward and backward propagation of RLQ. Blue notations are learnable parameters.
$$
Q _ {\mathrm {R L Q}} ^ {b} (\boldsymbol {x}, \hat {v} _ {b}, \hat {\tau}) = \operatorname {r o u n d} \left(\phi \left(\frac {\operatorname {c l i p} (\boldsymbol {x} + \hat {\tau})}{\hat {v} _ {b}}\right)\right) \hat {v} _ {b}, \quad \phi (\boldsymbol {x}) = \frac {\tanh  (2 (\boldsymbol {x} - \lfloor \boldsymbol {x} \rfloor) - 1)}{\tanh  1} + \lfloor \boldsymbol {x} \rfloor + 2 ^ {- 1}, \tag {5}
$$
where the  $\hat{v}_b$  and  $\hat{\tau}$  denote the learnable interval and mean-shifting parameters, respectively, which are initialized as  $\frac{a}{2^{b-1}-1}$  and 0. The function  $\phi(\cdot)$  is embedded within each quantization interval as a transformation function. This function does not alter the rounded value but rather reduces the gradient of elements that are distant from the interval center. The aim of this function is to incorporate the information reflecting the quantizer's actual behavior while maintaining optimization stability.
The derivative w.r.t. the input and learnable parameters used in the backward pass are
$$
\frac {\partial Q _ {\mathrm {R L Q}} ^ {b} \left(\boldsymbol {x} , \hat {v} _ {b} , \hat {\tau}\right)}{\partial \boldsymbol {x}} = \left\{ \begin{array}{l l} \frac {\partial \phi (\boldsymbol {x} + \hat {\tau})}{\partial \boldsymbol {x}} & \text {i f} \boldsymbol {x} \in (- a, a), \\ 0 & \text {o t h e r w i s e} \end{array} , \quad \frac {\partial Q _ {\mathrm {R L Q}} ^ {b} \left(\boldsymbol {x} , \hat {v} _ {b} , \hat {\tau}\right)}{\partial \hat {\tau}} = 1 + \frac {\partial \phi (\boldsymbol {x} + \hat {\tau})}{\partial \hat {\tau}}, \right. \tag {6}
$$
$$
\frac {\partial Q _ {\mathrm {R L Q}} ^ {b} (\boldsymbol {x} , \hat {v} _ {b} , \hat {\tau})}{\partial \hat {v} _ {b}} = \left\{ \begin{array}{l l} \text {r o u n d} \left(\frac {\boldsymbol {x} + \hat {\tau}}{\hat {v} _ {b}}\right) + \frac {\partial \phi ((x + \hat {\tau}) \hat {v} _ {b} ^ {- 1})}{\partial \hat {v} _ {b}} & \text {i f} \boldsymbol {x} \in (- a, a) \\ - a \text {o r} a & \text {o t h e r w i s e} \end{array} \right..
$$
Our proposed  $Q_{\mathrm{RLQ}}^b (\pmb {x},\hat{\upsilon}_b,\hat{\tau})$  quantizer is embedded in the entire quantized SR network for quantization-aware training to compress weights and activations to low bit-width.
After applying RLQ, both the forward and backward representations of QuantSR are greatly enhanced. (1) For the forward propagation, since introducing the learnable mean-shifting and quantization scale parameters to RLQ, the quantizers are gradually diversified during training. The learnable parameters of the quantizers in the entire network diversify continuously throughout the training process, which significantly reduces the impact of parameter representation degradation caused by parameter discretization. (2) For the backward propagation, the embedded transformation function  $\phi(\cdot)$  can introduce information in the gradient to help accurate updates. Compared with using STE directly, RLQ provides additional gradient-guided information, that is, in each quantization interval, the farther away from the center of the region, the smaller the gradient. The transformation function is embedded in the round function and can be merged at inference time without additional burden.
3.3 Depth-dynamic Quantized Architecture
3.3.1 Latent Architecture Bounds for Quantized SR Network
Quantization is a highly effective compression technique that employs low-bit parameters and efficient bitwise operations, making it ideal for fast inference on resource-constrained devices. One of the key benefits of quantization is that it can be applied to SR models without affecting their
Figure 4: The architecture and training pipeline of DQA, which uses a two-stage strategy to train weight-shared variants with different depths.
underlying architecture. However, in our investigation, we observed that the quantized SR network's accuracy is typically limited by that of its full-precision counterpart and that quantization inevitably leads to some loss in accuracy. Furthermore, since SR models come in various sizes to meet different accuracy requirements, striking a balance between accuracy and efficiency becomes critical in practical applications. To address this challenge, we propose a dynamic depth quantization architecture that allows for immediate and adaptive accuracy-efficiency trade-offs in SR applications. This architecture also generates quantization models with greater accuracy at the same depth.
3.3.2 Depth-dynamic Quantized Architecture for Flexible and Accurate Inference
We propose a Depth-dynamic Quantized Architecture (DQA) for the quantized high-level feature extractor  $\mathcal{E}_{\mathrm{H}}^{\prime}$  (Fig. 4), which selects a more lightweight model with fewer layers at runtime, directly reducing the computational consumption. We first build the main quantized architecture, which is computationally expensive with learnable short connections. The main architecture  $\mathcal{E}_{\mathrm{H}}^{\mathrm{DQA}}$  containing  $2N$  blocks ( $N \in \{2^n, n \in \mathbf{Z}^+\}$ ), where the initialized  $i$ -th block  $\hat{\Phi}_i^{\mathrm{DQA}}$  is expressed as
$$
\hat {\Phi} _ {i} ^ {\mathrm {D Q A}} \left(\boldsymbol {x} _ {i}\right) = \varphi_ {i} \left(\boldsymbol {x} _ {i}\right) + \alpha_ {i} \boldsymbol {x} _ {i}, \tag {7}
$$
where  $\pmb{x}_i$  denotes the input feature of  $i$ -th block,  $\varphi(\cdot)$  denotes the quantized feature extractor consisting of convolution and activation layers, and  $\alpha$  denotes the learnable scaling of the skip connection.
The training strategy of our architecture is divided into two stages. The first stage is a warm-up, which takes 1/5 of the training iterations. In this stage, the entire structure is optimized according to the original loss, and the skip connection is updated during the training process. In the next stage, the architecture is derived into a dynamic-depth version. Different derivatives are jointly trained. Specifically, the dynamic-depth architecture is allowed to be derived into different variants with the  $100\%$ ,  $50\%$ , and  $25\%$  number of blocks. The variants of DQA can be expressed as
$$
\mathcal {E} _ {\mathrm {H}} ^ {\mathrm {D Q A}} (x, \operatorname {v a r}) = \prod_ {i = 1} ^ {n} \Phi_ {i} ^ {\mathrm {D Q A}} \left(\boldsymbol {x} _ {i}, \operatorname {v a r}\right) = \prod_ {i = 1} ^ {n} \left(b _ {i} (\operatorname {v a r}) \varphi_ {i} \left(\boldsymbol {x} _ {i}\right) + \alpha_ {i} \boldsymbol {x} _ {i}\right), \tag {8}
$$
where  $\prod$  denotes composition,  $\mathrm{var} \in (0, 100\%]$  and  $b_i(\mathrm{var}) \in \{0, 1\}$ . The coefficient  $b_i(\mathrm{var})$  of  $i$ -th block is determined based on the value of its scaling  $\alpha_i$  of each block. The  $b[\mathrm{var}]$  is determined based on the value of its short connection  $\alpha$  in each block.
For a specific variant var, we select the corresponding proportion of blocks with the smallest short connection values (meaning that the block is less likely to be skipped) after warming up and set their coefficient  $b[\text{var}]$  to 1. The coefficients corresponding to other blocks are set to 0 and their feature extraction processes are skipped. The reserved short connections can also be merged into the previous blocks to achieve acceleration during inference.
3.4 Training Pipeline of QuantSR