SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To optimize the SR model quantized by the proposed QuantSR, we adopt the joint training strategy to update multiple DQA variants, where the RLQ is applied to quantized weight and activation. On the given training dataset  $D = \{I_{\mathrm{LR}}^i, I_{\mathrm{HR}}^i\}_{i=1}^K$  with  $K$  low-resolution inputs  $I_{\mathrm{LR}}$  and corresponding high-resolution ground-truth  $I_{\mathrm{HR}}$ , for a specific variant var of quantized SR model  $\mathcal{M}$ , the conventional pixel-wise  $\mathcal{L}_{\mathrm{var}}^{\mathrm{PIX}}$  loss can be expressed as
$$
\mathcal {L} _ {\text {v a r}} ^ {\mathrm {P I X}} = \frac {1}{n} \sum_ {i = 1} ^ {k} \left\| I _ {\mathrm {H R}} ^ {i} - \mathcal {M} \left(I _ {\mathrm {L R}} ^ {i}, \text {v a r}\right) \right\| _ {\ell_ {1}}. \tag {9}
$$
And the losses of different variants of DQA are accumulated and used to jointly update the network:
$$
\mathcal {L} _ {\mathrm {tot}} = \sum_ {\operatorname {var} \in \operatorname {var}} \frac {\mathcal {L} _ {\operatorname {var}} ^ {\mathrm {PIX}}}{| \operatorname {var} | _ {\ell_ {0}}}, \quad \operatorname {var} = \{100 \%, 50 \%, 25 \%\}, \tag{10}
$$
where  $|\cdot|_{\ell_0}$  denotes  $\ell_0$  normalization. With the well-designed quantizer, architecture, and training pipeline, our QuantSR enjoys accuracy and efficiency with CNN- and Transformer-based backbones.
Dataset. We adhere to the standard procedure in image SR, training on DIV2K [32] and evaluating on Set5 [2], Set14 [37], B100 [29], Urban100 [13], and Manga109 [30].
Evaluation. We report the reconstruction performance measured by PSNR and SSIM [34] on the Y channel of the YCbCr space. And we also evaluate the amounts of parameters and computation for the quantized SR model and the full-precision one.
Proposed Quantization Baselines. We quantize image SR models with our proposed method, including the CNN- and Transformer- based architectures. For CNN-based SR models, we follow PAMS [22] and CADyQ [10], and use SRResNet as the backbone. For Transformer-based SR models,
Method	#Bit (w/a)	#Blk	Set5	Set14	B100	Urban100	Manga109
PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
SRResNet	32/32	16	38.00	0.9605	33.59	0.9171	32.19	0.8997	32.11	0.9282	38.56	0.9770
DoReFa	4/4	16	37.60	0.9589	33.10	0.9133	31.87	0.8954	30.87	0.9151	37.65	0.9746
RLQ	4/4	16	37.72	0.9594	33.25	0.9147	31.98	0.8971	31.15	0.9187	37.99	0.9755
DAQ	4/4	32	37.77	0.9596	33.25	0.9150	32.00	0.8973	31.30	0.9200	38.12	0.9758
16	37.69	0.9593	33.15	0.9140	31.92	0.8963	30.65	0.9163	37.80	0.9751
8	37.51	0.9584	32.99	0.9122	31.78	0.8943	30.50	0.9111	37.38	0.9736
QuantSR	4/4	32	37.95	0.9603	33.59	0.9177	32.17	0.8996	31.98	0.9274	38.62	0.9771
16	37.88	0.9600	33.45	0.9166	32.10	0.8988	31.72	0.9249	38.37	0.9766
8	37.74	0.9595	33.25	0.9150	31.99	0.8973	31.27	0.9199	37.98	0.9757
Table 1: Ablation study ( $\times 2$  scale) about our proposed Redistribution-driven Learnable Quantizer for Representation Recovery (RLQ) and Depth-dynamic Quantized Architecture (DQA) for flexible and accurate inference.  $w / a$  denotes the weight/activation bits. #Blk' means residual block number.
we quantize lightweight SwinIR [25]. The implementations of comparison methods follow the official codes [11, 23] and are trained with the same settings as ours. We quantize both the weight and activation of the body part in all the models with low bit-width (e.g., 2, 4, or 8). We denote  $w$ -bit weight and  $a$ -bit activation as  $w / a$ . To ensure a fair comparison, QuantSR reports the performance of the same-depth variant to other networks (e.g., 16 blocks for SRResNet) in the comparison. The complete efficiency performance (32, 16, and 8 blocks) is demonstrated in Sec. 4.3.
Training Strategy. In our training process, we follow the practices of previous studies [26, 40, 36, 25] by conducting data augmentation, which involves random rotations of  $90^{\circ}$ ,  $180^{\circ}$ ,  $270^{\circ}$ , and horizontal flipping. The models are trained for 300K iterations, with each training batch consisting of 32 image patches. The input size of each patch is  $64 \times 64$ . To optimize our model, we utilize the Adam optimizer [19]. The learning rate is initially set to  $2 \times 10^{-4}$  and is then halved at the 250K-th iteration. All experiments are conducted on NVIDIA RTX A6000 GPUs with PyTorch [31].
To showcase the efficacy of the techniques employed in our QuantSR, we conduct ablation studies on RLQ and DQA. We employ SRResNet [20] as the image SR backbone and trained it for 200K iterations with a 4-bit setting. To establish a quantization baseline, we used the vanilla quantization method, DoReFa [44]. Subsequently, we incorporate RLQ and/or DQA into SRResNet and binarized it. The results of our experiments are presented in Tab. 1, where we report the PSNR/SSIM values on five test benchmarks. Additionally, we provide detailed analyses of all variants (e.g., 32, 16, and 8 blocks) for the depth-dynamic DQA and QuantSR.
Redistribution-driven Learnable Quantizer. The vanilla version of the quantized SR model, DoReFa [44], has demonstrated basic SR performance. However, we have improved upon this by incorporating RLQ for activations, which allows us to learn the shift and numerical scale. Our proposed RLQ has significantly enhanced the performance of the quantized network while reducing the performance drop. In Tab. 1, it is clear that our RLQ achieves remarkable improvements in PSNR and SSIM with little additional computation overhead, particularly with the 4-bit setting, where we achieve around  $0.12 - 0.28$  dB and 0.0005-0.0036 improvements, respectively. Our enhanced quantizer significantly improves the representation of information during both forward and backward propagation, leading to improved performance in quantized SR networks. This enhancement results in enhanced representation capabilities and more precise optimization.
Depth-dynamic Quantized Architecture. The proposed dynamic quantization algorithm (DQA) can adaptively operate on different versions of a model, based on the energy budget of real-world application scenarios. As shown in Tab. 1, we can use the proposed optimization scheme to jointly train models with varying degrees of quantization, where the variable in Eq. (10) can take values of  $100\%$ ,  $50\%$ , and  $25\%$ . The 16-block DQA has achieved an improvement of around  $0.05\mathrm{dB}$  and  $0.0010$  over the baseline, surpassing the upper limit of accuracy achievable by full-precision models. Notably, the DQA can effectively reduce computational complexity and memory usage by selectively ranking and skipping blocks based on a percentage threshold. Consequently, the 8-block model with the 4/4 setting has the smallest computational and storage footprint. The 32-block variant exhibits a noteworthy accuracy improvement at the cost of increased computational overhead, e.g., the improvement in Set5 is  $0.17\mathrm{dB}$  and  $0.0005$ . The accuracy results of QuantSR also demonstrate that employing both RLQ and DQA techniques can enhance the performance jointly and even elevate it to full precision levels, accomplished by utilizing an efficient 4-bit width.
Method	Scale	#Bit (w/a)	Set5	Set14	B100	Urban100	Manga109
PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM	PSNR	SSIM
Bicubic	×2	-/-	33.66	0.9299	30.24	0.8688	29.56	0.8431	26.88	0.8403	30.80	0.9339
SRResNet [20]	×2	32/32	38.00	0.9605	33.59	0.9171	32.19	0.8997	32.11	0.9282	38.56	0.9770
SwinIR_S [25]	×2	32/32	38.14	0.9611	33.86	0.9206	32.31	0.9012	32.76	0.9340	39.12	0.9783
DoReFa [44]	×2	8/8	37.32	0.9520	32.90	0.8680	31.69	0.8504	30.32	0.8800	37.01	0.9450
CADyQ [10]	×2	8/8	37.79	0.9590	33.37	0.9150	32.02	0.8980	31.53	0.9230	38.06	0.9760
DoReFa [44]	×2	4/4	37.31	0.9510	32.48	0.9091	31.64	0.8901	30.18	0.8780	36.95	0.9440
PAMS [22]	×2	4/4	37.67	0.9588	33.19	0.9146	31.90	0.8966	31.10	0.9194	37.62	0.9400
CADyQ [10]	×2	4/4	37.58	0.9580	33.14	0.9140	31.87	0.8960	30.94	0.9170	37.31	0.9740
QuantSR-C (ours)	×2	4/4	37.80	0.9597	33.35	0.9158	32.04	0.8979	31.46	0.9221	38.25	0.9762
QuantSR-T (ours)	×2	4/4	38.10	0.9604	33.65	0.9186	32.21	0.8998	32.20	0.9295	38.85	0.9774
DoReFa [44]	×2	2/2	36.91	0.9470	32.55	0.9071	31.41	0.8868	29.60	0.8740	36.132	0.9410
PAMS [22]	×2	2/2	34.04	0.8270	30.91	0.8751	30.11	0.8592	27.57	0.8400	31.79	0.9110
CADyQ [10]	×2	2/2	19.44	0.5610	18.51	0.4810	19.70	0.4760	17.97	0.4550	17.346	0.5830
QuantSR-C (ours)	×2	2/2	37.57	0.9589	33.09	0.9136	31.84	0.8954	30.77	0.9149	37.60	0.9745
QuantSR-T (ours)	×2	2/2	37.55	0.9587	33.12	0.9143	31.89	0.8958	30.96	0.9172	37.61	0.9745
Bicubic	×4	-/-	28.42	0.8104	26.00	0.7027	25.96	0.6675	23.14	0.6577	24.89	0.7866
SRResNet [20]	×4	32/32	32.16	0.8951	28.60	0.7822	27.58	0.7364	26.11	0.7870	30.46	0.9089
SwinIR_S [25]	×4	32/32	32.44	0.8976	28.77	0.7858	27.69	0.7406	26.47	0.7980	30.92	0.9151
DoReFa [44]	×4	4/4	29.57	0.8369	26.82	0.7352	26.47	0.6971	23.75	0.6898	27.89	0.8634
PAMS [22]	×4	4/4	31.59	0.8851	28.20	0.7725	27.32	0.7220	25.32	0.7624	28.86	0.8805
CADyQ [10]	×4	4/4	31.48	0.8830	28.05	0.7690	27.21	0.7240	25.09	0.7520	28.82	0.8840
QuantSR-C (ours)	×4	4/4	32.00	0.8924	28.50	0.7799	27.52	0.7342	25.88	0.7807	30.15	0.9040
QuantSR-T (ours)	×4	4/4	32.18	0.8941	28.63	0.7822	27.59	0.7367	26.11	0.7871	30.49	0.9087
DoReFa [44]	×4	2/2	30.54	0.8610	27.50	0.7538	26.90	0.7098	24.44	0.7242	27.31	0.8502
PAMS [22]	×4	2/2	29.20	0.8239	26.61	0.7273	26.36	0.6934	23.58	0.6812	25.59	0.8012
CADyQ [10]	×4	2/2	19.67	0.5380	19.30	0.4740	19.80	0.4620	17.97	0.4360	17.30	0.5640
QuantSR-C (ours)	×4	2/2	31.30	0.8819	28.08	0.7694	27.23	0.7246	25.13	0.7537	28.81	0.8844
QuantSR-T (ours)	×4	2/2	31.53	0.8845	28.16	0.7715	27.28	0.7274	25.26	0.7609	29.06	0.8898
Table 2: Quantitative results. SRResNet and SwinIR-S are used as full-precision backbones.  $w / a$  denotes the weight/activation bits. The best and second best results are colored with red and cyan.
4.3 Image Super-Resolution (SR)
We select SRResNet [20] (i.e., 1,367K  $(\times 2)$  and 1,515K  $(\times 4)$  Params) and SwinIR_S [25] (i.e., lightweight SwinIR with  $910\mathrm{K}(\times 2)$  and  $930\mathrm{K}(\times 4)$  Params) as CNN and Transformer backbones, respectively. The corresponding QuantSR variants are QuantSR-C and QuantSR-T, respectively. And we then compare with recent quantization methods, e.g., DoReFa [44], PAMS [22], and CADyQ [10].
Quantitative Results. In Tab. 2, we provide PSNR and SSIM. Our 4-bit QuantSR-C achieves comparable or superior PSNR/SSIM scores than 8-bit DoReFa and CADyQ with scale  $\times 2$ . In the 4-bit case, our QuantSR-C achieves  $0.52\mathrm{dB} / 0.0051$  ( $\times 2$ ) and  $0.79\mathrm{dB} / 0.0287$  ( $\times 4$ ) higher PSNR/SSIM values than CADyQ [10] on Urban100. The case of the Transformer is more challenging, as we observe a larger performance gap between the quantized QuantSR-T and the full-precision model compared to the quantized SRResNet. However, our QuantSR effectively narrows this gap, with a constant improvement compared to previous techniques, particularly at lower bit widths. For instance, with a 2-bit setting on SRResNet, our QuantSR-C outperforms PAMS by a significant  $3.20\mathrm{dB} / 0.0749$  ( $\times 2$ ) and  $1.55\mathrm{dB} / 0.0725$  ( $\times 4$ ) on Urban100. It also is significantly higher than the performance obtained with the SOTA CADyQ and approaches that of the full-precision model. Similarly, on Urban100, our QuantSR-T improves  $1.26\mathrm{dB} / 0.0125$  ( $\times 2$ ) over CADyQ in the 4-bit setting.
Compression Ratio. By utilizing a combination of quantization and dynamic, lightweight architecture, compression ratio of image SR models is significantly increased. In Tab. 3, we present the compression ratio and speedup in terms of Params and Ops, respectively. By quantizing the
full-precision SRResNet to 2 and 4 bit-widths, we can reduce both the model size (i.e., Params) and operations (i.e., Ops) considerably. Following PAMS [22], we only quantize the weights and activations in the high-level feature extractor part, but we calculate the compression ratio and speedup for the entire model. With a 4-bit setting, our QuantSR-C achieves approximately  $77.8\%$  and  $77.5\%$  compression ratio for parameters and operations, respectively, through quantization alone. By integrating DQA for the 8-block variant, we were able to push the compression ratios to  $83.1\%$  and  $82.9\%$ , respectively. Furthermore, with a 2-bit setting, the operation compression ratios of 16- and 8-block variants can even reach  $87.9\%$  and  $88.3\%$ , respectively. Regarding the variant with 32
Method	#Bit (w/a)	#Blk	Params (K) (↓ Ratio)	Ops (G) (↓ Ratio)	Urban100
PSNR	SSIM
SRResNet	32/32	16	1,367 (0%)	90.1 (0%)	32.16	0.8951
QuantSR-C	4/4	32	451 (↓ 67.0%)	29.9 (↓ 66.9%)	32.17	0.8943
16	303 (↓ 77.8%)	20.2 (↓ 77.5%)	32.00	0.8924
8	230 (↓ 83.1%)	15.4 (↓ 82.9%)	31.75	0.8894
QuantSR-C	2/2	32	170 (↓ 87.6%)	11.5 (↓ 87.2%)	31.48	0.8849
16	161 (↓ 88.2%)	10.9 (↓ 87.9%)	31.30	0.8819
8	156 (↓ 88.6%)	10.6 (↓ 88.3%)	31.04	0.8771
Table 3: Compression ratio of 2-bit and 4-bit SRResNet  $(\times 2)$ , and their input sizes are  $3\times 256\times 256$  for calculating Ops.
Urban100: img_098  $(\times 4)$
Bicubic / -
SRResNet [20] / 32-bit
DoReFa [44] / 4-bit
Urban100: img_047  $(\times 4)$
DoReFa [44] / 2-bit
(a)  $\hat{v}_b$  for weight quantizer.
Figure 6: The statistics of RLQ demonstrate its representation improvement in forward and backward.
(b)  $\hat{\tau}$  for activation quantizer.
Figure 5: Visual comparison  $(\times 4)$  with lightweight SR in terms of 4-bit and 2-bit.
(c) Gradient effect of  $\phi (\cdot)$
blocks, 4-bit QuantSR-C model also achieves notable parameter and operation savings of  $67.0\%$  and  $66.9\%$ , respectively, while obtaining comparable accuracy with the 16-block full-precision model.
Visual Results. In Fig. 5, we provide visual results of representative methods with scale  $\times 4$  in terms of 2-bit and 4-bit cases. For each case, we compare with several quantization methods, like DoReFa [44], PAMS [22], and CADyQ [10]. Our QuantSR-C and QuantSR-T recover more structural details and alleviate more blurring artifacts than other quantization methods. Consequently, the visual difference between our QuantSR and the full-precision one is small. These visual comparisons further demonstrate the effectiveness of our QuantSR, which is consistent with the observations in Tab. 2.
Training Statistics. In Fig. 6, we present statistics of the learnable parameters and function effects in our RLQ for every 100 iterations of training. Specifically, we report the statistics of  $\hat{v}_b$ , which represents the activations,  $\hat{\tau}$ , which represents the weights, and the gradient effects of  $\phi(\cdot)$ . Figures 6 (a) and (b) illustrate that the learnable parameters are initially initialized in close proximity to each other. However, as training progresses, the optimized parameters diverge in various directions. This indicates that the quantizers containing these parameters in QuantSR are becoming more diverse, thus recovering the lost representation information due to discretization. In Fig. 6 (c), we depict the gradient change induced by a specific embedded transformation function  $\phi(\cdot)$  during backpropagation. While the forward results and training stability remain unaffected, the gradient guidance information provided by this function is significant. Consequently, it promotes the optimization of QuantSR, leading to an overall improvement in the results.
In this paper, we propose QuantSR, a novel quantized image super-resolution (SR) network that overcomes the limitations of existing low-bit quantized SR models. The proposed network employs a Redistribution-driven Learnable Quantizer (RLQ) to improve the representation ability of quantized networks and a Depth-dynamic Quantized Architecture (DQA) to achieve flexible inference and break the upper limit of accuracy. The authors demonstrate that QuantSR outperforms existing state-of-the-art quantized SR networks in both accuracy and computational efficiency. Additionally, we provide QuantSR-C and QuantSR-T for both convolution and Transformer versions, respectively, to demonstrate the scheme's satisfactory architecture generality.
Acknowledgements. This work was supported by the National Natural Science Foundation of China (No. 62022009), the State Key Laboratory of Software Development Environment (SKLSDE-2022ZX-23), and the Huawei Technologies Oy (Finland) Project.