Title: 2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution
Venue: NeurIPS 2024
Accepted: yes
Keywords: Quantization

Image super resolution

Low bit

Post-training quantization.
Abstract: Low-bit quantization has become widespread for compressing image super-resolution (SR) models for edge deployment, which allows advanced SR models to enjoy compact low-bit parameters and efficient integer/bitwise constructions for storage compression and inference acceleration, respectively. However, it is notorious that low-bit quantization degrades the accuracy of SR models compared to their full-precision (FP) counterparts. Despite several efforts to alleviate the degradation, the transformer-based SR model still suffers severe degradation due to its distinctive activation distribution. In this work, we present a dual-stage low-bit post-training quantization (PTQ) method for image super-resolution, namely 2DQuant, which achieves efficient and accurate SR under low-bit quantization. The proposed method first investigates the weight and activation and finds that the distribution is characterized by coexisting symmetry and asymmetry, long tails. Specifically, we propose Distribution-Oriented Bound Initialization (DOBI), using different searching strategies to search a coarse bound for quantizers. To obtain refined quantizer parameters, we further propose Distillation Quantization Calibration (DQC), which employs a distillation approach to make the quantized model learn from its FP counterpart. Through extensive experiments on different bits and scaling factors, the performance of DOBI can reach the state-of-the-art (SOTA) while after stage two, our method surpasses existing PTQ in both metrics and visual effects. 2DQuant gains an increase in PSNR as high as 4.52dB on Set5 (x2) compared with SOTA when quantized to 2-bit and enjoys a 3.60x compression ratio and 5.08x speedup ratio. The code and models are available at https://github.com/Kai-Liu001/2DQuant.

2DQuant: Low-bit Post-Training Quantization for Image Super-Resolution

Abstract

Low-bit quantization has become widespread for compressing image superresolution (SR) models for edge deployment, which allows advanced SR models to enjoy compact low-bit parameters and efficient integer/bitwise constructions for storage compression and inference acceleration, respectively. However, it is notorious that low-bit quantization degrades the accuracy of SR models compared to their full-precision (FP) counterparts. Despite several efforts to alleviate the degradation, the transformer-based SR model still suffers severe degradation due to its distinctive activation distribution. In this work, we present a dual-stage lowbit post-training quantization (PTQ) method for image super-resolution, namely 2DQuant , which achieves efficient and accurate SR under low-bit quantization. The proposed method first investigates the weight and activation and finds that the distribution is characterized by coexisting symmetry and asymmetry, long tails. Specifically, we propose Distribution-Oriented Bound Initialization (DOBI), using different searching strategies to search a coarse bound for quantizers. To obtain refined quantizer parameters, we further propose Distillation Quantization Calibration (DQC), which employs a distillation approach to make the quantized model learn from its FP counterpart. Through extensive experiments on different bits and scaling factors, the performance of DOBI can reach the state-of-the-art (SOTA) while after stage two, our method surpasses existing PTQ in both metrics and visual effects. 2DQuant gains an increase in PSNR as high as 4.52dB on Set5 ( × 2 ) compared with SOTA when quantized to 2-bit and enjoys a 3.60 × compression ratio and 5.08 × speedup ratio. The code and models are available at https://github.com/Kai-Liu001/2DQuant.

1 Introduction

As one of the most classical low-level computer vision tasks, image super-resolution (SR) has been widely studied with the significant development of deep neural networks. With the ability to reconstruct high-resolution (HR) image from the corresponding low-resolution (LR) image, SR has been widely used in many real-world scenarios, including medical imaging [13, 21, 19], surveillance [46, 39], remote sensing [1], and mobile phone photography. With massive parameters, DNN-based SR models always require expensive storage and computation in the actual application. Some works have been proposed to reduce the demand for computational power of SE models, like lightweight architecture design and compression. One kind of approach investigates lightweight and efficient models as the backbone for image SR. This progression has moved from the earliest convolutional neural network (CNNs) [10, 11, 25, 49] to Transformers [48, 32, 44, 42, 4, 3] and their combinations. The parameter number significantly decreased while maintaining or even enhancing performance. The other kind of approach is compression, which focuses on reducing the parameter ( e.g. , pruning and distillation) or bit-width (quantization) of existing SR models.

Model quantization [7, 9, 20, 31] is a technology that compresses the floating-point parameters of a neural network into lower bit-width. The discretized parameters are homogenized into restricted

candidate values and cause heterogenization between the FP and quantized models, leading to severe performance degradation. Considering the process, quantization approaches can be divided into quantization-aware training (QAT) and post-training quantization (PTQ). QAT simultaneously optimizes the model parameters and the quantizer parameters [6, 16, 26, 50], allowing them to adapt mutually, thereby more effectively alleviating the degradation caused by quantization. However, QAT often suffers from a heavy training cost and a long training time, and the burden is even much heavier than the training process of the FP counterparts, which necessitates a large amount of compatibility and makes it still far from practical in training-resource-limited scenarios.

Fortunately, post-training quantization emerges as a promising way to quantize models at a low training cost. PTQ fixes the model parameters and only determines the quantizer parameters through search or optimization. Previous researches [41, 26] on PTQ for SR has primarily focused on

HR

Bicubic

Percentile [27]

DBDC+Pac [41]

Ours

FP

CNN-based models such as EDSR [33] and SRResNet [24]. However, these quantization methods are not practical for deployment for two reasons. Firstly , these CNN-based models themself require huge space and calculation resources. Their poor starting point makes them inferior to advanced models in terms of parameters and computational cost, even after quantization. As shown in Table 1, the light version of SwinIR needs only 16.2% parameters and 15.9% FLOPs compared with quantized EDSR. But its PSNR metric is close to that of the FP EDSR. While the previous PTQ algorithm, DBDC+Pac, suffers from unacceptable degradation in both visual and metrics. Secondly , most of these methods can not adapt well to Transformer-based models because of the deterioration of self-attention in quantized transformers. As shown in Figure 1, when applied on SwinIR, the existing methods still suffer from distorted artifacts compared with FP or HR.

Therefore, we conducted a post-training quantization analysis on super-resolution with a classical Transformer-based model SwinIR [32]. The weight and activation

distribution is characterized by coexisting symmetry and asymmetry, long tails. Firstly, if the previous symmetric quantization method is applied for asymmetric distribution, at least half of the candidates are completely ineffective. Besides, the long tail effect causes the vast majority of floating-point numbers to be compressed into one or two candidates, leading to worse parameter homogenization. Furthermore, with such a small number of parameters, SwinIR's information has been highly compressed, and quantizing the model often results in significant performance degradation. Nevertheless, the excellent performance and extremely low computational requirements of Transformer-based models are precisely what is needed for deployment in real-world scenarios.

In this paper, we propose 2DQuant , a two-stage PTQ algorithm for image super-resolution tasks. To enhance the representational capacity in asymmetry scenarios, we employ a quantization method with two bounds. The bounds decide the candidate for numbers out of range and the interval of candidates in range. First , we propose distribution-oriented Bound Initialization (DOBI), a fast MSE-based searching method. It is designed to minimize the value heterogenization between quantized and FP models. Two different MSE [5] search strategies are applied for different distributions to avoid nonsense traversal. This guarantees minimum value shift while maintaining high speed and efficiency in the search process. Second , we propose Distillation Quantization Calibration (DQC), a trainingbased method. It is designed to adjust each bound to its best position finely. This ensures that the outputs and intermediate feature layers of the quantized model and that of the FP model should remain as consistent as possible. Thereby DQC allows the quantizer parameters to be finely optimized toward the task goal. The contributions of this paper can be summarized as follows:

(1) To the best of our knowledge, we are the first to explore PTQ with Transformer-based model in SR thoroughly. We design 2DQuant, a unique and efficient two-stage PTQ method (see Figure 2) for image super-resolution, which utilizes DOBI and DQC to optimize the bound from coarse to fine.

(2) In the first stage of post-quantization, we use DOBI to search for quantizer parameters, employing customized search strategies for different distributions to balance speed and accuracy. In the second stage, we design DQC, a more fine-grained optimization-based training strategy, for the quantized model, ensuring it aligns with the FP model on the calibration set.

FP Model

Moving Simultaneously

Moving Upper Bound Only

FP weights and activations

Distribution Oriented Bound Initialization

Distillation Quantization Calibration

Searching

Direction

Initial Bound

DQC Loss

Forward

Full-precision

Layer

Quantized

Layer

(3) Our 2DQuant can compress Transformer-based model to 4,3,2 bits with the compression ratio being 3.07 × , 3.31 × , and 3.60 × and speedup ratio being 3.99 × , 4.47 × , and 5.08 × . No additional module is added so 2DQuant enjoys the theoretical upper limit of compression and speedup.

(4) Through extensive experiments, our 2DQuant surpasses existing SOTA on all benchmarks. We gain an increase in PSNR by as high as 4.52dB in Set5 ( × 2 ) when compressed to 2 bits, and our method has a more significant increase when compressed to lower bits.

2 Related work

Image super-resolution. Deep CNN networks have shown excellent performance in the field of image super-resolution. The earliest SR-CNN [10, 11] method adopted a CNN architecture. It surpassed previous methods in the image super-resolution domain. In 2017, EDSR [33] won the NTIRE2017 [40] championship, becoming a representative work of CNNs in the SR by its excellent performance. Thereafter, with the continuous development of Vision Transformers (ViT) [12], models based on the ViT architecture have surpassed many CNN networks. These Transformerbased models achieve significant performance improvements and they have fewer parameters and lower computational costs. Many works have modified the ViT architecture, achieving continuous improvements. A notable example is SwinIR [32]. With a simple structure, it outperforms many CNNbased models. However, previous explorations of post-quantization in the super-resolution domain have been limited to CNN-based models. They focus on models like EDSR [33] or SRResNet [24]. It is a far cry from advanced models no matter in parameters, FLOPs, or performance. Currently, there is still a research gap in post-quantization for Transformer architectures.

Model quantization. Model quantization is used to compress large models [30, 28, 29] and can be divided into QAT and PTQ. QAT is widely accepted due to its minimal performance degradation. PAMS [26] utilizes a trainable truncated parameter to dynamically determine the upper limit of the quantization range. DAQ [17] proposed a channel-wise distribution-aware quantization scheme. CADyQ [16] is designed for SR networks and optimizes the bit allocation for local regions and layers in the input image. However, QAT usually requires training for as long as or even longer than the original model, which becomes a barrier for real scenarios deployment. Instead of training the model from scratch, existing PTQ methods use the pre-trained models. PTQ algorithms only find the clipping bound for quantizers, saving time and costs. DBDC+Pac [41] is the first to optimize the post-training quantization for image super-resolution task. It outperforms other existing PTQ algorithms. Whereas, they only focus on EDSR [33] and SRResNet [24]. Their 4-bit quantized version is inferior to advanced models in terms of parameters and computational cost, let alone performance. It reveals a promising result for PTQ on SR, and using a more advanced model could bridge the gap between high-performance models and limited calculation resource scenarios.

3 Methodology

To simulate the precision loss caused by quantization, we use fake-quantize [22], i.e. quantizationdequantization, for activations and weights. and the process can be written as

LayerNorm

X

WQ

WK

WV

Q

K

V

BMM

Scale

BMM

Projection

LayerNorm

FC2

GELU

FC1

FP32

where v denotes the value being fake quantized, which can be weight or activation. l and u are the lower and upper bounds for clipping, respectively. Clip ( v, l, u ) = max ( min ( v, u ) , l ) , and Round rounds the input value to the nearest integer. v c denotes the value after clipping, and v r denotes the integer representation of v , and v q denotes the value after fake quantization. The Clip and Round operations contribute to reducing the parameters and FLOPs but also introduce quantization errors.

Figure 3 shows the basic structure of the Transformer block. We have quantized all the modules with a significant computational load within them, effectively reducing the model's FLOPs. Table 2 shows the FLOPs needed for each module. The Linear layers and matrix multiplication account for approximately 86% of the computation load, which are all transformed into integer arithmetic. When

performing gradient backpropagation, we follow the Straight-Through Estimator [8] (STE) style:

where ∂v c ∂u = H ( u -v ) and v c ∂l = H ( l -v ) , H ( · ) denotes Heaviside step function [47]. This formula approximates the direction of gradient backpropagation, allowing training-based optimization to proceed. The derivation of the formula can be found in the supplementary material.

Figure 2 shows the whole pipeline of 2DQuant, which is a two -stage coarse-to-fine post-training quantization method. The first stage is D OBI, using two strategies to minimize the value shift while the second stage is D QC, optimizing two bound of each quantizer towards the task goal.

3.1 Analysis of data distribution

To achieve better quantization results, we need to analyze the distribution of the model's weights and activations in detail. We notice that the data distribution shows a significantly different pattern from previous explorations, invalidating many of the previous methods. The weights and activations distribution of SwinIR is shown in Figure 4. More can be found in supplemental material. Specifically, the weights and activations of SwinIR exhibit noticeable long-tail, coexisting symmetry and asymmetry.

Weight. The weights of all linear layers are symmetrically distributed around zero, showing clear symmetry, and are generally similar to a normal distribution. This is attributed to the weight decay applied to weights, which provides quantization-friendly distributions. From the value shift perspective, both symmetric and asymmetric quantization are tolerable. Whereas, from the vantage point of task objectives, asymmetric quantization possesses the potential to offer a markedly enhanced information density, thus elevating the overall precision of the computational processes involved.

Activations. As for activations, they exhibit obvious periodicity in different Transformer Blocks. For V or the input of FC1, the obtained activation values are symmetrically distributed around 0. However, for the attention map or the input of FC2 in each Transformer Block, due to the Softmax calculation or the GELU [14] activation function, the minimum value is almost fixed, and the overall distribution is similar to an exponential distribution. Therefore, the data in SwinIR's weights and activations exhibit two distinctly different distribution characteristics. Setting asymmetric quantization and different search strategies for both can make the search rapid and accurate.

3.2 Distribution-oriented bound initialization

Because the data distribution exhibits a significant long-tail effect, we must first clip the range to avoid low effective bits. Common clipping methods include density-based, ratio-based, and MSE-based approaches. The first two require manually specifying the clipping ratio, which significantly affects the clipping outcome and necessitates numerous experiments to determine the optimal ratio. Thus we proposed the Distribution-Oriented Bound Initialization (DOBI) to search the bound for weight and

0.0

0.2

0.4

0.6

0.8

1.0





2

0

2





5.0

2.5

0.0

2.5

5.0

7.5





0

2

4

6





0.2

0.0

0.2





0.50

0.25

0.00

0.25

0.50





0.75

0.50

0.25

0.00

0.25

0.50





0.5

0.0

0.5





The collection of all quantizers' bounds { ( l i , u i ) } N i =1 is the linchpin of quantized model performance as it indicates the candidate value for weights and activations. We note that the data distribution falls into two categories: one resembling a bell-shaped distribution and the other resembling an exponential distribution. For the bell-shaped distribution, we use a symmetric boundary-narrowing search method. Whereas, for the exponential distribution, we fix the lower bound to the minimum value of the data and only traverse the right bound. The specific search method is shown in Algorithm 1. The time complexity of Algorithm 1 is O ( MK ) , where M is the number of elements in data v and K is the number of search points. The condition v is symmetrical is obtained by observing the visualization of v and the activations are from the statistics on a small calibration set.

3.3 Distillation quantization calibration

To further fine-tune the clipping range, we propose distillation quantization calibration (DQC) to transfer the knowledge from the FP model to the quantized model. It leverages the knowledge distillation [15] where the FP model acts as the teacher while the quantized model is the student. Specifically, for the same input image, the student model needs to continuously minimize the discrepancy with the teacher model on the final super-resolution output. The loss for the final output can be written as

where O and O q are the final outputs of the teacher and student models, C O , H O , and W O represent the number of output channels, height, and width, respectively. we adopt the L1 loss for the final output, as it tends to converge more easily compared to the L2 loss [33]. As the quantized model shares the same structure with the FP model and is quantized from the FP model, the student model also need to learn to extract the same feature of the teacher model, which can be written as

where F i and F qi are the intermediate features of the teacher and student models respectively and i is the index of the layer. In the field of super-resolution, there is a clear correspondence between the feature maps and the final reconstructed images, making training on feature maps crucial. since the quantized network

Algorithm 1: DOBI pipeline

and the full-precision network have identical structures, we do not need to add extra adaptation layers for feature distillation. The final loss function can be written as

where λ is the co-efficient of L F . In the second stage, based on training optimization methods, the gap between the quantized model and the full-precision model will gradually decrease. The performance of the quantized model will progressively improve and eventually converge to the optimal range.

4 Experiments

4.1 Experimental settings

Data and evaluation. We use DF2K [40, 34] as the training data, which combines DIV2K [40] and Flickr2K [34], as utilized by most SR models. During training, since we employ a distillation training method, we do not need to use the high-resolution parts of the DF2K images. For validation, we use the Set5 [2] as the validation set. After selecting the best model, we tested it on five commonly used benchmarks in the SR field: Set5 [2], Set14 [45], B100 [36], Urban100 [18], and Manga109 [37]. On the benchmarks, we input low-resolution images into the quantized model to obtain reconstructed images, which we then compared with the high-resolution images to calculate the metrics. We do not use self-ensemble in the test stage as it increases the computational load eightfold, but the improvement in metrics is minimal The evaluation metrics we used are the most common metrics PSNR and SSIM [43], which are calculated on the Y channel ( i.e. , luminance) of the YCbCr space.

Implementation details. We use SwinIR-light [32] as the backbone and provide its structure in the supplementary materials. We conduct comprehensive experiments with scale factors of 2, 3, and 4 and with 2, 3, and 4 bits, where Our hyperparameter settings remain consistent. During DOBI, we use a search step number of K = 100, and the statistics of activations are obtained from 32 images in DF2K being randomly cropped to retain only 3 × 64 × 64. During DQC, we use the Adam [23] optimizer with a learning rate of 1 × 10 -2 , betas set to (0.9, 0.999), and a weight decay of 0. We employ CosineAnnealing [35] as the learning rate scheduler to stabilize the training process. Data augmentation is also performed. We randomly utilize rotation of 90°, 180°, and 270° and horizontal flips to augment the input image. The total iteration for training is 3,000 with batch size of 32. Our code is written with Python and PyTorch [38] and runs on an NVIDIA A800-80G GPU.

4.2 Comparison with state-of-the-art methods

The methods we compared include MinMax [22], Percentile [27], and the current SOTA postquantization method in the super-resolution field, DBDC+Pac [41]. For a fair comparison, we report the performance of DBDC+Pac [41] on EDSR [33], as the authors performed detailed parameter adjustments and model training on EDSR. We directly used the results reported by the authors, recorded in the table as EDSR † . It should be noted that the EDSR method uses self-ensemble in the final test, which can improve performance to some extent but comes at the cost of 8 times the computational load. Additionally, we applied DBDC+Pac [41] to SwinIR-light [32], using the same hyperparameters as those set by the authors for EDSR, recorded in the table as DBDC+Pac [41]. The following are the quantitative and qualitative results of the comparison.

Quantitative results. Table 3 shows the extensive results of comparing different quantization methods with bit depths of 2, 3, and 4, as well as different scaling factors of × 2 , × 3 , and × 4 .

DBDC+Pac [41] performs poorly mainly because 1. The DBDC process requires manually specifying the clipping ratio, which significantly affects performance. 2. DBDC does not prune weights, and the learning rate in the Pac process is too low, causing slow con-

0

20

40

60

80

100

120

140

0.00

0.05

0.10

0.15

0.20

0.25

0.30



DOBI

DOBI+DQC

0

20

40

60

80

100

120

140

0.70

0.75

0.80

0.85

0.90

0.95

1.00



DOBI

DOBI+DQC

Figure 5: The bound percentile of DOBI and DQC.

vergence of weight quantizer parameters. However, both adverse factors are eliminated in our 2DQuant algorithm. When using only DOBI algorithm, our performance has already reached a level comparable to that of DBDC+Pac algorithms. Upon applying DQC, our performance experienced a remarkable and discernible enhancement, elevating it to new heights. In the case of × 2 , 4-bit on Set5 and Urban100, DOBI has an improvement of 1.11dB and 0.39 dB compared to EDSR, while

2DQuant has an improvement of 0.69 dB and 1.18 dB compared to the SOTA method. All these results indicate that our two-stage PTQ method can effectively mitigate the degradation caused by quantization and ensure the quality of the reconstructed images.

Figure 5 shows the bound percentile of DOBI searching and DQC. Overall, the bound of DQC is tighter as the values around the zero point enjoy greater importance. Besides, the shallow layers'

Urban100: img_004 (

×

4)

HR

Bicubic

MinMax [22]

Percentile [27]

DBDC+Pac [41]

DOBI (Ours)

2DQuant (Ours)

FP

Urban100: img_046 (

×

4)

HR

Bicubic

MinMax [22]

Percentile [27]

DBDC+Pac [41]

DOBI (Ours)

2DQuant (Ours)

FP

Urban100: img_023 (

×

4)

HR

Bicubic

MinMax [22]

Percentile [27]

DBDC+Pac [41]

DOBI (Ours)

2DQuant (Ours)

FP

(a) Learning rate

(b) Batch size

(c) DOBI and DQC

bounds vary more significantly due to the elevated significance of these layers within the neural network. Detailedly, the bound for the second MLP fully connected layer's weight in Layer 0 Block 1 only remains 46% data in its range. It has the second-highest lower bound percentile and the smallest upper bound percentile among the network. Its percentiles are 0.2401 and 0.7035 respectively while its bound values are -0.062 and 0.047 and its distribution is visualized in Figure 4. In conclusion, only through task-oriented optimization of each bound at a fine-grained level can redundant information be maximally excluded and useful information be maximally retained.

Qualitative results. We show the visual comparison results for × 4 in Figure 6. Since quantized models are derived from full-precision models with information loss, their global performance will rarely exceed that of full-precision models. As seen in the three images for Minmax, after quantization, if no clipping is performed, the long tail effect will lead to a large number of useless bits, resulting in a significant amount of noise and repeated distorted patterns in the reconstructed images. In these challenging cases, our training method allows the model to retain edge information of objects better, preventing blurring and distorted effects. For example, in img_046 and img_023, we have the highest similarity to the full-precision model, while other methods show varying degrees of edge diffusion, significantly affecting image quality. Compared to the DBDC+Pac method, our DOBI and DQC allow for better representation of edge and texture information in the images and effectively avoid distortions and misalignments in the graphics. The visual results demonstrate that our proposed DQC is essential for improving performance in both metric and visual comparisons.

4.3 Ablation study

Learning rate and batchsize. We first study the performance variations of the model under different hyperparameters. From Tables 4a and 4b, it can be seen that our DQC enables the model to

converge within a range of outstanding performance for most learning rates and batch sizes. Due to the non-smooth impact of quantization parameters on the model, the quantized model is more prone to local optima compared to the full-precision model, resulting in a noticeable performance drop when the learning rate is too low. Additionally, as shown in Table 4b, the larger the batch size, the better the model's performance, and the smoother the convergence process. However, even with a smaller batch size, we can still achieve a performance of 37.82dB on Set5, indicating that our two-stage method has good robustness to different hyperparameters.

DOBI and DQC. Moreover, we also study the impact of different stages on performance, with the results shown in Table 4c, from which we can draw the following conclusions: Firstly , the goal of DOBI is to minimize the value shift for weights and activations. Although it is not the task goal, it can still enjoy significant enhancement due to better bit representational ability. Secondly , DQC alone cannot achieve the optimization effect of DOBI. This is because the impact of quantizer parameters on model performance is oscillatory, and training alone is prone to converge to local optima. In contrast, search-based methods can naturally avoid local optima. So it's necessary to use results from the search-based method to initialize training-based method in PTQ. Thirdly , when DOBI and DQC are combined, namely our 2DQuant, the 4-bit quantized model has only a 0.28dB decrease on Set5 compared to the FP model, which maximally mitigates the accuracy loss caused by quantization.

5 Discussion

Why our results surpass FP outcomes. While our method's performance metrics do not yet fully match those of full-precision models, visual results reveal a compelling advantage. As observed in image img_092 of Figure 1 of Urban100, our approach correctly identifies the direction of the stripes in the image. Whereas the full-precision model erroneously selects the wrong direction. This discrepancy arises because the lower-resolution image, affected by aliasing, creates an illusion of slanted stripes, misleading the FP model's reconstruction. This phenomenon demonstrates that our PTQ algorithm allows more accurate restored results in certain localized and challenging tasks without being misled. More examples are in the supplementary materials.

It suggests that full-precision models contain not only redundant knowledge but also incorrect information. The latter is hard to get rid of by training the FP model. Our quantization method can effectively reduce model parameters and computational demands while eliminating erroneous information, achieving multiple benefits simultaneously. This also suggests that the FP model doesn't represent the pinnacle of what a quantized model can achieve.

Limitations. Despite achieving excellent results, this study still has some limitations. During the DOBI process, the data distribution of activations and weights is required to approximate a bell curve or exponential distribution; otherwise, the DOBI method cannot find the most suitable positions. Additionally, increasing the number of search points for a single tensor in MSE does not necessarily guarantee better performance. However, the second-stage training can somewhat alleviate this issue. Moreover, our method requires a calibration set; without which, the first-stage DOBI and the second-stage DQC cannot be carried out at all.

Societal impacts. Our super-resolution quantization method effectively saves computational resources, facilitating the deployment of super-resolution models at the cutting edge

6 Conclusion

This paper studies the post-training quantization in the field of image super-resolution. We first conducted a detailed analysis of the data distribution of Transformer-based model in SR. These data exhibit a clear long-tail effect and symmetry and asymmetry coexisting effect. We designed 2DQuant, a dual-stage PTQ algorithms. In the first stage DOBI, we designed two different search strategies for the two different distributions. In the second stage DQC, we designed a distillationbased training method that let the quantized model learn from the FP model, minimizing the accuracy loss caused by quantization. Our 2DQuant can compress Transformer-based model to 4,3,2 bits with the compression ratio being 3.07 × , 3.31 × , and 3.60 × and speedup ratio being 3.99 × , 4.47 × , and 5.08 × . No additional module is added so 2DQuant enjoys the theoretical upper limit of compression and speedup. Extensive experiments demonstrate that 2DQuant surpasses all existing PTQ methods in the field of SR and even surpasses the FP model in some challenging cases. In the future, recognizing the significant impact of the model on performance, we will conduct PTQ research on more advanced super-resolution models and attempt to deploy quantized super-resolution algorithms to actual photography tasks, providing a more detailed evaluation of the performance of PTQ algorithms.

Acknowledgments and Disclosure of Funding

This work is supported by the National Natural Science Foundation of China (62141220, 62271414), Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102), the Fundamental Research Funds for the Central Universities, Zhejiang Provincial Distinguish Young Science Foundation (LR23F010001), Zhejiang 'Pioneer' and 'Leading Goose' R&D Program (2024SDXHDX0006, 2024C03182), the Key Project of Westlake Institute for Optoelectronics (2023GD007), and Ningbo Science and Technology Bureau, 'Science and Technology Yongjiang 2035' Key Technology Breakthrough Program (2024Z126).

References

Wele Gedara Chaminda Bandara and Vishal M. Patel. Hypertransformer: A textural and spectral feature fusion transformer for pansharpening. In CVPR , 2022. 1

Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC , 2012. 6

Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggregation transformer for image super-resolution. In CVPR , 2023. 1

Zheng Chen, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xin Yuan. Cross aggregation transformer for image restoration. In NeurIPS , 2022. 1

Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). arXiv , 2018. 2

Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.06085 , 2018. 2

Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In ICCVW , 2019. 1

Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830 , 2016. 4

Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai, Junjie Liu, Xiaolin Wei, and Xianglong Liu. Towards accurate post-training quantization for vision transformer. In ACM MM , 2022. 1

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. In ECCV , 2014. 1, 3

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI , 2016. 1, 3

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv , 2020. 3

Hayit Greenspan. Super-resolution in medical imaging. The Computer Journal , 2008. 1

Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR , 2016. 4

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. In NeurIPS Workshop , 2014. 5

Cheeun Hong, Sungyong Baik, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Cadyq: Content-aware dynamic quantization for image super-resolution. In ECCV , 2022. 2, 3

Cheeun Hong, Heewon Kim, Sungyong Baik, Junghun Oh, and Kyoung Mu Lee. Daq: Channel-wise distribution-aware quantization for deep image super-resolution networks. In WACV , 2022. 3

Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR , 2015. 6

Yawen Huang, Ling Shao, and Alejandro F Frangi. Simultaneous super-resolution and cross-modality synthesis of 3d medical images using weakly-supervised joint convolutional sparse coding. In CVPR , 2017. 1

Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In ICML , 2021. 1

Jithin Saji Isaac and Ramesh Kulkarni. Super resolution techniques for medical image processing. In ICTSD , 2015. 1

Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integerarithmetic-only inference. In CVPR , 2018. 3, 6, 7, 8

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR , 2015. 6

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe Shi. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR , 2017. 2, 3

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR , 2017. 1

Huixia Li, Chenqian Yan, Shaohui Lin, Xiawu Zheng, Baochang Zhang, Fan Yang, and Rongrong Ji. Pams: Quantized super-resolution via parameterized max scale. In ECCV , 2020. 2, 3

Rundong Li, Yan Wang, Feng Liang, Hongwei Qin, Junjie Yan, and Rui Fan. Fully quantized network for object detection. In CVPR , 2019. 2, 6, 7, 8

Yuchen Li, Haoyi Xiong, Linghe Kong, Qingzhong Wang, Shuaiqiang Wang, Guihai Chen, and Dawei Yin. S2phere: Semi-supervised pre-training for web search over heterogeneous learning to rank data. In ACM SIGKDD , 2023. 3

Yuchen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Fanqin Xu, Guihai Chen, and Minglu Li. Mhrr: Moocs recommender service with meta hierarchical reinforced ranking. IEEE TSC , 2023. 3

Yuchen Li, Haoyi Xiong, Qingzhong Wang, Linghe Kong, Hao Liu, Haifang Li, Jiang Bian, Shuaiqiang Wang, Guihai Chen, Dejing Dou, et al. Coltr: Semi-supervised learning to rank with co-training and over-parameterization for web search. IEEE TKDE , 2023. 3

Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. In ICLR , 2021. 1

Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCVW , 2021. 1, 2, 3, 6, 7

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW , 2017. 2, 3, 5, 6, 7

Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW , 2017. 6, 7

Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv , 2016. 6

David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV , 2001. 6

Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications , 2017. 6

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS , 2019. 6

Pejman Rasti, Tõnis Uiboupin, Sergio Escalera, and Gholamreza Anbarjafari. Convolutional neural network super resolution for face recognition in surveillance monitoring. In AMDO , 2016. 1

Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, Kyoung Mu Lee, et al. Ntire 2017 challenge on single image super-resolution: Methods and results. In CVPRW , 2017. 3, 6

Zhijun Tu, Jie Hu, Hanting Chen, and Yunhe Wang. Toward accurate post-training quantization for image super resolution. In CVPR , 2023. 2, 3, 6, 7, 8

Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li. Uformer: A general u-shaped transformer for image restoration. In CVPR , 2022. 1

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP , 2004. 6

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In CVPR , 2022. 1

Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representations. In Proc. 7th Int. Conf. Curves Surf. , 2010. 6

Liangpei Zhang, Hongyan Zhang, Huanfeng Shen, and Pingxiang Li. A super-resolution reconstruction algorithm for surveillance images. Elsevier Signal Processing , 2010. 1

Weihong Zhang and Ying Zhou. Chapter 2 - level-set functions and parametric functions. In The FeatureDriven Method for Structural Optimization . Elsevier. 4

Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super-resolution using very deep residual channel attention networks. In ECCV , 2018. 1

Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR , 2018. 1

Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160 , 2016. 2

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to our abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations in the main paper.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided implementation details in the experiments section. We will also release all the code and models.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Firstly, all data in the paper can be accessed publicly. Secondly, we provide very detailed instructions ( e.g. , method descriptions and implementation details) to reproduce our results. Thirdly, we promise to release code and all models.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided implementation details, which cover the above questions.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Please refer to the experiment part.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to experiment part.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to the supplementary file.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This work poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have credited most previous works in the paper. The license and terms are respected properly.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will release code and models. In the paper, we have provided implementation details and other contents to reproduce our results.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents 2DQuant, an innovative low-bit post-training quantization technique for transformer-based image super-resolution that significantly advances the state-of-the-art by introducing a two-stage optimization process. This process includes a novel Distribution-Oriented Bound Initialization strategy and a Distillation Quantization Calibration method, resulting in exceptional performance with minimal loss of accuracy for SwinIR-light. The 2DQuant approach demonstrates the potential to compress and accelerate transformer-based super-resolution models effectively for edge deployment.\n\nSTRENGTHS:\n1. This paper is well-written and easy to understand.\n\n2. Transformer-based SR is much popular in recent year, this method is much helpful for obtaining accurate quantized transformer-based SR models. \n\n3. The experimental results and visualizations are sufficient.\n\nWEAKNESSES:\n1. The proposed DOB and DQC are much similar with DBDC and PaC, but only change weight compression to symmetric quantization, could you give more details about the differences?\n\n2. The motivation in the introduction are confusing. Usually, the deterioration of self-attention in quantized transformer causes sever performance degradation instead of the unadaptable changes in weight and activation distributions, which is also appeared in CNN-based SR networks. \n  \n3. The selected baselines in this paper are rough and the experimental results are not convinced. This paper does not compare with transformer-based post-training quantization methods, such as PTQ4ViT[1], FQ-ViT[2], NoiseQuant[3] and RepQ-ViT[4], these methods are open-source, the author should apply these PTQ methods for super resolution for that the basic transformer blocks are not different for super resolution and image classification.\n\n[1] Z. Liu et al. Post-training quantization for vision transformer. Advances in Neural Information Processing Systems, 34:28092–28103, 2021.\n\n[2] Y. Lin et al. Fq-vit: Post-training quantization for fully quantized vision transformer. arXiv preprint arXiv:2111.13824, 2021.\n\n[3] Y. Liu et al. Noisyquant: Noisy bias-enhanced post-training activation quantization for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20321– 20330, 2023.\n\n[4] Z. Li et al. Repq-vit: Scale reparameterization for post-training quantization of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17227–17236, 2023.\n\nQUESTIONS:\nPlease refer to the weaknesses above.\n\nLIMITATIONS:\nThe authors have discussed the limitations in this paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel two-stage post-training quantization (PTQ) method aimed at compressing image super-resolution (SR) models for efficient deployment on edge devices. The authors address the challenge of accuracy degradation in low-bit quantization by proposing the 2DQuant method, which combines Distribution-Oriented Bound Initialization (DOBI) and Distillation Quantization Calibration (DQC) to achieve efficient and accurate super-resolution under low-bit quantization.\n\nSTRENGTHS:\n1. The writing of this paper is very clear.\n2. The figures in the paper are simple and easy to understand.\n3. This paper conducted some experiments to verify the effectiveness of the proposed method.\n\nWEAKNESSES:\n1. The contribution lacks novelty.\nThis paper does not bring any new insights. For example, the distribution of weights and activations is well studied by previous works[1][2][3]. Some papers have used searchable upper/lower bounds for quantization [2] (although these methods are based on QAT, the essence of QAT and PTQ is not different, with only slight differences in training strategies) and NO references were given in this manuscript. The proposed distillation approach is a widely used routine operation in quantization tasks [1][2][3][4][5][6].\n\n2.  Experiments are insufficient.\nThe PTQ for transformers has been extensively studied in many other papers (e.g. [4][5]), but it seems that the authors have not compared it with these methods.\n\n3.  Experimental settings are not consistent.\nOnly 100 images are used in [7], which is inconsistent with this paper.\n\n\nReferences:\n[1] Li, Huixia et al. “PAMS: Quantized Super-Resolution via Parameterized Max Scale.” ArXiv abs/2011.04212 (2020): n. pag.\n\n[2] Zhong, Yunshan et al. “Dynamic Dual Trainable Bounds for Ultra-low Precision Super-Resolution Networks.” European Conference on Computer Vision (2022).\n\n[3] Hong, Chee and Kyoung Mu Lee. “Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks.” ArXiv abs/2307.13337 (2023): n. pag.\n\n[4] Li, Yanjing et al. “Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer.” ArXiv abs/2210.06707 (2022): n. pag.\n\n[5] Liu, Shi et al. “Oscillation-free Quantization for Low-bit Vision Transformers.” International Conference on Machine Learning (2023).\n\n[6] Tu, Zhaopeng et al. “Toward Accurate Post-Training Quantization for Image Super Resolution.” 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2023): 5856-5865.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper present a practical post-training quantization method for SR transformer, namely 2DQuant. The 2dquant mainly rely one two techniques, the first is Distribution-Oriented Bound Initialization (DOBI) determine the quantization range initially, and the second one is distillation Quantization calibration (DQC) to further finetune the quantizer for accurate quantization. Both of the proposed two techniques significantly improving the performance of PTQ for SR transformer, and the quantization can even be pushed to as low as 2-bit without retraining.\n\nSTRENGTHS:\n1. The proposed method is a PTQ method without full retraining instead of the usually studied quantization-aware training methods for SR tasks, which can be seen as resource-saving in real applications. Considering the targeted architectures are SR transformers is always used as well-pretrained models, the PTQ pipeline is significant and practical.\n\n2. The proposed quantization method for SR transformer are clear and effective. DOBI allows the optimization of quantization ranges start from a statistical search-based optimal, and then use DQC to fully optimized the quantizer. It’s brings improvements of both weight and activation quantization, especially for the activation with dynamic distribution that requires more robust and powerful quantizer.\n\n3. Experiments show that the proposed method achieve SOTA results on ultra-low 2-4 bit Sr networks, which allows SR transformer enjoys very good efficiency with little accuracy loss. And the visualization also show the good performance of the networks quantized by the proposed methods.\n\n4. The paper is easy to follow and with good writing and presentation, and with comprehensive visualization for the distribution of both weights and distributions.\n\nWEAKNESSES:\n1. As presented in section 3, the authors use fake quantization during the design of the method, but didn’t mention if it can be replaced by the real quant, which can be implemented and bring real acceleration in hardware. The author should discuss if the proposed method can achieve real quantized inference when deployment.\n\n2. The algorithm of DQC is missed and make the description not very clear, I suggest to add it or merge it to the algorithm of DOBI. And Figure 3 not highlight the application of the proposed techniques, which is also suggested to revise.\n\n3. Some details of writing should be improved and polished carefully, e.g., the subtitles of section 2 and 3, some explain of the proposed equations are missing.\n\nQUESTIONS:\nPlease respond for the issues raised in weaknesses part.\n\nLIMITATIONS:\nThe authors have discussed the limitation in their paper.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a low-bit post-training quantization (PTQ) method, 2DQuant, for image super-resolution. 2DQuant is a dual-stage low-bit PTQ method. They first investigate the weight and activations. They propose Distribution-Oriented Bound Initialization (DOBI) by using different searching strategies to get a coarse bound for quantizers. They further propose Distillation Quantization Calibration (DQC) to refine the quantizer parameters with a distillation approach. They provide extensive experiments on different bits and scaling factors. When compared with SOTA methods, there proposed methods achieve superior performance with 3.6X compression ratio and 5.08X speedup ratio in the 2-bit case.\n\nSTRENGTHS:\n1)The authors explore PTQ to Transformer based image super-resolution (SR), which is one of the first works in this research field. This topic is also very practical and is favored in real-world applications.\n\n2)The authors investigate this topic with some observations and visualizations. For example, the visualizations in Figures 4, 5, 7-11, are inspiring.\n\n3)They propose distribution-oriented Bound Initialization (DOBI) to minimize the value heterogenization between quantized and the FP models. They DOBI to search for quantizer parameters by employing customized search strategies for different distributions to balance speed and accuracy.\n\n4)They propose Distillation Quantization Calibration (DQC) to adjust each bound to its best position finely. DQC is used to ensure the quantized model align with the FP model on the calibration set.\n\n5)In the ablation study, the authors investigate the effect of some key components, like DOBI and DQC.\n\n6)In the main comparisons with other SOTA methods, the proposed method obtain the best performance and visual results. I believe this low-bit quantization work in Transformer achieves excellent permance and has good potential in future research.\n\n7)The authors give detailed derivation of their backward gradient propagation formula, which makes the method more convincing.\n\nWEAKNESSES:\nThe authors did not give results about inference time. The quatization can reduce params and ops largely. Then, how about the acceleration when deploying the quantized models into device?\n\nSome parts in the experiments are confusing. For example, in Table 3, the results of EDSR confuse me somehow. I am not very sure how the EDSR results are obtained. The original EDSR is full-precision model. Here, the authors report its quantized version. The baseline in Table 3 is not very clear. Please clarify them.\n\nIn Table 4 (b), when the batch size becomes larger, the performance changes slightly. This is not very consistent with the common observation, where larger batch size usually improves performance.\n\nThe writing can be further improved. Some details should be given attention and revised. For example, Section 3.2, the first letter should be capital.\n\nQUESTIONS:\nIn Figure 1, the proposed method performs even better than the full-precision (FP) version. What are the potential reasons behind this observation?\n\nCan this method be applied to diffusion based image SR models? If so, please give some key ideas. It will be much better if the authors can provide some preliminary results.\n\nIn Table 3, what is the model size for the original EDSR? I do not which version of EDSR did the authors use in the paper.\n\nWhat is the motivation by providing two versions, i.e., DOBI (ours) and 2DQuant (ours), in Table 3?\n\n------------After Rebuttal and  Discussions----------\nAfter rebuttal and disscussions, my concerns have been well solved.\n\nLIMITATIONS:\nThe authors have discussion the limitation in their paper.\n\nFlag For Ethics Review: No ethics review needed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}