Binarized Diffusion Model for Image Super-Resolution
Authors: Zheng Chen
Haotong Qin
Yong Guo
Xiongfei Su
Xin Yuan
Linghe Kong
Yulun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion model
binarization
image super-resolution
Primary area: machine_vision
TL;DR: A binarized diffusion model, BI-DiffSR, for image SR.

Abstract: Advanced diffusion models (DMs) perform impressively in image super-resolution (SR), but the high memory and computational costs hinder their deployment. Binarization, an ultra-compression algorithm, offers the potential for effectively accelerating DMs. Nonetheless, due to the model structure and the multi-step iterative attribute of DMs, existing binarization methods result in significant performance degradation. In this paper, we introduce a novel binarized diffusion model, BI-DiffSR, for image SR. First, for the model structure, we design a UNet architecture optimized for binarization. We propose the consistent-pixel-downsample (CP-Down) and consistent-pixel-upsample (CP-Up) to maintain dimension consistent and facilitate the full-precision information transfer. Meanwhile, we design the channel-shuffle-fusion (CS-Fusion) to enhance feature fusion in skip connection. Second, for the activation difference across timestep, we design the timestep-aware redistribution (TaR) and activation function (TaA). The TaR and TaA dynamically adjust the distribution of activations based on different timesteps, improving the flexibility and representation alability of the binarized module. Comprehensive experiments demonstrate that our BI-DiffSR outperforms existing binarization methods. Code is released at: https://github.com/zhengchen1999/BI-DiffSR.

Binarized Diffusion Model for Image Super-Resolution
Zheng Chen', Haotong Qin Yong Guo?, Xiongfei Su',
Xin Yuan", Linghe Kong', Yulun Zhang"
'Shanghai Jiao Tong University, 'ETH Zürich, Max Planck Institute for Informatics, *Westlake University
Abstract
Advanced diffusion models (DMs) perform impressively in image super-resolution (SR), but the high memory and computational costs hinder their deployment. Bi- narization, an ultra-compression algorithm, offers the potential for effectively accelerating DMs. Nonetheless, due to the model structure and the multi-step iterative attribute of DMs, existing binarization methods result in significant perfor- mance degradation. In this paper, we introduce a novel binarized diffusion model, BI-DifSSR, for image SR. First, for the model structure. we design UNet archi- tecture optimized for binarization. We propose the consistent consistent-pixel--eeeeee pixel-downsampp (CP-Down) and consistent- pinpseenttpxell-upaappll (CP-Up) to maintain dimension consis- tent and facilitate the full-precision information transfer. Meanwhile, we design the channel-shuffle-fusse fusion (CS-Fusion) to enhance feature fusion in skip connection. Second, for the activation difference across timestep. design the timestep-aware redistribution (TaR) and activation function (TaA). The TaR and TaA dynamically adjust the distribution of activations based on different timesteps, improving the flexibility and representation alability of the binarized module. Comprehensive ex- periments demonstrate that BI-DiffSR outperforms existing binarization meth- ods. Code released at: https: // github. com/ wwwegchenn999/// -iifSSS
Introduction
Image super-resolution (SR) is a fundamental task in low-level vision and image processing. aims to reconstruct high-resolution (HR) images from low-resolution (LR) counterparts. Currently, the mainstream methods for this task are deep neural networks, which employ learning- based techniques to map LR images HR images [10, 70, 31, 54, 6, 68]. Among these methods, generative models [62, 9, 44] have garnered widespread attention for their ability to restore more realism results.
Especially, the diffusion model (DM) [16, 58, 52], newly proposed generative model, exhibits impressive performance. With its superior generation quality and more stable training, diffusion model is widely used in various image tasks, including image SR [54, 63]. Specifically, the diffusion model transforms a standard Gaussian distribution into high-quality image through a stochastic iterative denoising process. In image SR, it further constrains the generation scope by conditioning on the LR image to produce the targeted HR image.
However, produce high-quality results, diffusion models require thousands of iterative steps, leading to slow inference processes and high computational costs. Some methods [58, 40, 37] implement faster sampling strategies via learning sample trajectories, effectively reducing the number of iterations to tens. Yet, a single inference step still demands substantial memory usage and floating- point computations, especially for SR tasks involving high-resolution images. Mcanwhile, most edge devices (e.g., mobile and loT devices), have limited storage and computational resources. This hampers the deployment of diffusion models these platforms and limits their application. Therefore, it is essential to compress diffusion models to accelerate inference speed and reduce computational costs while maintaining model performance.
"Conrsponding authors: Haotong Qin, qinhaotong @ gmail.com; Yulun Zhang, yulun100c 100@ gmmil.mmoommm
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

HR
Bicubic
SR3 (FP) [S4]
BNN [19]
DoReFa [71]
Urban100: ing_074 074
XNOR [50]
IRNet [48]
ReActNet [38]
BBCU [66]
BI-DHISR (ours)
Figure I: Visual comparison (x4) of binarization methods. Some methods (e.g. BNN (19)) cannot work diffusion models. Several methods (e.g., BBCU [66]) suffer from blurring and artifacts. In contrast, our proposed BI-DiffSR outperforms other methods with accurate results.
Common compression approaches include pruning [11], distillation [61], and quantization [45, 66, 26]. Among these, 1-bit quantization (i.e., binarization) stands out for its effectiveness. As the most aggressive form of bit-width reduction, binarization significantly reduces memory and computational costs by quantizing the weights and activations of full-precision (32-bit) models to 1-bit.
Nonetheless, existing binarization research primarily deals with higher-level tasks (e.g.. classification) and end-to-end models [49, 19, 39J. Applying existing binarization methods directly to current diffusion model architectures results in a significant performance drop. This is primarily due two aspects: (1) Model Structure. Diffusion models typically apply the UNet architecture [53] for noise estimation, which is not easy to binarize directly. I. Dimension Mismatch: The identity shortcut is crucial for the binarized SR model, since it facilitates the transfer of full full-precision (FP) information, compensating for the binarized model [66]. However, in UNet, the feature dimensions change since hwwgaammii///pppamplingggg The dimension mismatch prevents the usage of shortcuts, cutting off the full-p full-precision propagation. II. Fusion Difficulty: The UNet structure uses skip connections to transfer information from encoder to decoder. However, the typical fusion method, concatenation, leads to the dimension mismatch. Alternatively, other methods (e.g., addition) also struggle to achieve effective fusion due to significant differences in value ranges between encoder and decoder features. (2) Activation Distribution. Due to the multi-step iterative nature of diffusion models, the activation distribution dramatically changes with timesteps. Furthermore, the activation binarization will even amplify activation differences [50]. The difference increases the learning challenges for binarized modules (e.g., binarized convolution). thereby hindering the effective representation of features. Consequently, the SR performance of the binarized diffusion model is limited.
Based on the above analysis, we propose novel binarized diffusion model, BI-DiffSR, to achieve effective image SR. Our design comprises two main aspects: structure and activation, (1) Structure. We develop simple yet effective convolutional UNet architecture, which is suitable for binarization. I. Dimension Consistency: We propose
pinsstett-pixll--owwssappppp (CP-Down) and consistent- pixel-upsample (CP-Up) to ensure dimensional consistency in binarized computation. CP-Down and CP-Up maintain the full-pre precision information transfer during feature scaling. II. Feature Fusion: We develop the channel-shuffle fhuffle-fusion (CS- Fusionn)) to facilitate the fusion of different features within skip connections and suit binarized modules. Through channel shufflc, we combine two input features into two shuffled features to balance their activation value ranges.
(2) Activation. Considering the activation differences at different timesteps, we design the timestep-aware redistribution (TaR) and timestep-aware activation function (TaA). The TaR and TaA adjust the binarized module input and output activations according to different timesteps. This timestep-aware adjustment improves the flexibility and representational ability of the binarized module various activation distributions.
Extensive experiments demonstrate that our proposed BI-DifiSR significantly outperforms existing binarization methods. As shown in Fig. 1, our BI- BI-DiffSR restores more perceptually pleasing results than other methods. Overall, our contributions are as follows:
We design the novel binarized model, BI-DiffSR, for image SR. To the best of our knowledge, this is the first binarized diffusion model applied to SR.
We develop a UNet architecture optimized for binarization, with consistent-pixel- downsample (CP-Down) and upsample (CP-Up), and channel-shuffle shuffle-fusion (CS-Fusion).
We introduce the timestep-an aware redistribution (TaR) and activation function (TaA) to adapt activation distributions by timestep, enhancing the capabilities of the binarized module.
Our BI-DiftSR surpasses current state-of-the-art binarization methods, and offers compara- ble perceptual performance to full full-precision diffusion models.

---

2 Related Work
2.1 Image Super-Resoluticn
Since the advent of SRCNN [10], deep neural networks have gradually become the mainstream for image SR. Numerous architectures [33, 70, 46, 31, 5] are designed to advance reconstruction accuracy. Concurrently, generative methods are widely applied improve the quality of restored image details. This includes autoregressive model [23, 9], normalizing flow [51, 41, 321, and generative adversarial network (GAN) [13, 24]. For instance, SRFlow [41] utilizes normalizing flows to transform a Gaussian distribution into the HR image space. Meanwhile, SRGAN [24] employs GAN as supervision loss and combines it with perceptual loss produce visually pleasing results. Subsequent methods [62, 4] further refine the network and loss to yield more natural results. Recently, the diffusion model (DM) [16, 8] has been introduced into SR, displaying impressive performance, especially regarding perception. Thereby, DM has been attracting widespread attention [54, 25, 65].
2.2 Diffusion Model
Through the Markov chain, the diffusion model (DM) generates images from the Gaussian distri- bution [16]. It has demonstrated exceptional performance in various tasks [3, 17, 52, 7, 14, 30, 29, 36, 35, 28, 15]. Naturally, DM has also been extensively researched in the field of image SR [54, 21, 63, 34, 65]. For instance, SR3 [54] achieves conditional diffusion by concatenating the resized LR image with the noise image as the input of the noise estimation network. Meanwhile, some methods, e.g., DDNM [63], utilize an unconditional pre-trained diffusion model as a prior for zero- shot SR. Additionally, some approaches [34, 65] employ text-to-image diffusion models to achieve realistic and controllable SR. Despite promising results, these methods require hundreds or thousands of sampling steps to generate HR images. Although some accelcration algorithms [58, 37, 28] reduce the inference steps to tens, each denoising step still demands substantial resources. The high memory and computational costs hinder the practical application of DMs on resource- -imited platforms (e.g.. mobile devices). To address this issue, we design practical binarized SR diffusion model.
2.3 Binarization
Binarization is a popular model compression approach [49]. As an extreme case of quantization, it reduces the weights and activations of full-precision neural network to l-bit. This significantly decreases the model size and computational complexity, making it widely used in both high-level [19, 39, 48. 38, 67] and low-level [20, 66, 66, 69] vision tasks. For example, BNN [19] directly binarizes weights and activations during forward and backward processes. IRNet [48] retains information accurately through the proposed information retention network. ReActNet [38] proposes the RSign and RPReLU to enable explicit distribution reshape and shift of activations. Meanwhile, in the image SR field, BBCU [66] introduces a meticulously designed basic binary conv unit, which removes batch normalization (BN) in the binarized model. However, for DM, though some methods realize low-bit (e.g.. or 8) quantization [55, 26, 27], implementing binarization remains challenging. Due to the structure of the noise estimation network and the multi-step iterative attribute, existing binarization methods often result in significant SR performance degradation.
3 Method
In this section, we introduce our proposed BI-DiftSR. First, we describe the structural designs suitable for binarization: wewpiisten--ixll-dowweee chamel--shufle-fusie module (CS- (CS-Fusion). The CP-Down and CP-Up achieve dimension adjustment and ensure the transfer of full-precision information. The CS-Fusion effectively integrates different features within the skip connection. Secondly, we present the dynamic designs tailored for varying activations: timestep-aware redistribution (TaR) and activation function (TaA). The TaR and TaA enhance the representational learning of the binarized modules across multiple timesteps.
(CP-Down), consistent-pixel-upaaan uusamplee (CP-Up), and
3.1 Model Structure
Overall. We employ convolutional UNet [53] as the noise estimation network. Details of the diffusion model for SR are provided in the supplementary materials. As the common choice within DMs, using UNet as the backbone for binarization offers generalizability. Moreover, for binarized models, the design should be compact and well-defined. Compared to the noc-l self-attention operations, convolution simpler and easier to implement. Our architecture is shown Fig. 2a, featuring encoder- mncoder-bottleneekkddee bottlenecc--decoeer (E-B-D) design.
3

---

Res Block
GP- Down
Res Block
GP. Down
Res Block
CP- Down
Res Block
Fem
for
Len
Res Block
Res Block
CS- Fusion
Res Block
CS- Fusion
Fusion
Up
Up
Up
(a) U-Net (Noise Estimation Network)
Group Norm
Swish
Bl-Connv
Block
Group
Norm
Swish
Bli Bli-Comv
Block
-
Swish
Linear
Eyox
(c) TE
fm
Linear
(b) ResBlock
Concaten
Elemeni-wise Additon
Posl Encoding
Figure 2: The overall structure of the noise estimation network. (a) UNet: The model consists of ResBlock, CP-Down, CP-Up, and CS-Fusion. It predicts noise 6, with the upscaled LR image y, noise image x,, and timestep L. (b) ResBlock: Residual block, utilizes the binarized convolution (BI-Conv) block. The input and output dimensions of the block remain consistent, making it suitable for binarization. (c) TE: Time encoding, encoders timestep / to produce the timestep embedding tem-
Given the noise image x, & at I-th timestep, and the LR image yERHxw* x3 (bicubic to HR resolution), two images are concatenated along the channel dimension as the UNet input, where H X is the resolution. For timestep t, the sinusoidal position encoding [60] is applied to obtain the timestep embedding tem =RC The input images first pass through a convolutional layer to produce the shallow feature F,ER" C. w.WCC xC, where C' is the channel number. Then, the shallow Each level of the
feature F, are further refined by the E-B-D into the deepe feature F #x E-B-D is composed of multiple (N₂ in E and N₂ in D) residual blocks (ResBlocks), with details illustrated in Fig. 2b. Within the ResBlocks, the timestep embedding tem is incorporated to provide temporal information. In the encoder &, downsample module (i.e., CP-Down) progressively reduces feature resolution and increases channel number. Conversely, in the decoder D, upsample module (i.e., CP-Up) gradually restores the higl-resolution representation. Moreover, to compensate for information loss during downsampling, the skip connection is used link features between the encoder and decoder. Finally, through one convolution, the predicted noise E x X x3 is obtained.
Structure Analysis. Although the UNet architecture is suitable for diffusion models, its unique structure poses challenges for direct binarization, which results in a substantial accuracy decrease compared to full- pellipreniiion models. We identify two main issues/challenges that contribute to the problem: dimension mismatch fusion difficulty.
Challenge I: Dimension Mismatch. In the binarized model, 1-bit quantization leads to significant information loss, limiting the capability for feature representation and the ultimate SR performance. Compared to binary activations, full-precisio activations contain more information. Therefore, we can apply the identity shortcut preserve the full-precision information. This operation effectively compensates for the information loss caused by binarization. However, in UNet, the frequent changes in feature resolution and channel size lead to dimension mismatches. This prevents the effective use of the identity shortcut and cuts off the propagation of full-precision information.
Challenge II: Fusion Difficulty. Another crucial structure of UNet is the skip connection, which links encoder and decoder features. The typical approach is to concatenate these features along the channel dimension and pass them subsequent layers. However, concatenate causes dimension mismatch. As analyzed in Challenge I, it is unsuitable for binarization. Furthermore, we find that there is significant difference in the activation ranges between the two inputs (from encoder and decoder) of the skip connection (Fig. 3d). This imbalance makes other fusion methods, e.g., addition, also unsuitable, since the smaller range activation masked by the larger one, illustrated in Fig. 3d.
To better adapt binarization for the UNet architecture, we propose two structures: Consistent- Downsample/Upsaappl and Channel-Shufle Fusion, as illustrated in Fig. 3.
Consistent- D psamppe. To address the dimension mismatch in the UNet structure, we first confine all feature reshaping operations to the Upsample and Downsample modules. That is to ensure that the dimension of the main module, i.e., ResBlock, remains matched. Meanwhile, we propose the consistent-pixet pixel -ixcl-downsample (CP-Down) and consistent-pixel pixpeuutppl (CP-Up).

---

Bi-Conv Block
Bl-Connv Block
Bl-Conv
Channel Shuffle
Block
Split
Bi-Conv Block
UnShuffte
Bi-Comy
Shuftle
Bi-Canv
CP-Down
CP-Up
(c) CS-Fusion
Spei
Concall
3
X1
Distibution
Distriutioon
sour puuree
Disibcttion Compars
Concot
Disrb.tion
X2
(d) Distribution Compare
(e) Channel Shuffle
(b) CP-Up: Consistent-pixee sesiiitent-pixlllpsaaappp (c) CS-Fusion: Channel-shuffle fusion. (d) In the skip connection, the value ranges of two features (X1. X2) may be significant differences, which impedes effective fusion. (e) The illustration of channel shuffle. the shuffled features (xf", x²") have closely matched value ranges.
Figure 3: (a) CP-Down:
(I) CP-Down: We evenly split the input features X ER XC along the channel dimension and process them through two convolutions with identical input and output dimensions. The stable (match- ing) dimension allows the usage of identity shortcuts. Finally, by applying Pixel-UnShuffl [57], we reduce the resolution of the features while increasing the channel number. The formula is:
X
do
x E RR#x
PS (c, C2(x))
x2C the output of CP-Down; (-)- and C2(-) represent two (binarized) convo- where x ER x lutions; PS- denotes the Pixel-UnShuffle operation.
(2) CP-Up: Similarly, feature upsampling is achieved through two convolutions combined with Pixel-Shuffle. The operation can be mathematically expressed follows:
ouu PS (Concat (C1 (xxr)) ,2 (x))))
where, x HRRWxx and x but 21 2 x2W & denotes the input and output of CP-Up; Concat, (.)
represents the channel concatenation operation; PS is the Pixel-Shuffle operation.
With the above design, we ensure the flow of full-precision information throughout the UNet, effectively improving feature representation and enhancing SR performance.
Channel-Shuftle Fusion. To effectively fuse the features in the skip connection while meeting the requirements for dimension matching in binarization, we propose the channel-shuffl fusion (CS- (CS-Fusion), as shown in Fig. 3c. Given two features X1. X xC, we first employ the channel-shufe operation to mitigate the differences in their value ranges, as illustrated in Fig. 3e. Specifically, split the two features according to the odd and and concatenate features along the channel dimension, based on odd and even indexes, to produce two shuffle features x x5 EIR ER Hx
channel indexes. Then, pair
This process be formulated as follows:
Xy [x]] x X C-1 2 n (1,2),
5/ Concat (fx 1m - i (/..,,%/,j--11,
m E (1,2),
Through visualization in Fig. 3e, becomes balanced. Subsequently, we process the shuffled features through two convolutions and addition to produce the final fused feature xth CR Hx
observe that the value range of features after channel shuffle
in manner similar Eq. (1),
(4) where (-) and Cgh(-) (-) are two (binarized) convolutions. This process realizes the fusion of two features, ensuring that dimensions are matched within the fusion process and in subsequent modules (e.g., ResBlock), Meanwhile, the matched dimension allows the usage the identity shortcut, thus effectively transferring full- pell-precision information. Overall, our proposed CS- SS--uusion achieves effective feature integration the skip connection. Therefore, binarized model can better represent features and improve SR performance. Furthermore, our CS-Fusion does not introduce additional memory or computational overhead since the channel shuffle only involves feature transformation operations. Experiments in Sec. 4.2 further reveal the impacts of CS-Fus CS-Fusion.
sh Cx (x**) )+ + cah (x²²),
5

---

Modu e: ups.8.res block.b lock block.3
Module: ups res ock.b ock2 block.3
Aciaaton Varue
2002
200.0
Figure 4: Visualization of the changes in actívation distribution across 50 timesteps.
TaR
TaA
x
x x b
Sign
f-bit
RPReLU
+
Nout
x
1-bit
Nout
RPReLU
(8) Basic BI-Conv Block
(b) BI-Conv Block with TaR&TaA
Figure 5: (a) The basic binarized convolutional (BI-Conv) block. The learnable bias and the activation function RPReLU adjust the activations. (b) In timestep-aware redistribution (TaR) and activation function (TaA), multiple pairs of b and RPRcLU are applied to adapt to the multi-step in DM. At each step t, only one pair of b and RPReLU is used (the darker modules with solid lines).
3.2 Activation Distribution
Basic Binarized Convolutional Block, We first introduce the basic binarized module, as illustrated in Fig. 5a. For the full-precision activation x/ "R""xxx binarize the shifted activation to 1-bit activations with sign function Sign(-). The process is:
initially shift its distribution and
Sign (x") = [+1, x' x²20
= xx
(-1, x" <0 (Vx" Ex", Vx E xx").
(5)
where bERC is a learnable parameter; HW the 1-bit activation. Meanwhile, for the binarized convolution, the full-precision weight w/ ERRC XC.. X.a xK.. is also binarized to 1-bit weight w' Cous X xK,XKw To compensate for the differences between binary and full-precision weights, scale w using the mean absolute value of w/ [50]. The total operation is:
w/lll
w
Sign(w)), Vwl E w., Vw" E w",
where is the number of w values. Subsequently, the floating pointg matrix multiplication in full-precision convolution can be replaced by logical XNOR and bit-counting operations as:
X = x * w bit-count (XNOR (x", w²))
where * means the convolutional operation; x ER XXXC is the output of 1-bit convolution. Then, we adjust d with the activation function RPReLU [38], resulting in x ER xWxC
Finally, we combine xb with full-precision activation x/ via identity shorteut get the final output = XX H xcxt Moreover, since the sign function Sign(-) is non-differentiable, we use the ttraight-through estimator (STE) [1] for backpropagation to train binarized models.
Distribution Analysis. In diffusion models, the multi-step iterative design leads to changes in the activation distribution the timestep changes. By visualizing the activation distributions different timesteps in Fig. 4, we can observe that activation distributions of adjacent timesteps are similar, whereas those separated by larger intervals show significant differences.
For full-precision models, the impact of these variations may be small due to the real-valued weight and activation. In contrast, for binarized modules, the activation distribution has a substantial impact on feature representation, and consequently, affects the SR performance. This is because 1-bit modules, due to the binary weights, struggle to effectively learn representations from different distri- butions, thereby limiting their modeling capabilities. Meanwhile, during the activation binarization, the sign function further amplifies activation differences, particularly for values around zero [38].
The basic binarized module utilizes the learnable biase and the activation function RPReLU to adjust the input and output activations. This approach mitigates the representational challenges posed by activation distribution differences across timestep some extent. However, these static designs are insufficient to cope with the extreme activation changes across multiple timesteps in diffusion models. Consequently, the SR performance of the binarized diffusion model limited. Experiments in Sec. 4.2, further demonstrate the above analyses.

---

Timestep-aware PedistributiionAAttiaatt Function. To cope with the variability of activation distribution with timestep, we propose the timestep-award redistribution (TaR) and timestep-aware activation function (TaA). The module details are illustrated in Fig. 5b. The design of TaR and TaA inspired by the mixture of experts (MoE) [56]. applying a set of learnable biases and RPReLU activation functions to accommodate different timesteps.
Specifically, we apply K pairs of bias and RPReLU for TaR (b()R)O) and TaA (R where iE [1, 2..... ........ K. Given the total timesteps (e.g., 21,2.... T)), we evenly divide them into K groups in sequence. For the input activation H Xx at t-th timstep (1E 2, ... T),, select the corresponding pair of bias and RPReLU based on the group associated with Ł, to adjust its input and output activation. The process can be formulated as:
K
x² = TaR(x))n) = -X,, +
+E 1 [ - b(O)
2=1
(8)
b4 Xact
TaA(x 6.b
Σ
[xtr]] RPReLU ( (x
where (() the indicator function; x"',, Xout- Xact EIR HX represent, at f-th timestep, the shifted input activation, the output of the 1-bit convolution, the output of the RPReLU activation function, respectively. Since the activations at adjacent timesteps exhibit a certain degree of similarity (as shown in Fig. 4), we employ the fixed grouping sampling strategy (defined in Eq. (8)).
b,t bet
Essentially, the TaR and TaA segment the multi-step process into smaller groups, limiting the range of activation changes. This reduces the difficulty of adjusting activations, allowing the binarized module to better adapt to changing activations. Therefore, the proposed TaR and TaA can effectively enhance the representation ability of the binarized module and ultimately improve SR performance. Meanwhile, compared to the basic module, there are no additional computational costs in our TaR and TaA. This is because, for each timestep, only one pair of bias and RPReLU are selected for use.
4 Experiments
4.1 Experimental Settings
Data and Evaluation. We take DIV2K [59] and Flickr2K [33] as the training dataset. Meanwhile, we evaluate the models with four benchmark datasets: Set5 [2], B100 [42], Urban [18], and Manga109 109 [43]. Experiments are conducted under two upscale factors: x2 and x4. The LR images are generated from HR images through bicubic downsampling degradation. We apply two distortion- based metrics, PSNR and SSIM [64], which are calculated on the Y channel (i.e., luminance) of the YCbCr space. We also use the perceptual metrics: LPIPS [12]. Following previous work [66, 49], the total parameters (Params) of the model are calculated as Params Params-PPrams" + Params and the overall operations (OPs) as OPs= * +OPS where P /32 and OPs" /64; the superscripts f and b denote full-precision and binarized modules, respectively.
Implementation Details. For the noise estimation network, we set the encoder and decoder level to 4. In each level of the encoder, we use 2 Residual Blocks (ResBlocks), while in the decoder, we set to 64. We set the number of bias and RPReLU
apply 3 ResBlocks. The number of channels in TaR and TaA as 5=5. For the diffusion model, we set the total number of timesteps to T 7220000. During the inference phase, we employ the DDIM sampler with 50 timesteps.
Training Settings. We train models with the L1 loss. We employ the Adam optimizer [22] with 3 =09 and 32=0.99, and learning rate of x 10-4 The batch size is set to 16, with total of 1.000K iterations. Input LR images are randomly cropped to size 64x64. Random rotations of 90", 180%, and 270" and horizontal flips are used for data augmentation. Our model is implemented based on PyTorch [47] with two Nvidia A 1100-80G GPUs.
4.2 Ablation Study
In this section, we conduct all experiments on the x2 scale factor. We apply DIV2K [59] and Flickr2K [33] as the training dataset, and Manga109 [43] as the testing dataset. The training iterations are set to 500K. Other settings are the same as defined in Sec. 4.1. We test the computational complexity (i.e., OPs) of one single sampling step on the output size 3 2556
7

---

Method
Baseline
+ldenbly +CP- +CP.Dowa&Up +CS Fusion
+TaR&Taa
Method
Params (M)
OPs
PSNR (dB)
LPIPS
Params (M)
4.29
4.29
4.30
4.58
4.10
0.1695
OPs (G)
36.67
36.67
36,67
36.67
36.67
Concat
4.29
36.67
31,08
0.0327
PSNR (dB)
27.66
29.29
31,08
31.99
32.66
Split
4.30
36.67
29.67
0.0384
LPIPS
0,0780
0.0658
0.0327
0.0261
0.0200
CS-Fusion
4.30
36.67
31.99
0.0261
(a)
Break-down ablation.
(b) Ablation on feature fusion.
Method
TaR
TaA
Params (M)
Ops (G)
PSNR (dB)
LPIPS
Pair
5
4.30
36.67
31.99
0.0261
Params (M)
4.30
4.58
In
4.37
36.67
29.27
0.0337
OPs (G)
36.67
36.67
36.67
4.51
29.13
PSNR (dB)
31.99
32.66
All
4.58
36.67
32,66
0.0200
L
0.0261
0.0229
0.0200
(c) Ablation time aware module (TaR and TaA).
(d) Numbers (#) of bias and RPReLU pair.
Table 1: Ablation study. We train models on DIV2K and Flickr2K, and evaluate on Mangal 109 (x2).
Addition (Module: upe
Channel Fusion (Module: ups.21
T3R (Module: downs.S.res blockb lockl come
Sum
Fuakin
Figure 6: Activation distribution in the skip connection. Input 1(2): X1, X2. Sum: X1 xy.. Fusion 1(2): x x²"
Arteatts was sembee
Artoon Wake tomtanee
Figure 7: Weights of biases by
(1..,, 5)) in TaR.
Break Down. We first execute break-down ablation on different components of our method. The results are listed in Tab. la. The baseline is established by using binarized convolution (BI-Conv) and SUn)S-eUnSShuffle for dimension scaling in the downsample, upsample, and fusion (skip connection) modules of the UNet. Meanwhile, the basic BI-Conv block (Fig. 5) is employed without the identity shorteut. The baseline performance is poor, with the PSNR of 27.66 dB. Then, we add identity shortcut, wewpsstett-iixl-ddveeee module (CS- (CS-Fusion), and timestep- redistribution (TaR) and activation function (TaA) in sequence. We can find that the performance gradually increases. Ultimately, the final model achieves gains of dB in PSNR and 0.0580 in LPIPS, compared to the baseline.
(CP-Down) and upsample (CP-Up), channel-shuffle-fune
Channel-Shufii Fusion. We experiment on the fusion module for the skip connection. We attempt four methods: directly add two features (Add); concatenation and adjust dimension by binarized convolution (Concat); process each feature via binarized convolution and add them; and our proposed CS-Fusion. The results shown in Tab. 1b. Due the differences between features, direct addition (Add) can hardly work, even with convolution (Split). Moreover, since the concatenation changes the dimensions, the Method (Concat) also degrades the performance. In contrast. our proposed CS-Fusion, eliminates the distribution imbalances by channel fusion, thereby achieving effective fusion. The visualization in Fig. 6, further indicates that addition cannot fuse data with narrow value distributions, whereas channel shuffle can effectively integrate.
Timestep-aware Module. We conduct experiments on the time-aware redistribution (TaR) and activation function (TaA). Firstly, we experiment with the combinations of TaR and TaA in Tab. Ic. We find that effective improvements are only achieved when both TaR and TaA are employed. This may be because both input and output activation impact the learning of the binarized module. Then, in Tab. 1d, we experiment with the pair number (#Pair) bias and RPReLU. The experiments show that 5 pairs already lead to effective improvements. Considering the additional parameters, we adopt 5 as the pair number in BI-DiffSR. Moreover, we present the weights of five learnable biases in the TaR (module position shown at the image top) in Fig. 7. The difference in weights indicates that TaR can effectively adapt to the varying activation distributions at different timesteps.
4.3 Comparison with State-of-the-Art Methods
We compare our proposed BI-DiffSR with recent binarization methods, including BNN [19], DoReFa [71], XNOR [50], IRNet [48], ReActNet [38], and BBCU [66]. To ensure fair com- parison, we set the parameters (Params) and complexity (OPs) of all binarization methods to be similar. We also compare our BI-DiffSR with the full-precision (FP) model, SR3 [54].
8

---

Method
Scale
Params (M)
Ops
(G)
SSIM
Set5
B100
LPIPS
Urban100
LPIPS
Manga109
L.PIPS
Bicubic
x2
N/A
N/A
33:67
0.9303
0. 1274
29.55
0.8431
0.2508
26.87
0.8403
0.2064
30.82
0.9349
0.1025
SR3
176.41
0.9513
0.0310
0.8683
0.0700
0.9060
0.0430
0.9682
0.0161
BNN [19]
37.93
13.97
0.5210
0.4529
13:73
0.453
0.5784
12:75
04236
0.5575
9.29
0.3035
0.7489
DoReFa [7]]
x2
4.78
37.93
16.43
0,6553
0.2662
16.11
0,5912
0.3972
15.09
0.5495
0,4055
12.35
0.4609
0.5047
XNOR [50]
x2
37.93
32.34
0.8661
0.0782
27.94
0.7548
0.1665
27.47
0.8225
0.1
31,99
0.9428
0.0326
IRNet [+8]
X2
4.78
37.93
32.55
0,9340
0.0446
27.76
0.8199
0.1115
26.34
0.8452
0,0913
23,89
0.7621
0.1820
ReActNet 1381
X2
4.85
37.93
34,30
0.9271
0.0351
28.36
0.8158
0,0943
27.43
0.8563
0.0731
32.16
0.9441
0.0379
BBCU [66]
X2
4.82
37.75
34.31
0,9281
0.0393
28.39
0.8202
0.0905
28.05
0.8669
0,0620
32.88
0.9508
0.0272
BI-DiTSSR (ours)
X2
4.58
36.67
35.68
0.9414
0.0277
29.73
0.8478
0.0682
28.97
0.8815
0.0522
33.99
0.9601
0.0172
Bicuhic
x4
N/A
NIA
28.43
0.8111
0.3398
25.95
0.6678
0.5244
23.14
0.6579
0.4729
24.90
0.7876
0.3210
[54]
176.41
31.03
0.8798
0.1127
0.6933
0.2247
25.52
0.7702
0.1438
28.77
0.8854
0.0646
BNN [19]
x4
4.78
37.93
12:21
0.3103
0.8310
12.30
0,2128
0.9519
11.30
0.2191
0.9592
8.96
0.1833
1.0117
DoReFa 1711
4.78
37.93
10.40
0.246
0.9855
9.78
0,1709
1,0793
8.79
0.1614
1.1186
7.52
0.1464
1.1169
XNOR [50]
x4
4.78
37.93
28.06
0.8274
0.1381
25.25
0.6552
0.3101
23.13
0.6647
0.2564
23.84
0.7839
0.1559
IRNet [48]
4.78
37.93
15.52
0.3514
0.7548
16.38
0,3121
0.7072
15.23
0,3043
0.7068
11.82
0.2442
0.8354
ReActNet [38]
x4
4.85
37.93
29.23
0,8362
0.1472
23.56
0.5670
0.3339
22.32
0.6440
0.2276
25.32
0.7854
0.1721
BRCU [16]]
4.82
37.75
25.44
0.7795
0.1650
21,46
0,5472
0.3206
20.52
0.6293
0.2290
23.02
0.7966
0.1496
BI-DifISR (ours)
X4
4.58
36.67
29.63
0,8374 0.1109
25.84
0.6719 0,2754
24.11
0.7177 0.1823
26.95
0.8548
0.0889
Table 2: Quantitative comparison with state-of-thee oftte--ftthheart binarization methods. The best and second best results are coloured with red and blue. Our method surpasses current approaches.
HR
Ricuhic
SR3 (FP) [54]
XNOR [50]
Urban 100: imp_023
IRNer
ReActNer [38]
BBCU [66]
BI-DinSR (ours)
Bieubic
SR3 [54]
XNOR [50]
Urban100: img_033 033
IRNct
ReActNer [38]
BBCU [66]
BI-DiN'S (ours)
Figure& 8: Visual comparison (x4) in some challenge cases,
Quantitative Results. We provide the quantitative comparisons in Tab. 2. We test OPs of single-step sampling on the output size xx256x 256. Compared to other binarization methods, our BI-DifiSR Urban 100 and Manga (x2), BI-DiffSR surpasses
achieves the best performance. Specifically, the second-bor -bott method, BBCU, with PSNR gain of 0.92 and 1.11 dB. respectively. Moreover, compared to the full-precision model, SR3, our method achieves comparable or even better perceptual performance with only 8.3% Params and 20.8% OPs. For instance, BI-DifTSR achieves 93.6% LPIPS results of SR3 on Manga109, These results demonstrate the superiority of our method.
Visual Results. We present visual comparisons ( (x4) in Fig. 8. Previous binarization methods struggle recover image details in challenging cases. In contrast, our method can restore clearer results with more texture details. Meanwhile, the difference between our BI-DiffSR and the full-precision model results is small. More visual results are provided in the supplementary material.
Conclusion
In this paper, we propose the BI-DifiSR, a novel binarized diffusion model for image SR. Specifically, we first design the UNet structure suitable for binarization. To ensure dimension consistency and full- precision information transfer, we design the pewpistent-pixl-lownssaee
(CP-Down) and upsample (CS- (SS-Fusion to enhance information
(CP-Up). Meanwhile, we develop the fusion within the skip connection. Furthermore, in response to the multi-step mechanism of diffusion models, we design the timestep- redistribution (TaR) and activation functions (TaA) to adapt to the varying activation distributions. The TaR and TaA enhance the representational capabilities of the binarized modules under multiple timesteps. Extensive experiments indicate that our method outperforms current binarization methods, and achieves comparable perceptual performance to the full full-precision model, demonstrating substantial potential.

---

Acknowledgments. This work is supported by the National Natural Science Foundation of China (62141220, 62271414), Shanghai Municipal Science and Technology Major Project (2021SH22ZX01022, the Fundamental Research Funds for the Central Universities, Zhejiang Provin- cial Distinguish Young Science Foundation (IR2230100011), Zhejjang "Pioneer" and "leeaing Goose" R&D Program (2045DDXDXX0006 2024C03182) the Key Project of Westlake Institute for Opto- electronics (2023GD007) and Ningbo Science and Technology Bureau, "Science and Technology Yongjiang 2035" Key Technology Breakthrough Program (2024Z126).
References
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3444 2013.
[2] Marco Bevilaequa, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolutio based on nonnegative neighbor embedding. In BMVC, 2012. 7
[3] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. In ICLR, 2020. 3
[4] Weimin Chen, Yuqing Ma, Xianglong Liu, and Yi Yuan. Hierarchical generative adversarial networks for single image super resouuiono In CVPR, 2021. 3
[5] Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggregation transformer for image super-resolution. In ICCV, 2023. 3
[6] Zheng Chen, Yulun Zhang, Jinjin Gu. Yongbing Zhang, Linghe Kong, and Xin Yuan. Cross aggregation transformer for image restoration. In NeurIPS, 2022. 1
[7] Zheng Chen, Yulun Zhang, Ding Liu, Bin Xia, Jinjin Gu, Linghe Kong, and Xin Yuan. Hicrarchical integration diffusion model for realistic image deblurring. In NeurIPS, 2023.
[8] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah. Ditfusion models in vision: survey, TPAMI. 2023.
Ryan Dahl, Mohammad Norouzi, and Jonathon Shlens. Pixel recursive super resolution. In ICCV, 2017. 1,
[10] Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 2016. 1.3
[11] Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. In NeurlPS, 2024.
[12] Abhijay Ghildyal and Feng Liu. Shift-tolerant perceptual similarity metric. In ECCV, 2022. 7
[13] lan Goodfellow, Jean Pouget-Abadin Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
[14] Chumming Hc, Chengyu Fang, Yulun Zhang, Kai Li, Longxiang Tang, Chenyu You, Fengyang Xiao, Zhenhua Guo, and Xiu Li. Reti-diff: Illumination degradation image restoration with retinex-based latent diffusion model. arXiv preprint arXiv:2311.11638 2023. 3
[15] Chunming He, Yuqi Shen, Chengyu Fang, Fengyang Xiao, Longxiang Tang. Yulun Zhang, Wangmeng Zuo, Zhenhua Guo, and Xiu Diffusion models in lowelevel vision: survey, arXiv preprint arXiv:2406.1111 2024. 3
[16] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020. 1.
[17] Shengehao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End-to-end vision-based autonomous driving via spatial-temporal feature learning. In ECCV, 2022. 3
[18] Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, 2015. 7
[19] Ita Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniy, and Yoshua Bengio, Binarized neural networks. In NeurIPS, 2016. 2, 8,9
10

---

[20] Xinrui Jiang, Nannan Wang, Jingwei Xin, Keyu Li, Xi Yang, and Xinbo Gao. Training binary neural network without batch normalization for image super-tesolution. In AAAI, 2021.
[21] Bahjat Kawar, Jiaming Song, Stefano Ermon, and Michael Elad. Jpeg artifact correction using denoising diffusion restoration models. NeurlPS Workshop. 2022. 3
[22] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 7
[23] Diederik P Kingma and Max Welling. Auto-cncoding variational bayes. In ICLR, 2014. 3
[24] Christian Ledig, Lucas Theis, Ferene Huszár, Jose Cahallero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz. Zehan Wang. and Wenzhe Shi. Photo-realiste single image super-resolution using a generative adversarial network. CVPR, 2017."
[25] Haoying Li, Yifan Yang, Meng Chang, Shiqi Chen, Huajun Feng, Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffussio probabilistic models. Nearocomputing, 2022. 3
[26] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In ICCV, 2023.
2023. 2,
[27] Yanjing Li, Sheng Xu, Xianbin Cao, Xiao Sun, and Banchang Zhang. Q-dm: An efficient low-bit quantized diffusion model. In NeurlPS, 2023. 3
[28] Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. In NeurIPS, 2024. 3
[29] Yuchen Li, Haoyi Xiong, Linghe Kong, Zeyi Sun. Hongyang Chen, Shuaiqiang Wang, and Dawei Yin. Mpgraf: a modular and pre-traines graphformer for learning to rank web-scale. In ICDM, 2023.
[30] Yuehen Li, Haoyi Xiong, Linghe Kong, Rui Zhang, Fanqin Xu, Guihai Chen, and Minglu Li. Mhrr: Mooes recommender service with meta hierarchical reinforced ranking. TSC, 2023. 3
[31] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In ICCVW, 2021. 1.3
[32] Jingyun Liang. Andreas Lugmayr, Kai Zhang, Martin Danelljan, Luc Van Gool, and Radu Timofte. Hierarchical conditional flow: A unified framework for image super-resolution and image rescaling. In ICCV, 2021. 3
[33] Bee Lim, Sanghyun Son, Heewon Kim, Scungjun Nah, and Kyoung Mu Lec. Enhanced deep residual networks for single image super-resolution. In CVPRW, 2017. 3,7
[34] Xinqi Lin, Jingwen He, Ziyan Chen, Zhaoyang Lyu. Ben Fei, Bo Dai, Wanli Ouyang. Yu Qiao, and Chao Dong. Diffbir: Towards blind image restoration with generative diffusion prior. In ECCV, 2024. 3
[35] Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, Yanfeng Wang, and Weidi Xie. Intelligent grimm--pen-ended visual storytelling via latent diffusion models. In CVPR, 2024. 3
[36] Chang Liu, Haoning Wu, Yujie Zhong, Xiaoyun Zhang, and Weidi Xie. Intelligent grimm open-ended visual storytelling via latent diffusion models. In CVPR, 2024. 3
[37] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. ICLR, 2022. 1,3
[38] Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV, 2020. 2, 6, 8,
[39] Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang- Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cans with improved representational capability and advanced training algorithm. In ECCV, 2018. 2, 3
[40] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022.
[41] Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Srflow: Learning the super- resolution space with normalizing flow. In ECCV, 2020. 3

---

[42] David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001. 7
[43] Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa. Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga 109 dataset. MTAP, 2017. 7
[44] Sachit Menon, Alexandru Damian, Shijia Hu, Nikhil Ravi, and Cynthia Rudin, Pulse: sufe-superviee photo upsampling via latent space exploration of generative models. In CVPR, 2020. 1
[45] Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint 2rX0v:200608225,, 2021. 2
[46] Ben Niu, Weilei Wen, Wenqi Ren. Xiangde Zhang, Lianping Yang, Shuzhen Wang, Kaihao Zhang. Xiaochun Cao, and Haifeng Shen. Single image super-resolution via a holistic attention network. In ECCV, 2020. 3
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019. 7
[48] Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In CVPR, 2020. 2, 2, 8,
[49] Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Zhongang Cai, Ziwei Liu, Fisher Yu, and Xianglong Liu. Bibench: Benchmarking and analyzing network binarization. In ICMI, 2023. 3,7,
[50] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classifica- tion using binary convolutional neural networks. In ECCV, 2016. 2, 6, 8,9
[51] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. ICML, 2015. 3
[52] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022. 1,3
[53] Olaf Ronneherger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015. 3
[54] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. TPAMI, 2022. 2, 8,
[55] Yuzhang Shang. Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan. Post-training quantization on diffusson models. In CVPR, 2023. 3
[56] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely- gated mixture-of of--eeertss layer. arXiv preprint arXiv:1001.06338, 2017. 7
[57] Wenzhe Shi, Jose Caballero, Ferene Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR, 2016. 5
[58] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2020. 1,
[59] Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, Lei Zhang, Bee Lim, Sanghyun Son, Heewon Kim, Scungjun Nah, Kyoung Mu Lee, et Ntire 2017 challenge on single image super-resolution Methods and results. In CVPRW, 2017.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017. 4
[61] Huan Wang. Suhas Lohit, Michael N Jones, and Yun Fu. What makes a" good" data augmentation in knowledge distillation-a statistical perspective. In ICLR, 2022. 2
[62] Xintao Wang. Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In ECCVW, 2018. 1,3
12

---

[63] Yinhuai Wang. Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In ICLR, 2023. 1.3
[64] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. TIP, 2004. 7
[65] Rongyuan Wu, Tao Yang. Lingchen Sun, Zhengqiang Zhang, Shuai Li, and Lei Zhang. Seesr: Towards semantics-aware real-world image super-resolution. In CVPR, 2024. 3
[66] Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, and Luc Van Gool. Basic binary convolution unit for binarized image restoration network. In ICLR, 2022. 2, 7,
[67] Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. In NeurlPS, 2021. 3
[68] Eduard Zamfir, Zongwei Wu, Nancy Mchta, Yulun Zhang, and Radu Timofte. See more details: Efficient image super-resolution experts mining. In ICML, 2024.
[69] Yulun Zhang, Haotong Qin, Zixiang Zhao, Xianglong Liu, Martin Danelljan, and Fisher Yu. Flexible residual binarization for image super-resolution. In ICMI., 2024.
[70] Yulun Zhang. Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolation. In CVPR, 2018. 1,3
[71] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. ICLR, 2016. 2, 8, 9
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to our abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the supplementary file.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided implementation details in the experiments section. We release all the code and models at: https: //tp://gitttt com/ 2thencchen1999/555 BBII9//IIIIffS
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We provide the code and pre-trained models at: https://github. com/ zhengchen1999 99I
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. ec/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: halliii/ggiee//ee
/ /nips...
sssissiin wwwssssoiiiiyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided implementation details, which cover the above questions.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Please refer to the experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neee httpa/://////p/////eeeeeeee cc cc/public/Bthiccciuiie
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer to the supplementary file.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This work poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have credited most previous works in the paper. The license and terms are respected properly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release code and pre-trained models at: https://githu: com/ zhengchen1 999 19t1/9999/IIIDD BBI-DiffSRR In the paper, have provided implementation details and other contents to reproduce our results.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "sWXfiOGDuH", "reviewer": "Reviewer_mEsa", "summary": "This paper introduce a novel binarized diffusion model, BI-DiffSR, for image SR. A UNet architecture optimized for binarization, channel shuffle fusion, and time-step-aware redistribution and activation functions are designed. The experimental results proved the effectiveness of the method.", "questions": "See Weaknesses.", "limitations": "Limitations were discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. This paper is well written, nicely presented, and well organized.\n\n2. Binarized diffusion networks are promising.\n\n3. The performance improvement over other binary SR networks is significant.", "weaknesses": "1. Lack of discussion with some related works[1, 2, 3, 4], in particular [1] which is also for binary SR networks. Please analyze and discuss the differences with [1,2].\n\n2. Ablation experiments are not convincing enough. Comparisons with some other activation function or fusion methods [1, 2, 3, 4] should be included.\n\n3. It is always known that diffusion models are slow. Although binarization will speed up the operation, can it achieve a better trade-off in performance and efficiency than a real-valued efficient SR network. It is suggested to compare with some efficient SR networks [5, 6, 7] in terms of Params, FLOPs, inference time and performance.\n\n> 1. Flexible Residual Binarization for Image Super-Resolution. ICML24.\n\n> 2. Q-DM: An Efficient Low-bit Quantized Diffusion Model. NIPS23. \n\n> 3. Binarized Low-light Raw Video Enhancement. ICCV23.\n\n> 4. Binarized Spectral Compressive Imaging. NIPS23.\n\n> 5. Efficient long-range attention network for image super-resolution. ECCV22.\n\n> 6. DLGSANet: lightweight dynamic local and global self-attention networks for image super-resolution. ICCV23.\n\n> 7. Feature modulation transformer: Cross-refinement of global representation via high-frequency prior for image super-resolution. ICCV23.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer mEsa (denoted as R1)\n\n\n\n`Q1-1` Lack of discussion with some related works[1, 2, 3, 4], in particular [1] which is also for binary SR networks. Please analyze and discuss the differences with [1,2].\n\n`A1-1` Thanks for your advice. We add more analyses and discussions of related works and incorporate them into our paper.\n\n**FRB [1]**\n\n- **Differences:**\n  1. ***Application Model:*** FRB targets one-step models (*e.g.*, Transformer); our BI-DiffSR suits diffusion models with multiple sampling steps.\n  2. ***Binarization Approach:*** FRB proposes a new binarization operation (SRB) to reduce performance gaps; our method uses regular binarization operation, while focusing on dimension matching and timestep awareness.\n  3. ***Training Approach:*** FRB uses full-precision distillation (DBT) to guide the binarized network; we train our model using a basic L1 loss for generalizability.\n- **Analysis:** FRB and our approach improve binarized SR models from different aspects. The SRB and DBT proposed in FRB could potentially be integrated with our method.\n\n\n\n**Q-DM [2]**\n\n- **Differences:**\n  1. ***Task:*** Q-DM focuses on low-bit quantization (*e.g.*, 2/4-bit) for generative tasks; we address 1-bit quantization for image SR.\n  1. ***Time Aware Approach:*** Q-DM stabilizes activations over timesteps by training adaptations; we design timestep awareness TaA and TaR to adjust activation distributions.\n  1. ***Training Strategy:*** Q-DM uses distillation to fine-tune binary DM; we train the network with no special training strategy.\n\n\n- **Analysis:** Q-DM focuses on different aspects from our method but could be integrated to explore better quantization diffusion models.\n\n\n\n**BRVE [3]**\n\n- **Differences:**\n  1. ***Fusion Module:*** BRVE uses real-value 1×1 Convs in its fusion block, adding parameters; our CS-Fusion is parameter-efficient.\n  2. ***Temporal Awareness:*** BRVE enhances restoration using temporal redundancy; our approach adjusts activation according to different sampling timesteps.\n\n\n\n\n**BiSRNet [4]**\n\n- **Differences:**\n  1. ***Structure:*** BiSRNet addresses dimension mismatch; our method further considers activation distribution in the skip connections.\n  2. ***Conv Modules:*** BiSR-Conv considers dimensional characteristics in SCI; our TaR&TaA in BI-Conv are timestep-oriented designs.\n\n\n\n`Q1-2` Ablation experiments are not convincing enough. Comparisons with some other activation function or fusion methods [1, 2, 3, 4] should be included.\n\n`A1-2` Thank you for the suggestions. We conduct more ablation studies on activation functions and fusion methods, with settings consistent with Sec. 4.2 (ablation study) of our paper.\n\n**Activation Function:**\n\nWe compare our timestep-aware activation function (TaA) with RPReLU and LeakyReLU, which are applied in previous methods.\n\n| Method     | Params (M) | OPs (G) | PSNR (dB) | LPIPS  |\n| :--------- | :--------: | :-----: | :-------: | :----: |\n| LeakyReLU  |    4.32    |  36.67  |   24.92   | 0.0903 |\n| RPReLU     |    4.37    |  36.67  |   29.27   | 0.0337 |\n| TaA (ours) |    4.58    |  36.67  |   32.66   | 0.0200 |\n\nResults demonstrate the superior performance of our TaA.\n\n\n\n**Fusion Method:**\n\nWe compare our channel-shuffle fusion (CS-Fusion) with BFB in BRVE [3] and BiFD in BiSRNet [4]. The structures of these fusion modules are detailed in the **attached PDF (Figure 1)**.\n\n| Method    | Params (M) | OPs (G) | PSNR (dB) | LPIPS  |\n| :-------- | :--------: | :-----: | :-------: | :----: |\n| BFB       |    6.38    |  43.11  |   30.53   | 0.0303 |\n| BiFD      |    4.30    |  36.67  |   29.67   | 0.0384 |\n| CS-Fusion |    4.30    |  36.67  |   31.99   | 0.0261 |\n\n1. Compared to BiFD, our CS-Fusion promotes fusion without increasing the Params.\n2. Compared to BFB, our CS-Fusion has smaller Params and OPs.\n\n\n\n`Q1-3` It is always known that diffusion models are slow. Although binarization will speed up the operation, can it achieve a better trade-off in performance and efficiency than a real-valued efficient SR network. It is suggested to compare with some efficient SR networks [5, 6, 7] in terms of Params, FLOPs, inference time and performance.\n\n`A1-3` Thank you for your suggestion. We compare our proposed BI-DiffSR with ELAN [5] and DLGSANet [6]. \n\nELAN, lacking official results, is retrained under the same settings as our method, yielding **ELAN***. We test the OPs and latency (simulated by daBNN) with outputs of 3×256×256, and assess LPIPS on Manga109 (×2).\n\n| Method                          | Step | Params (M) | Per-Step OPs (G) | Total OPs (G) | Per-Step Latency (s) | Total Latency (s) | LPIPS  |\n| :------------------------------ | :--: | :--------: | :--------------: | :-----------: | :------------------: | :---------------: | :----: |\n| ELAN* (**Transformer**)         |  1   |    8.25    |      161.24      |    161.24     |        50.61         |       50.61       | 0.0206 |\n| DLGSANet (**Transformer**)      |  1   |    4.73    |      73.52       |     73.52     |        20.07         |       20.07       | 0.0210 |\n| BI-DiffSR (**Diffusion**, ours) |  50  |    4.58    |      36.67       |    1833.44    |        13.00         |      650.09       | 0.0172 |\n\n1. Compared to real-valued efficient SR methods, our BI-DiffSR obtains superior **perceptual** performance (**LPIPS**).\n2. Our method achieves comparable **parameters**. Meanwhile, the **per-step** OPs (**36.67 G**) and latency (**13.00 s**) are lower, although the multi-step nature of DM increases total OPs and latency.\n3. Model compression (like binarization) and sampling acceleration are two **parallel** speedup strategies for DMs. As mentioned in the main paper, our method focuses on the former to accelerate **one inference step**. Thus, we apply the regular sampler (*i.e.*, DDIM, **50 steps**) to show its generality. \n4. Furthermore, our method has the potential to be used with more advanced samplers to further enhance the efficiency of DMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vX22UUCWT5", "reviewer": "Reviewer_Z33c", "summary": "The authors propose BI-DiffSR to binarize diffusion based image super-resolution (SR) model. They design a UNet architecture for the whole binarized model structure. To maintain dimension consistency, they propose two modules, CP-Down and CP-Up, which can further help transfer full-precision information. To enhance feature fusion, they propose the channel-shuffle-fusion (CS-Fusion). They also propose TaR and TaA to dynamically adjust activation distribution cross different timesteps. The authors provide extensice experiments to demonstrate the effectiveness of their proposed method.", "questions": "Can this method be applied to other diffusion models, like stable diffusion? If so, can the authors give some suggestions to binarize stable diffusion?\n\nCan we apply this binarization method to other related image restoration tasks? Like image denoising, deblurring?\n\nHow long did the authors to train the models?", "limitations": "Please refer to weaknesses", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The topic is very important and practical. Diffusion models have shown excellent performance for image super-resolution (SR). It is very practical to quantize the models before deploying them into devices. Binarization is an extreme tool to compress the SR model. Few works have been proposed to investigate such an important problem in image SR. \n\nThe authors give several insights for the specific topic. Namely, there are some key aspects in diffusion based image SR binarization, like dimension mismatch, fusion difficulty, and activation distraction. Those problems hinder the performance of binarized image SR diffusion models. The observation and analyses given in the introduction section are insightful and motivate readers well.\n\nTo alliveate the problems in binarized diffuision based SR models, the authors propose consistent-pixel-downsample (CP-Down) and consistent-pixel-upsample (CP-Up) to ensure dimensional consistency. They propose the channel-shuffle-fusion (CS-Fusion) to facilitate the fusion of different features within skip connections and suit binarized modules. They propose the timestep-aware redistribution (TaR) and timestep-aware activation function (TaA) to adjust the binarized module input and output arross different timesteps.\n\nThey provide extensive ablation study experiments (including quantitative results in Table 1 and visualization analyses in Figures 6 and 7.) to show the effects of each proposed components. Those experiments are convincing.\n\nThe authors provide comparions with SOTA methods. According to the main quantitive and visual comparisons, they show that their proposed BI-DiffSR achieves superior performance over others.\n\nThe overall writing and organization are pretty good. I think the work is well-prepared. The supplementary file further provides more details. The paper is easy to follow and they promise to release the code, which makes this work more convincing.", "weaknesses": "When binarizing full-precision model from 32-bit to 1-bit, ideally we can reduce the parameters by 32 times. But, as shown in Table 2, the authors reduce parameters from 55.41M to 4.58M (for scale 2). There is a gap between ideal case and practical one. Please give some analyese about the reasons for this gap. Also, are there any idea to further narrow the gap?\n\nThe parameters and Ops are reduced obviously from full-precision to binary one. But the authors did not give results about inference time on real devices or give some analyses. I am curious how fast the binarized model will be.\n\nThe writing can further refine in some cases. For example, in the abstract part (Line 9-10), “… to maintain dimension consistent” should be changed to “… to maintain dimension consistency”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer Z33c (denoted as R2)\n\n\n\n`Q2-1` When binarizing full-precision model from 32-bit to 1-bit, ideally we can reduce the parameters by 32 times. But, as shown in Table 2, the authors reduce parameters from 55.41M to 4.58M (for scale 2). There is a gap between ideal case and practical one. Please give some analyese about the reasons for this gap. Also, are there any idea to further narrow the gap?\n\n`A2-1` **Reasons:** \n\n1. We use a partial number (*i.e.*, 6) full-precision (FP) ResBlocks in the UNet decoder part to trade off parameters and performance.\n2. The proposed timestep-aware redistribution (TaR) and activation function (TaA) add ~0.3M parameters.\n\n\n\n**Strategy to Narrow the Gap:** In practice, we can adjust the number of FP ResBlocks according to the situation to suit different environments. For instance, on platforms with significant resource constraints, using fewer FP ResBlocks (and correspondingly more binarized ResBlocks) to reduce Params is acceptable, even if it slightly lowers performance.\n\n\n\n`Q2-2` The parameters and Ops are reduced obviously from full-precision to binary one. But the authors did not give results about inference time on real devices or give some analyses. I am curious how fast the binarized model will be.\n\n`A2-2` Thank you for your suggestion. In **Supplementary Material A.2**, we have compared the inference time of SR3 (FP) and BI-DiffSR (ours).\n\n| Method           | Params (M) | OPs (G) | Simulated Time (s) |\n| ---------------- | :--------: | :-----: | :----------------: |\n| SR3              |   55.41    | 176.41  |       55.37        |\n| BI-DiffSR (ours) |    4.58    |  36.67  |       13.00        |\n\nOur method operates faster compared to the full-precision approach (SR3).\n\n**Note:** Real device testing for binary models is limited by specific hardware requirements. Hence, we estimate inference times using the daBNN inference frame. The correlation between the operation count and running time on an **ARM64 CPU** is: the FP module costs 313.85 ms per GOP, and the BI module costs 354.56 ms per GOP.\n\n\n\n`Q2-3` The writing can further refine in some cases. For example, in the abstract part (Line 9-10), “… to maintain dimension consistent” should be changed to “… to maintain dimension consistency”.\n\n`A2-3` Thank you for the suggestion. We will check the entire paper and improve the writing.\n\n\n\n`Q2-4` Can this method be applied to other diffusion models, like stable diffusion? If so, can the authors give some suggestions to binarize stable diffusion?\n\n`A2-4` Other diffusion models, such as stable diffusion (SD), can be binarized by our method. The SD also employs the UNet noise estimation network, which is composed of ResBlocks and AttentionBlocks.\n\n**Methodology:**\n\n1. **Dimension Matching:** Adjust the number of channels in different layers of the UNet to match dimensions via our proposed CP-Down, CP-Up, and CS-Fusion.\n2. **ResBlock Binarization:** Directly binarize it using our proposed binary convolution (BI-Conv).\n3. **AttentionBlock Binarization:** For this block, the core is to binarize self-attention (**SA**). This can be extended from BI-Conv. The SA obtains $Q, K, V$ by three linear projections, and gets the output by another linear. We can use 1x1 BI-Conv to binarize the linear layer. And for the attention computation $\\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)V$, the activations $Q, K, V$, and the intermediate attention map $\\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)$ can be binarized by the sign function Sign(·).\n\n\n\n`Q2-5` Can we apply this binarization method to other related image restoration tasks? Like image denoising, deblurring?\n\n`A2-5` Yes, our method can be applied to other tasks.\n\n1. **Direct Application:** Our BI-DiffSR model can be directly trained on specific tasks like deblurring using datasets such as GoPro, without structural modifications.\n2. **Binarization Task-Specific Model:** Our binarization method can also binarize diffusion models specific to these tasks.\n\n\n\n`Q2-6` How long did the authors to train the models?\n\n`A2-6` It takes approximately 84 hours to train the BI-DiffSR (×2) on two NVIDIA A100 GPUs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q4uZ6xam7v", "reviewer": "Reviewer_KWS7", "summary": "This work present a novel binarized diffusion model for improving the efficiency of super resolution tasks. Compared with the existing works, this work first pointed out the specific challenges of binarized DMs for SR, including the dimension mismatch and fusion difficulty of representations. Then this work present several techniques: consistent-pixel down/upsampleing, channel-shuffle fusion, and Time-step-aware redistribution function for the aforementioned challenges. Comprehensive results show that the provided binarized DMs for SR not only significantly outperform the binarized models with existing SOTA binarization methods, but also achieve floating-point level performance. And for the efficiency, the statistics of params and flops show the advantage of proposed method, and the paper also present the real inference time on edge, which seems important and encouraged in the binarization community.", "questions": "1. Compared with the quantization of DMs for SR, can authors provide discussion about the advantage and motivation of binarization?\n2. What is the type of ARM hardware for the evaluation of inference?\n3. If the proposed method have potential generalized to more generative tasks and architectures?", "limitations": "The authors have addressed the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. As far as I know, this is the first work to present the specific binarization method for diffusion model of SR. Since the good performance has been achieved by DMs in various SR tasks, it’s important to present novel insight to compress these models, especially considering the severe drop still exists after binarizing by existing SOTA methods.\n2. The motivation is intuitive and techniques are novelty, especially considering the features of DMs. The proposed CP-Up/down and channel shuffle are highly specified to the architecture of the diffusion models, which is novel and cannot be achieved by previous methods, including binarization function and binarized structures. And the computation is also small, allowing minor burden with significant performance improvement. And the proposed activation function also focus on the high dynamic of activation range during  time-step, which is one of the most critical problem for the quantization of DMs. \n3. The proposed method achieve SOTA results in accuracy. Comprehensive comparison has been included in this paper, including SOTA binarization methods and various evaluation datasets. The results show that the proposed outperforms than previous binarized DMs for SR with significant improvements.\n4. In this paper, diverse analysis, including quantitative, statistical, and visual results are presented in detail. More important, the paper shows the efficiency evaluation based on real inference libraries and edge hardware, which is of great significance for practical application.", "weaknesses": "Though it’s a good paper, some issues should be addressed.\n1. The writing and presentation of the paper should be improved, including but not limited to the grammar and description. For example, some basic knowledge about quantization, SR, and DMs seems to be summarized as a preliminaries section; and let the proposed techniques be highlighted in Figure 2.\n2. As for the efficiency, I suggest the authors present the computation more detailed, such as present the computation of each part in the whole network before and after the binarization. This will show the efficiency advantage of the proposed method much clearer.\n3. The proposed challenge I and II are insightful, but more further discussion (such as visual, quantitative, or theoretical analysis) are presented after proposing. I suggest authors do more discussion about that.\n4. Some recent binarization methods for SR [1] are suggested to be compared and some quantized DMs [2] are suggested to be discussed the differences to make the comparison more comprehensive.\n[1] Flexible Residual Binarization for Image Super-Resolution. ICML 2024\n[2] EfficientDM: Efficient Quantization-Aware Fine-Tuning of Low-Bit Diffusion Models. ICLR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer KWS7 (denoted as R3)\n\n\n\n`Q3-1` The writing and presentation of the paper should be improved, including but not limited to the grammar and description. For example, some basic knowledge about quantization, SR, and DMs seems to be summarized as a preliminaries section; and let the proposed techniques be highlighted in Figure 2.\n\n`A3-1` Thank you for your suggestions. We will check and improve our manuscript.\n\nWe have added a **preliminaries section**:\n\n>**SR Pipeline.** SR network aims to reconstruct a low-resolution image into a corresponding high-resolution image. The process can be represented as follows:\n>$$\n>I_{SR} = \\mathcal{SR} (I_{LR} ; \\Theta),\n>$$\n>where $\\mathcal{SR} (\\cdot)$ denotes the image SR network, and $\\Theta$​ represents the network parameters.\n>\n>**Binarization Framework.** In binarized networks, weights and activations are converted using the sign function:\n>$$\n>\\operatorname{Sign}\\left(x\\right)= \\begin{cases}+1, & x \\geq 0 \\\\\\ -1, & x < 0 \\end{cases}.\n>$$\n>As the sign function Sign(⋅) is non-differentiable, we use the straight-through estimator (STE) for backpropagation to train binarized models:\n>$$\n>\\frac{\\partial \\operatorname{sign}}{\\partial \\boldsymbol{x}}= \\begin{cases}1 & \\text { if }|\\boldsymbol{x}| \\leq 1 \\\\\\ 0 & \\text { otherwise }\\end{cases}.\n>$$\n>Binarization reduces storage and accelerates computation through XNOR and bit-counting operations.\n\nFor **Figure 2**, we have modified the image to highlight our proposed techniques and added it to the **attached PDF (Figure 2)**.\n\n\n\n`Q3-2` As for the efficiency, I suggest the authors present the computation more detailed, such as present the computation of each part in the whole network before and after the binarization. This will show the efficiency advantage of the proposed method much clearer.\n\n`A3-2` Thank you for your suggestion. We present the Params and OPs for each part of the UNet network: encoder, bottleneck, and decoder, before and after binarization. OPs are tested with an output of 3×256×256.\n\n| Part       | Params$^f$ (M) | Params$^d$ (M) | OPs$^f$ (G) | OPs$^d$ (G) |\n| ---------- | :------------: | :------------: | :---------: | :---------: |\n| Encoder    |     13.09      |      0.65      |    42.66    |    1.04     |\n| Bottleneck |     10.59      |      1.45      |    13.97    |    2.31     |\n| Decoder    |     27.30      |      2.46      |    79.29    |    33.31    |\n\n1. Params$^f$ and OPs$^f$ correspond to the full-precision modules, while Params$^d$ and OPs$^d$ are for the binarized versions. \n\n2. The encoder and bottleneck show high compression ratios. To balance parameters and performance, we use a partial number full-precision ResBlocks in the decoder, which reduces the compression ratio.\n\n\n\n`Q3-3` The proposed challenge I and II are insightful, but more further discussion (such as visual, quantitative, or theoretical analysis) are presented after proposing. I suggest authors do more discussion about that.\n\n`A3-3` Thank you for your suggestion. We add more visual results for the two challenges in the **attached PDF (Figure 3)**. Here are the analyses:\n\n**Challenge I: Dimension Mismatch.** The binarized modules struggle with detailed features. Using full-precision features through residual connections helps, as shown in columns three and four of Figure 3. However, in UNet, up/down sampling makes feature shape mismatches, making it unable to use residual connections. We solve this problem via CP-Down/Up.\n\n**Challenge II: Fusion Difficulty.** The activation distributions of skip connections vary greatly, resulting in information loss in the decoding stage and a lack of details in the restored image. Therefore, we design CS-Fusion to address this problem, as shown in columns four and five of Figure 3.\n\n\n\n`Q3-4` Some recent binarization methods for SR [1] are suggested to be compared and some quantized DMs [2] are suggested to be discussed the differences to make the comparison more comprehensive. \n\n`A3-4` Thanks for your advice. We discuss the differences with FRB [1] and EfficientDM [2] below.\n\n**FRB [1]**\n\n1. **Network Design:** FRB targets one-step networks; our method includes dimension and timestep considerations in DMs.\n2. **Training Approach:** FRB uses full-precision distillation to guide the training; we use a basic L1 loss to train our model.\n\n**EfficientDM [2]**\n\n1. **Quantization Level:** EfficientDM implements low-bit (2/4/8) quantization; we consider the extreme case of 1-bit (binarization).\n2. **Activation Management:** EfficientDM employs temporal activation LSQ to handle changes in activation distribution; we propose timestep-aware techniques (TaA and TaR).\n\n\n\n`Q3-5` Compared with the quantization of DMs for SR, can authors provide discussion about the advantage and motivation of binarization?\n\n`A3-5` We discuss them below.\n\n**Advantage:**\n\n1. **Higher Compression Ratio:** Binarization (1-bit) offers the highest parameter reduction compared to 2/4/8-bit quantizations.\n2. **Efficient Computation:** Binarization allows the model to perform inference via bitwise operations, a capability not inherent to other bit-level models.\n\n**Motivation:** Diffusion models (DMs) have excellent generation abilities but are resource-intensive. Binarization greatly reduces this overhead, enhancing usability on limited-capacity devices.\n\n\n\n`Q3-6` What is the type of ARM hardware for the evaluation of inference?\n\n`A3-6` The evaluation is conducted on a **Raspberry Pi 3 Model B+** (BCM2837B0, Cortex-A53 (ARMv8) 64-bit SoC @ 1.4GHz).\n\n\n\n`Q3-7` If the proposed method have potential generalized to more generative tasks and architectures?\n\n`A3-7` Yes, it has the potential.\n\n1. **Other Tasks:** The model can be directly applied to unconditional generative tasks by removing the LR input.\n2. **Other Architectures:** Many generative diffusion models, like stable diffusion, can be binarized by our proposed approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WbAyBpXqW8", "reviewer": "Reviewer_MRyP", "summary": "The paper introduces BI-DiffSR, a novel binarized diffusion model for image super-resolution, designed to accelerate the inference speed and reduce computational costs of diffusion models while maintaining high performance. It proposes a UNet architecture optimized for binarization, featuring consistent-pixel downsampling/upsampling and channel-shuffle fusion to address dimension mismatch and fusion difficulty, alongside a timestep-aware redistribution and activation function to adapt to varying activation distributions across different timesteps. The model demonstrates superior results over existing binarization methods, approaching the perceptual quality of full-precision models with significantly reduced memory and computational requirements.", "questions": "Please refer to the weaknesses above.", "limitations": "The authors have addressed the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is well-written and easy to understand.\n\n- This paper designs a novelty 1-bit UNet for accurate binarized diffusion model, including:\n   - New downsample module and upsample module for Dimension Consistency.\n   - Channel shuffle module to balance the activation value ranges of two input features.\n   - The timestep-aware redistribution (TaR) and timestep-aware activation function (TaA)\n\n- Experiments achieve the state-of-the-art in super resolution with diffusion.", "weaknesses": "- The basic BI-Conv block lacks novelty, which is as the same as the binarized module in ReActNet that contains RSign and RPReLU.\n\n- TaR uses different parameters for different time steps, but in the mean while, the normal time embedding is projected into the resblock, it is also a time-aware on feature maps, what is the differences or why TaR works?\n\n- SR3 is not a new diffusion baseline for super resolution, ResShift[1], SinSR[2] should be better, and the metrics of PSNR, SSIM, LPIPS is much old, the CLIPIQA, MUSIQ, MANIQA should be better for evaluating the performance of generative super resolution.\n\n- Self-attention and MLP are common modules in diffusion, such as LDM[3] and ResShift[1], which require a lot of computation. How can the method in this paper be extended to self-attention and MLP?\n\n\n[1] Yue, Zongsheng, Jianyi Wang, and Chen Change Loy. \"Resshift: Efficient diffusion model for image super-resolution by residual shifting.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Wang, Yufei, et al. \"SinSR: diffusion-based image super-resolution in a single step.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Rombach, Robin, et al. \"High-resolution image synthesis with latent diffusion models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to Reviewer MRyP (denoted as R4)\n\n\n\n`Q4-1` The basic BI-Conv block lacks novelty, which is as the same as the binarized module in ReActNet that contains RSign and RPReLU.\n\n`A4-1` Thanks for pointing it out. We clarify it below.\n\n1. Indeed, our basic BI-Conv block utilizes RSign and RPReLU for the learnable bias and activation function. However, this is **not** an innovative aspect of our method.\n\n2. Our **innovation** lies in timestep-aware operations (*i.e.*, TaR and TaA). Inspired by the mixture of experts (MoE), we employ different RSign and RPReLU according to different timesteps.\n\n3. We demonstrate in **Table 1c** of the main paper that using timestep-aware operations effectively improves the performance of binarized DM.\n\n   | Method       | Params (M) | OPs (G) | PSNR (dB) | LPIPS  |\n   | :----------- | :--------: | :-----: | :-------: | :----: |\n   | RSign&RPReLU |    4.30    |  36.67  |   31.99   | 0.0261 |\n   | TaA&TaR      |    4.58    |  36.67  |   32.66   | 0.0200 |\n\n4. To enhance transparency, we will clarify in the manuscript that RSign and RPReLU are from ReActNet.\n\n\n\n`Q4-2` TaR uses different parameters for different time steps, but in the mean while, the normal time embedding is projected into the resblock, it is also a time-aware on feature maps, what is the differences or why TaR works?\n\n`A4-2` We explain the differences below.\n\n**Differences:**\n\n1. **Location:** Time embedding acts on the **ResBlock**; TaR operates within the **BI-Conv** (inside the ResBlock).\n2. **Purpose:** Time embedding operates at **the network level**, enabling the model to be aware of different timesteps and enhance feature modeling; TaR operates at **the module level** (such as Conv), adjusting the activation distribution based on the timestep to improve module computations.\n\n**Functionality of TaR:**\n\n1. **Adapting Activation Distribution:** TaR dynamically adjusts Conv input activation according to different timesteps, adapting changing distributions.\n2. **Enhancing Conv Representation:** TaR divides multiple steps into smaller groups, limiting the changing range of activations, thereby reducing the representational difficulty of BI-Conv and enhancing feature extraction.\n3. **Complementarity with Time Embedding:** TaR and time embedding serve different purposes and are not contradictory. Using TaR (along with TaA) in DM further enhances model performance. This is evidenced in **Table 1c** (as detailed in `A4-1`).\n\n\n\n`Q4-3` SR3 is not a new diffusion baseline for super resolution, ResShift[1], SinSR[2] should be better, and the metrics of PSNR, SSIM, LPIPS is much old, the CLIPIQA, MUSIQ, MANIQA should be better for evaluating the performance of generative super resolution.\n\n`A4-3` Thanks for your suggestion. We apply the new baseline **ResShift** [1] (official code), and compare our method, BI-DiffSR, with the full-precision (FP) model and the previous binarized model, BBCU.\n\n**Setup:**\n\n1. We train on the bicubic-×4 task on the DF2K dataset, batch size 32, iteration 400,000, sample step 15, maintaining other settings consistent with the official setup.\n2. As our method involves adjusting the channels of UNet, we modify the UNet structure (maintaining the core module unchanged) to create **ResShift\\***. We then apply **BBCU** and **BI-DiffSR** (quantizing both Conv and SA) on ResShift\\* to ensure fairness.\n3. We measure the FLOPs on an input size of 128×128. All metrics are evaluated on Manga109.\n\n| Method                      | Params (M) | OPs (G) | CLIPIQA $\\uparrow$ | MANIQA $\\uparrow$ | MUSIQ $\\uparrow$ |\n| :-------------------------- | :--------: | :-----: | :----------------: | :---------------: | :--------------: |\n| ResShift* (full-precision)  |   58.12    | 170.77  |       0.7662       |      0.5543       |     67.3729      |\n| BBCU (binarized)            |    3.89    |  20.19  |       0.7611       |      0.5175       |     64.0867      |\n| BI-DiffSR (binarized, ours) |    3.65    |  18.57  |       0.7659       |      0.5638       |     66.5730      |\n\n**Results Analyses:**\n\n1. **Comparison with BBCU:** Our method performs better across all new metrics.\n2. **Comparison with ResShift\\*:** Our method achieves similar performance while significantly reducing parameters (**93.72%**) and operations (**89.13%**).\n3. **General Effectiveness:** These results further demonstrate the generalizability and effectiveness of our method.\n\n\n\n**Further Consideration:**\n\n1. **Baseline Choice:** In our paper, we use **SR3** as the baseline, considering it is a normal DM SR method, which can demonstrate the generality of our approach. Classic metrics such as PSNR, SSIM, and LPIPS are used for similar reasons.\n2. **Inclusion of New Baseline:** As the reviewer suggested, using new baselines and metrics better reflects the effect of our method. We will include ResShift-based experiments in our paper for a thorough assessment.\n\n\n\n`Q4-4` Self-attention and MLP are common modules in diffusion, such as LDM[3] and ResShift[1], which require a lot of computation. How can the method in this paper be extended to self-attention and MLP?\n\n`A4-4` The methods are below.\n\n**Extension to Self-Attention (SA):**\n\n1. **Linear Layers:** SA involves three linear layers for $Q, K, V$, and a linear projection for the output. These parts can be replaced with 1×1 BI-Conv to achieve binarization.\n2. **Matrix Operations:** The matrix operation, $\\text{Attention}(Q, K, V) = \\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)V$, can be binarized by binarizing the activations: $Q, K, V, \\text{softmax}\\left(\\frac{QK^T}{\\sqrt{d_k}}\\right)$, through the sign function, Sign(·).\n\n**Extension to MLP:**\n\n1. **Linear Layers:** The MLP consists of several linear layers, which can be directly replaced with 1×1 BI-Conv for binarization.\n\n**Experiment:** We binarize **ResShift [1]** with our method, effectively reducing Params and OPs while maintaining performance, as detailed in `A4-3`.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Response to all reviewers and area chairs



Dear Reviewers and Area Chairs,



We thank all reviewers (**R1-mEsa**, **R2-Z33c**, **R3-KWS7**, **R4-MRyP**) and area chairs for their insightful comments and valuable time.

We are pleased that:

- R2 and R3 appreciate our intuitive motivation and diverse and insightful analyses.
- R2, R3, and R4 acknowledge the novelty of our techniques.
- All reviewers recognize the impressive performance of our method, which surpasses existing approaches.



We have responded individually to each reviewer to address any concerns. Here, we offer a summary:

- We discuss the differences with more **related methods**, including binarized SR and quantized DM.
- We add more experiments, including **ablation studies** and comparisons on the new baseline (**ResShift**) with new metrics.
- We compare our approach with **efficient SR networks** and provide an analysis.
- We add a **preliminary section** to our paper and commit to improving our manuscript.
- We introduce methods to **extend our approach** to new modules, models, and tasks.
- We clarify the **gap** between theory and practice, the **advantages** of the binarization method, the **innovative** aspects of BI-Conv, and the **difference** between TaR and time embedding.
- We address various **detailed questions** raised by reviewers, including actual running speeds, training times, module efficiencies, and ARM hardware types.
- Finally, we include an **attached PDF** as a supplement for some questions.



Thanks again to all the reviewers and area chairs. We appreciate you taking the time to review our responses and hope to **discuss further** whether the issues have been resolved. If you need any clarification, please let us know.



Best Regards,

Authors

Meta-review: The paper received all accept recommendations. The area chairs agree with this recommendation.
Decision: Accept (poster)