Basic Binary Convolution Unit for Binarized Image Restoration Network
Authors: Bin Xia
Yulun Zhang
Yitong Wang
Yapeng Tian
Wenming Yang
Radu Timofte
Luc Van Gool
Venue: ICLR 2023 poster (2023)
Keywords: Imgae Restoration； Low-Level Vision

Abstract: Lighter and faster image restoration (IR) models are crucial for the deployment on resource-limited devices. Binary neural network (BNN), one of the most promising model compression methods, can dramatically reduce the computations and parameters of full-precision convolutional neural networks (CNN). However, there are different properties between BNN and full-precision CNN, and we can hardly use the experience of designing CNN to develop BNN. In this study, we reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks. We conduct systematic analyses to explain each component's role in binary convolution and discuss the pitfalls. Specifically, we find that residual connection can reduce the information loss caused by binarization; BatchNorm can solve the value range gap between residual connection and binary convolution; The position of the activation function dramatically affects the performance of BNN. Based on our findings and analyses, we design a simple yet efficient basic binary convolution unit (BBCU). Furthermore, we divide IR networks into four parts and specially design variants of BBCU for each part to explore the benefit of binarizing these parts. We conduct experiments on different IR tasks, and our BBCU significantly outperforms other BNNs and lightweight models, which shows that BBCU can serve as a basic unit for binarized IR networks. All codes and models will be released.

Published as a conference paper at ICLR 2023
BASIC BINARY CONVOLUTION UNIT FOR BINARIZED IMAGE RESTORATION NETWORK
Bin Xia', Yulun Zhang', Yitong Wang', Yapeng Tian", Wenming Yang's Radu Timofte', and Luc Van Gool 'Tsinghua University 2ETH Zürich ByyteDance Inc *University of Texas at Dallas University of Würzburg
ABSTRACT
Lighter and faster image restoration (IR) models are crucial for the deployment on cesource-limited devices. Binary neural network (BNN), one the most promising model compression methods, can dramatically reduce the computations and parameters of full-precision convolutional neural networks (CNN). However, there different properties between BNN and full-precision CNN, and hardly use the experience of designing CNN to develop BNN. In this study, we reconsider components in binary convolution, such residual connection, Batch- Norm, activation function, and structure, for IR tasks. We conduct systematic analyses to explain each component's 's role in binary convolution and discuss the pitfalls. Specifically. we find that residual connection can reduce the informa- tion loss caused by binarization; BatchNorm can solve the value range gap be- tween residual connection and binary convolution; The position of the activa- tion function dramatically affects the performance of BNN. Based on our find- ings and analyses, design simple yet efficient basic binary convolution unit (BBCU). Furthermore, we divide IR networks into four parts and specially de- sign variants of BBCU for each part explore the benefit of binarizing these parts. We conduct experiments on different IR tasks, and our BBCU signifi- cantly outperforms other BNNs and lightweight models, which shows that BBCU can serve as a basic unit for binarized IR networks. The code is available at https: http:/ggtttaaa thub com Biiiiiiiii/BBBe
INTRODUCTION
Image restoration (IR) aims to restore a high-quality (HQ) image from its low (LQ) coun- terpart corrupted by various degradation factors. Typical IR tasks include image denoising, super- resolution (SR), and compression artifacts reduction. Due to its ill-posed nature and high practical values, image restoration is active yet challenging research topic in computer vision. Recently, the deep convolutional neural network (CNN) has achieved excellent performance by learning a mapping from LQ HQ image patches for image restoration (Chen & Pock, 2016; Zhang et al., 201 8a; Tai et al., 2017; Xia et al., 2023). However, most IR tasks require dense pixel prediction and the powerful performance of CNN-based models usually relies on increasing model size and computational complexity. That requires extensive computing and memory resources. While, most hand-held devices and small drones are not equipped with GPUs and enough memory to store and run the computationally expensive CNN models. Thus, it is quite essential to largely reduce its computation and memory cost while preserving model performance to promote IR models.
Binary neural network (Courbariaux et al., 2016) (BNN, also known 1-bit CNN) has been rec- ognized as one of the most promising neural network compression methods (He et al., 2017; Jacob et al., 2018: Zoph & Le, 2016) for deploying models onto assource-limited devices. BNN could achieve 32x memory compression ratio and up to 64x computational reductions on specially de- signed processors (Rastegari et al., 2016). Nowadays, the researches of BNN mainly concentrate on high-level tasks, especially classification (Liu et al., 2018; 2020), but do not fully explored in low- level vision, like image denoising. Considering the great significance of BNN for the deployment of IR deep networks and the difference between high-level and low-level vision tasks, there is an
"Corresponding Author: Wenming Yang, yang. wwng..weninn@@.sssnnnuaaamm

---

Published as a conference paper at ICLR 2023
25.6
EUCUL TOuns
EBCU (Ouns)
30.0
BUCO (Duns
25.4
-BCCU-M(OUS)
ReActhet
VERe (85)
31.5
25.3
ITM
29.8
25.2
IRNeC
31.0
STM
29.7
RaketNet
EIM
305
29.5
25.0
a Rl--Real
30
0.76 9.77
0.63
0.54
5.7 0.68
Number of DDfs (F)
Number of ats (F)
Number of DaN 06)
(a) Super Resolution. Take SRResNet backbons and best on 1x Urban100.
(b) Denoising. Take DeCNN
(c) Deblocking Take DaCNN3
backbonc and test on = 15 Urban100.
backbone and test g 10 Classics.
Figure I: Our BBCU achieves the SOTA performance on IR tasks with efficient computation.
urgent need to explore the property of BNN on low-level vision tasks and provide a simple, strong, universal, and extensible baseline for latter researches and deployment.
Recently, there have been several works exploring the application of BNN image SR networks. Specifically, Ma et al. (Ma et al., 2019) tried to binarize the convolution kernel weight to decrease the SR network model size. But, the computational complexity is still high due to the preservation of full-precision activations. Then BAM (Xin et al., 2020) adopted the bit accumulation mechanism to approximate the coll-precision convolution for the SR network. Zhang et al. (Zhang et al., 202 designed a compact uniform prior to constrain the convolution weight into a highly narrow range centered at zero for better optimization. BTM (Jiang al., 2021) further introduced knowledge distillation (Hinton et al., 2015) to boost the performance of binarized SR networks.
However, the above-mentioned binarized SR networks can hardly achieve the potential of BNN. In this work, we explore the properties of three key components of BNN, including residual connec- tion (He et al., 2016), BatchNorm (BN) (loffe Szegedy, 2015), and activation function (Glorot et al., 201 1) and design a strong basic binary Conv unit (BBCU) based on our analyses.
(1) For the IR tasks, we observe that residual connection is quite important for binarized IR networks That is because that BNN will binarize the input full-precision activations to 1 or 11 before binary convolution (BC). It means that BNN would lose a large amount of information about the value range of activations. By adding the full-precision residual connection for each binary convolution (BC), BNN can reduce the effect of value range information loss.
(2) Then, we explore the BN for BBCU BNN methods (Liu et al., 2020) for classification always adopt BN in BC. However, IR tasks, EDSR (Lim et al., 2017) has demonstrated that BN is harmful to SR performance. We find that BN in BNN for IR tasks is useful and can be used to balance the value range of residual connection and BC. Specifically, as shown in Fig. 2 (b), the values of the full full-precision residual connection are mostly in the range of to 1 because the value range of input images is around O to or -1 to I, while the values of the BC without BN are large and ranges from -15 to 15 for the bit counting operation (Fig. ??). The value range of BC is larger than that of the residual connection, which covers the preserved full-precision image information in the residual connection and limits the performance of BNN. In Fig. (a), the BN in BNN for image restoration can realize the value range alignment of the residual connection and BC.
(3) Based on these findings, to remove BN, we propose a residual alignment (RA) scheme by mul- tiplying the input image by an amplification factor k to increase the value range of the residual connection rather than using BN to narrow the value range of the BC (Fig. 2 (c)). Using this scheme can improve the performance of binarized IR networks and simplify the BNN structure (Sec. 4.5).
(4) In Fig. 2 (d), different from BNNs (Liu al., 2020; 2018) for classification, we further move the activation function into the residual connection and can improve performance (Sec. 4.5). That because activation function would narrow the negative value ranges of residual connection. Its information would be covered by the next BC with large negative value ranges (Fig. 2 (c)).
(5) Furthermore, we divide IR networks into four parts: head, body, upsampling, and tail (Fig. (a)). These four parts have different input and output channel numbers. Previous binarized SR networks (Xin et al., 2020; Jiang et al., 2021) merely binarize the body part. However, upsampling part accounts for 52.3% total calculations and needs be binarized. Besides, the binarized head and tail parts are also worth exploring. Thus, we design different variants of the BBCU to binarize these four parts (Fig. 3 (b)). Overall, contributions can be mainly summarized in threefold:
2

---

Published as a conference paper at ICLR 2023
We believe work is timely. The high computational and memory cost of IR networks hinder their application on shoource-limited devices. BNN, as one of the most promising compression methods, can help IR networks to solve this dilemma. Since the BNN-based networks have different properties from full-precision CNN networks, we reconsider, ana- lyze, and visualize some essential components of BNN to explore their functions.
According to our findings and analyses on BNN, we specially develop a simple, strong, universal, and extensible basic binary Conv unit (BBCU) for IR networks. Furthermore, we develop variants of BBCU and adapt it to different parts of IR networks.
Extensive experiments on different IR tasks show that BBCU can outperform the SOTA BNN methods (Fig. 1). BBCU can serve as strong basic binary convolution unit for future binarized IR networks, which is meaningful to academic research and industry.
RELATED WORK
2.1 IMAGE RESTORATION
As pioneer works, SRCNN (Dong et al., 2015b), DrCNN (Zhang et al., 2017), and ARCNN (Dong et al., 20) use the compact networks for image super-resolution, image denoising, and compres- sion artifacts reduction, respectively. Since then, researchers had carried out its study with different perspectives and obtained more elaborate neural network architecture designs and learning strate- gies, such as residual block (Kim et al., 2016; Zhang et al., 2021a; Cavigelli et al., 2017), dense block (Zhang et al., 20: 2020; Wang et al., 2018), attention mechanism (Zhang et al., 2018b; Xia et al., 2022a; Dai et al., 2019), GAN (Gulrajani et al., 2017; Wang et al., 2018), and others (Wei et al., 2021; Peng et al., 2019; Jia et al., 2019; Fu et al., 2019; Kim et al., 2019; Fu et al., 2021; Soh et al., 2020; Xia et al., 2022d;c;b), to improve model representation ability. However, these IR networks require high computational and memory costs, which hinders practical application on edge devices. To address this issue, we explore and design BBCU for binarized IR networks.
2.2 BINARY NEURAL NETWORKS
Binary neural network (BNN) is the most extreme form of model quantization as it quantizes convo- lution weights and activations to only I bit, achieving great speed-up compared with its full-precision counterpart. As pioneer work, BNN (Courbariaux et al., 2016) directly applied binarization to full-precision model with a pre-defined binarization function. Afterward, XNOR-Net (Rastegari et al., 2016) adopted gain term compensate for lost information to improve the performance of BNN (Courbariaux et al., 2016). After that, Ri-Real (Liu et al., 2018) introduced residual connection to preserve full-precision scheme to retain the information in forward and backward propagations. Recently, ReActNet (Liu al., 2020) proposed generalized activation functions learn more representative features. For super-resolution task, Ma et al. (2019) binarized convolution weights to save model size. However, it can hardly speed up inference enough for retaining full fuel-ireciiion activations. Then, Xin et al. (2020) further binarized activations and used the bit accumulation mechanism to approximate the full-precision convolution for SR networks. Besides, Zhang et al. (2021b) introduced a compact uniform prior for better optimization. Subsequently, Jiang et al. (2021) designed binary training mechanism by adjusting the feature distribution. In this paper, we reconsider the function of cach basic component in BC and develop strong, simple, and efficient BBCU.
information in forward propagation. IRNet (Qin et al., 2020) developed
3 PROPOSED METHOD
3.1 BASIC BINARY CONY UNIT DESIGN
As shown in Fig. 2(a), we first construct the BBCU-VI. Specifically, the full-precision convolu- tion xf @ Wf (Xf. Wf, and & are full-precision activations, weights, and Conv respectively) is approximated by the binary convolution X2b Wb. For binary convolution, both weights and acti- vations binarized to -1 and +1. Efficient bitwise XNOR and bit counting operations can replace computationally heavy Koating-point matrix multiplication, which can be defined as:
x3b W/ bitcount (XNOR (2zh, wh)),
{ +1, if ff > oi.j
2 = Sign (44
-1
. E x5. Ihs E EX i [0, C)
(2)
11, if x I ais
3

---

Published as a conference paper at ICLR 2023
Sign
Binary
Conv
Berchhlomm
RPRAUU
Sign
Cony
APReLU
(x] Basic Binary Conv Unit (BBCU-I1)
(b) Basic Birory Conv Unit V2 (BBCU-V2)
1.8
1.55
1.53
RA
RA
RA
Sggn
RPRALU
Xi
Sign
APRALU
Rasic Binery Conv Unit V3 (RBCU-V3)
Baskc Binary Conv Unit V4 (BECU-V, Adopted)
Figure 2: The illustration of the improvement process of our BBCU. (a) The initial BBCU design. (b) We remove BatchNorm explore its actual function in IR tasks. We find that BatchNorm essential because it can balance the value range gap between residual connection and binary convolution. (c) We further propose the residual aligmment (RA) by multiplying an amplification factor & on the input image to address the value range gap. (d) Based on BBCU-V3, we move the activation function into the residual connection to reduce the negative value information loss.
|
|wi'll
w
Sign whi)) =
+
if Tww > 0 ,u/ E WWf
w/ E Wh i (0,
[will
if w/ 00
(3)
where X5 E RCCHXW and W.f E &RooutCin x are full-precision activations and convolu- tion weights in j-th layer, respectively. Similarly, X/ RCCHHW and W/h IR.Cout Kú K binarized activations and convolution weights in j-th layer separately. 2/15- w a?. and w are the elements of i-th channel of Xf. Wf. xh. and W/ respectively. as the learnable coeffi- cient controlling the threshold of sign function for i-th channel of Xf. It is notable that the weight
binarization method is inherited from XONR-Net (Rastegari et al., 2016), of which | is the average of absolute weight values and acts as a scaling factor to minimize the difference between binary and full-precision convolution weights.
Then, we use the RPReLU (Liu et al., 2020) as our activation function, which is defined as follows:
f (1i.)) {{
Vi. Y. + Si.j if yhu Yij 3r, (yi.) - 7.,)) + Cigs yi.j S Th. Mhi. E V3, (0,C),
(4)
where J/j RC is the input feature maps RPReLU function f(.) in j-th layer. Yi.j the element of i-th channel of D). K. and Gi, learnable shifts for moving the distribution. Ba, is learnalle coefficient controlling the slope of the negative part, which acts on i-th channel of )/-
Different from the common residual block, which consists of two convolutions used in full- full-precision IR network (Lim et al., 2017), we find that residual connection is essential for binary convolution to supplement the information loss caused by binarization. Thus, we residual connection for each binary convolution. Therefore, the BBCU-VI can be expressed mathematically as:
xff f (BatchNorm (x)h With) + x;5) = f (rg (2² @ wh) + + + 259)
(5)
where Kij Tj E R are learnable parameters of BatchNorm in j-th layer.
4

---

Published as a conference paper at ICLR 2023
In BNNs, the derivative of the sign function in Eq. 2 is an impulse function that cannot be utilized in training. Thus, we adopt the approximated derivative function as the derivative of the sign function. It can be expressed mathematically as:
a Sign
0z (x!)
+ 2
1 0i) 2
if o4 - o < 0z
Approx
x ai
if oz < of < az +1
(6)
0
otherwise
However, EDSR (Lim et al., 2017) demonstrated that BN changes the distribution of images, which is harmful to accurate pixel prediction in SR. So, can we also directly remove the BN in BBCU- VI to obtain BBCU-V2 (Fig. 2 (b))? In BBCU-VI (Fig. 2 (a)), we can see that the bit counting operation of binary convolution tends to output large value ranges (from -15 to 15). In contrast, residual connection preserves the full-precision information, which flows from the front end of IR network with small value range from around -1 to 1. By adding BN, its learnable parameters can narrow the value range of binary convolution and make close to the value range of residual connection avoid full-precision information being covered. The process can be expressed as:
Mean (1 (kg (xg) & Wh)) + ml) + Mean ((xff))
where Kg. J3 E RC are learnable parameters of BN in j-th layer. Thus, compared with BBCU-V1, BBCU-V2 simply removes BN and suffers a huge performance drop.
After the above exploration, we know that BN is essential for BBCU, because it can balance the value range of binary convolution and residual connection. However, BN changes image distribu- tions limiting restoration. In BBCU-V3, we propose residual alignment (RA) scheme by multiplying the value range of input image by an amplification factor (k: > 1) to remove BN Figs. 2(c) and 3(b): Mean (|>5h)) <- Mean ( (Ix22f))
(8)
We can see from the Eq. 8, since the residual connection flows from the amplified input image, the value range of Xf also is amplified, which we define as k2 Meanwhile, the values of binary con- volution are almost not affected by RA, because X3b filters amplitude information of k2 Different from BatchNorm, RA makes the value range of residual connection close to binary convolution (-60 to 60). Besides, we find that using RA to remove the BatchNorm has two main benefits: (1) Similar to full-precision IR networks, the binarized IR networks would have better performance without BatchNorm. (2) The structure of BBCU becomes more simple, efficient, and strong.
Based the above findings, are aware that the activation function (Eq. 4) in BBCU-V3 (Fig. 2(c)) narrows the negative value range of residual connection, which means it loses negative full- full-precision information. Thus, we further develop BBCU-V4 by moving the activation function into the residual connection to avoid losing the negative full nell-precision information. The experiments (Sec. 4.5) show that our design is accurate. We then take BBCU-V4 as the final design of BBCU.
3.2 ARM IR NETWORKS WITH BBCU
As shown in Fig. 3(a), the existing image restoration (IR) networks could be divided into four parts: head H. body B, upsampling U, and tail T. If the IR networks do not need increasing resolu- tion (Zhang et al., 2017), it can remove the upsampling U/ part. Specifically, given an LQ input IrQ. the process of IR network restoring HQ output í Hq can be formulated as:
= i=q==
=T(U(((H((TT)))
Previous BNN SR works (Xin et al., 2020; Jiang et al., 2021; Zhang et al.. 2021b) concentrate on binarizing the body B part. However, upsampling 24 part accounts for 52.3% total calculations and is essential to be binarized. Besides, the binarized head H. and tail T parts are also worth exploring.
As shown in Fig. 3(b), we further design different variants of BBCU for these four parts. (1) For the head H. part, its input is T E R3 and the binary convolution output with C' channels. Thus, we cannot directly add IrQ to the binary convolution output for the difference in the number of channels. To address the issue, we develop BBCU-H by repeating InQ to have C' channels. (2) For body B part, since the input and output channels are the same, we develop BBCU-B by directly adding the input activation to the binary convolution output. (3) For upsampling 24 part, develop BBCU-U by repeating the channels of input activations to add with the binary convolution. (4) For the tail T part, we develop BBCU-T by adopting T T.Q as the residual connection. To better evaluate
5

---

Published as a conference paper at ICLR 2023
Hoad SC
Body 9
Upasmpling TI
Tal
110
Full-Presision
Cony
Relu
(e) The archtocture Procissnn Network
Binary Conv
1u0
Residual Algimment (RA)
Ussamplin
RPReLU
ge
W, F geoceaoee
Wi T
Wi
nncuu4
BBCU-9
BRCUAU
M00
BEOUUT
Sign
ShweShulfle
for read (BECU))
for Body
Urppariiing (860L-4)
(BRCU-1)
Iippu & Output
archilactura Binarized IR Network
Figure 3: The illustration of full-precision and binary IR networks. (a) The IR network can be generally divided into four parts: head, body, upsampling, and tail. Notably, for IR tasks whose resolution remains unchanged, such as denoising and deblocking, we can ignore the upsampling part. (b) We equip different parts of binarized networks with the variants of BBCU. the benefits of binarizing each part on computations and parameters, we define two metrics:
Vc (PSNR* PSNR*) (OPs' (OPs'-OPs") -OPs")
(10)
Vp (PSNR* - PSNR*) (Parms' Parms")
(11)
where PSNR*, OPs", and Parms denote the performance, calculations, and parameters after bina- rizing one part of networks. Similarly, PSNR OPs/, and Parms measure full full-precision networks.
We adopt reconstruction loss Liec guide the image restoration training, which is defined as:
Liec
Tnq i TQQ
(12)
where JiQ and In are the real and restored HQ images, respectively. || denotes the T, norm.
4 EXPERIMENTS
4.1 EXPERIMENT SETTINGS
Training Strategy. We apply our method to three typical image restoration tasks: image super- resolution, color image denoising, and image compression artifacts reduction. We train all models DIV2K (Agustsson & Timofte, 2017), which contains 800 high-quality images. Besides, we adopt widely used test sets for evaluation and report PSNR and SSIM. For image super-resolution, simple and practical SRResNet (Ledig et al., 2017) as the backbone. The mini-batch contains 16 images with the size of 192x 192 randomly cropped from training data. We set the initial learning rate to x 10 train models with 300 epochs, and perform halving every 200 epochs. For image denoising and compression artifacts reduction, we take DrCNN and DrCNN3 as backbone (Zhang et al., 2017). The mini-batch contains 32 images with the size of 64x 64 randomly cropped from training data. We set the initial learning rate x 10-4, train models with 300,000 iterations, and perform halving per 200,000 iterations. The amplification factor & in the residual alignment is set to 130. We implement our models with a Tesla V100 GPU.
take
OPs and Parameters Calculation of BNN. Following Rastegari et al. (2016); Liu et al. (2018), the operations of BNN (OPs") calculated by OPs" = OPs 64 (OPsf indicates FLOPs), and the parameters of BNN (Parms") " is calculated by Parms Parms/ 32..
4.2 EVALUATION ON IMAGE
Following BAM (Xin et al., 2020), take SRResNet (Ledig et al., 2017) as backbone. We binarize body B part SRResNet with some SOTA BNNs including BNN (Courbariaux et al., 2016). Bi- Real (Liu et al., 2018), IRNet (Qin al., 2020), ReActNet (Liu et al., 2020), and BAM (Xin et al., 2020), BTM (Jiang et al., 2021). For comparison, we adopt our BBCU to binarize body part to ob- tain BBCU-L. Furthermore, we binarize body B and upsampling 24 parts simultaneously to develop

---

Published as a conference paper at ICLR 2023
Table 1: Quantitative comparison (average PSNR/SSIM) with BNNs for classical image Super- Resolution on benchmark datasets. Best and second best performance among BNNs are in red and blue colors, respectively. OPs are computed based on LQ images with a resolution of 320x 180.
Set5
Setl4
8100
Urban100
Manga109
Methods
Scale
Ops (G)
Parums (K)
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
SRResNei
85:43
1367
38.00
0.9605
33.59
0.9171
32.19
0.8997
32.11
0.9282
38.56
0.9770
SRResNet Lite
3.08
49
37.21
0.9578
32,86
0.9113
31.67
0.8936
30.48
0.9109
36.70
0.9722
Bicuhic
33.97
0.9330
30.55
0.8750
29.73
018494
27.07
0.8456
31.24
0.938
BNN
18.55
225
32.25
0.9118
29.25
0.8406
28.68
0.8104
25.96
0,8088
29.16
0.9127
Bi Bi-Real
18.55
225
32.32
0.9123
29.47
0.8424
28.74
0.8111
26.35
0.8161
29.65 99.64
0.9167
BAM
X2
20.52
226
37.21
0.9560
32.74
0.8910
30.20
0.9064
IRNe
18:55
225
37,27
0.9579
32.92
0.9115
31.76
0.8941
30.63
0.9122
36.77
0,9724
BTM
18.55
224
37,22
0,9575
32.93
0.9118
31.77
0.8945
30.79
0.9146
36.76
0,9724
RcActNet
0.9579
0,9728
BBCU-M (Ouns)
1.83
46
37.44
0.9584
33.04
0.9127
31.81
0.8946
30.84
0.9149
37 20
0.9738
BBCU-L (Owx)
18.55
225
37,58
0,9590
33.18
0.9143
31.91
0.8962
31.12
0.9179
37 50
0.9746
SRRt
SRResNes-Lite
5.39
31,40
0.8843
28.11
0.7692
27.26
0.7243
25.19
0.7544
28.92
0.8863
Bicubic
28.63
0.8128
26.21
0.7087
26.04
0.6719
23.24
0,6114
25.07
0,7904
0.7896
Bi-Real
79.20
372
27.75
0.7935
25.
D.68799
25.59
0.6478
22.91
0.6450
24.57
0.7752
BAM
81.17
373
31.24
0.8780
27.97
0.7650
27.15
0.7190
24.95
0.7450
27.24
0.8863
BTM
79.20
372
31,43
0.8810
28.16
0.7706
21.29
0.7255
25.
0.7605
29.19
0.8912
ReActNet
79.20
372
31.54
0.8859
28.19
0.7705
27.31
0.7252
25.35
0.7608
29.25
0.8912
BBCU-M (Ous)
3.95
31.54
0.8862
28.
0.7718
27.31
0.7263
0.7602
29.22
0.8918
BBCU-L (Ours)
79.20
372
31.79
0.8905
28.38
0,7762
27.41
0.7303
25.62
0,7696
69
0.8992
rearearaare rea
reare rea
rearearearee
Bhacke
BRCUAL
Boaks
SURONe
BOCL-L
Bilte
BBCUM
Bllic
BBCUUM
Figure 4: Visual comparison of BNNs for 4x image super-resolution.
BBCU-M. Besides, we reduce the number of channels of SRResNet to 12, obtaining SRResNet- Lite. In addition, we use Set5 (Bevilacqua et al., 2012), Set14 (Zeyde et al., 2010). B100 (Martin et al., 2001), Urban100 (Huang et al., 2015), and Manga109 (Matsui et al., 2017) for evaluation.
The quantitative results (PSNR and SSIM), the number of parameters, and operations of different methods are shown in Tab. 1. Compared with other binarized methods, our BBCU-L achieves the best results all benchmarks and all scale factors. Specifically, for 4x SR, BBCU-L surpasses ReActNet by 0.25dB, 0.27dB, and 0.44dB on Set5, Urban100, and Manga109, respectively. For 2x SR. BBCU-L also achieves 0.32dB, 0.27dB, and 0.58dB improvement on these three benchmarks compared with ReActNet. Furthermore, our BBCU-M achieves the second best performance on most benchmarks consuming 5% of operations and 13. 13.7% parameters ReActNet (Liu al., 2020) on 4x SR. Besides, BBCU-L significantly outperforms SRResNet-Lin by 0.16dB and 0.3dB with less computational cost, showing the superiority of BNN. The qualitative results are shown in Fig. 4, and our BBCU-L has the best visual quality containing more realistic details close to respective ground-truth HQ images. More qualitative results are provided in appendix.
4.3 EVALUATION ON IMAGE DENOISING
For image denoising, we use DrCNN (Zhang et al., 2017) as the backbone and binarize its body B part with BNN methods, including BNN, Bi-Real, IRNet, BTM, ReActNet, and our BBCU. We also develop DnCNN-Lite by reducing the number of channels of DnCNN from 64 to 12. The standard benchmarks: Urban 100 (Huang et al., 2015), BSD68 (Martin al., 2001), and Set12 (Shan al.,
7

---

Published as a conference paper at ICLR 2023
Table 2: Quantitative comparison (average PSNR) with BNNs for classical image denoising on benchmark datasets. Best and second best performance among BNNs are in red and blue colors, respectively, OPs is computed based on LQ images with a resolution of 320x 180.
CBSD68
Kodak24
Urhan 100
Methods
OPs (G)
Params (K)
0
15
0 25
a
50
15
a 25
0
50
15
0
25
a 50
DrCNN
38.42
667
33.90
31.24
27.95
34,60
32.14
28.95
32.98
30.81
27.59
DnCNN
1,38
24
32.26
29.64
26.49
32.73
30.25
27.22
30.97
28.46
25.21
BNN
0.8
24
26.90
22.67
16.41
27.12
22.58
16.25
26.67
22.67
16.54
Bi-Real
0.8
24
30.73
28.72
25.63
30.97
29.17
26.11
30.18
28.18
25.11
IRNet
0.8
24
31.37
29.01
26.75
31.61
29.54
27.48
30.57
28.35
26.00
BTM
0.8
24
32.25
29.91
26.79
32.75
30.64
27.62
30.99
29.05
25.86
ReActNet
0.8
24
32.26
29.95
26.93
32.78
30.65
27.70
31.27
29.20
26.01
BBCU (Ours)
33,08
27.33
33.66
31.28
28.15
32.27
29.96
Table 3: Quantitative comparison (average PSNR/SSIM) with BNNs for classical JPEG compres-
sion artifact reduction. Best and second best performance among BNNs are in red and blue colors, respectively. OPs is computed based on LQ images with resolution of 320x 180.
Livel
Classic
Methods
OPs (G)
Params (K)
20
300
-00
10
20
-0
60
DrCNK:3
38.29
665
29, 1910,8125
31.50 11.59008802
91.96.0.000 9090
33.96
$3.9600-933 92.47
29.400 29,40/0.80 5025
31 31,5300 5.10005510
32.91/0,5330
33.77/0 9003
Dnenn 3-Lite
1.36
24
9000 8061
31,200,875 6746
12,6200.00 9029
33.5000 91700.9179
5000 8000
32.06 32.0600.8611
33.36/0.8866
34,25/00.00
BNN
23,4580
33.1200.9128
29.41A1.7775
31.71A1,8537
33.048188417
35.89
Real
0.66
28,6700.801
31,0602.706
32.4200.899
33.36A091158
29.54A1.9934
31.5841,875
25,21,1,8345
34,06611.3996
IRNet
31.130,
32,4900
33
11.96018570
38.26/1.8836
34.12/1.008
BTM
52.51A0.000
22.02AL85997
34.31.01.0 5191
ReActhNet
6581
09.0600808
31,43006780
32.80099061
30.0001.000
32.2500.6655
33,50018003
34.45/1.0044
2019) are applied to evaluate each method. Additive white Gaussian noises (AWGN) with different noise levels a (15, 25, 50) are added to the clean images.
The quantitative results of image denoising are shown in Tab. 2, respectively. As one can see, our BBCU achieves the best performance among compared BNNs. In particular, our BBCU surpasses the state-of-the-ar BNN model ReActNet by 0.82dB, 0.88 0.88dB, and d CBSD68, Kodak24, and Urban datasets respectively as a = 15. Compared with DnCNN-Lite, our BBCU surpasses it by 0.92dB, 1.03dB, and 1. 1. on these three benchmarks as 15 consuming 58% computations of DrCNN-Lite. Qualitative results are provided in appendix.
4.4 EVALUATION ON JPEG COMPRESSION ARTIFACT REDUCTION
For this JPEG compression deblocking, we use practical DnCNN-3 (Zhang et al., 2017) as the back- bone and replace the full- aill-precision body B part of DnCNN3 with some competitive BNN methods, including BNN, Ri-Real, IRNet, BTM, ReActNet, and our BBCU. The compressed images are gen- erated by Matlab standard JPEG encoder with quality factors q E {10, 20, 30, 40). We take the widely used LIVEI (Sheikh, 2005) and Classic5 (Foi et al., 2007) as test sets to evaluate the per- formance of each method. The quantitative results are presented in Tab. 3. As we can see, our BBCU achieves the best performance on all test sets and quality factors among all compared BNNs. Specifically, our BBCU surpasses the state-of-the-a on the Livel 1 and Classic5 as = 40. In addition, our BBCU surpasses DrCNN-3-Lin by 0.16dB and 0.2dB on benchmarks as g = 10. The visual comparisons are provided in appendix.
BNN model ReActNet by 0.38dB and 0.34dB
4.5 ABLATION STUDY
Basic Binary Convolution Unit. To validate BBCU for the binarized IR network, we bina- rize the body part of SRResNet with four vari- ants of BBCU (Fig. 2) separately. The results are shown in Tab, (1) Compared with BBCU- V1, BBCU-V2 declines by 0.14 and 0.24dB on Urban 100 and Manga 109. This is because BBCU-V2 simply removes the BN making the
Table 4: PSNR (dB) values (4x) on four types of basic binary convolution unit (BBCU).
Methods
(P)
Se15
Se114
B100
Urhun 100
Manga109
BBCU-VI
79.20
31.54
28. 19
27.31
25.35
29.25
BRCULVV
79.20
51.37
28.07
27.22
25.21
29.01
BBCULV
79.20
31.71
28.31
27.37
25.51
29.54
BBCU-V4
79.20
31.79
27.41
25.62
29.69
value range of binary Conv far larger than residual connection and cover full-precision informa- tion (Fig. 2). (2) BBCU-V3 adds residual alignment scheme on BBCU-V2, which addresses the value range imbalance between binary Conv and residual connection and removes the BN. Since the BN is harmful for IR networks, BBCU-V3 surpasses BBCU-V1 by 0. 0.17dB, 0.16 and 0.29dB on Set5, Urban100, and Mangal09 respectively. (3) BBCU-V4 moves the activation function into the residual connection, which preserves the full-precision negative values of residual connection (Fig. 2). Thus, BBCU-V4 outperforms BBCU-V3.
8

---

Published as a conference paper at ICLR 2023
Table 5: The breakpoint posi-
Table The binarization benefit different parts IR net-
tion of residual connection.
works. We test models on Set14, and PSNR is 28.00
Position Idx
PSNR; (dB)
Head 7
Body
Upsampling 24
Tail T
25.43
OPs (M)
99.53
67947.73
76441.19
1592.53
5
25.44
OPs" (M)
1.56
1061.68
1194.39
24.88
25.48
Params, (K)
1.73
1179.65
294.91
1.73
Params; (K)
0,05
36.86
9.22
0,05
15
25.46
20
25.45
PSNR (dB)
28.58
28.38
28.59
27.76
25
25.44
Ve 9
204.14
3.29
0.13
535.83
25.62
Vo Lx10
11.91
0.19
0.04
500.00
26.0
26.00
25.6
25.50
PSNR
23.20
Kovwr betworc 12 channes Chamab
24.50
Ful Peviica Nerwock 128 charnik
10
25
1.0
500
Binery
Figure 5: The effect of number of binary convo- lution in residual connection on SRResNet.
Figure 6: The effect of amplification factor on SRResNet with various number of channels.
Residual Connection. We take SRResNet the backbone with 32 Convs in body part and explore the relationship between performance and the number of binary convolution residual connection. Specifically, for both full-precision and binarized SRResNet, we 1,2,4,8,14 8,16, and 32 Convs with residual connection as basic convolution units, respectively. We evaluate SRResNet equipped with these basic convolution units 4x Urban (see Fig. 5). (1) For full-precision networks, it is best to have Convs with a residual connection to form a basic convolution unit. Besides, if there are more than 4 Convs rized networks, it is best to equip each binary convolution with a residual connection. In addition, compared with the full-precision network, the binarized network is more insensitive the growth of the Convs number in the basic convolution unit. (2) For binarized SRResNet, we delete residual connection of a BBCU in certain position. In Tab. 5, seems that once the residual connection is removed (the full-precision information is lost) at any position, the binarized SRResNet would suffer severe performance drop.
basic convolution unit, its performance would sharply drop. For bina-
Amplification Factor. As shown in Fig. 6, the performance of full-precision remain basically un- changed with the variation of amplification factor k. However, the binarized network is sensitive to the k. The best 'n is related to the number of channels n, which empirically fits A = 130n /64. Intuitively, the larger number of channels makes binary convolution count elements and have larger results, which needs larger to increase the value of residual connection for balance.
The Binarization Benefit for Different Parts of IR Network. We binarize one part in SRResNet with BBCU in Fig. 3 while keeping other parts full-precision. The results are shown in Tab. 6. We use Ve (Eq. 10) and Vp (Eq. 11) as metric to value binarization benefit of different parts. We can see that Upsampling part most worth be binarized. However, is ignored by previous works (Xin et al., 2020). The binarization benefit of first and last convolution is relatively low.
5 CONCLUSION
This work devotes attention to exploring the performance of BNN low-level tasks and search of generic and efficient basic binary convolution unit. Through decomposing and analyzing existing elements, propose BBCU, simple yet effective basic binary convolution unit, that outperforms existing state-of-the-arte with high efficiency. Furthermore, we divide IR networks to four parts, and specially develop the variants of BBCU for them to explore the binarization benefits. Overall, BBCU provide insightful analyses on BNN and can serve as strong basic binary convolution unit for future binarized IR networks, which is meaningful academic research and industry.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was partly supported by the Alexander von Humboldt Foundation, the National Nat- ural Science Foundation of China(No. 62171251), the Natural Science Foundation of Guangdong the Special Foundations for the Development of Strategic Emerg- ing Industries JCtaeeennoossJJ222200000 19100002272 and CGGJ2D20021040002244441 1) and Oversea Cooperation Foundation of Tsinguua
REFERENCES
Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. CVPRW, 2017.
Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie line Alberi Morel. Low- complexity single-image super-resolution based on nonnegative neighbor embedding. In BMVC, 2012. 7
Lukas Cavigelli, Pascal Hager, and Luca Benini. Cas-cnn: A deep convolutional neural network for image compression artifact suppression. In IJCNN, 2017.
Yunjin Chen and Thomas Pock. Trainable nonlinear reaction diffusion: A flexible framework for fast and effective image restoration. TPAMI, 2016. 1
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arXiv: 02830, 2016. 3,
Tao Dai, Jianrui Cai, Yonghing Zhang, Shu-Tao Xia, and Lei Zhang. Second-order attention network for single image super-resolution. In CVPR, 2019.
Chao Dong, Yubin Deng, Chen Change Loy, and Xiaoou Tang. Compression artifacts reduction by a deep convolutional network. In ICCV, 2015a.
Chao Dong. Chen Change Loy. Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. TPAMI, 2015b.
Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Pointwise shape-adaptive det for high- quality denoising and deblocking of grayscale and color images. TIP, 2007.
Xueyang Fu, Zheng-Jun Zha, Feng Wu, Xinghao Ding, and John Paisley. Jpeg artifacts reduction via deep convolutional sparse coding. In ICCV, 2019.
Xueyang Fu, Menglu Wang, Xiangyong Cao, Xinghao Ding, and Zheng-Jun Zha. A model-driven deep unfolding method for jpeg artifacts removal. TNNLS, 2021. 3
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In
Proceedings of the fourteenth international conference on artificial intelligence and statisties.
JMLR Workshop and Conference Proceedings, 2011. 2
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. Im- proved training of wasserstein gans. arXiv preprint arXiv: 00028, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural net- works. In ICCV, 2017.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253: 2015. 2
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from trans- formed self-exemplars. In CVPR, 2015.
10

---

Published as a conference paper at ICLR 2023
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient mnneger-aithmmeti-onf inference. In CVPR, 2018. 1
Xixi Jia, Sanyang Liu, Xiangchu Feng, and Lei Zhang. Focnet: fractional optimal control network for image denoising. In CVPR, 2019. 3
Xinrui Jiang, Nannan Wang, Jingwei Xin, Keyu Li, Xi Yang, and Xinbo Gao. Training binary neural network without batch normalization for image super-resolution. In AAAI, 2021. 2, 3, 5,
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resolution using very deep convolutional networks. In CVPR, 2016. 3
Yoonsik Kim, Jae Woong Soh, Jaewoo Park, Byeongyong Ahn, Hyun-Seung Lee, Young-Su Moon, and Nam Cho. A pseudo-blind convolutional neural network for the reduction of compression artifacts. TCSVT, 2019.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic single image super-resolution using generative adversarial network. In CVPR, 2017. 6
Bee Lim, Sanghyun Son, Heewon Kim, Scungjun Nah, and Kyoung Mu Lee, Enhanced deep resid- ual networks for single image super-resolution. In CVPRW, 2017. 2, 4,
Zechun Liu, Baoyuan WU, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV, 2018. 1, 2, 3,
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In ECCV, 2020. 1, 2, 3, 4, 6,
Yinglan Ma, Hongyu Xiong, Zhe Hu, and Lizhuang Ma. Efficient super resolution using binarized neural network. In CVPRW, 2019. 2,
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In ICCV, 2001. 7
Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using mangal 109 dataset. Multimedia Tools and Applications, 2017.
Yali Peng, Lu Zhang, Shigang Liu, Xiaojun Wu, Yu Zhang, and Xili Wang. Dilated residual net- works with symmetric skip connection for image denoising. Neurocomputing. 2019.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In CVPR, 2020. 3,6 6
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016. 1, 4, 6
Chuanhui Shan, Xirong Guo, and Jun Ou. Residual learning of deep convolutional neural networks for image denoising. Journal of Intelligent Fuzzy Systems, 2019. 7
HR Sheikh. Live image quality assessment database release 2.
http://live ece. utexas.
edu/research/qualityy 2005. 8
Jae Woong Soh, Sunwoo Cho, and Nam Ik Cho. Meta-transfer learning for zero-shot super- resolution. In CVPR, 2020.
11

---

Published as a conference paper at ICLR 2023
Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. Memnet: A persistent memory network for image restoration. ICCV, 2017.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In ECCVW, pp. 0-0, 2018. 3
Yunxuan Wei, Shuhang Gu, Yawei Li, Radu Timofte, Longeun Jin, and Hengjie Song. Unsupervised real-world image super resolution via domain -istancc aware training. CVPR, 2021.
Bin Xia, Yucheng Hang, Yapeng Tian, Wenming Yang, Qingmin Liao, and Jie Zhou. Efficient non-local contrastive attention for image super-resolution. AAAI, 2022a.
Bin Xia, Jingwen He, Yulun Zhang, Yucheng Hang. Wenming Yang, and Luc Van Gool. Structured sparsity learning for efficient video super-resolution. arXiv preprint arXiv:2206. 07687, 2022b. 3
Bin Xia, Yapeng Tian, Yucheng Hang, Wenming Yang, Qingmin Liao, and Jie Zhou. Coarse-to-fie fine embedded patchmatch and multi-scale dynamic aggregation for reference-based super-resolution. In AAAI, 2022c. 3
Bin Xia, Yapeng Tian, Yulun Zhang, Yucheng Hang, Wenming Yang, and Qingmin Liao. Meta-learnig based degradation representation for blind super-resolution. arXiv preprint 27Xiv:2207.13963, 2022d. 3
Bin Xia, Yulun Zhang, Yitong Wang, Yapeng Tian, Wenming Yang, Radu Timofte, and Luc Van Gool. Knowledge distillation based degradation estimation for blind super-resolution. ICLR, 2023. 1
Jingwei Xin, Nannan Wang, Xinrui Jiang, Jie Li, Heng Huang, and Xinbo Gao. Binarized neural network for single image super resolution. In ECCV, 2020. 2, 3. 6.
Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse- representations. International conference on curves and surfaces, 2010.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng. and Lei Zhang. Beyond gaussian denoiser: Residual learning deep enn for image denoising. TIP. 2017. 5. 5. 7,8
Kai Zhang, Wangmeng Zuo, and Lei Zhang. Ffdnet: based image denoising. TIP, 201 Sa.
Toward a fast and flexible solution for cnn-
Kai Zhang, Yawei Li, Wangmeng Zuo, Lei Zhang, Lue Van Gool, and Radu Timofte. Plug-and-play image restoration with deep denoiser prior. TPAMI, 2021a.
Lei Zhang, Zhiqiang Lang, Wei Wei, and Yanning Zhang. Embarrassingly simple binarization for deep single imagery super-resolution networks. TIP, 2021b. 3.
Yulun Zhang, Kunpeng Li, Kai Li, Lichen Wang, Bineng Zhong, and Yun Fu. Image super- resolution using very deep residual channel attention networks. In ECCV, 201 18. 3
Yulun Zhang, Yapeng Tian, Yu Kong. Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In CVPR, 2018c.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu, Residual dense network for image restoration. TPAMI, 2020.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578, 2016. 1
12


Reviews:
[{"review_id": "gv8Pr5BIQ1D", "reviewer": "Reviewer_WV2K", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the components in binary convolution, such as residual connection, Batch Norm, activation function, and structure, for image restoration (IR) tasks. The aim is to conduct systematic analyses to explain each component’s role in binary convolution and discuss the pitfalls of such mechanisms.", "summary_of_the_review": "1) The section 1 can be rearranged and written better. For example, analysis are provided in numerate bullet points but the paragraphing for each analysis is not consistent. Please rectify that for better readability.\n2) It would be better to include some form of content table for better readability at the end of section 1 to describe the progression of the paper texts. Readability of the paper is not good and needs to be worked on.\n3) Please specify each variables in Xjf⊗Wjf in section 3. Additionally, please specify what each avertible in the section is and why are they important. Please justify the design choices.\n4) Section 4, though has a lot of experimental evaluation but lacks justification and reproducibility. Please explain why such evaluation are chosen and how can be reproduced successfully. Explanation is lacking.", "strength_and_weaknesses": "Strengths::\n1) The motivation case study in section 1 is good.\n2) The experimental section is good but can be improved based on the point mentioned in the weaknesses section.\n\nWeaknesses::\nThe paper can be improved based on the following points.\n1) The section 1 can be rearranged and written better. For example, analysis are provided in numerate bullet points but the paragraphing for each analysis is not consistent. Please rectify that for better readability.\n2) It would be better to include some form of content table for better readability at the end of section 1 to describe the progression of the paper texts. Readability of the paper is not good and needs to be worked on.\n3) Please specify each variables in Xjf⊗Wjf in section 3. Additionally, please specify what each avertible in the section is and why are they important. Please justify the design choices.\n4) Section 4, though has a lot of experimental evaluation but lacks justification and reproducibility. Please explain why such evaluation are chosen and how can be reproduced successfully. Explanation is lacking.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z0ti5kLrru", "reviewer": "Reviewer_SdPY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to improve upon binary neural network for image restoration (IR). It shows that batch normalization can help to obtain a good accuracy performance when binarizing IR models. Based on this observation, the authors introduce a basic binary convolution unit that benefits from batch norm properties using residual alignment. It has been shown that IR binary models outperform existing models across different tasks and datasets.\n", "summary_of_the_review": "In general, this paper introduces a simple and effective binary unit that further closes the gap between the binary model and its full-precision counterpart. The reasoning and development of the idea is based on empirical results, which its technical novelty is limited.", "strength_and_weaknesses": "Strengths:\n\n-- The basic binary convolution unit is very simple and effective. IR models equipped with such units outperform existing works.\n\n-- The binary IR models were tested on a wide range of tasks and datasets.\n\n-- The paper is well-motivated, easy to read and well-structured.\n\nWeaknesses:\n\n-- The weight of technical novelty of this work is much lower than that of its empirical novelty. The proposed unit is obtained based on empirical experiments rather than a concrete mathematical foundation.\n\n--  It is not very clear if the improvement comes from RA or RPReLU. What I am curious to see is an ablation study for BBCU-V4 without RA.\n\n-- There is a small degradation in performance when binarizing body and upsampling parts in image super-resolution. I am wondering why it was not explored by prior works. If it was, then please compare.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "RZ6QCRxorok", "reviewer": "Reviewer_zzdZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper evaluated some essential design choices of existing binary neural networks for image restoration tasks. The authors further propose a network design and evaluate it on three different tasks. Experimental results show that the proposed binary network outperforms all the reference methods.", "summary_of_the_review": "The paper is well structured, and the experimental result seems to be strong. But, I still have several concerns regarding novelty, a more complete comparison, and adding inference speed evaluation.", "strength_and_weaknesses": "**Strength**\n\n- The paper is easy to follow.\n- Experimental results seem to be strong. The proposed method outperforms all reference methods across all three tasks.\n\n**Weaknesses**\n\n- Limited novelty: The presented findings regarding residual connection, BN and the position of RPrelu are not supervising. Similar observations have already been reported and discussed in Bireal, Real2binaryNet and ReActNet papers. In terms of network design, there are only minor adaptions on top of ReActNet block design making the proposed BBCU block incremental. The current design lacks significant innovation.\n\n- Current design choices are more based on empirical attempts and lack of refinement of optimization methods specified on IR tasks. For example, BNN-specific training methods such as two-stage training (real2binaryNet), progressive sign function ([1]), knowledge distillation training, etc., have not been verified in this work. However, these methods show significant improvement in the image classification task.\n\n- If we want to demonstrate the practical effectiveness of BNN-based IR methods, it will be better to also compare with more compression methods, such as pruning-based methods and compact model designs. Also, is it possible to use a binary inference engine like larq or Dabnn to verify the real inference speed on Arm cpu? We know that the test results on hardware can sometimes be quite different from the theoretical computational complexity OPs. Arms cpu is also an important computing platform for IR tasks.\n\n[1] Gradient matters: Designing binarized neural networks via enhanced information-flow", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "3S4hrGGIym", "reviewer": "Reviewer_2H4s", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Since the different properties in CNN and BNN, it hardly applies the experience of designing CNN to develop BNN. In this paper, the author rethinks and analyzes the actual function of components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for IR tasks. Based on their observations and analysis, they designed a general basic binary convolution unit for binarized image restoration networks. Extensive results can support their claimed contributions.", "summary_of_the_review": "The paper is well written. The paper deeply analyzes the components of BNN and designs a BBCU. The ablation studies, like the visualization of feature distribution, demonstrate the effect of the method. The main comparison results with others are impressive and general. Overall, this paper is interesting, insightful, and practical.", "strength_and_weaknesses": "Strengths: The paper is easy to follow and provides many insightful analyses, and the results are impressive and show promising potential for BNNs.\n1. IR model compression is an important and urgent research direction, and BNN is a promising compression method. This paper tackles this problem and thus potentially can have a big impact.\n2. This paper rethinks and analyzes the component of BNN. It is interesting and practical. The visualizations, experiments, and analyses are sound and interesting. \n3. The experiment results are impressive and general. The proposed model can significantly improve performance on different low-level vision tasks compared with SOTA methods. \n4. The ablation studies are insightful. It shows the binarization benefit of different parts and the performance of CNN and BNN to the residual connection, etc. \n5. The paper is well written and clearly organized.\n\nWeakness： I have the following questions about the method and result details.\n1. About the method and experiments, do you use the BBCU-V4 in Fig.2 to arm the IR networks? It is better to clarify this point.\n2. In Tab.6, are the OPs computed based on 320×180 LQ images?  Besides, I hope the author provides OPs in Tab.4.  \n3. Typos: In Fig.3 (b), the “BBCM-U“ seems not correct.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster