Binarized Spectral Compressive Imaging
Authors: Yuanhao Cai
Yuxin Zheng
Jing Lin
Xin Yuan
Yulun Zhang
Haoqian Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Applications
Computer Vision
Low-level Vision
Image Restoration
Snapshot Compressive Imaging
Hyperspectral Image Reconstruction
TL;DR: This work is the first attempt to study the binarized spectral compressive imaging reconstruction problem

Abstract: Existing deep learning models for hyperspectral image (HSI) reconstruction achieve good performance but require powerful hardwares with enormous memory and computational resources. Consequently, these methods can hardly be deployed on resource-limited mobile devices. In this paper, we propose a novel method, Binarized Spectral-Redistribution Network (BiSRNet), for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems. Firstly, we redesign a compact and easy-to-deploy base model to be binarized. Then we present the basic unit, Binarized Spectral-Redistribution Convolution (BiSR-Conv). BiSR-Conv can adaptively redistribute the HSI representations before binarizing activation and uses a scalable hyperbolic tangent function to closer approximate the Sign function in backpropagation. Based on our BiSR-Conv, we customize four binarized convolutional modules to address the dimension mismatch and propagate full-precision information throughout the whole network. Finally, our BiSRNet is derived by using the proposed techniques to binarize the base model. Comprehensive quantitative and qualitative experiments manifest that our proposed BiSRNet outperforms state-of-the-art binarization algorithms. Code and models are publicly available at https://github.com/caiyuanhao1998/BiSCI

Binarized Spectral Compressive Imaging
Yuanhao Cai 1 Yuxin Zheng ', ,Jing Lin Xin Yuan 2, Yulun Zhang 3.. Haoqian Wang 1,-
Tsinghua University, Weetlake University, ETH Zürich
Abstract
Existing deep learning models for hyperspectral image (HSI) reconstruction achieve good performance but require powerful hardwares with enormous memory and computational resources. Consequently, these methods can hardly be deployed resource-limited mobile devices. In this paper, we propose novel method, Binarized Opectral-Redistribuue
Network (BiSRNet), for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems, Firstly, we redesign a compact and easy-to-deploy base model to be binarized. Then we present the basic unit, Binarized Spectral- Redistribution Convo- lution (BiSR-Cove Conv). BiSR-Conv can adaptively redistribute the HSI representations before binarizing activation and uses a scalable hyperbolic tangent function closer approximate the Sign function in backpropagation. Based on our BiSR-Conv, we customize four binarized convolutional modules to address the dimension mismatch and propagate full-precision information throughout the whole network. Finally, our BiSRNet is derived by using the proposed techniques to binarize the base model. Comprehensive quantitative and qualitative experiments manifest that proposed BiSRNet outperforms state-of-the-ar binarization algorithms. Code and models are publicly available https: : //gi //ggttu..... com/ caiyuanhao
1 Introduction
Compared to normal RGB images, hyperspectral images (HSIs) have more spectral bands to capture richer information of the desired scenes. Thus, HSIs have wide applications in agriculture [1, 2, 3], medical image analysis [4, 5, 6], object tracking [7, 9], remote sensing [10, 11, 12], etc.
To capture HSIs, conventional imaging systems leverage ID or 2D spectrometers to scan the desired scenes along the spatial or spectral dimension. Yet, this process is very time-consuming and thus fails in measuring dynamic scenes. In recent years, snapshot compressive imaging (SCT) systems [13, 14, 15, 16, 17] have been developed to capture HSI cubes in real time. Among these SCI systems, the coded aperture snapshot spectral imaging (CASSI) [14, 18, 19] demonstrates its outstanding effectiveness and efficiency. The CASSI systems firstly employ coded aperture (physical mask) to modulate the 3D HSI cube, then use a disperser to shift spectral information of different wavelengths, and finally integrate these HSI signals We study the inverse problem, i... restoring the original 3D HSI cube from the 2D measurement.
detector array to capture 2D compressed measurement.
Existing ofate-of sate-oofthhe-rr (SOTA) SCI reconstruction methods are based on deep learning. Convolu- tional neural network (CNN) [18, 20, 21. 22, 23, 24, 25] and Transformer [26, 27, 28, 29] have been used to implicitly learn the mapping from compressed measurements to HSIs. Although superior performance achieved, these LNNN-TTransforme-baasee methods require powerful hardwares with abundant computing and memory resources, such as high-end graphics processing units (GPUs). However, edge devices (e.g., mobile phones, hand-held cameras, small drones, etc.) evidently cannot meet the requirements of these expensive algorithms because edge devices have very limited memory, computational power, and battery. As mobile devices are more and more widely used, the demands of running and storing HSI restoration models edge devices grow significantly. This motivates us to reduce the memory and computational burden of HSI reconstruction methods while preserving the performance much as possible so that the algorithms can be deployed resource- limited devices.
Yulun Zhang and Haoqian Wang are the corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

30-
28
7.57 dB
5.88
3.50 dB
3.46 dB
3.39 dB
3.25 dB
2.55 dB
24-
22
BiConnect (NeurPS 15)
Bi-Real (ECCV
IRNet (CVPR920)
ReActNet (ECCV
BBCU (CLR
8TM (AAA121)
BiSRNet (Ours)
(NeudPS 16)
Figure I: Comparison between our BiSRNet (in red color) and state-of-the-art BNNs (in blue color). BiSRNet significantly advances BTM [30], BBCU [31], ReActNet [32], IRNet [33]. Bi-Real [34], BNN [35], and BiConnect [36] 2.55, 3.25, 3.39, 3.46, 3.50, 5.88, and 7.57 dB on the simulation HSI reconstruction task.
The studies on neural network compression and acceleration [37, 38] can be divided into four categories: quantization [31, 32, 33, 34, 35, 391, pruning [40, 41, 42], knowledge distillation [43, 44], and compact network design [45, 46, 47, 48]. Among these methods, binarized neural network (BNN) belonging to quantization stands out because it extremely compress the memory and computational costs by quantizing the weights and activations of CNN to only 1 bit. In particular, BNN could achieve 32x memory compression ratio and up 58x practical computational reduction on central processing units (CPUs) [49]. In addition, the pure logical computation (i.e., XNOR and bit-count operations) of BNN is highly energy- eefeggeefficient for embedded devices [50, 51]. However, directly applying model binarization for HSI reconstruction algorithms may encounter three issues. (i) The HSI representations have different density and distribution different spectral bands. Equally binarizing the activations of different spectral channels may lead to the collapse of HSI features. (ii) Previous model binarization methods mainly adopt a piecewise linear [33, 35, 36] or quadratic [31, 32, 34] function to approximate the non -on-differentiable Sign function. Nonetheless, there still remain large approximation errors between them and Sign. (iii) How to tackle the dimension mismatch problem during feature reshaping while allowing fuee-prccision information propagation in BNN has not been fully explored, Previous model binarization methods [32, 33, 34, 35, 36) mainly consider the feature downsampling situation in backbone network for high-level vision tasks.
Bearing the above considerations in mind, we propose novel BNN-based method, namely Binarized Opectral-Redistrbuttie Network (BiSRNet) for efficient and practical HSI reconstruction. Firstly, we redesign a compact and easy-to-deploy base model to be binarized. Different from previous CNN- Trranfformer-basee methods, this base model does not include complex computations like unfolding inference and non-local self-attention that are difficult to implement on edge devices. Instead, our base model only uses convolutional units that can be easily replaced by XNOR and bit-count logical operations on resource- limited devices. Secondly, we develop the basic unit, Binarized Spectral- Redistribution Convolution (BiSR-CC -oSs), used in model binarization. Specifically, BiSR-Conv can adapt the density and distribution of HSI representations in spectral dimension before binarizing the activation. Besides, BiSR-Conv employs scalable hyperbolic tangent function to closer approximate the non-differentiable Sign function by arbitrarily reducing the approximation error. Thirdly, as the full-precision information is very critical in BNN and the input HSI is the only full-precision source, we use BiSR-Conv to build up four binarized convolutional modules that can handle the dimension mismatch issue during feature reshaping and simultancously propagate full full-precision information through all layers. Finally, we derive our BiSRNet by using the proposed techniques to binarize the base model. As shown in Fig. 1, BiSRNet outperforms SOTA BNNs by large margins, over 2.5 dB.
In nutshell, our contributions can summarized as follows:
(i) We propose novel BNN-based algorithm BiSRNet for HSI reconstruction. To the best of our knowledge, this the first work study the binarized spectral compressive imaging problem.
(ii) We customize a new binarized convolution unit BiSR-Com that can adapt the density and distribution of HSI representations and approximate the Sign function better in backpropagation.
(iii) We design four binarized convolutional modules to address the dimension mismatch issue during feature reshaping and propagate full- null-precision information through all convolutional layers.
(iv) Our BiSRNet dramatically surpasses SOTA BNNs and even achieves comparable performance with full-precision CNNs while requiring extremely lower memory and computational costs.

---

2 Related Work
2.1 Hyperspectral Image Reconstruction
Traditional HSI reconstruction methods [14, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61] are mainly based on hand-crafted image priors. Yet, these traditional methods achieve unsatisfactory performance and generality due to their poor representing capacity. Recently, deep CNN [18, 20, 21, 22, 23, 24, 25, 62] and Transformer [26, 27, 28, 29] have been employed as powerful models to learn the underlying mapping from compressed measurements to HSI data cubes. For example. TSA-Net [18] employs three
self-attention layers at the decoder of a U-shaped CNN. Cai et al. propose a series of Transformer-basee -based algorithms (MST [26]. MST++ [27], CST [28], and DAUHST [29]), pushing the performance boundary from 32 dB to 38 dB. Although impressive results are achieved, these LNN-/Traasformer--e based methods rely on powerful hardwares with enormous computational and memory resources, which are unaffordable for mobile devices. How to develop HSI restoration algorithms toward resource-limited platforms under-explored. Our goal is to fill this research gap.
2.2 Binarized Neural Network
BNN [35] is the extreme case of model quantization as it quantizes the weights and activations into only I bit. Due to its impressive effectiveness in memory and computation compression, BNN has been widely applied in high-level vision [32, 33, 34, 35, 49] and low-level vision [30, 31, 39J. For example, Jiang et al. [30] train BNN without batch normalization for image super-resolution. Xia er al. [31] design binarized convolution unit BBCU for image super-resolution, denoising, and JPEG compression artifact reduction. Yet, the potential of BNN for SCI reconstruction has not been studied.
3 Method
3.1
Base Model
The full-pre full-precision model to be binarized should be compact and its computation should be easy to deploy on edge devices. However, previous TONN-TTranfoomerbbaae algorithms are computationally expensive or have large model sizes. Some of them exploit complex operations like unfolding inference [21, 22, 23, 29, 63] and non-local self-attention computation [18, 20, 26, 27, 28) that are challenging to binarize and difficult to implement on mobile devices. Hence, we redesign a simple, compact, and easy-to-deploy base model without using complex computation operations.
Inspired by the success of MST [26] and CST [28], we adopt a U-shaped structure for the base model as shown in Fig. It consists of an encoder E, bottleneck B, and decoder D. Please refer the supplementary for the CASSI mathematical model. Firstly, we reverse the dispersion of CASSI by shifting back the measurement Y E H x
to derive the input H E R H NN as
H(z, y. mx) = Y Y(E,, y d(An -x)),,
(I)
where H1, W, and N^ denote the HSI's height, width, and number of wavelengths. d represents the shifting step. The concatenation of H and the 3D mask M E R XxXxN is fed into feature embedding module to produce the shallow feature X, E H The feature embedding module is a conel x I (convolutional layer with kernel size 1x Subsequently, X. undergoes the encoder E, bottleneck B, and decoder D to generate the deep feature X, E R x'''''',,, E consists of two convolutional blocks and two downsample modules. The details of the convolutional block are depicted Fig. (b). The fusion up and down modules are both conel maps and modify the channels. The downsample module is strided com4 x layer that downscales the feature maps and doubles the channels. is a convolutional block. D consists of two convolutional blocks and two upsample modules. The upsample module is a bilinear interpolation followed by conv3 x3 to upscale the feature maps and halve the channels. Skip connections between È and D are employed alleviate the information loss during rescaling. Finally, the sum of X, and X is fed into the feature mapping module (comvl 1 X 1) to produce the reconstructed HSI H' E RBNWWX,
to aggregate the feature
3.2 Binarized Opectral-Redistriing
Convolution
The details of BiSR-Conv are illustrated in Fig. 2 (c). We define the input full-precision activation as X E R XXXC We notice that HSI signals have different density and distribution along the spectral dimension due the constraints of specific wavelengths. To adaptively fit this HSI nature, we propose to redistribute the HSI representations in channel wise before binarizing the activation as
X, k X + b
(2)
3

---

anan
+
add
concatenation
backpropaggation
Embedding
Mopping
K.
Sigox,)
Comn Layyrr
Encoder
Decoder
Tamnjax
Full precision Information
Corvv Block
Fusion Down
Conv Block
Layer Norm
(c) Binarized Specctral-Rediistbbbul Redistrbulion Convolution
Heals
DownSample
UpSample
BiSR-Conv
Li
Average Pooling
BiSR-Corv
Layer
Conw Block
C
Fusion Down
Conv Block
(d) Binarized DownSample Binarized Fusion AvgPool
DownSample
UpSemple
Fusion Down
Bottlenack
BISR-ConV
Charnel Spiit
Upscale
Conv Block
BISR-Corw
(b) Conv Block
(8) Base Model
Binarized UpSample Binarized Fusion Down Bilinear
Figure 2: The overall diagram of our method. (a) The proposed base model to be binarized adopts a U-shaped architecture. (b) The components of the convolutional block. (c) The details of our Binarized Spectral- Redistribution Convolution (BiSR-Conv). (d) The structure of our binarized downsample module. The binarized fusion up module derived by removing the average pooling operation. (e) The architecture of our binarized upsample module, which includes one more hilinear upscaling operation than the binarized fusion down module.
where X, E #xxxC denotes the redistributed activation of X, f- k and b E RC are learnable parameters. le rescales the density of HSIs while b shifts the bias. Then X, undergoes a Sign function to be binarized into 1-bit activation X,,
xC, where Ta +1 or -1 for V E X,, as
+ 1, 24 > 0
Th Sign(z,)) = 1 11 24 0
where x, E X,.. As shown in Fig. 3 (b) and (c), since the Sign function non-differentiable, previous methods either adopt piecewise linear function Clip(z) [33, 35, 36, 49, 64] a piecewise quadratic function Quad(r) [31, 32, 34] to approximate the Sign function during the backpropagation as
+ 1,
x 2 1
+ 1, x;
x 2 <1<
x² 2x
0 0< x <1
Clip(z)
Quad (x)
(4)
2x x2
-1<050
1, x -1
1,
x 5
Nonetheless, the Clip function is rough estimation and there is a large approximation error between Clip and Sign. The shaded areas in Fig. 3 reflect the differences between the Sign function and its approximations. The shaded area corresponding to the Clip function is 1. Besides, once the absolute values of weights or activations are outside the range of [-1, 1]. they are no longer updated. Although the piecewise quadratic function is a closer approximation (the shaded area is 2/3) than Clip, the above two problems have not been fundamentally resolved. To address the two issues, we redesign a scalable hyperbolic tangent function to approximate the Sign function in the backpropagation as
00h 0Z2,
%6 Tanh(ax,)) Ca + a
where a R+ is a learnable parameter adaptively adjusting the distance between Tanh( and Sign(a). e denotes the natural constant. We prove that when - 400, Tanh(ox) - Sign(r) as
6oz 0
lim a+++
+1,
x >> 0
+ 0
lim Tanh
lim
x = 0 0,
(6)
a 0000
a+++00
6 + el
0 e -0
lim
11 <<< 0
000+000
0 + oz

---

Sgr00)
Clpp))
Quaduo
Tanh
(a)
(b)
(d)
(Sign(e)
aQuelte)
(uab((or)
8x
dx
8x
3k
Figure 3: The upper line shows (a) Sign(x) and its three approximation functions including (b) piecewise lincar function Clip(:)), (c) piccewise quadratic function Quad(æ), and (d) our scalable hyperbolic tangent function Tanh(ax). The area of the shaded region reflects the approximation error. The lower line depicts the derivatives.
If strictly following the mathematical definition, Sign(0) = 0 + #1. However, in BNN, the weights and activations are binarized into 1-bit, i.e., only two values (+1)). Hence, Sign(O) is usually set to 11. Similar to this common setting, we also define lim Tanh(a 0) = -1 BNN. Then we have a-+oc
lim Tanh(ar) = Sign(x).
We compute the area the shaded region between our Tanh(ax) and Sign(æ) in Fig. 3 (d) as T--ann((x)) dr
+oo
(i) Tanh(aa)) dr =2 2 // +00
= 2(x x + log (og(Tan((axr) + 1))
x==
#=+00
(8)
(log(2) - log ()) 2 log(2)
Different from previous Clip(z) and Quad(x), our Tanh(ax) can arbitrarily reduce the approximation eroor with Sign(a) when a in Eq. (8) is large enough. Besides, our Tanh( neither piecewise nor unchanged when x is outside the range of [-.. On the contrary, the weights and activations can still updated when their absolute values are larger than In addition, as depicted in the lower line of Fig. 3, the value ranges [0, 1] and fixed shapes of @Cii((a) and (Quad(()) are fundamentally different from those of @Sign(a)) 8x E [0, +oc). In contrast, our (Tnnh(ax) can change its value range (0, a) and 8e
dx
shape by adapting the parameter It is more flexible and can approximate @Sign(æ)
Ba better.
In the binarized convolutional layer, the 32-bit weight W, is also binarized into 1-bit weight Wa as wh Ewyew, (Iwy)) Sign(wy),
where E represents computing the mean value. Multiplying the mean absolute value of 32-bit
weight value wy E W f can narrow down the difference between binarized and full-precision weights. Subsequently, the computationally heavy operations of floating point matrix multiplication in full-precision convolution can be replaced by pure logical XNOR and bit-count operations [49] as
Y, = X, W, = bit-coun(XNNO((X.. (X,, W)),,
(10)
where Y, represents the output and * denotes the convolution operation. Since the value range of full-precision activation X, largely varies from that of 1-bit convolution output Y,, directly employing an identity mapping to aggregate them may cover up the information of Y, To cope with this problem, we first fed 6 into RPReI.U [32] activation function to change its value range and then add it with X by residual connection to propagate full-precision information as
X. X + RPReLU(Y
(11)
where X,, denotes the output feature and RPReI.U is formulated for the i-th channel of Y,
y/ % + Se.
3/4 %
RPReLU((::
(12)
{ B, (y1 - 76) + Cin
y S 7
5

---

Sign
AvgPool = 2.
Bilncar
Bilinear
Sign
Split
Binertzed Conv
BISR-Conv
BiSR-Corv 3.
k=3,5
Binarized Conv k= 3
BisR-Conv k- 3.
BiSR-Conv 3,
RPReLU
RPReLU
Normal
Ours
Normal
(a) Binarized DownSample
(b) Binarized UpSample
Sign
Channel Split
Sign
Binarbred Comv
BisR-Canv 1,
BISR-Conv k 1,5=1
Binarized Conv k 1,
BISR-Conv
k 1-8--
k 1,
1.
RPReLU
RPReLU
Normal
Ours
Normal
Ours
(c) Binarized Fusion Down
(d) Binarized Fusion Up
Figure 4: Comparison between normal and our binarized convolutional modules, including (a) downsample. (b) upsample, (c) fusion down to half the channels, and (d) fusion up to double the channels. The red arrow + indicates the full- dull-precision information flow, while the blue arrow | denotes the binarized signal flow.
where E R indicates single pixel values belonging to the i-th channel of Y,. 8i, Yr, and Ça E I represents learnable parameters. Please note that the full-precision information is not blocked by the binarized convolutional layer in the proposed BiSR- -onv. Instead, it is propagated by the bypass identity mapping, as shown in the red arrow - in Fig. 2 (c). Based on this important property of BiSR-Conv, we design the four binarized convolutional modules, as illustrated in Fig. 2 (d) and (c).
3.3 Binarized Convolutional Modules
In model binarization, the identity mapping is critical to propagate full-precision information and ease the training procedure. The only source of full full-precision information is the input end (X, in Fig. 2) of the binarized part. However, the dimension mismatch during the feature downsampling, upsampling, and aggregation processes blocks the residual connections, which degrades the HSI reconstruction performance. To tackle this problem, we use BiSR-Cor Conn to build up four binarized convolutional modules including downsample, upsample, fusion up, and fusion down with unblocked identity mappings to make sure the full-precision information can flow through all binarized layers.
Specifically, Fig. 4 compares the normal and our binarized modules. The red arrow J indicates the full- cill--recision information flow, while the blue arrow denotes the binarized signal flow. The downsample modules in Fig. 4 (a) downscale the input feature maps and double the channels. The upsample modules in Fig. (b) upscale the input spatial dimension and half the channels. The fusion down modules in Fig. 4 (c) maintain the spatial size of the input feature and half the channels. The fusion up modules in Fig. 4 (d) keep the spatial dimension the input feature maps while doubling the channels. In the normal modules, the full-precision information is blocked by the Sign function and binarized into I-bit signal. Meanwhile, the intermediate feature maps are directly reshaped by the binarized convolutional layers. For example, in the normal binarized downsample module, the intermediate feature is directly reshaped from R xxxW W xC to IR # x 4 x2C by a strided 1-bit conv4 44 The spatial and channel dimension mismatch of the input and output feature maps impede the identity mapping to propagate full- preccirceiioon information from previous layers. In contrast, our binarized modules rely on the proposed BiSR-Cor that has bypass identity connection for full-precision information flow, as shown in Fig. (c). By using channel-wis concatenating and splitting operations, the intermediate feature maps the input and output ends of BiSR-Com are free from being reshaped.
Therefore, the full-preed precision information can flow through our binarized modules. Finally, we derive our BiSRNet by using BiSR-Cony and the four modules to binarize E, B, and D of the base model.

---

Algurihiss
Calegory
0P%
51
57
.68
59
TwIST (59)
Minkl
25,16
0.700
0.503
25.02
0.644
0.643
22.20
0,650
21.82
0,690
22.42
0.669
23.12
GAP TV (96)
Model
26.81
0.754
0.610
12.89
0.802
26.51
0.852
30.65
0 201
23,64
0.663
21.85
0.688
23.96
21 91.98
0.655
0.682
12.63
23,10
0.564
0.969
24.36
DJSCI (53)
64
Mudel
0.745
27.13
0.620
23.04
26,62
0.118
34.96
0.897
0.706
23,94
0,685
22.38
0.743
24.45
0.673
22.03
24.56
0.732
0.567
23,99
0.721
25.27
A Net (20)
32
CNN
62640
117.98
0.840
30,10
0.805
28.49
0.870
27.23
0.914
57.01
0.817
26.19 19
0.855
28.66
0.806
26.47
0.831
26.09
27.50
0.826
0816
27.13
0.641
28.55
TSA-Ner (18)
32
CNN
44230
110.06
0.592
32.03
0.858
31.00
0.955
32.25
0.953
39.19
29.35
0.884
0.508
31.44
0.878
30.32
0.385
29.35
0.890
30.01
0.874
29.59
0.893
31.46
BiCoznect [36]
BNN
38
1.18
0.676
25.85
22.07
0.530
0.558
18.93
0.836
25.18
0.588
21.21
21.82
0.547
0,570
21.34
22.25
0.580
0.556
19.57
0.524
23.18
22.19
0.575
(NS)
BNN
26.09
0.661
0.551
0.679
22.96
0.584
24.12
0.599
0.568
23.30
23.29
0.590
21.65
0.588
0.547
0.593
Real [54)
0.701
0.840
31.04
25.33
0.644
0.611
25.47
0.678
24.59
0.644
BNN
35
27.91
0.700
0.620
35.84
0.661
0.663
25.13
0.685
0.665
25.20
0.662
25.14
25.07
0.608
0.600
25.20
0.665
ReActNet [12)
BNN
36
1.18
27.91
0.707
0.633
26.17
0.682
25.40
0.725
31.58
0.675
25.43
0.670
26.43
0.703
25.85
0,650
25.50
0.677
15.47
0.583
25.11
0.671
26.48
BBCU [31]
BNN 36
1.18
27.91
0.705
0.628
26.21
0.654
25.04
0.741
31.35
0.677
25.30
0,704
26.68
0.668
25.42
25.59
0,671
0.670
25,09
0.615
25.59
0.675
26.51
BTM (30]
INN 36
1.18
0.700
25
0.674
76.01
26.14
0.708
0.794
32.74
0.692
25.80
0.789
27.97
0.707
26.36
26.20
0.718
26. 10
0.717
0.671
25.74
0.716
27.11
BiSMNet (Ours)
BNN
36
1.18
0.347
38,95
0.791
29.21
29.11
0.825
35.91
0,903
0.827
25.19
0,863
30.22
27.85
0.800
28.83
0.843
29.46
0.832
0.800
27,85
0.837
29.76
Table 1: Quantitative results of BiSRNet, seven 1-bit BNN-based methods, two 32-bit CNN-based algorithms, and three 64-bit model-based methods on 10 scenes (SI~S10) of the KAIST [65] dataset. Params, OPs, PSNR (upper entry in each cell), and SSIM (lower entry in each cell) are reported.
4 Experiment
4.1 Experimental Settings
Following [18, 26, 28, 29, 63], we select 28 wavelengths from 450nm to 650nm by using spectral interpolation manipulation to derive HSIs. We conduct experiments on simulation and real datasets.
Simulation Data. Two simulation datasets, CAVE [66] and KAIST [65], are adopted. The CAVE dataset provides 32 HSIs with a spatial size of 512x512. The KAIST dataset includes 30 HSIs at a spatial size of 2704 x 3376. We use CAVE for training and select 10 scenes from KAIST for testing.
Real Data. We adopt the five real HSIs captured by the CASSI system developed in [18] for testing.
Evaluation Metrics. The peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) are adopted as metries to evaluate the HSI reconstruction performance. Similar to [31, 32, 33, 34, 35], the operations per second of BNN (OPs*) is computed as OPs" OPs / 64 (OPs = FLOPs) to measure the computational complexity and the parameters of BNN calculated as Params Params 322, where the superscript b and of refer to the binarized and full- precision models. The total computational and memory costs of a model are computed as OPs 0P5 + OPs/ and Params = Params Params/.
Implementation Details. The proposed BiSRNet is implemented by PyTorch [67]. We use Adam [68] optimizer (81 0.9 and 3g- 0.999) and Cosine Annealing [69] scheduler to train BiSRNet for 300 epochs on a single RTX 2080 GPU. Training samples are patches with spatial sizes of 256 x256 and 96x96 randomly cropped from 28-channel 3D HSI data cubes for simulation and real experiments. The shifting step d is 2. The batch size is 2. We set the basic channel C = NN = 28 to store HSI information. We use random flipping and rotation for data augmentation. The training loss function is the root mean square error (RMSE) between reconstructed and ground-truth HSIs
4.2 Quantitative Results
We compare our BiSRNet with 12 SOTA algorithms, including seven 1-bit BNN-based methods
(BiConnect [36]. BNN [35], Ri-Real [34], IRNet [33]. ReActNet [32], BTM [30], and BBCU [31]), two 32- full-precision CNN- methods (A-Net [20] and TSA-Net [18]), and three b4 double-precision model-based methods (TwIST [59], GAP-TV [56], and DeSCI [53]). For fair comparison, we set the Params and OPs of based methods to the same values.
Directly applying the seven SOTA BNN-based methods to HSI reconstruction task achieves unsatis- factory performance, ranging from 22.19 dB to 27.21 dB. Our BiSRNet surpasses these methods by large margins. More specifically, BiSRNet significantly outperforms BTM, BBCU, ReActNet, IRNet,
7

---

Image
Spectral Density Curves
BiConnect
BNN
Bi-Real
IRNet
ReActNet
BBCU
BTM
BiSRNet
Ground Truth
Figure 5: Reconstructed simulation HSIs of Scene with 4 out 28 spectral channels. Seven SOTA BNN-based algorithms and our proposed BiSRNet are compared, The spectral density curves (bottom-left) are corresponding the selected region of the green box in the RGB image (Top-left). Please zoom in for a better view.
Bi-Real, BNN, and BiConnect by 2.55, 3.25, 3.39, 3,46, 3.50, 5.88, and 7.57 dB. This evidence suggests the significant effectiveness advantage of our BiSRNet HSI restoration.
The proposed BiSRNet with extremely lower memory and computational complexity yields compara- ble results with 32-bit full-precision CNN-based methods. Surprisingly, our BiSRNet outperforms A-Net by 1.23 dB while only costing 0.06 % (36/62640) Params and 1.0% (1.18/117. 117119 OPs. In con- trast, the previous best BNN-based method BTM is still 1.33 dB lower than X-Net. When compared with TSA-Net, our BiSRNet only uses 0.08% Params and 1.1% OPs but achieves 94.65 44.6% (29.76/3) 76/31.46) performance. These results demonstrate the efficiency superiority of the proposed method.
The three model-based algorithms are implemented by MATLAB, where the default type of variable is double-preed precision floating poatingg-pintt number. Although they use more accurate 64-bit data type, 1-bit BiSRNet dramatically outperforms DeSCI, GAP-TV, and TwIST by 4.06, 5.40, and 6.64 dB.
4.3 Qualitative Results
Simulation HSI Restoration. Fig. 5 depicts the simulation HSIs on Scene with 4 out of 28 spectral channels reconstructed by the SOTA BNN-based algorithms and BiSRNet. Previous BNNs are less favorable to restore HSI details. They generate blurry HSIs while introducing undesirable artifacts. In contrast, BiSRNet reconstructs more visually pleasing HSIs with more structural contents and sharper edges. Additionally, we plot the spectral density curves (bottom- corresponding to the selected regions of the green box in the RGB image (Top-left). BiSRNet achieves the highest correlation score with the ground truth, suggesting the advantage of Net in spectral-wise consistency restoration.
Real HSI Restoration. Fig. 6 visualizes the reconstructed HSIs of the seven SOTA BNN-based algorithms and our BiSRNet. We follow the setting of [18, 26, 28, 29, 63] to re-train the models with all samples of the CAVE and KAIST datasets. To simulate the noise disturbance real imaging scenes, we inject 1-bit shot noise into measurements during training. It can be observed that our BiSRNet is more effective detailed content reconstruction and real imaging noise suppression.
4.4 Ablation Study
Break-down Ablation. We adopt baseline-1 to conduct break-down ablation towards higher performance. Baseline-1 is derived by using vanilla 1-bit convolution to replace BiSR-Com and normal binarized convolutional modules (see Fig. 4) to replace our binarized convolutional modules. As shown in Tab. 2a, baseline-1 yiclds 23.90 dB in PSNR and 0.594 in SSIM. When apply BiSR-Cony the model achieves 3.90 dB improvement. Then we successively use our binarized downsample (BiDS), upsample (BiUS), fusion down (BiFD), and fusion up (BiFU) modules, the model gains by 1.96 dB in total. These results verify the effectiveness of the proposed techniques.
To study the effects of BiSR-Conv components, we adopt baseline-2 to conduct an ablation. Baseline-2 obtained by removing Spectral-Redd Redistribution (SR) operation and Sign approximation Tanh(ox)) from BiSRNet. As reported in Tab. 2b, baseline-2 achieves 27.68 dB in PSNR and 0.723 in SSIM. When we respectively apply SR and Tanh(oa)), baseline-2 gains by 1.29 and 1.06 dB. When we exploit SR and Tanh(oz) jointly, the model achieves 2.08 dB improvement.
BiSR-Conv.
8

---

5365 nm
Ser.5 pm
594.5 mm
648.0 nm
8GB Image
Measurement
BiConnect
Bi-Real
IRNet
RoActNet
BBCU
BTM
BiSRNet
Figure 6: Reconstructed real HSIs of seven SOTA ANs-based algorithms and our BiSRNet on four scenes with 4 out of 28 wavelengths. BiSRNet is more effective in reconstructing detailed contents and suppressing noise.
Method
Baseline-1
+BiSR-Conv
+BIDS
+BiUS
+BIFD
+BIFU
Baseline-2
Tamh(ax))
PSNR
SSIM
PSNR
23.90
27,80
27.97
28.07
28.31
29.76
68
0.723
SSIM
0.737
0.729
0.758
0.776
0.837
0.783
OPs (M)
1176
1176
1176
1176
1176
1176
74
0.782
Params (K)
35.48
35.49
35.51
35.72
35.81
29.76
0.837
(a)
freak-dowa ablation study towards higher performance
(b) Ablation study of BiSR-Com
Method
PSNR
SSIM
Rinarized Part
OPs/ (M)
OPs* (M)
Params/
Params
PSNR
SSIM
Clip(r)
28.97
0.783
Encoder E
3390
177878
5559
32.28
0,905
Quad(ar)
29.02
0.794
Bottleneck
1096
278889
8715
33,80
0.932
Tanh(azz)
29.76
0.837
Decoder D
5005
78
186562
5830
33.03
0.919
(c) Study approximation
(d) Ablation study of binarizing different parts the base model
Table 2: Ablations on the simulation datasets. In table (a), BiUS, BiDS, BiFU, and BiFD denote the binarized upsample, downsample, fusion up. and fusion down modules of Fig. 4. In table (b), SR refers to the Spectral- Redistribution of Eq. (11). In table (d), the full- cull-precision model yields 34. 11 dB in PSNR and 0.936 in SSIM.
Sign Approximation. We compare our scalable hyperbolic tangent function with previous Sign approximation functions. The experimental results listed in Tab. 2c. Our Tanh(ax:) dramatically surpasses the piecewise linear function Clip(z) and quadratic function Quad(a)) by 0.79 and 0.74 dB, suggesting the superiority of the proposed Tanh(aæ). This advantage can be explained by the analysis in Sec. 3.2 that our Tanh(ax) is more flexible and can adaptively reduce the difference with Sign(r).
part of the base model while keeping the other parts full-precision to study the binarization benefit. The experimental results are reported in Tab. 2d. The base model yields 34.11 1 dB in PSNR and 0.936 in SSIM while costing 10.52 G OPs and 634 K Params. can be observed from Tab. 2d: (i) Binarizing the bottleneck B reduces the Params the most (270174) with the smallest performance drop (only 0.31 dB). (ii) Binarizing the decoder D achieves the largest OPs reduction (4927 M) while the performance degrades by a moderate margin (1.08 dB).
Binarizing Different Parts. We binarize
5 Conclusion
In this paper, we propose a novel BNN-based method BiSRNet for binarized HSI restoration. To the best of our knowledge, this is the first work to study the binarized spectral compressive imaging reconstruction problem. We first redesign a compact and easy-to-deploy base model with simple computation operations. Then we customize the basic unit BiSR-Conv for model binarization. BiSR- Conv can adaptively adjust the density and distribution of HSI representations before binarizing the activation. Besides, EiSR-Conv employs a scalable hyperbolic tangent function to closer approach Sign by arbitrarily reducing the approximation error. Subsequently, we use BiSR- Cony to build up four binarized convolutional modules that can handle the dimension mismatch issue during feature reshaping and propagate full-precision information through all layers. Comprehensive quantitative and qualitative experiments demonstrate that our BiSRNet significantly outperforms SOTA BNNs and even achieves comparable performance with full- full-precision CNN-based HSI reconstruction algorithms.

---

Acknowledgement
This research was funded through National Key Research and Development Program of China (Project No. 2022YFB36066,, in part by the Shenzhen Science and Technology Project under National Natural Science Founda-
Grant (JCY1202208 UCC1120220881000000000 1100001004 ISGG202108021511 tion of China (62271414) 1414,, Science Fund for Distinguished Young Scholars of Zhejiang Province (LR23F01000)), and Research Center for Industries of the Future at Westlake University.
References
[1] A. A. Gowen, C. P. O' O'Donnell, P. J. Cullen, G. Downey, and J. M. Frias, "Hyperspectral imaging-an emerging process analytical tool for food quality and safety control," Trends in food science & technology, 2007.
[2] D. Lorente, N. Aleixos, J. Cómez-Sanchij S. Cubero, O. L. Garcia-Navarrete and J. Blasco, "Recent advances and applications of hyperspectral imaging for fruit and vegetable quality assessment," Food and Bioprocess Technology, 2012.
[3] R. Lu and Y.-R. Chen, "Hyperspectral imaging for safety inspection of food and agricultural products," in Pathogen Detection and Remediation for Safe Eating, 1999.
[4] V. Backman, M. B. Wallace. L. Perelman, J. Arendt, R. Gurjar, M. Muller, Q. Zhang, G. Zonios,
E. Kline, and McGillican, "Detection of preinvasive cancer cells," " Nature, 2000.
[5] G. Lu and B. Fei, "Medical hyperspectral imaging: a review," Journal of Biomedical Optics, 2014.
[6] Z. Meng. M. Qiao, J. Ma, Z. Yu, K. Xu, and X. Yuan, "Snapshot multispectral endomicroscopy." Optics Letters, 2020.
[7] B. Uzkent, M. J. Hoffman, and A. Vodacek, "Real-time vehicle tracking in acrial video using hyperspectral features," in CVPRW, 2016.
[8] B. Uzkent, A. Rangnekar, and M. Hoffman, "Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps," in CVPRW, 2017.
[9] M. H. Kim, T. A. Harvey, D. S. Kittle, H. Rushmeier, J. Dorsey, R. O. Prum, and D. J. Brady, "3d imaging spectroscopy for measuring hyperspectral patterns on solid objects." TOG, 2012.
[10] M. Borengasser, W. S. Hungate, and R. Watkins, "Hyperspectral remote sensing: principles and applications," CRC press, 2007.
[11] Melgani and Bruzzone, "Classification of hyperspectral remote sensing images with support vector machines," IEEE Transactions Geoscience and Remote Sensing, 2004.
[12] Y. Yuan, X. Zheng, and X. Lu, "Hyperspectral image superresolution by transfer learning," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2017.
[13] P. Llull, X. Liao, X. Yuan, J. Yang. D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, "Coded aperture compressive temporal imaging," Optics Express, 2013.
[14) A. Wagadarikar, John, R. Willett, and Brady, "Single disperser design for coded aperture snapshot spectral imaging," Applied Optics, 2008.
[15] A. A. Wagadarikar, N. P. Pitsianis, X. Sun, and D. J. Brady, "Video rate spectral imaging using a coded aperture snapshot spectral imager," Optics Express, 2009.
[16] X. Yuan, T.-H. Tsai, R. Zhu, P. Llull, D. Brady, and L. Carin, "Compressive hyperspectral imaging with side information." IEEE Journal of selected topics in Signal Processing, 2015.
[17] X. Ma, X. Yuan, C. Fu, and Arce, "Led-based compressive spectral-temporal imaging." " Optics Express, 2021.
[18] Z. Meng, J. Ma, and X. Yuan, "End-to-end low cost compressive spectral imaging with spatial- spectral self-attention," in ECCV, 2020.
[19] M. E. Gehm, R. John, J. J. Brady, R. M. Willett, and T. J. Schulz, "Single-shot compressive spectral imaging with a dual-disperser architecture," Optics express, 2007.
[20] X. Miao, X. Yuan, Pu, and V. Athitsos, "I-net: Reconstruct hyperspectral images from snapshot measurement," in ICCV, 2019.
10

---

[21] L. Wang, C. Sun, M. Zhang, Y. Fu, and H. Huang, "Dnu: Deep non-local unrolling for computational spectral imaging. " in CVPR, 2020.
[22] L. Wang, C. Sun, Fu, M. H. Kim, and H. Huang, "Hyperspectral image reconstruction using a deep spatial-spectie prior," in CVPR, 2019.
[23] J. Ma, X.-Y. Liu, Z. Shou, and X. Yuan, "Deep tensor admm-net for snapshot compressive imaging." in ICCV, 2019.
[24] X. Hu, Y. Cai, J. Lin, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, "Hdnet: High-resolution dual-domain learning for spectral compressive imaging," in CVPR, 2022.
[25] Z. Meng, 7. Yu, K. Xu, and X. Yuan, "Self-supervised neural networks for spectral snapshot compressive imaging." in ICCV, 2021.
[26] Y. Cai, Lin, X. Hu, H. Wang. X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, "Mask-guided
spectral-wise transformer for efficient hyperspectral image reconstruction," " in CVPR, 2022.
[27] Y. Cai, J. Lin, Z. Lin, H. Wang Y. Zhang, H. Pfister, R. Timofte, and L. V. Gool, "Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction." in CVPRW, 2022.
[28] Y. Cai, J. Lin, X. Hu, H. Wang. X. Yuan, Y. Zhang. R. Timofte, and I.. V. Gool, "Coarse-to-fine sparse transformer for hyperspectral image reconstruction," in ECCV, 2022.
[29] Y. Cai, Lin, H. Wang. X. Yuan, H. Ding, Zhang, R. Timofte, and L.. V. Gool, "Degradation- aware unfolding half-shuffle transformer for spectral compressive imaging," in NeurlPS, 2022.
[30] X. Jiang, N. Wang, J. Xin, K. Li, X. Yang, and X. Gao, "Training binary neural network without batch normalization for image super- super-resolution." " in AAAI, 2021.
[31] B. Xia, Y. Zhang, Y. Wang, Y. Tian, W. Yang, R. Timofte, and L. V. Gool, "Basic binary convolution unit for binarized image restoration network," in ICLR, 2023.
[32] Z. Liu, Z. Shen, M. Savvides, and K.-T. Cheng, "Reactnet: Towards precise binary neural network with generalized activation functions," in ECCV, 2020.
[33] H. Qin, R. Gong, X. Liu, M. Shen, Z. Wei, F. Yu, and J. Song, "Forward and backward information retention for accurate binary neural networks," in CVPR, 2020.
[34] Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng, "Bi-real net: Enhancing the performance of 1-bit enns with improved representational capability and advanced training algorithm," in ECCV, 2018.
[35] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, "Binarized neural networks," in NeurIPS, 2016.
[36] M. Courbariaux, Y. Bengio, and J.-P. David, "Binaryconnect: Training deep neural networks with binary weights during propagations," NeurlPS, 2015.
[37] V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, "Efficient processing of deep neural networks:
A tutorial and survey," Proceedings of the IEEE, 2017.
[38] H. Qin, R. Gong, X. Liu, X. Bai, J. Song, and N. Sebe, "Binary neural networks: A survey," Puttern Recognition, 2020.
[39] Xin, N. Wang, X. Jiang, Li, H. Huang, and X. Gao, "Binarized neural network for single image super resolution." in ECCV, 2020.
[40] H. Wang and Y. Fu, "Trainability preserving neural structured pruning," in ICLR, 2023.
[41] Y. Zhang, H. Wang, C. Qin, and Y. Fu, "Learning efficient image super-resolution networks via structure-regullrrized pruning," in ICLR, 2022.
[42] H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, "Pruning filters for efficient convnets." in ICLR, 2017.
[43] H. Wang. S. Lohit, M. Jones, and Y. Fu, "What makes a "good" data augmentation in knowledge distillation statistical perspective," in NeurIPS, 2022.
[44] G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in neural network," in NewrlPSW. 2014.
[45] X. Zhang, X. Zhou, M. Lin, and Sun, "Shufflenet: An extremely efficient convolutional neural network for mobile devices," in CVPR, 2018.

---

[46] N. Ma. X. Zhang, H.-T. Zheng, and J. Sun, "Shufflenet v2: Practical guidelines for efficient enn architecture design." in ECCV, 2018.
[47] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "Mobilenets: Efficient convolutional neural networks for mobile vision applications." " arXiv preprint
2017.
[48] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, "Mobilenetv2: Inverted residuals and linear bottlenecks," in CVPR, 2018.
[49] M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, "Xnor-net: Imagenet classification using binary convolutional neural networks," in ECCV, 2016.
[50] R. Ding, T.-W. Chin, Z. Liu, and D. Marculescu, "Regularizing activation distribution for training binarized deep networks," in CVPR, 2019.
[51] J. Zhang. Y. Pan, T. Yao, H. Zhao, and T. Mei, "dabnn: A super fast inference framework for binary neural networks on arm devices," ," in ACM MM, 2019.
[52] D. Kittle, K. Choi, A. Wagadarikar, and D. J. Brady, "Multiframe image estimation for coded aperture snapshot spectral imagers," Applied optics, 2010.
[53] Y. Liu, X. Yuan, J. Suo, D. Brady, and Q. Dai, "Rank minimization for snapshot compressive imaging." TPAMI, 2019.
[54] L. Wang. Z. Xiong, G. Shi, F. Wu, and W. Zeng. "Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging," TPAMI, 2016.
[55] S. Zhang, L. Wang. Y. Fu, X. Zhong, and H. Huang, "Computational hyperspectral imaging based on dimension-diseriminaave low-rank tensor recovery," in ICCV, 2019.
[56] X. Yuan, "Generalized alternating projection based total variation minimization for compressive sensing," " in ICIP, 2016.
[57] J. Tan, Y. Ma, H. Rueda, D. Baron, and G. R. Arce, "Compressive hyperspectral imaging via approximate message passing," " IEEE Journal of Selected Topics in Signal Processing, 2016.
[58) M. A. Figueiredo, R. D. Nowak, and S. J. Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems," " IEEE Journal of selected topics in signal processing, 2007.
[59] J. Bioucas-Dias and M. Figueiredo., "A new twist: Two-step iterative shrinkagetthresholddngg algorithms for image restoration.." " TIP, 2007.
[60] J. Yang, X. Liao, X. Yuan, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, "Compressive sensing by learning a gaussian mixture model from measurements," TIP, 2014.
[61] J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro. and L. Carin, "Video compressive sensing using gaussian mixture models," " TIP, 2014.
[62] X. Zhang, Y. Zhang. R. Xiong. Q. Sun, and J. Zhang, "Herosnet: Hyperspectral explicable
reconstruction and optimal sampling deep network for snapshot compressive imaging," in CVPR, 2022.
[63] T. Huang, W. Dong, X. Yuan, J. Wu, and G. Shi, "Deep gaussian scale mixture prior for spectral compressive imaging." " in CVPR, 2021.
[64] X. Lin, C. Zhao, and W. Pan, "Towards accurate binary convolutional neural network," in NeurlPS, 2017.
[65] I. Choi, D. S. Jeon, G. Nam, D. Gutierrez, and M. H. Kim, "High-quality hyperspectral reconstruction using a spectral prior," TOG, 2017.
[66] J.-I. Park, M.-H. Lee, M. D. Grossberg, and K. Nayar, "Multispectral imaging using multi- plexed illumination," in ICCV, 2007.
[67] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang. Z. Vito,o M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in NeurlPS, 2019.
[68] D. P. Kingma and J. L. Ba, "Adam: A method for stochastic optimization," in ICLR, 2015.
[69] I. Loshchilov and F. Hutter, "Sgdr: Stochastic gradient descent with warm restarts," in ICLR,
2017.
12


Reviews:
[{"review_id": "JmKf31QUiS", "reviewer": "Reviewer_F7DC", "summary": "This paper proposes a Binarized Neural Network based approach known as BiSRNet for binarized HSI restoration. The main motivation of the paper stems from the fact that any CNN or transformer-based architectures are computationally heavy for devices with low computing power and hence need extremely fast and light weight networks like binary neural networks. This paper redesigns the binary conv layer (BiSR-Conv) to exploit the distributed nature of HSI representations. They also employed a scalable Tanh function to decrease the approximation error. Overall, this paper is well-written and reasons well for various changes to the design of BCNNs.", "questions": "- I am not very familiar with HSI work so I do not have many questions at this point.", "limitations": "Please see the weakness. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Well-motivated problem and design of the solution\n- Good empirical evidence\n- Good presentation \n- Clear to understand", "weaknesses": "- Main weakness of this paper is their redesigned conv modules are only applicable to the HSI domain? If so this limits the scope and impact of this work.\n- Lack of theory\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\n### Response to Reviewer F7DC\n\n&nbsp;\n\nThanks for your valuable comments.\n\nCode and models will be released to the public.\n\n&nbsp;\n\n`Q-1:` Are the redesigned conv modules only applicable to the HSI domain?\n\n`A-1:` No. Although this work mainly studies the binarized spectral compressive imaging reconstruction problem in HSI domain, the proposed method can be generalized to other natural and even medical image domains.\n\n(1)\tFirst of all, in Line 131 – 138 of the main paper, we notice that HSI signals have different density and distribution along the spectral dimension due to the constraints of specific wavelengths. To adaptively fit this HSI nature, we propose to redistribute the HSI representations along the spectral dimension before binarizing the activation. When this technique is used to other image domains, it shifts and reshapes the feature maps in channel wise to preserve more information in activation binarization. \n\n(2)\tSecondly, in Line 138 – 160 and Fig. 3 of the main paper, our designed scalable hyperbolic tangent function is not only applicable to the HSI domain. Instead, it is a very basic and general technique to approximate the Sign function in any BNN for any image domain.\n\n(3)\tThirdly, allowing full-precision information flow is a critical insight for us to design BiSR-Conv unit in Fig. 2 (c) and the four binarized convolution modules in Fig. 4. The identity paths propagate the full-precision information to all layers of the network. By this means, the quantization error between BNN and CNN can be narrowed down.\n\nThe proposed BiSR-Conv and the four binarized convolution modules in Fig.4 can replace the vanilla binarized convolution layer and normal downsample / upsample / fusion convolution modules (in Fig. 4) in other BNNs and can be seamlessly generalized to other image domains.\n\nWe conduct experiments of RGB image denoising ($\\sigma = 25$). The BNNs are trained on DIV2K [73] and tested on CBSD68 [74], Kodak24 [75], and Urban100 [76]. Besides, we also conduct experiments of medical image enhancement on Real Fundus [77] dataset. The PSNR (dB) results are shown in the following table.\n\n| Datasets| BNN | Bi-Real | IRNet | BTM | ReAcNet | BBCU | BiSRNet |      \n| :- | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| CBSD68 | 22.67 | 28.72 | 29.01 | 29.91 | 29.95 | 30.56 | **31.15** |\n| Kodak24 | 22.58 | 29.17 | 29.54 | 30.64 | 30.65 | 31.28 | **32.06** |\n| Urban100 | 22.67 | 28.18 | 28.35 | 29.05 | 29.20 | 29.96 | **30.21** |\n| Real Fundus | 16.89 | 23.94 | 24.03 | 25.58 | 24.16 | 24.25 | **26.31** |\n\nOur method still significantly outperforms other BNNs. These results demonstrate the generality and effectiveness of our method on RGB/Medical image domains. The reason why we focus on studying binarized spectral compressive imaging reconstruction is that this problem has not been studied until now. We aim to fill this research gap. All compared methods in Tab. 1 are re-implemented by us.\n\n&nbsp;\n\n`Q-2:` Lack of theory\n\n`A-2:` This work mainly studies an application problem, i.e., binarized spectral compressive imaging reconstruction, instead of theory. The first keyword labeled on this paper is “Applications” and the primary area of this submission is “Machine Vision” instead of “Machine Learning Theory”. \n\nHowever, there are still some theoretical analysis and derivation in our paper.\nIn Line 138 – 160 of the main paper, we first theoretically prove that the proposed scalable hyperbolic tangent function can arbitrarily approach the Sign function in Eq. (6) and (7). Then we theoretically compute and compare the approximation error of previous picewise linear and quadratic functions and our scalable hyperbolic tangent function in Line 143 – 145, Fig. 3, and Eq. (8). Finally, we theoretically analyze the differentiability and flexibility of previous and our approximation functions to show the advantages of our method.\n\nBesides, we also provide detailed theoretical derivation of the CASSI mathematical model in Sec. 1 (Line 8 – 48) of the supplementary. \n\n&nbsp;\n\n**References**\n\n[73] NTIRE 2017 Challenge on Single Image Super-Resolution: Dataset and Study. CVPRW 2017.\n\n[74] A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. ICCV 2001.\n\n[75] Residual learning of deep convolutional neural networks for image denoising. Journal of Intelligent & Fuzzy Systems, 2019.\n\n[76] Single image super-resolution from transformed self-exemplars. CVPR 2015.\n\n[77] Rformer: Transformer-based generative adversarial network for real fundus image restoration on a new clinical benchmark. JBHI 2022.\n"}, {"review_id": "574i2bZeWm", "reviewer": "Reviewer_aGHs", "summary": "In this paper, binarized neural network is first utilized in hyperspectral image reconstruction. For model binarization, authors propose the Binarized Spectral-Redistribution Convolution (BiSR-Conv), which adaptively redistributes the HSI representations before binarizing activation. Since the Sign function is non-differentiable, a scalable hyperbolic tangent function is applied in backpropagation, which has less approximation error. In BiSR-Conv, the additional residual connection encourages the full-precision information to propagate throughout the whole network. Based on BiSR-Conv, four binarized convolutional modules are designed to address the dimension mismatch issue during feature reshaping.", "questions": "In Sign Approximation experiment, the proposed scalable hyperbolic tangent function and previous Sign approximation functions should be compared in other BNNs, to further verify the generalization of the proposed function. ", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "An application to a new domain;\nClear and consecutive writing;\nA feasible idea for deploying spectral compression imaging in edge devices.\n", "weaknesses": "Experiments are insufficient. How does the ‘full-precision information’ affect the whole network? Is the ‘full-precision information’ play a dominant role in BiSR-Conv or not? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\n### Response to Reviewer aGHs\n\n&nbsp;\n\nThanks for your valuable comments.\n\nCode and models will be released to the public.\n\n&nbsp;\n\n`Q-1:` How does the 'full-precision information' affect the whole network? Does the 'full-precision information' play a dominant role in BiSR-Conv or not?\n\n`A-1:` In BNNs, the weights and activations are binarized into 1 bit. Thus, there is a large gap between the outputs of the binarized and full-precision convolution, which severely degrades the HSI reconstruction performance. Hence, it is very important to allow the full-precision information flow to compensate for this quantization error. \n\n(1) To this end, we do not binarize the first (embedding) and last (mapping) convolution modules of the base model in Fig. 2 (a) of the main paper. Then the full-precision representations can be input into the binarized encoder and full-precision derivatives can be backpropagated to the binarized decoder.\n\n(2) In addition, to allow full-precision information flow through all binarized convolution layers, we design BiSR-Conv unit and the four binarized convolution modules, as shown in Fig. 4 of the main paper. More specifically, we add a bypass identity path in BiSR-Conv unit to propagate full-precision information, as shown in Fig. 2 (c). Besides, in Sec. 3.3, our binarized convolution modules cleverly use channel-wise concatenating and splitting operations to free the intermediate feature maps at the input and output ends of BiSR-Conv from being reshaped. In this case, the full-precision information flow is not blocked as the normal convolution modules do, as shown in Fig. 4.\n\n(3) The experiments to verify the importance of the full-precision flow are reported in Tab. 2 (a) of the main paper. Baseline-1 adopts vanilla 1-bit convolution and the normal convolution modules in Fig. 4. The full-precision information in the binarized parts is impeded. As a result, Baseline-1 only achieves poor results, 23.90 dB. When we successively apply the proposed BiSR-Conv and the four binarized modules, the full-precision information can be propagated through all layers of the BNN, leading to significant improvements of 3.90 and 1.96 dB.\n\n(4) Besides, we also conduct more experiments to verify the importance of full-precision information in the following table. When we binarize the feature embedding and mapping blocks in Fig. 2 (a), BiSRNet degrades by 2.13 and 1.06 dB respectively. When we jointly binarize the two blocks, the full-precision information can not be forward and backward propagated. As a result, the reconstruction performance degrades by 2.87 dB.\n\nThe above results and analysis reveal the significant importance of full-precision information. This is also the key insight and motivation to design our BiSRNet and the four binarized convolution modules in Fig. 4.\n\n&nbsp;\n\n`Q-2:` In Sign approximation experiment, the proposed scalable hyperbolic tangent function and previous Sign approximation functions should be compared in other BNNs, to further verify the generalization of the proposed function.\n\n`A-2:` Thanks for your suggestion. Following your advice, we conduct experiments of Sign approximation in other BNNs. The PSNR (dB) results are reported in the following table.\n\n| Methods | BiConnect | BNN | Bi-Real | IRNet | ReAcNet | BBCU | BTM | BiSRNet | Avg |      \n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Clip($x$) | 22.19 | 23.88 | 26.20 | 26.16 | 26.41 | 26.43 | 27.15 | 28.97 | 25.92 |\n| Quad($x$) | 22.34 | 23.95 | 26.26 | 26.30 | 26.48 | 26.51 | 27.21 | 29.02 | 26.01 |\n| Tanh($\\alpha x$) | 22.93 | 24.46 | 26.94 | 26.95 | 27.23 | 27.25 | 27.97 | 29.76 | 26.69 |\n\nOur scalable hyperbolic tangent function achieves 0.77 and 0.68 dB improvements on average than the piecewise linear and quadratic functions, showing the effectiveness of our proposed technique."}, {"review_id": "e7U5rX1oA6", "reviewer": "Reviewer_nobf", "summary": "This work focuses on studying the binarized spectral compressive imaging reconstruction problem. A Binarized Spectral-Redistribution Network (BiSRNet) is proposed. The authors first design a basic U-Net as the base model to begin the binarization. Then a Binarized Spectral-Redistribution Convolutional (BiSR-Conv) unit is proposed to replace the 32-bit convolutional layer of the base model to derive BiSRNet. The BiSR-Conv has two advantages than vanilla 1-bit convolutional layer: (i) can redistribute the spectral distributions before binarization. (ii) has a scalable hyperbolic tangent function to approach the Sign function more closely.", "questions": "Will you also plan to release the code of other BNNs compared in Table 1? ", "limitations": "See the above comments", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": " The novelty and motivation are good. First of all, the research topic is new. Nobody explores binarized spectral compressive imaging before. The authors not only contribute a good method but also conduct experiments using before BNNs designed for other topics. We should respect this. Secondly, the redesigned base model is well motivated. The architecture shows the insight of “full-precision information flow”. The idea of redistributing HSI representations before binarization is interesting and reasonable. The Tanh(\\alpha x) is also amazing. Flexibly controlling the gap with Sign function is really cool.\n\n+ The performance is good and solid. The proposed BiSRNet outperforms the SOTA BNNs widely used in image classification. Meanwhile, the computational and memory complexity of BiSRNet is much lower than full-precision CNNs while the performance is comparable. In the real HSI experiments, it seems that the proposed BiSCI performs better than CNN-based methods in noise suppression. \n\n+ The experiments are comprehensive, not only quantitative but also qualitative comparisons shows the advantages of the proposed method. The ablation study is sufficient to verify the effectiveness of the proposed techniques.\n\n+ The codes and pre-trained weights are provided in the supplementary. The reproducibility can be checked and ensured.\n\n+ The writing is good and easy-to-follow. The figures, tables, and mathematical notations are very clear.", "weaknesses": "- The definition is a little confusing. Sign(0) = 0 and Tanh(0) = 0. But in this paper, Sign(0) and Tanh(0) is defined as -1, as shown in Eq.(3) and line 152. Why? More explanation should be provided.\n\n- Although the proposed BNN achieves good performance with CNN. But compared with Transformer-based methods, there is a large gap. I understand that binarizing Transformer is not easy because of the self-attention mechanism. But it is good idea to keep trying.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\n### Response to Reviewer nobf\n\n&nbsp;\n\nThanks for your valuable comments. \n\nCode and models will be release to the public.\n\n&nbsp;\n\n`Q-1:` Why Sign(0) and Tanh(0) are defined as -1?\n\n`A-1:` As explained in Line 150 -152 of the main paper, if strictly following the mathematical definition, Sign(0) $= 0 \\neq \\pm 1$. However, in BNN, the weights and activations are binarized into 1-bit, i.e., only two values ($\\pm 1$). Hence, Sign(0) is usually set to $\\pm 1$, like [33, 34, 35, 36, 37, 38, 52]. Following this common setting, we also define $\\underset{\\alpha \\rightarrow +\\infty}{lim}~\\text{Tanh}(\\alpha \\cdot 0) = -1$ in BNN.\n\n&nbsp;\n\n`Q-2:` It is a good idea to keep trying on binarizing Transformer.\n\n`A-2:` Thanks for your reminding. As you mentioned, Transformer is tough to binarize because of the computation scheme of self-attention. According to the experience, preserving at least 8-bit weights and activations can considerably control the performance gap between quantized and full-precision (32-bit) Transformers. We would like to set your advice as a future research direction and have a try.\n\n&nbsp;\n\n`Q-3:` Will we plan to release the code of other BNNs compared in Table 1?\n\n`A-3:` Yes, of course. We will release all of them. Our goal is to establish a toolbox and baseline for further research in this topic.\n"}, {"review_id": "00PKswqnmb", "reviewer": "Reviewer_ZUfp", "summary": "This paper proposes a novel method, Binarized Spectral-Redistribution Network (BiSRNet), for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems. This paper redesigns the base model and presents the basic unit (BiSR-Conv) for model binarization. Specifically, this convolutional layer of BiSRNet is tailored for hyperspectral image processing. Comprehensive quantitative and qualitative experiments show that the proposed BiSR-Net outperforms state-of-the-art binarization methods and achieves comparable performance with full-precision algorithms.", "questions": "Please refer to Weaknesses", "limitations": "This paper proposes BiSRNet for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems. But this paper has not discussed the limitations of the proposed method. I suggest the author discuss the potential and limitations of the proposed method in more detail, which will make the contributions of the proposed method more significant.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(1) This paper redesigns a U-Net consisting of the four BCMs as shown in Figure 4 to let the full-precision information flow pass by the all network. This point is critical since previous BNNs do not consider all the situations in feature reshaping. Then in the basic unit, BiSR-Conv, with the insight of treating different HSIs with different densities and distribution, this paper proposes to shift HSIs before binarization to allow more binarized HSI activation. \n\n(2) Comprehensive experiments have been conducted including synthetic and real experiments to demonstrate the superiority of BiSRNet. The ablation study is also extensive. The performance of BiSRNet is impressive, which surpasses existing SOTA BNNs by huge margins, over 2.5 dB.", "weaknesses": "(1) Some details are missing. For example, how did you get the 3D mask $\\mathbf{M}^*$? I remember the coded aperture used in CASSI is 2D. Its shape should be $H \\times W$. Why does it have an additional dimension here? And how?\n\n(2) Some critical experimental analyses are lacking. For instance, at the end of section 4.4, “Binarizing Different Parts”, why binarizing the bottleneck can reduce the most parameters? And why binarizing the decoder can achieve the largest Ops reduction? An analysis should be provided to explain this.\n\n(3) I remember Binary Connect [39] binarizes the weights of CNN. But it seems that the activations of Binary Connect [39] are also binarized in Table 1. Why did you do that? Could you please explain this?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\n### Response to Reviewer ZUfp\n\n&nbsp;\n\nThanks for your valuable comments.\n\nCode and models will be released to the public.\n\n&nbsp;\n\n`Q-1:` How to derive the 3D mask? Why does it have an additional dimension?\n\n`A-1:` The original coded aperture $\\mathbf{M}^* \\in \\mathbb{R}^{H \\times W}$ is a 2D mask. However, as analyzed in Sec. 1 (Line 9 - 33) of the supplementary, the 3D HSI cube is point-wisely modulated by $\\mathbf{M}^*$ in each spectral channel before dispersion. To simply describe the process of modulation and dispersion, we derive a 3D Mask $\\mathbf{M} \\in \\mathbb{R}^{H \\times (W + d(N_{\\lambda}-1)) \\times N_{\\lambda}}$ by shifting  $\\mathbf{M}^*$ as\n\n$\\mathbf{M} (u,v, n_{\\lambda}) = \\mathbf{M}^* (x, y + d(\\lambda_n - \\lambda_c))$\n\n\nSimilarly, we obtain $\\tilde{\\mathbf{F}} \\in {\\mathbb R}^{H \\times (W + d(N_{\\lambda}-1)) \\times N_{\\lambda}}$ by shifting the original HSI signal $\\mathbf{F}$ as\n\n$\\tilde{\\mathbf{F}} (u,v, n_{\\lambda}) = \\mathbf{F} (x, y + d(\\lambda_n - \\lambda_c), n_{\\lambda})$\n\n\nFollowing this, we can simply reformulate the measurement $\\mathbf{Y}$ as\n\n$\\mathbf{Y} = \\sum\\nolimits_{n_{\\lambda}=1}^{N_{\\lambda}}   \\tilde{\\mathbf{F}} (:,:, n_{\\lambda})  \\odot \\mathbf{M} (:,:, n_{\\lambda}) + \\mathbf{E}$.\n\n\n&nbsp;\n\n\n`Q-2:` Why binarizing the bottleneck can reduce the most parameters? And why binarizing the decoder can achieve the largest Ops reduction? An analysis should be provided to explain this.\n\n`A-2:` Thanks for reminding. We will add the following explanation in the revision.\n\n(1) As shown in Fig. 2 (a) of the main paper, when the feature map is downscaled in the encoder, its channel number is doubled. Thus, the feature map in the bottleneck has the most channel number, leading to the most parameters in the bottleneck part of the network.\n\n(2) As shown in Fig. 4 (a) and (b) of the main paper, the convolution of downsample modules in encoder follows the downscaled operation while the convolution of upsample modules in decoder follows the upscaled operation. This implies the input spatial size of the convolution in upsample modules at the same level is larger than that of the convolution in downsample modules, which requires more computational costs. Thus, binarizing decoder achieves the largest Ops reduction.\n\n&nbsp;\n\n`Q-3:` Why binarizing the activations of Binary Connect in Tab. 1?\n\n`A-3:` The activations and weights of all compared BNN methods in Tab. 1 of the main paper are binarized for fair comparison. Thus, we also binarize the activations of Binary Connect.\n\n&nbsp;\n\n`Q-4:` Limitations of our work\n\n`A-4:` In fact, we have discussed the limitations in Sec. 3 (Line 59 - 64) of the supplementary. The main limitation of our work is that the model binarization sacrifices the HSI reconstruction performance. More specifically, compared to the full-precision counterpart, our BiSRNet is 4.35 (34.11 - 29.76) dB lower in PSNR and 0.099 (0.936 - 0.837) lower in SSIM. The PSNR and SSIM are reduced by 12.8\\% and 10.6\\%, respectively. However, this performance drop is smaller than that of other model binarization methods. To handle this issue, we will study how to preserve more performance while reducing the memory and computational complexity as much as possible in model binarization."}, {"review_id": "kA3u3k4Hfm", "reviewer": "Reviewer_kC2s", "summary": "This paper presents a deep neural network with binarized operations for low-cost spectral compressive imaging of hyperspectral images (HSI). The network introduces a basic unit for model binarization which adaptively redistributes HSI representations before binarization activation and uses a scalable hyperbolic tangent function to approximate the Sign function in backpropagation. Four binarization convolution modules are designed to solve the dimensional mismatch problem during feature reshaping and propagate full precision information throughout the network. Experiments on both synthetic and real data are conducted for performance evaluation.", "questions": "1. The paper argues that, as mobile devices are more and more widely used, the demands of running and storing HSI restoration models on edge devices grow significantly. Could the authors provide practical cases where mobile edge devices are used for HSI restoration?\n\n2.  In the binarized convolution layer, the formula for converting the model parameters from 32 bits to 1 bit is not clearly written. the average of the 32-bit absolute value multiplied by the sign function cannot be turned into a 1-bit number.\n\n3. It is not clear how the proposed method deals with the dimensional mismatch feature remodeling process, and how to solve the dimensional mismatch with binarized convolution modules.\n\n", "limitations": "Approximation error bounds of the proposed binarization scheme are not discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. An effective binarized deep neural network is proposed for HSI reconstruction.\n2. Four binarized convolution modules are designed to enable the propagation of full precision information through all convolution layers.\n3. Based on the ablation experiments in the article, the proposed scalable hyperbolic tangent function provides inspiration for approximating the symbolic function in back propagation. ", "weaknesses": "1. The proposed binarization scheme is general, without specific design optimized for HSI restoration. Then, it lacks comparison to other general binarized NNs or comparison in other reconstruction tasks.\n2. The proposed scheme is quite straightforward and engineering.\n3. Approximation error bounds are not discussed, but this is quite important for binarized NNs.\n4. Some key technical parts are not clearly/detailly described. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n&nbsp;\n\n### Response to Reviewer kC2s\n\n&nbsp;\n\nThanks for your valuable comments.\n\nCode and models will be released to the public.\n\n&nbsp;\n\n`Q-1:` The proposed binarization scheme is general, without specific design for HSI restoration. It lacks comparison to other BNNs or in other tasks.\n\n`A-1:` In fact, our BiSR-Conv based on HSI nature is tailored for HSI restoration. In Line 132 – 138, we notice that HSI signals have different density and distribution along the spectral dimension due to the constraints of specific wavelengths. To fit this HSI nature, BiSR-Conv redistributes the HSI representations before binarization.\n\nWe compare our method with other BNNs in Tab. 1. Our method outperforms other BNNs by over 2.55 dB.\n\nFollowing your advice, we also conduct experiments of RGB image denoising ($\\sigma = 25$). The BNNs are trained on DIV2K and tested on CBSD68, Kodak24, and Urban100. We also conduct experiments of medical image enhancement on Real Fundus [77]. The PSNR (dB) results are shown in the following table. Our method still outperforms other BNNs.\n\n| Datasets | BNN | Bi-Real | IRNet | BTM | ReAcNet | BBCU | BiSRNet |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| CBSD68 | 22.67 | 28.72 | 29.01 | 29.91 | 29.95 | 30.56 | **31.15** |\n| Kodak24 | 22.58 | 29.17 | 29.54 | 30.64 | 30.65 | 31.28 | **32.06** |\n| Urban100 | 22.67 | 28.18 | 28.35 | 29.05 | 29.20 | 29.96 | **30.21** |\n| Real Fundus | 16.89 | 23.94 | 24.03 | 25.58 | 24.16 | 24.25 | **26.31** |\n\n&nbsp;\n\n`Q-2:` The proposed scheme is straightforward and engineering.\n\n`A-2:` In fact, all the proposed techniques have strong motivations and deep insights.\n\n(1) See Line 107 – 112, previous CNN-/Transformer-based methods suffer from heavy computational burden or massive parameters. Meanwhile, these methods often exploit complex computations that are difficult to binarize. Hence, we redesign a simple, compact, and easy-to-deploy base model in Fig. 2 (a).\n\n(2) Our BiSR-Conv block is based on the HSI nature. See Line 131 – 134. To handle different density and distribution along the spectral dimension, BiSR-Conv reshapes the HSI features before binarization.\n\n(3) In Line 138 – 160, previous Sign approximation functions are inflexible and have large estimation errors. To solve this problem, we propose Tanh($\\alpha x$).\n\n(4) Full-precision information is critical in BNN. Bearing this key insight into consideration, we design the bypass path in BiSR-Conv and four binarized convolution modules in Fig. 4 to make sure the full-precision information can be propagated into all layers of BNN.\n\n&nbsp;\n\n`Q-3:` Approximation error bounds are not discussed.\n\n`A-3:` Unfortunately, existing BNN works [33 - 39] do not provide theoretical analysis of approximation error bound. This is because the deep neural network is a huge and complex black-box system with a large number of parameters.\n\nWe compute the approximation error in the following table. Our method achieves the smallest error.\n\n| Methods | BiConnect | BNN | Bi-Real | IRNet | ReAcNet | BBCU | BTM | BiSRNet |\n| :- | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Error ( $\\times 10^{-4}$ ) | 75.3 | 49.2 | 16.5 | 15.7 | 14.5 | 13.2 | 10.6 | **8.1** |\n\n&nbsp;\n\n`Q-4:` Some technical parts are not clearly described.\n\n`A-4:` Please be specific about which part.\n\nWe not only describe the details of our method in the paper but also provide code and pre-trained weights to reproduce our method.\n\n&nbsp;\n\n`Q-5:` Practical cases about mobile edge devices used for HSI restoration.\n\n`A-5:` The HSI reconstruction techniques can be applied in hyperspectral (HS) cameras. For example, Motoki et al. [78] develops a video-rate HS camera using compressive sensing techniques. This HS camera can equip with AI-based HSI reconstruction algorithms to accelerate the imaging process. This HS camera can be used in real-world scenarios, including consumer applications such as smartphones and drones.\n\n&nbsp;\n\n`Q-6:` The formula for converting the model parameters from 32 bits to 1 bit is not clearly written. \n\n`A-6:` First of all, the values of the binarized weights used for 1-bit convolution (bit-count and XNOR operations in Eq. 10) are still $\\pm 1$. The binarized weights are obtained by Eq. 3. The multiplied 32-bit mean absolute value does not affect the 1-bit convolution. It is outside the binarized weights. Multiplying this scale factor is to reduce the quantization error between 1-bit and 32-bit weights, as analyzed in XNOR-Net [52]. Many later works [33 - 37] also follow this multiplication.\n\n&nbsp;\n\n`Q-7:` Questions about dimension mismatch.\n\n`A-7:` In Line 189 – 201, the feature maps are directly reshaped by the binarized convolution in the normal binarized modules of Fig. 4. As aforementioned, the full-precision information is critical in BNN. Yet, the dimension of the input end of 1-bit conv does not match that of the output end. For example, in the normal downsample module (Fig. 4a), the input shape $\\mathbb{R}^{H\\times W\\times C}$ does not match the output shape $\\mathbb{R}^{\\frac{H}{2}\\times \\frac{W}{2}\\times 2C}$. This mismatch issue impedes the identity connection from input to output, blocking 32-bit information flow.\n\nTo address this issue, we first design a bypass identity path in BiSR-Conv, and then use concatenating and splitting operations to free the feature map from being reshaped at the input and output ends of BiSR-Conv. See Fig. 4 (a), the feature map keeps its shape $\\mathbb{R}^{\\frac{H}{2}\\times \\frac{W}{2}\\times 2C}$ when passing BiSR-Conv in our downsample module. By this means, the full-precision information can flow through our BiSR-Conv and four binarized modules, see the red arrows of Fig. 2 (c) and 4.\n\n&nbsp;\n\n**References**\n\n[77] Rformer: Transformer-based generative adversarial network for real fundus image restoration on a new clinical benchmark. JBHI 2022\n\n[78] Video-rate hyperspectral camera based on a CMOS-compatible random array of Fabry–Pérot filters. Nature Photonics 2023"}]

Author rebuttal:

Meta-review: All reviewers found some merit in this work.
Decision: Accept (poster)