Title: Binarized Spectral Compressive Imaging
Venue: NeurIPS 2023
Accepted: yes
Keywords: Applications

Computer Vision

Low-level Vision

Image Restoration

Snapshot Compressive Imaging

Hyperspectral Image Reconstruction
Abstract: Existing deep learning models for hyperspectral image (HSI) reconstruction achieve good performance but require powerful hardwares with enormous memory and computational resources. Consequently, these methods can hardly be deployed on resource-limited mobile devices. In this paper, we propose a novel method, Binarized Spectral-Redistribution Network (BiSRNet), for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems. Firstly, we redesign a compact and easy-to-deploy base model to be binarized. Then we present the basic unit, Binarized Spectral-Redistribution Convolution (BiSR-Conv). BiSR-Conv can adaptively redistribute the HSI representations before binarizing activation and uses a scalable hyperbolic tangent function to closer approximate the Sign function in backpropagation. Based on our BiSR-Conv, we customize four binarized convolutional modules to address the dimension mismatch and propagate full-precision information throughout the whole network. Finally, our BiSRNet is derived by using the proposed techniques to binarize the base model. Comprehensive quantitative and qualitative experiments manifest that our proposed BiSRNet outperforms state-of-the-art binarization algorithms. Code and models are publicly available at https://github.com/caiyuanhao1998/BiSCI

Binarized Spectral Compressive Imaging

Abstract

Existing deep learning models for hyperspectral image (HSI) reconstruction achieve good performance but require powerful hardwares with enormous memory and computational resources. Consequently, these methods can hardly be deployed on resource-limited mobile devices. In this paper, we propose a novel method, Binarized Spectral-Redistribution Network (BiSRNet), for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems. Firstly, we redesign a compact and easy-to-deploy base model to be binarized. Then we present the basic unit, Binarized Spectral-Redistribution Convolution (BiSR-Conv). BiSR-Conv can adaptively redistribute the HSI representations before binarizing activation and uses a scalable hyperbolic tangent function to closer approximate the Sign function in backpropagation. Based on our BiSR-Conv, we customize four binarized convolutional modules to address the dimension mismatch and propagate full-precision information throughout the whole network. Finally, our BiSRNet is derived by using the proposed techniques to binarize the base model. Comprehensive quantitative and qualitative experiments manifest that our proposed BiSRNet outperforms state-of-the-art binarization algorithms. Code and models are publicly available at https://github.com/caiyuanhao1998/BiSCI

1 Introduction

Compared to normal RGB images, hyperspectral images (HSIs) have more spectral bands to capture richer information of the desired scenes. Thus, HSIs have wide applications in agriculture [1, 2, 3], medical image analysis [4, 5, 6], object tracking [7, 8, 9], remote sensing [10, 11, 12], etc.

To capture HSIs, conventional imaging systems leverage 1D or 2D spectrometers to scan the desired scenes along the spatial or spectral dimension. Yet, this process is very time-consuming and thus fails in measuring dynamic scenes. In recent years, snapshot compressive imaging (SCI) systems [13, 14, 15, 16, 17] have been developed to capture HSI cubes in real time. Among these SCI systems, the coded aperture snapshot spectral imaging (CASSI) [14, 18, 19] demonstrates its outstanding effectiveness and efficiency. The CASSI systems firstly employ a coded aperture (physical mask) to modulate the 3D HSI cube, then use a disperser to shift spectral information of different wavelengths, and finally integrate these HSI signals on a detector array to capture a 2D compressed measurement. We study the inverse problem, i.e. , restoring the original 3D HSI cube from the 2D measurement.

Existing state-of-the-art (SOTA) SCI reconstruction methods are based on deep learning. Convolutional neural network (CNN) [18, 20, 21, 22, 23, 24, 25] and Transformer [26, 27, 28, 29] have been used to implicitly learn the mapping from compressed measurements to HSIs. Although superior performance is achieved, these CNN-/Transformer-based methods require powerful hardwares with abundant computing and memory resources, such as high-end graphics processing units (GPUs). However, edge devices ( e.g., mobile phones, hand-held cameras, small drones, etc. ) evidently cannot meet the requirements of these expensive algorithms because edge devices have very limited memory, computational power, and battery. As mobile devices are more and more widely used, the demands of running and storing HSI restoration models on edge devices grow significantly. This motivates us to reduce the memory and computational burden of HSI reconstruction methods while preserving the performance as much as possible so that the algorithms can be deployed on resource-limited devices.

(

NeurIPS

15)

(

NeurIPS

16)

(

ECCV

18)

(

CVPR

20)

(

ECCV

20)

(

ICLR

23)

(

AAAI

21)

(

Ours

)

2.55 dB

3.25 dB

3.39 dB

3.46 dB

3.50 dB

5.88 dB

7.57 dB

PSNR

The studies on neural network compression and acceleration [37, 38] can be divided into four categories: quantization [31, 32, 33, 34, 35, 39], pruning [40, 41, 42], knowledge distillation [43, 44], and compact network design [45, 46, 47, 48]. Among these methods, binarized neural network (BNN) belonging to quantization stands out because it can extremely compress the memory and computational costs by quantizing the weights and activations of CNN to only 1 bit. In particular, BNN could achieve 32 × memory compression ratio and up to 58 × practical computational reduction on central processing units (CPUs) [49]. In addition, the pure logical computation ( i.e. , XNOR and bit-count operations) of BNN is highly energy-efficient for embedded devices [50, 51]. However, directly applying model binarization for HSI reconstruction algorithms may encounter three issues. (i) The HSI representations have different density and distribution in different spectral bands. Equally binarizing the activations of different spectral channels may lead to the collapse of HSI features. (ii) Previous model binarization methods mainly adopt a piecewise linear [33, 35, 36] or quadratic [31, 32, 34] function to approximate the non-differentiable Sign function. Nonetheless, there still remain large approximation errors between them and Sign. (iii) How to tackle the dimension mismatch problem during feature reshaping while allowing full-precision information propagation in BNN has not been fully explored. Previous model binarization methods [32, 33, 34, 35, 36] mainly consider the feature downsampling situation in a backbone network for high-level vision tasks.

Bearing the above considerations in mind, we propose a novel BNN-based method, namely Binarized Spectral-Redistribution Network (BiSRNet) for efficient and practical HSI reconstruction. Firstly , we redesign a compact and easy-to-deploy base model to be binarized. Different from previous CNN/Transformer-based methods, this base model does not include complex computations like unfolding inference and non-local self-attention that are difficult to implement on edge devices. Instead, our base model only uses convolutional units that can be easily replaced by XNOR and bit-count logical operations on resource-limited devices. Secondly , we develop the basic unit, Binarized SpectralRedistribution Convolution (BiSR-Conv), used in model binarization. Specifically, BiSR-Conv can adapt the density and distribution of HSI representations in spectral dimension before binarizing the activation. Besides, BiSR-Conv employs a scalable hyperbolic tangent function to closer approximate the non-differentiable Sign function by arbitrarily reducing the approximation error. Thirdly , as the full-precision information is very critical in BNN and the input HSI is the only full-precision source, we use BiSR-Conv to build up four binarized convolutional modules that can handle the dimension mismatch issue during feature reshaping and simultaneously propagate full-precision information through all layers. Finally , we derive our BiSRNet by using the proposed techniques to binarize the base model. As shown in Fig. 1, BiSRNet outperforms SOTA BNNs by large margins, over 2.5 dB .

In a nutshell, our contributions can be summarized as follows:

(i) We propose a novel BNN-based algorithm BiSRNet for HSI reconstruction. To the best of our knowledge, this is the first work to study the binarized spectral compressive imaging problem.

(ii) We customize a new binarized convolution unit BiSR-Conv that can adapt the density and distribution of HSI representations and approximate the Sign function better in backpropagation.

(iii) We design four binarized convolutional modules to address the dimension mismatch issue during feature reshaping and propagate full-precision information through all convolutional layers.

(iv) Our BiSRNet dramatically surpasses SOTA BNNs and even achieves comparable performance with full-precision CNNs while requiring extremely lower memory and computational costs.

2 Related Work

2.1 Hyperspectral Image Reconstruction

Traditional HSI reconstruction methods [14, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61] are mainly based on hand-crafted image priors. Yet, these traditional methods achieve unsatisfactory performance and generality due to their poor representing capacity. Recently, deep CNN [18, 20, 21, 22, 23, 24, 25, 62] and Transformer [26, 27, 28, 29] have been employed as powerful models to learn the underlying mapping from compressed measurements to HSI data cubes. For example, TSA-Net [18] employs three spatial-spectral self-attention layers at the decoder of a U-shaped CNN. Cai et al. propose a series of Transformer-based algorithms (MST [26], MST++ [27], CST [28], and DAUHST [29]), pushing the performance boundary from 32 dB to 38 dB. Although impressive results are achieved, these CNN-/Transformer-based methods rely on powerful hardwares with enormous computational and memory resources, which are unaffordable for mobile devices. How to develop HSI restoration algorithms toward resource-limited platforms is under-explored. Our goal is to fill this research gap.

2.2 Binarized Neural Network

BNN [35] is the extreme case of model quantization as it quantizes the weights and activations into only 1 bit. Due to its impressive effectiveness in memory and computation compression, BNN has been widely applied in high-level vision [32, 33, 34, 35, 49] and low-level vision [30, 31, 39]. For example, Jiang et al. [30] train a BNN without batch normalization for image super-resolution. Xia et al. [31] design a binarized convolution unit BBCU for image super-resolution, denoising, and JPEG compression artifact reduction. Yet, the potential of BNN for SCI reconstruction has not been studied.

3 Method

3.1 Base Model

The full-precision model to be binarized should be compact and its computation should be easy to deploy on edge devices. However, previous CNN-/Transformer-based algorithms are computationally expensive or have large model sizes. Some of them exploit complex operations like unfolding inference [21, 22, 23, 29, 63] and non-local self-attention computation [18, 20, 26, 27, 28] that are challenging to binarize and difficult to implement on mobile devices. Hence, we redesign a simple, compact, and easy-to-deploy base model without using complex computation operations.

Inspired by the success of MST [26] and CST [28], we adopt a U-shaped structure for the base model as shown in Fig. 2. It consists of an encoder E , a bottleneck B , and a decoder D . Please refer to the supplementary for the CASSI mathematical model. Firstly, we reverse the dispersion of CASSI by shifting back the measurement Y ∈ R H × ( W + d ( N λ -1)) × N λ to derive the input H ∈ R H × W × N λ as

where H , W , and N λ denote the HSI's height, width, and number of wavelengths. d represents the shifting step. The concatenation of H and the 3D mask M ∈ R H × W × N λ is fed into a feature embedding module to produce the shallow feature X s ∈ R H × W × N λ . The feature embedding module is a conv 1 × 1 (convolutional layer with kernel size = 1 × 1). Subsequently, X s undergoes the encoder E , bottleneck B , and decoder D to generate the deep feature X d ∈ R H × W × N λ . E consists of two convolutional blocks and two downsample modules. The details of the convolutional block are depicted in Fig. 2 (b). The fusion up and down modules are both conv 1 × 1 to aggregate the feature maps and modify the channels. The downsample module is a strided conv 4 × 4 layer that downscales the feature maps and doubles the channels. B is a convolutional block. D consists of two convolutional blocks and two upsample modules. The upsample module is a bilinear interpolation followed by a conv 3 × 3 to upscale the feature maps and halve the channels. Skip connections between E and D are employed to alleviate the information loss during rescaling. Finally, the sum of X s and X d is fed into the feature mapping module ( conv 1 × 1) to produce the reconstructed HSI H ′ ∈ R H × W × N λ .

3.2 Binarized Spectral-Redistribution Convolution

The details of BiSR-Conv are illustrated in Fig. 2 (c). We define the input full-precision activation as X f ∈ R H × W × C . We notice that HSI signals have different density and distribution along the spectral dimension due to the constraints of specific wavelengths. To adaptively fit this HSI nature, we propose to redistribute the HSI representations in channel wise before binarizing the activation as

Embedding

Conv Block

Conv Block

DownSample

DownSample

Mapping

Conv Block

Conv Block

UpSample

UpSample

Conv Block

𝐗"

𝐗#

𝐻×𝑊×2𝑁)

𝐻×𝑊×𝐶

𝐻

2

× 𝑊

2

×2𝐶

𝐻

4

× 𝑊

4

×4𝐶

(a) Base Model

𝐻×𝑊×𝐶

𝐻

2

× 𝑊

2

×2𝐶

𝐻

2

× 𝑊

2

×2𝐶

𝐻

4

× 𝑊

4

×4𝐶

𝐻×𝑊×𝐶

𝐻

2

× 𝑊

2

×2𝐶

𝐻×𝑊×𝐶

𝐻×𝑊×𝑁)

Bottleneck

Encoder

Decoder

x2

x2

(b) Conv Block

c

Fusion Down

c

Fusion Down

𝐌

c

shift

(c) Binarized Spectral-Redistribution Convolution

+

Average Pooling

BiSR-Conv

BiSR-Conv

c

(d) Binarized DownSample = Binarized Fusion Up + AvgPool

(e) Binarized UpSample = Binarized Fusion Down + Bilinear

+

Bilinear

Upscale

BiSR-Conv

BiSR-Conv

Sign(

𝐗-

)

Binarized

α𝐗-

)

𝐇

𝐘

+

𝐻×𝑊×𝐶

Full-precision Information

+ add

c

concatenation

backpropagation

Fusion Down

Fusion Up

ReLU

ReLU

Layer Norm

+

𝒌 2 𝐗 3 +𝒃

𝐗-

=

𝐗3

𝐘6

RPReLU

𝐗7

𝐇′

where X r ∈ R H × W × C denotes the redistributed activation of X f . k and b ∈ R C are learnable parameters. k rescales the density of HSIs while b shifts the bias. Then X r undergoes a Sign function to be binarized into 1-bit activation X b ∈ R H × W × C , where x b =+1 or -1 for ∀ x b ∈ X b as

where x r ∈ X r . As shown in Fig. 3 (b) and (c), since the Sign function is non-differentiable, previous methods either adopt a piecewise linear function Clip( x ) [33, 35, 36, 49, 64] or a piecewise quadratic function Quad( x ) [31, 32, 34] to approximate the Sign function during the backpropagation as

Nonetheless, the Clip function is a rough estimation and there is a large approximation error between Clip and Sign. The shaded areas in Fig. 3 reflect the differences between the Sign function and its approximations. The shaded area corresponding to the Clip function is 1. Besides, once the absolute values of weights or activations are outside the range of [ -1 , 1] , they are no longer updated. Although the piecewise quadratic function is a closer approximation (the shaded area is 2/3) than Clip, the above two problems have not been fundamentally resolved. To address the two issues, we redesign a scalable hyperbolic tangent function to approximate the Sign function in the backpropagation as

where α ∈ R + is a learnable parameter adaptively adjusting the distance between Tanh( αx ) and Sign( x ). e denotes the natural constant. We prove that when α → + ∞ , Tanh( αx ) → Sign( x ) as

Sign(𝑥)

∂Sign 𝑥

𝜕𝑥

Clip(𝑥)

𝜕𝑥

Quad(𝑥)

𝜕Quad 𝑥

𝜕𝑥

Tanh(𝛼𝑥)

𝜕Tanh 𝛼𝑥

𝜕𝑥

0

1

2

-1

-2

-1

1

-1

1

2

0

-1

-2

1

2

3

4

5

0

1

2

-1

-2

0

1

2

-1

-2

0

1

2

-1

-2

1

2

0

-1

-2

1

2

0

-1

-2

1

2

0

-1

-2

𝛼 = 5

𝛼 = 2

𝛼 = 1

𝛼 = 5

𝛼 = 2

𝛼 = 1

(a)

(b)

(c)

(d)

̸

If strictly following the mathematical definition, Sign(0) = 0 = ± 1 . However, in BNN, the weights and activations are binarized into 1-bit, i.e. , only two values ( ± 1 ). Hence, Sign(0) is usually set to -1 . Similar to this common setting, we also define lim α → + ∞ Tanh ( α · 0) = -1 in BNN. Then we have

We compute the area of the shaded region between our Tanh( αx ) and Sign( x ) in Fig. 3 (d) as

Different from previous Clip( x ) and Quad( x ), our Tanh( αx ) can arbitrarily reduce the approximation error with Sign( x ) when α in Eq. (8) is large enough. Besides, our Tanh( αx ) is neither piecewise nor unchanged when x is outside the range of [ -1 , 1] . On the contrary, the weights and activations can still be updated when their absolute values are larger than 1 . In addition, as depicted in the lower line of Fig. 3, the value ranges [0 , 1] and fixed shapes of ∂ Clip ( x ) ∂x and ∂ Quad ( x ) ∂x are fundamentally different from those of ∂ Sign ( x ) ∂x ∈ [0 , + ∞ ) . In contrast, our ∂ Tanh ( αx ) ∂x can change its value range (0 , α ) and shape by adapting the parameter α . It is more flexible and can approximate ∂ Sign ( x ) ∂x better.

In the binarized convolutional layer, the 32-bit weight W f is also binarized into 1-bit weight W b as

where E represents computing the mean value. Multiplying the mean absolute value of 32-bit weight value w f ∈ W f can narrow down the difference between binarized and full-precision weights. Subsequently, the computationally heavy operations of floating-point matrix multiplication in full-precision convolution can be replaced by pure logical XNOR and bit-count operations [49] as

where Y b represents the output and ∗ denotes the convolution operation. Since the value range of full-precision activation X f largely varies from that of 1-bit convolution output Y b , directly employing an identity mapping to aggregate them may cover up the information of Y b . To cope with this problem, we first fed Y b into a RPReLU [32] activation function to change its value range and then add it with X f by a residual connection to propagate full-precision information as

where X o denotes the output feature and RPReLU is formulated for the i -th channel of Y b as

(a) Binarized DownSample

Sign

Binarized Conv

k = 3, s = 2

RPReLU

Normal

𝐻×𝑊×𝐶

𝐻×𝑊×𝐶

𝐻

2

× 𝑊

2

×2𝐶

𝐻

2

× 𝑊

2

×2𝐶

Ours

k = 3, s = 1

BiSR-Conv

BiSR-Conv

k = 3, s = 1

c

𝐻×𝑊×𝐶

𝐻

2

× 𝑊

2

×𝐶

𝐻

2

× 𝑊

2

×𝐶

𝐻

2

× 𝑊

2

×𝐶

𝐻

2

× 𝑊

2

×𝐶

𝐻

2

× 𝑊

2

×2𝐶

AvgPool

k = 2, s = 2

Binarized Conv

k = 3, s = 1

Bilinear

Sign

RPReLU

𝐻×𝑊×𝐶

2𝐻×2𝑊×𝐶

2𝐻×2𝑊×𝐶

2𝐻×2𝑊×

𝐶

2

2𝐻×2𝑊×

𝐶

2

Normal

𝐻×𝑊×𝐶

2𝐻×2𝑊×𝐶

Bilinear

k = 3, s = 1

BiSR-Conv

k = 3, s = 1

BiSR-Conv

+

2𝐻×2𝑊×

𝐶

2

2𝐻×2𝑊×

𝐶

2

2𝐻×2𝑊×

𝐶

2

2𝐻×2𝑊×

𝐶

2

2𝐻×2𝑊×

𝐶

2

Ours

(b) Binarized UpSample

(c) Binarized Fusion Down

Sign

Binarized Conv

k = 1, s = 1

RPReLU

Normal

𝐻×𝑊×𝐶

𝐻×𝑊×𝐶

𝐻×𝑊× 𝐶

2

𝐻×𝑊× 𝐶

2

Ours

k = 1, s = 1

BiSR-Conv

BiSR-Conv

k = 1, s = 1

+

𝐻×𝑊×𝐶

𝐻×𝑊× 𝐶

2

𝐻×𝑊× 𝐶

2

𝐻×𝑊× 𝐶

2

𝐻×𝑊× 𝐶

2

𝐻×𝑊× 𝐶

2

(d) Binarized Fusion Up

Binarized Conv

k = 1, s = 1

Sign

RPReLU

𝐻×𝑊×𝐶

𝐻×𝑊×𝐶

𝐻×𝑊×2𝐶

𝐻×𝑊×2𝐶

Normal

k = 1, s = 1

BiSR-Conv

k = 1, s = 1

BiSR-Conv

𝐻×𝑊×𝐶

𝐻×𝑊×𝐶

𝐻×𝑊×2𝐶

Ours

𝐻×𝑊×𝐶

c

𝐻×𝑊×𝐶

𝐻×𝑊×𝐶

where y i ∈ R indicates single pixel values belonging to the i -th channel of Y b . β i , γ i , and ζ i ∈ R represents learnable parameters. Please note that the full-precision information is not blocked by the binarized convolutional layer in the proposed BiSR-Conv. Instead, it is propagated by the bypass identity mapping, as shown in the red arrow → in Fig. 2 (c). Based on this important property of BiSR-Conv, we design the four binarized convolutional modules, as illustrated in Fig. 2 (d) and (e).

3.3 Binarized Convolutional Modules

In model binarization, the identity mapping is critical to propagate full-precision information and ease the training procedure. The only source of full-precision information is the input end ( X s in Fig. 2) of the binarized part. However, the dimension mismatch during the feature downsampling, upsampling, and aggregation processes blocks the residual connections, which degrades the HSI reconstruction performance. To tackle this problem, we use BiSR-Conv to build up four binarized convolutional modules including downsample, upsample, fusion up, and fusion down with unblocked identity mappings to make sure the full-precision information can flow through all binarized layers.

Specifically, Fig. 4 compares the normal and our binarized modules. The red arrow ↓ indicates the full-precision information flow, while the blue arrow ↓ denotes the binarized signal flow. The downsample modules in Fig. 4 (a) downscale the input feature maps and double the channels. The upsample modules in Fig. 4 (b) upscale the input spatial dimension and half the channels. The fusion down modules in Fig. 4 (c) maintain the spatial size of the input feature and half the channels. The fusion up modules in Fig. 4 (d) keep the spatial dimension of the input feature maps while doubling the channels. In the normal modules, the full-precision information is blocked by the Sign function and binarized into 1-bit signal. Meanwhile, the intermediate feature maps are directly reshaped by the binarized convolutional layers. For example, in the normal binarized downsample module, the intermediate feature is directly reshaped from R H × W × C to R H 2 × W 2 × 2 C by a strided 1-bit conv 4 × 4. The spatial and channel dimension mismatch of the input and output feature maps impede the identity mapping to propagate full-precision information from previous layers. In contrast, our binarized modules rely on the proposed BiSR-Conv that has a bypass identity connection for full-precision information flow, as shown in Fig. 2 (c). By using channel-wise concatenating and splitting operations, the intermediate feature maps at the input and output ends of BiSR-Conv are free from being reshaped. Therefore, the full-precision information can flow through our binarized modules. Finally, we derive our BiSRNet by using BiSR-Conv and the four modules to binarize E , B , and D of the base model.

4 Experiment

4.1 Experimental Settings

Following [18, 26, 28, 29, 63], we select 28 wavelengths from 450nm to 650nm by using spectral interpolation manipulation to derive HSIs. We conduct experiments on simulation and real datasets.

Simulation Data. Two simulation datasets, CAVE [66] and KAIST [65], are adopted. The CAVE dataset provides 32 HSIs with a spatial size of 512 × 512. The KAIST dataset includes 30 HSIs at a spatial size of 2704 × 3376. We use CAVE for training and select 10 scenes from KAIST for testing.

Real Data. We adopt the five real HSIs captured by the CASSI system developed in [18] for testing.

Evaluation Metrics. The peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) are adopted as metrics to evaluate the HSI reconstruction performance. Similar to [31, 32, 33, 34, 35], the operations per second of BNN (OPs b ) is computed as OPs b = OPs f / 64 (OPs f = FLOPs) to measure the computational complexity and the parameters of BNN is calculated as Params b = Params f / 32, where the superscript b and f refer to the binarized and full-precision models. The total computational and memory costs of a model are computed as OPs = OPs b + OPs f and Params = Params b + Params f .

Implementation Details. The proposed BiSRNet is implemented by PyTorch [67]. We use Adam [68] optimizer ( β 1 = 0.9 and β 2 = 0.999) and Cosine Annealing [69] scheduler to train BiSRNet for 300 epochs on a single RTX 2080 GPU. Training samples are patches with spatial sizes of 256 × 256 and 96 × 96 randomly cropped from 28-channel 3D HSI data cubes for simulation and real experiments. The shifting step d is 2. The batch size is 2. We set the basic channel C = N λ = 28 to store HSI information. We use random flipping and rotation for data augmentation. The training loss function is the root mean square error (RMSE) between reconstructed and ground-truth HSIs.

4.2 Quantitative Results

We compare our BiSRNet with 12 SOTA algorithms, including seven 1-bit BNN-based methods (BiConnect [36], BNN [35], Bi-Real [34], IRNet [33], ReActNet [32], BTM [30], and BBCU [31]), two 32-bit full-precision CNN-based methods ( λ -Net [20] and TSA-Net [18]), and three 64-bit double-precision model-based methods (TwIST [59], GAP-TV [56], and DeSCI [53]). For a fair comparison, we set the Params and OPs of BNN-based methods to the same values.

Directly applying the seven SOTA BNN-based methods to HSI reconstruction task achieves unsatisfactory performance, ranging from 22.19 dB to 27.21 dB. Our BiSRNet surpasses these methods by large margins. More specifically, BiSRNet significantly outperforms BTM, BBCU, ReActNet, IRNet,

RGB Image

Measurement

Spectral Density Curves

BiConnect

BNN

Bi

-

Real

IRNet

ReActNet

BTM

BiSRNet

Ground Truth

476.5

nm

536.5

nm

575.5

nm

648.0

nm

450

500

550

600

650

Wavelength (nm)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Density

Ground Truth

BiSRNet, corr: 0.9938

BBCU, corr: 0.9813

IRNet, corr: 0.9802

BTM, corr: 0.9752

Bi-Real, corr: 0.9713

ReActNet, corr: 0.9681

BiConnect, corr: 0.9333

BNN, corr: 0.9067

BBCU

Bi-Real, BNN, and BiConnect by 2.55, 3.25, 3.39, 3.46, 3.50, 5.88, and 7.57 dB. This evidence suggests the significant effectiveness advantage of our BiSRNet in HSI restoration.

The proposed BiSRNet with extremely lower memory and computational complexity yields comparable results with 32-bit full-precision CNN-based methods. Surprisingly, our BiSRNet outperforms λ -Net by 1.23 dB while only costing 0.06 % (36/62640) Params and 1.0% (1.18/117.98) OPs. In contrast, the previous best BNN-based method BTM is still 1.33 dB lower than λ -Net. When compared with TSA-Net, our BiSRNet only uses 0.08% Params and 1.1% OPs but achieves 94.6% (29.76/31.46) performance. These results demonstrate the efficiency superiority of the proposed method.

The three model-based algorithms are implemented by MATLAB, where the default type of variable is double-precision floating-point number. Although they use more accurate 64-bit data type, our 1-bit BiSRNet dramatically outperforms DeSCI, GAP-TV, and TwIST by 4.06, 5.40, and 6.64 dB.

4.3 Qualitative Results

Simulation HSI Restoration. Fig. 5 depicts the simulation HSIs on Scene 1 with 4 out of 28 spectral channels reconstructed by the 7 SOTA BNN-based algorithms and BiSRNet. Previous BNNs are less favorable to restore HSI details. They generate blurry HSIs while introducing undesirable artifacts. In contrast, BiSRNet reconstructs more visually pleasing HSIs with more structural contents and sharper edges. Additionally, we plot the spectral density curves (bottom-left) corresponding to the selected regions of the green box in the RGB image (Top-left). BiSRNet achieves the highest correlation score with the ground truth, suggesting the advantage of BiSRNet in spectral-wise consistency restoration.

Real HSI Restoration. Fig. 6 visualizes the reconstructed HSIs of the seven SOTA BNN-based algorithms and our BiSRNet. We follow the setting of [18, 26, 28, 29, 63] to re-train the models with all samples of the CAVE and KAIST datasets. To simulate the noise disturbance in real imaging scenes, we inject 11-bit shot noise into measurements during training. It can be observed that our BiSRNet is more effective in detailed content reconstruction and real imaging noise suppression.

4.4 Ablation Study

Break-down Ablation. We adopt baseline-1 to conduct a break-down ablation towards higher performance. Baseline-1 is derived by using vanilla 1-bit convolution to replace BiSR-Conv and normal binarized convolutional modules (see Fig. 4) to replace our binarized convolutional modules. As shown in Tab. 2a, baseline-1 yields 23.90 dB in PSNR and 0.594 in SSIM. When we apply BiSR-Conv, the model achieves 3.90 dB improvement. Then we successively use our binarized downsample (BiDS), upsample (BiUS), fusion down (BiFD), and fusion up (BiFU) modules, the model gains by 1.96 dB in total. These results verify the effectiveness of the proposed techniques.

BiSR-Conv. To study the effects of BiSR-Conv components, we adopt baseline-2 to conduct an ablation. Baseline-2 is obtained by removing Spectral-Redistribution (SR) operation and Sign approximation Tanh( αx ) from BiSRNet. As reported in Tab. 2b, baseline-2 achieves 27.68 dB in PSNR and 0.723 in SSIM. When we respectively apply SR and Tanh( αx ), baseline-2 gains by 1.29 and 1.06 dB. When we exploit SR and Tanh( αx ) jointly, the model achieves 2.08 dB improvement.

RGB Image

Measurement

BiConnect

BNN

Bi

-

Real

IRNet

ReActNet

BTM

BiSRNet

536.5

nm

567.5

nm

594.5

nm

648.0

nm

BBCU

8

(d) Ablation study of binarizing different parts of the base model

Sign Approximation. We compare our scalable hyperbolic tangent function with previous Sign approximation functions. The experimental results are listed in Tab. 2c. Our Tanh( αx ) dramatically surpasses the piecewise linear function Clip( x ) and quadratic function Quad( x ) by 0.79 and 0.74 dB, suggesting the superiority of the proposed Tanh( αx ). This advantage can be explained by the analysis in Sec. 3.2 that our Tanh( αx ) is more flexible and can adaptively reduce the difference with Sign( x ).

Binarizing Different Parts. We binarize one part of the base model while keeping the other parts full-precision to study the binarization benefit. The experimental results are reported in Tab. 2d. The base model yields 34.11 dB in PSNR and 0.936 in SSIM while costing 10.52 G OPs and 634 K Params. It can be observed from Tab. 2d: (i) Binarizing the bottleneck B reduces the Params the most (270174) with the smallest performance drop (only 0.31 dB). (ii) Binarizing the decoder D achieves the largest OPs reduction (4927 M) while the performance degrades by a moderate margin (1.08 dB).

5 Conclusion

In this paper, we propose a novel BNN-based method BiSRNet for binarized HSI restoration. To the best of our knowledge, this is the first work to study the binarized spectral compressive imaging reconstruction problem. We first redesign a compact and easy-to-deploy base model with simple computation operations. Then we customize the basic unit BiSR-Conv for model binarization. BiSRConv can adaptively adjust the density and distribution of HSI representations before binarizing the activation. Besides, BiSR-Conv employs a scalable hyperbolic tangent function to closer approach Sign by arbitrarily reducing the approximation error. Subsequently, we use BiSR-Conv to build up four binarized convolutional modules that can handle the dimension mismatch issue during feature reshaping and propagate full-precision information through all layers. Comprehensive quantitative and qualitative experiments demonstrate that our BiSRNet significantly outperforms SOTA BNNs and even achieves comparable performance with full-precision CNN-based HSI reconstruction algorithms.

References

A. A. Gowen, C. P. O'Donnell, P. J. Cullen, G. Downey, and J. M. Frias, 'Hyperspectral imaging-an emerging process analytical tool for food quality and safety control,' Trends in food science & technology , 2007.

D. Lorente, N. Aleixos, J. Gómez-Sanchis, S. Cubero, O. L. García-Navarrete, and J. Blasco, 'Recent advances and applications of hyperspectral imaging for fruit and vegetable quality assessment,' Food and Bioprocess Technology , 2012.

R. Lu and Y.-R. Chen, 'Hyperspectral imaging for safety inspection of food and agricultural products,' in Pathogen Detection and Remediation for Safe Eating , 1999.

V. Backman, M. B. Wallace, L. Perelman, J. Arendt, R. Gurjar, M. Muller, Q. Zhang, G. Zonios, E. Kline, and T. McGillican, 'Detection of preinvasive cancer cells,' Nature , 2000.

G. Lu and B. Fei, 'Medical hyperspectral imaging: a review,' Journal of Biomedical Optics , 2014.

Z. Meng, M. Qiao, J. Ma, Z. Yu, K. Xu, and X. Yuan, 'Snapshot multispectral endomicroscopy,' Optics Letters , 2020.

B. Uzkent, M. J. Hoffman, and A. Vodacek, 'Real-time vehicle tracking in aerial video using hyperspectral features,' in CVPRW , 2016.

B. Uzkent, A. Rangnekar, and M. Hoffman, 'Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps,' in CVPRW , 2017.

M. H. Kim, T. A. Harvey, D. S. Kittle, H. Rushmeier, J. Dorsey, R. O. Prum, and D. J. Brady, '3d imaging spectroscopy for measuring hyperspectral patterns on solid objects,' TOG , 2012.

M. Borengasser, W. S. Hungate, and R. Watkins, 'Hyperspectral remote sensing: principles and applications,' CRC press , 2007.

F. Melgani and L. Bruzzone, 'Classification of hyperspectral remote sensing images with support vector machines,' IEEE Transactions on Geoscience and Remote Sensing , 2004.

Y. Yuan, X. Zheng, and X. Lu, 'Hyperspectral image superresolution by transfer learning,' IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2017.

P. Llull, X. Liao, X. Yuan, J. Yang, D. Kittle, L. Carin, G. Sapiro, and D. J. Brady, 'Coded aperture compressive temporal imaging,' Optics Express , 2013.

A. Wagadarikar, R. John, R. Willett, and D. Brady, 'Single disperser design for coded aperture snapshot spectral imaging,' Applied Optics , 2008.

A. A. Wagadarikar, N. P. Pitsianis, X. Sun, and D. J. Brady, 'Video rate spectral imaging using a coded aperture snapshot spectral imager,' Optics Express , 2009.

X. Yuan, T.-H. Tsai, R. Zhu, P. Llull, D. Brady, and L. Carin, 'Compressive hyperspectral imaging with side information,' IEEE Journal of selected topics in Signal Processing , 2015.

X. Ma, X. Yuan, C. Fu, and G. R. Arce, 'Led-based compressive spectral-temporal imaging,' Optics Express , 2021.

Z. Meng, J. Ma, and X. Yuan, 'End-to-end low cost compressive spectral imaging with spatialspectral self-attention,' in ECCV , 2020.

M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, 'Single-shot compressive spectral imaging with a dual-disperser architecture,' Optics express , 2007.

X. Miao, X. Yuan, Y. Pu, and V. Athitsos, 'l-net: Reconstruct hyperspectral images from a snapshot measurement,' in ICCV , 2019.

L. Wang, C. Sun, M. Zhang, Y. Fu, and H. Huang, 'Dnu: Deep non-local unrolling for computational spectral imaging,' in CVPR , 2020.

L. Wang, C. Sun, Y. Fu, M. H. Kim, and H. Huang, 'Hyperspectral image reconstruction using a deep spatial-spectral prior,' in CVPR , 2019.

J. Ma, X.-Y. Liu, Z. Shou, and X. Yuan, 'Deep tensor admm-net for snapshot compressive imaging,' in ICCV , 2019.

X. Hu, Y. Cai, J. Lin, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, 'Hdnet: High-resolution dual-domain learning for spectral compressive imaging,' in CVPR , 2022.

Z. Meng, Z. Yu, K. Xu, and X. Yuan, 'Self-supervised neural networks for spectral snapshot compressive imaging,' in ICCV , 2021.

Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, 'Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction,' in CVPR , 2022.

Y. Cai, J. Lin, Z. Lin, H. Wang, Y. Zhang, H. Pfister, R. Timofte, and L. V. Gool, 'Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction,' in CVPRW , 2022.

Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, 'Coarse-to-fine sparse transformer for hyperspectral image reconstruction,' in ECCV , 2022.

Y. Cai, J. Lin, H. Wang, X. Yuan, H. Ding, Y. Zhang, R. Timofte, and L. V. Gool, 'Degradationaware unfolding half-shuffle transformer for spectral compressive imaging,' in NeurIPS , 2022.

X. Jiang, N. Wang, J. Xin, K. Li, X. Yang, and X. Gao, 'Training binary neural network without batch normalization for image super-resolution,' in AAAI , 2021.

B. Xia, Y. Zhang, Y. Wang, Y. Tian, W. Yang, R. Timofte, and L. V. Gool, 'Basic binary convolution unit for binarized image restoration network,' in ICLR , 2023.

Z. Liu, Z. Shen, M. Savvides, and K.-T. Cheng, 'Reactnet: Towards precise binary neural network with generalized activation functions,' in ECCV , 2020.

H. Qin, R. Gong, X. Liu, M. Shen, Z. Wei, F. Yu, and J. Song, 'Forward and backward information retention for accurate binary neural networks,' in CVPR , 2020.

Z. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng, 'Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm,' in ECCV , 2018.

I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv, and Y. Bengio, 'Binarized neural networks,' in NeurIPS , 2016.

M. Courbariaux, Y. Bengio, and J.-P. David, 'Binaryconnect: Training deep neural networks with binary weights during propagations,' in NeurIPS , 2015.

V. Sze, Y.-H. Chen, T.-J. Yang, and J. S. Emer, 'Efficient processing of deep neural networks: A tutorial and survey,' Proceedings of the IEEE , 2017.

H. Qin, R. Gong, X. Liu, X. Bai, J. Song, and N. Sebe, 'Binary neural networks: A survey,' Pattern Recognition , 2020.

J. Xin, N. Wang, X. Jiang, J. Li, H. Huang, and X. Gao, 'Binarized neural network for single image super resolution,' in ECCV , 2020.

H. Wang and Y. Fu, 'Trainability preserving neural structured pruning,' in ICLR , 2023.

Y. Zhang, H. Wang, C. Qin, and Y. Fu, 'Learning efficient image super-resolution networks via structure-regularized pruning,' in ICLR , 2022.

H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. P. Graf, 'Pruning filters for efficient convnets,' in ICLR , 2017.

H. Wang, S. Lohit, M. Jones, and Y. Fu, 'What makes a "good" data augmentation in knowledge distillation - a statistical perspective,' in NeurIPS , 2022.

G. Hinton, O. Vinyals, and J. Dean, 'Distilling the knowledge in a neural network,' in NeurIPSW , 2014.

X. Zhang, X. Zhou, M. Lin, and J. Sun, 'Shufflenet: An extremely efficient convolutional neural network for mobile devices,' in CVPR , 2018.

N. Ma, X. Zhang, H.-T. Zheng, and J. Sun, 'Shufflenet v2: Practical guidelines for efficient cnn architecture design,' in ECCV , 2018.

A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, 'Mobilenets: Efficient convolutional neural networks for mobile vision applications,' arXiv preprint arXiv:1704.04861 , 2017.

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, 'Mobilenetv2: Inverted residuals and linear bottlenecks,' in CVPR , 2018.

M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi, 'Xnor-net: Imagenet classification using binary convolutional neural networks,' in ECCV , 2016.

R. Ding, T.-W. Chin, Z. Liu, and D. Marculescu, 'Regularizing activation distribution for training binarized deep networks,' in CVPR , 2019.

J. Zhang, Y. Pan, T. Yao, H. Zhao, and T. Mei, 'dabnn: A super fast inference framework for binary neural networks on arm devices,' in ACM MM , 2019.

D. Kittle, K. Choi, A. Wagadarikar, and D. J. Brady, 'Multiframe image estimation for coded aperture snapshot spectral imagers,' Applied optics , 2010.

Y. Liu, X. Yuan, J. Suo, D. Brady, and Q. Dai, 'Rank minimization for snapshot compressive imaging,' TPAMI , 2019.

L. Wang, Z. Xiong, G. Shi, F. Wu, and W. Zeng, 'Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging,' TPAMI , 2016.

S. Zhang, L. Wang, Y. Fu, X. Zhong, and H. Huang, 'Computational hyperspectral imaging based on dimension-discriminative low-rank tensor recovery,' in ICCV , 2019.

X. Yuan, 'Generalized alternating projection based total variation minimization for compressive sensing,' in ICIP , 2016.

J. Tan, Y. Ma, H. Rueda, D. Baron, and G. R. Arce, 'Compressive hyperspectral imaging via approximate message passing,' IEEE Journal of Selected Topics in Signal Processing , 2016.

M. A. Figueiredo, R. D. Nowak, and S. J. Wright, 'Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems,' IEEE Journal of selected topics in signal processing , 2007.

J. Bioucas-Dias and M. Figueiredo., 'A new twist: Two-step iterative shrinkage/thresholding algorithms for image restoration.,' TIP , 2007.

J. Yang, X. Liao, X. Yuan, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, 'Compressive sensing by learning a gaussian mixture model from measurements,' TIP , 2014.

J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, 'Video compressive sensing using gaussian mixture models,' TIP , 2014.

X. Zhang, Y. Zhang, R. Xiong, Q. Sun, and J. Zhang, 'Herosnet: Hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging,' in CVPR , 2022.

T. Huang, W. Dong, X. Yuan, J. Wu, and G. Shi, 'Deep gaussian scale mixture prior for spectral compressive imaging,' in CVPR , 2021.

X. Lin, C. Zhao, and W. Pan, 'Towards accurate binary convolutional neural network,' in NeurIPS , 2017.

I. Choi, D. S. Jeon, G. Nam, D. Gutierrez, and M. H. Kim, 'High-quality hyperspectral reconstruction using a spectral prior,' TOG , 2017.

J.-I. Park, M.-H. Lee, M. D. Grossberg, and S. K. Nayar, 'Multispectral imaging using multiplexed illumination,' in ICCV , 2007.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, 'Pytorch: An imperative style, high-performance deep learning library,' in NeurIPS , 2019.

D. P. Kingma and J. L. Ba, 'Adam: A method for stochastic optimization,' in ICLR , 2015.

I. Loshchilov and F. Hutter, 'Sgdr: Stochastic gradient descent with warm restarts,' in ICLR , 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a deep neural network with binarized operations for low-cost spectral compressive imaging of hyperspectral images (HSI). The network introduces a basic unit for model binarization which adaptively redistributes HSI representations before binarization activation and uses a scalable hyperbolic tangent function to approximate the Sign function in backpropagation. Four binarization convolution modules are designed to solve the dimensional mismatch problem during feature reshaping and propagate full precision information throughout the network. Experiments on both synthetic and real data are conducted for performance evaluation.\n\nSTRENGTHS:\n1. An effective binarized deep neural network is proposed for HSI reconstruction.\n2. Four binarized convolution modules are designed to enable the propagation of full precision information through all convolution layers.\n3. Based on the ablation experiments in the article, the proposed scalable hyperbolic tangent function provides inspiration for approximating the symbolic function in back propagation.\n\nWEAKNESSES:\n1. The proposed binarization scheme is general, without specific design optimized for HSI restoration. Then, it lacks comparison to other general binarized NNs or comparison in other reconstruction tasks.\n2. The proposed scheme is quite straightforward and engineering.\n3. Approximation error bounds are not discussed, but this is quite important for binarized NNs.\n4. Some key technical parts are not clearly/detailly described.\n\nQUESTIONS:\n1. The paper argues that, as mobile devices are more and more widely used, the demands of running and storing HSI restoration models on edge devices grow significantly. Could the authors provide practical cases where mobile edge devices are used for HSI restoration?\n\n2.  In the binarized convolution layer, the formula for converting the model parameters from 32 bits to 1 bit is not clearly written. the average of the 32-bit absolute value multiplied by the sign function cannot be turned into a 1-bit number.\n\n3. It is not clear how the proposed method deals with the dimensional mismatch feature remodeling process, and how to solve the dimensional mismatch with binarized convolution modules.\n\nLIMITATIONS:\nApproximation error bounds of the proposed binarization scheme are not discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method, Binarized Spectral-Redistribution Network (BiSRNet), for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems. This paper redesigns the base model and presents the basic unit (BiSR-Conv) for model binarization. Specifically, this convolutional layer of BiSRNet is tailored for hyperspectral image processing. Comprehensive quantitative and qualitative experiments show that the proposed BiSR-Net outperforms state-of-the-art binarization methods and achieves comparable performance with full-precision algorithms.\n\nSTRENGTHS:\n(1) This paper redesigns a U-Net consisting of the four BCMs as shown in Figure 4 to let the full-precision information flow pass by the all network. This point is critical since previous BNNs do not consider all the situations in feature reshaping. Then in the basic unit, BiSR-Conv, with the insight of treating different HSIs with different densities and distribution, this paper proposes to shift HSIs before binarization to allow more binarized HSI activation. \n\n(2) Comprehensive experiments have been conducted including synthetic and real experiments to demonstrate the superiority of BiSRNet. The ablation study is also extensive. The performance of BiSRNet is impressive, which surpasses existing SOTA BNNs by huge margins, over 2.5 dB.\n\nWEAKNESSES:\n(1) Some details are missing. For example, how did you get the 3D mask $\\mathbf{M}^*$? I remember the coded aperture used in CASSI is 2D. Its shape should be $H \\times W$. Why does it have an additional dimension here? And how?\n\n(2) Some critical experimental analyses are lacking. For instance, at the end of section 4.4, “Binarizing Different Parts”, why binarizing the bottleneck can reduce the most parameters? And why binarizing the decoder can achieve the largest Ops reduction? An analysis should be provided to explain this.\n\n(3) I remember Binary Connect [39] binarizes the weights of CNN. But it seems that the activations of Binary Connect [39] are also binarized in Table 1. Why did you do that? Could you please explain this?\n\nQUESTIONS:\nPlease refer to Weaknesses\n\nLIMITATIONS:\nThis paper proposes BiSRNet for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems. But this paper has not discussed the limitations of the proposed method. I suggest the author discuss the potential and limitations of the proposed method in more detail, which will make the contributions of the proposed method more significant.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on studying the binarized spectral compressive imaging reconstruction problem. A Binarized Spectral-Redistribution Network (BiSRNet) is proposed. The authors first design a basic U-Net as the base model to begin the binarization. Then a Binarized Spectral-Redistribution Convolutional (BiSR-Conv) unit is proposed to replace the 32-bit convolutional layer of the base model to derive BiSRNet. The BiSR-Conv has two advantages than vanilla 1-bit convolutional layer: (i) can redistribute the spectral distributions before binarization. (ii) has a scalable hyperbolic tangent function to approach the Sign function more closely.\n\nSTRENGTHS:\nThe novelty and motivation are good. First of all, the research topic is new. Nobody explores binarized spectral compressive imaging before. The authors not only contribute a good method but also conduct experiments using before BNNs designed for other topics. We should respect this. Secondly, the redesigned base model is well motivated. The architecture shows the insight of “full-precision information flow”. The idea of redistributing HSI representations before binarization is interesting and reasonable. The Tanh(\\alpha x) is also amazing. Flexibly controlling the gap with Sign function is really cool.\n\n+ The performance is good and solid. The proposed BiSRNet outperforms the SOTA BNNs widely used in image classification. Meanwhile, the computational and memory complexity of BiSRNet is much lower than full-precision CNNs while the performance is comparable. In the real HSI experiments, it seems that the proposed BiSCI performs better than CNN-based methods in noise suppression. \n\n+ The experiments are comprehensive, not only quantitative but also qualitative comparisons shows the advantages of the proposed method. The ablation study is sufficient to verify the effectiveness of the proposed techniques.\n\n+ The codes and pre-trained weights are provided in the supplementary. The reproducibility can be checked and ensured.\n\n+ The writing is good and easy-to-follow. The figures, tables, and mathematical notations are very clear.\n\nWEAKNESSES:\n- The definition is a little confusing. Sign(0) = 0 and Tanh(0) = 0. But in this paper, Sign(0) and Tanh(0) is defined as -1, as shown in Eq.(3) and line 152. Why? More explanation should be provided.\n\n- Although the proposed BNN achieves good performance with CNN. But compared with Transformer-based methods, there is a large gap. I understand that binarizing Transformer is not easy because of the self-attention mechanism. But it is good idea to keep trying.\n\nQUESTIONS:\nWill you also plan to release the code of other BNNs compared in Table 1?\n\nLIMITATIONS:\nSee the above comments", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

comments: SUMMARY:
In this paper, binarized neural network is first utilized in hyperspectral image reconstruction. For model binarization, authors propose the Binarized Spectral-Redistribution Convolution (BiSR-Conv), which adaptively redistributes the HSI representations before binarizing activation. Since the Sign function is non-differentiable, a scalable hyperbolic tangent function is applied in backpropagation, which has less approximation error. In BiSR-Conv, the additional residual connection encourages the full-precision information to propagate throughout the whole network. Based on BiSR-Conv, four binarized convolutional modules are designed to address the dimension mismatch issue during feature reshaping.

STRENGTHS:
An application to a new domain;
Clear and consecutive writing;
A feasible idea for deploying spectral compression imaging in edge devices.

WEAKNESSES:
Experiments are insufficient. How does the ‘full-precision information’ affect the whole network? Is the ‘full-precision information’ play a dominant role in BiSR-Conv or not?

QUESTIONS:
In Sign Approximation experiment, the proposed scalable hyperbolic tangent function and previous Sign approximation functions should be compared in other BNNs, to further verify the generalization of the proposed function.

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper proposes a Binarized Neural Network based approach known as BiSRNet for binarized HSI restoration. The main motivation of the paper stems from the fact that any CNN or transformer-based architectures are computationally heavy for devices with low computing power and hence need extremely fast and light weight networks like binary neural networks. This paper redesigns the binary conv layer (BiSR-Conv) to exploit the distributed nature of HSI representations. They also employed a scalable Tanh function to decrease the approximation error. Overall, this paper is well-written and reasons well for various changes to the design of BCNNs.

STRENGTHS:
- Well-motivated problem and design of the solution
- Good empirical evidence
- Good presentation 
- Clear to understand

WEAKNESSES:
- Main weakness of this paper is their redesigned conv modules are only applicable to the HSI domain? If so this limits the scope and impact of this work.
- Lack of theory

QUESTIONS:
- I am not very familiar with HSI work so I do not have many questions at this point.

LIMITATIONS:
Please see the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06