SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuanhao Cai<sup>1</sup>, Yuxin Zheng<sup>1</sup>, Jing Lin<sup>1</sup>, Xin Yuan<sup>2</sup>, Yulun Zhang<sup>3,*</sup>, Haoqian Wang<sup>1,*</sup>
$^{1}$  Tsinghua University,  $^{2}$  Westlake University,  $^{3}$  ETH Zürich
Existing deep learning models for hyperspectral image (HSI) reconstruction achieve good performance but require powerful hardwares with enormous memory and computational resources. Consequently, these methods can hardly be deployed on resource-limited mobile devices. In this paper, we propose a novel method, Binarized Spectral-Redistribution Network (BiSRNet), for efficient and practical HSI restoration from compressed measurement in snapshot compressive imaging (SCI) systems. Firstly, we redesign a compact and easy-to-deploy base model to be binarized. Then we present the basic unit, Binarized Spectral-Redistribution Convolution (BiSR-Conv). BiSR-Conv can adaptively redistribute the HSI representations before binarizing activation and uses a scalable hyperbolic tangent function to closer approximate the Sign function in backpropagation. Based on our BiSR-Conv, we customize four binarized convolutional modules to address the dimension mismatch and propagate full-precision information throughout the whole network. Finally, our BiSRNet is derived by using the proposed techniques to binarize the base model. Comprehensive quantitative and qualitative experiments manifest that our proposed BiSRNet outperforms state-of-the-art binarization algorithms. Code and models are publicly available at https://github.com/caiyuanhao1998/BiSCI
Compared to normal RGB images, hyperspectral images (HSIs) have more spectral bands to capture richer information of the desired scenes. Thus, HSIs have wide applications in agriculture [1, 2, 3], medical image analysis [4, 5, 6], object tracking [7, 8, 9], remote sensing [10, 11, 12], etc.
To capture HSIs, conventional imaging systems leverage 1D or 2D spectrometers to scan the desired scenes along the spatial or spectral dimension. Yet, this process is very time-consuming and thus fails in measuring dynamic scenes. In recent years, snapshot compressive imaging (SCI) systems [13, 14, 15, 16, 17] have been developed to capture HSI cubes in real time. Among these SCI systems, the coded aperture snapshot spectral imaging (CASSI) [14, 18, 19] demonstrates its outstanding effectiveness and efficiency. The CASSI systems firstly employ a coded aperture (physical mask) to modulate the 3D HSI cube, then use a disperser to shift spectral information of different wavelengths, and finally integrate these HSI signals on a detector array to capture a 2D compressed measurement. We study the inverse problem, i.e., restoring the original 3D HSI cube from the 2D measurement.
Existing state-of-the-art (SOTA) SCI reconstruction methods are based on deep learning. Convolutional neural network (CNN) [18, 20, 21, 22, 23, 24, 25] and Transformer [26, 27, 28, 29] have been used to implicitly learn the mapping from compressed measurements to HSIs. Although superior performance is achieved, these CNN-/Transformer-based methods require powerful hardwares with abundant computing and memory resources, such as high-end graphics processing units (GPUs). However, edge devices (e.g., mobile phones, hand-held cameras, small drones, etc.) evidently cannot meet the requirements of these expensive algorithms because edge devices have very limited memory, computational power, and battery. As mobile devices are more and more widely used, the demands of running and storing HSI restoration models on edge devices grow significantly. This motivates us to reduce the memory and computational burden of HSI reconstruction methods while preserving the performance as much as possible so that the algorithms can be deployed on resource-limited devices.
* Yulun Zhang and Haoqian Wang are the corresponding authors.
Figure 1: Comparison between our BiSRNet (in red color) and state-of-the-art BNNs (in blue color). BiSRNet significantly advances BTM [30], BBCU [31], ReActNet [32], IRNet [33], Bi-Real [34], BNN [35], and BiConnect [36] by 2.55, 3.25, 3.39, 3.46, 3.50, 5.88, and 7.57 dB on the simulation HSI reconstruction task.
The studies on neural network compression and acceleration [37, 38] can be divided into four categories: quantization [31, 32, 33, 34, 35, 39], pruning [40, 41, 42], knowledge distillation [43, 44], and compact network design [45, 46, 47, 48]. Among these methods, binarized neural network (BNN) belonging to quantization stands out because it can extremely compress the memory and computational costs by quantizing the weights and activations of CNN to only 1 bit. In particular, BNN could achieve  $32 \times$  memory compression ratio and up to  $58 \times$  practical computational reduction on central processing units (CPUs) [49]. In addition, the pure logical computation (i.e., XNOR and bit-count operations) of BNN is highly energy-efficient for embedded devices [50, 51]. However, directly applying model binarization for HSI reconstruction algorithms may encounter three issues. (i) The HSI representations have different density and distribution in different spectral bands. Equally binarizing the activations of different spectral channels may lead to the collapse of HSI features. (ii) Previous model binarization methods mainly adopt a piecewise linear [33, 35, 36] or quadratic [31, 32, 34] function to approximate the non-differentiable Sign function. Nonetheless, there still remain large approximation errors between them and Sign. (iii) How to tackle the dimension mismatch problem during feature reshaping while allowing full-precision information propagation in BNN has not been fully explored. Previous model binarization methods [32, 33, 34, 35, 36] mainly consider the feature downsampling situation in a backbone network for high-level vision tasks.
Bearing the above considerations in mind, we propose a novel BNN-based method, namely Binarized Spectral-Redistribution Network (BiSRNet) for efficient and practical HSI reconstruction. Firstly, we redesign a compact and easy-to-deploy base model to be binarized. Different from previous CNN/Transformer-based methods, this base model does not include complex computations like unfolding inference and non-local self-attention that are difficult to implement on edge devices. Instead, our base model only uses convolutional units that can be easily replaced by XNOR and bit-count logical operations on resource-limited devices. Secondly, we develop the basic unit, Binarized Spectral-Redistribution Convolution (BiSR-Conv), used in model binarization. Specifically, BiSR-Conv can adapt the density and distribution of HSI representations in spectral dimension before binarizing the activation. Besides, BiSR-Conv employs a scalable hyperbolic tangent function to closer approximate the non-differentiable Sign function by arbitrarily reducing the approximation error. Thirdly, as the full-precision information is very critical in BNN and the input HSI is the only full-precision source, we use BiSR-Conv to build up four binarized convolutional modules that can handle the dimension mismatch issue during feature reshaping and simultaneously propagate full-precision information through all layers. Finally, we derive our BiSRNet by using the proposed techniques to binarize the base model. As shown in Fig. 1, BiSRNet outperforms SOTA BNNs by large margins, over 2.5 dB.
In a nutshell, our contributions can be summarized as follows:
(i) We propose a novel BNN-based algorithm BiSRNet for HSI reconstruction. To the best of our knowledge, this is the first work to study the binarized spectral compressive imaging problem.
(ii) We customize a new binarized convolution unit BiSR-Conv that can adapt the density and distribution of HSI representations and approximate the Sign function better in backpropagation.
(iii) We design four binarized convolutional modules to address the dimension mismatch issue during feature reshaping and propagate full-precision information through all convolutional layers.
(iv) Our BiSRNet dramatically surpasses SOTA BNNs and even achieves comparable performance with full-precision CNNs while requiring extremely lower memory and computational costs.
2.1 Hyperspectral Image Reconstruction
Traditional HSI reconstruction methods [14, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61] are mainly based on hand-crafted image priors. Yet, these traditional methods achieve unsatisfactory performance and generality due to their poor representing capacity. Recently, deep CNN [18, 20, 21, 22, 23, 24, 25, 62] and Transformer [26, 27, 28, 29] have been employed as powerful models to learn the underlying mapping from compressed measurements to HSI data cubes. For example, TSA-Net [18] employs three spatial-spectral self-attention layers at the decoder of a U-shaped CNN. Cai et al. propose a series of Transformer-based algorithms (MST [26], MST++ [27], CST [28], and DAUHST [29]), pushing the performance boundary from  $32\mathrm{dB}$  to  $38\mathrm{dB}$ . Although impressive results are achieved, these CNN-/Transformer-based methods rely on powerful hardwares with enormous computational and memory resources, which are unaffordable for mobile devices. How to develop HSI restoration algorithms toward resource-limited platforms is under-explored. Our goal is to fill this research gap.
2.2 Binarized Neural Network
BNN [35] is the extreme case of model quantization as it quantizes the weights and activations into only 1 bit. Due to its impressive effectiveness in memory and computation compression, BNN has been widely applied in high-level vision [32, 33, 34, 35, 49] and low-level vision [30, 31, 39]. For example, Jiang et al. [30] train a BNN without batch normalization for image super-resolution. Xia et al. [31] design a binarized convolution unit BBCU for image super-resolution, denoising, and JPEG compression artifact reduction. Yet, the potential of BNN for SCI reconstruction has not been studied.
3.1 Base Model
The full-precision model to be binarized should be compact and its computation should be easy to deploy on edge devices. However, previous CNN-/Transformer-based algorithms are computationally expensive or have large model sizes. Some of them exploit complex operations like unfolding inference [21, 22, 23, 29, 63] and non-local self-attention computation [18, 20, 26, 27, 28] that are challenging to binarize and difficult to implement on mobile devices. Hence, we redesign a simple, compact, and easy-to-deploy base model without using complex computation operations.
Inspired by the success of MST [26] and CST [28], we adopt a U-shaped structure for the base model as shown in Fig. 2. It consists of an encoder  $\mathcal{E}$ , a bottleneck  $\mathcal{B}$ , and a decoder  $\mathcal{D}$ . Please refer to the supplementary for the CASSI mathematical model. Firstly, we reverse the dispersion of CASSI by shifting back the measurement  $\mathbf{Y} \in \mathbb{R}^{H \times (W + d(N_{\lambda} - 1)) \times N_{\lambda}}$  to derive the input  $\mathbf{H} \in \mathbb{R}^{H \times W \times N_{\lambda}}$  as
$$
\mathbf {H} (x, y, n _ {\lambda}) = \mathbf {Y} (x, y - d (\lambda_ {n} - \lambda_ {c})), \tag {1}
$$
where  $H$ ,  $W$ , and  $N_{\lambda}$  denote the HSI's height, width, and number of wavelengths.  $d$  represents the shifting step. The concatenation of  $\mathbf{H}$  and the 3D mask  $\mathbf{M} \in \mathbb{R}^{H \times W \times N_{\lambda}}$  is fed into a feature embedding module to produce the shallow feature  $\mathbf{X}_s \in \mathbb{R}^{H \times W \times N_{\lambda}}$ . The feature embedding module is a conv1  $\times 1$  (convolutional layer with kernel size  $= 1 \times 1$ ). Subsequently,  $\mathbf{X}_s$  undergoes the encoder  $\mathcal{E}$ , bottleneck  $\mathcal{B}$ , and decoder  $\mathcal{D}$  to generate the deep feature  $\mathbf{X}_d \in \mathbb{R}^{H \times W \times N_{\lambda}}$ .  $\mathcal{E}$  consists of two convolutional blocks and two downsample modules. The details of the convolutional block are depicted in Fig. 2 (b). The fusion up and down modules are both conv1  $\times 1$  to aggregate the feature maps and modify the channels. The downsample module is a strided conv4  $\times 4$  layer that downscales the feature maps and doubles the channels.  $\mathcal{B}$  is a convolutional block.  $\mathcal{D}$  consists of two convolutional blocks and two upsample modules. The upsample module is a bilinear interpolation followed by a conv3  $\times 3$  to upscale the feature maps and halve the channels. Skip connections between  $\mathcal{E}$  and  $\mathcal{D}$  are employed to alleviate the information loss during rescaling. Finally, the sum of  $\mathbf{X}_s$  and  $\mathbf{X}_d$  is fed into the feature mapping module (conv1  $\times 1$ ) to produce the reconstructed HSI  $\mathbf{H}' \in \mathbb{R}^{H \times W \times N_{\lambda}}$ .
3.2 Binarized Spectral-Redistribution Convolution
The details of BiSR-Conv are illustrated in Fig. 2 (c). We define the input full-precision activation as  $\mathbf{X}_f\in \mathbb{R}^{H\times W\times C}$ . We notice that HSI signals have different density and distribution along the spectral dimension due to the constraints of specific wavelengths. To adaptively fit this HSI nature, we propose to redistribute the HSI representations in channel wise before binarizing the activation as
$$
\mathbf {X} _ {r} = \boldsymbol {k} \cdot \mathbf {X} _ {f} + \boldsymbol {b}, \tag {2}
$$
(b) Conv Block
(a) Base Model
(c) Binarized Spectral-Redistribution Convolution
(d) Binarized DownSample = Binarized Fusion Up + AvgPool
(e)Binarized UpSample  $=$  Binarized Fusion Down  $^+$  Bilinear
Figure 2: The overall diagram of our method. (a) The proposed base model to be binarized adopts a U-shaped architecture. (b) The components of the convolutional block. (c) The details of our Binarized Spectral-Redistribution Convolution (BiSR-Conv). (d) The structure of our binarized downsample module. The binarized fusion up module is derived by removing the average pooling operation. (e) The architecture of our binarized upsample module, which includes one more bilinear upscaling operation than the binarized fusion down module.
where  $\mathbf{X}_r\in \mathbb{R}^{H\times W\times C}$  denotes the redistributed activation of  $\mathbf{X}_f$ .  $k$  and  $b\in \mathbb{R}^C$  are learnable parameters.  $k$  rescales the density of HSIs while  $b$  shifts the bias. Then  $\mathbf{X}_r$  undergoes a Sign function to be binarized into 1-bit activation  $\mathbf{X}_b\in \mathbb{R}^{H\times W\times C}$ , where  $x_{b} = +1$  or  $-1$  for  $\forall x_{b}\in \bar{\mathbf{X}}_{b}$  as
$$
x _ {b} = \operatorname {S i g n} \left(x _ {r}\right) = \left\{ \begin{array}{l l} + 1, & x _ {r} > 0 \\ - 1, & x _ {r} \leq 0 \end{array} \right. \tag {3}
$$
where  $x_{r} \in \mathbf{X}_{r}$ . As shown in Fig. 3 (b) and (c), since the Sign function is non-differentiable, previous methods either adopt a piecewise linear function  $\mathrm{Clip}(x)$  [33, 35, 36, 49, 64] or a piecewise quadratic function  $\mathrm{Quad}(x)$  [31, 32, 34] to approximate the Sign function during the backpropagation as
$$
\operatorname {C l i p} (x) = \left\{ \begin{array}{l l} + 1, & x \geq 1 \\ x, & - 1 <   x <   1 \\ - 1, & x \leq - 1 \end{array} \right. \quad \operatorname {Q u a d} (x) = \left\{ \begin{array}{l l} + 1, & x \geq 1 \\ 2 x + x ^ {2}, & 0 <   x <   1 \\ 2 x - x ^ {2}, & - 1 <   x \leq 0 \\ - 1, & x \leq - 1 \end{array} \right. \tag {4}
$$
Nonetheless, the Clip function is a rough estimation and there is a large approximation error between Clip and Sign. The shaded areas in Fig. 3 reflect the differences between the Sign function and its approximations. The shaded area corresponding to the Clip function is 1. Besides, once the absolute values of weights or activations are outside the range of  $[-1, 1]$ , they are no longer updated. Although the piecewise quadratic function is a closer approximation (the shaded area is  $2/3$ ) than Clip, the above two problems have not been fundamentally resolved. To address the two issues, we redesign a scalable hyperbolic tangent function to approximate the Sign function in the backpropagation as
$$
x _ {b} = \operatorname {T a n h} \left(\alpha x _ {r}\right) = \frac {e ^ {\alpha x _ {r}} - e ^ {- \alpha x _ {r}}}{e ^ {\alpha x _ {r}} + e ^ {- \alpha x _ {r}}}, \tag {5}
$$
where  $\alpha \in \mathbb{R}^+$  is a learnable parameter adaptively adjusting the distance between  $\mathrm{Tanh}(\alpha x)$  and  $\operatorname{Sign}(x)$ .  $e$  denotes the natural constant. We prove that when  $\alpha \to +\infty$ ,  $\mathrm{Tanh}(\alpha x) \to \mathrm{Sign}(x)$  as
$$
\lim  _ {\alpha \rightarrow + \infty} \operatorname {T a n h} (\alpha x) = \left\{\begin{array}{l l}\lim  _ {\alpha \rightarrow + \infty} \frac {e ^ {\alpha x} - 0}{e ^ {\alpha x} + 0}&= + 1, \quad x > 0\\\lim  _ {\alpha \rightarrow + \infty} \frac {e ^ {0} - e ^ {0}}{e ^ {0} + e ^ {0}}&= 0, \quad x = 0\\\lim  _ {\alpha \rightarrow + \infty} \frac {0 - e ^ {- \alpha x}}{0 + e ^ {- \alpha x}}&= - 1, \quad x <   0\end{array}\right. \tag {6}
$$
Figure 3: The upper line shows (a)  $\operatorname{Sign}(x)$  and its three approximation functions including (b) piecewise linear function  $\operatorname{Clip}(x)$ , (c) piecewise quadratic function  $\operatorname{Quad}(x)$ , and (d) our scalable hyperbolic tangent function  $\operatorname{Tanh}(\alpha x)$ . The area of the shaded region reflects the approximation error. The lower line depicts the derivatives.
If strictly following the mathematical definition,  $\operatorname{Sign}(0) = 0 \neq \pm 1$ . However, in BNN, the weights and activations are binarized into 1-bit, i.e., only two values ( $\pm 1$ ). Hence,  $\operatorname{Sign}(0)$  is usually set to  $-1$ . Similar to this common setting, we also define  $\lim_{\alpha \to +\infty} \operatorname{Tanh}(\alpha \cdot 0) = -1$  in BNN. Then we have
$$
\lim  _ {\alpha \rightarrow + \infty} \operatorname {T a n h} (\alpha x) = \operatorname {S i g n} (x). \tag {7}
$$
We compute the area of the shaded region between our  $\mathrm{Tanh}(\alpha x)$  and  $\mathrm{Sign}(x)$  in Fig. 3 (d) as
$$
\begin{array}{l} \int_ {- \infty} ^ {+ \infty} | \operatorname {S i g n} (x) - \operatorname {T a n h} (\alpha x) | d x = 2 \int_ {0} ^ {+ \infty} (1 - \operatorname {T a n h} (\alpha x)) d x \\ = 2 (x - x + \frac {1}{\alpha} \log (\operatorname {T a n h} (\alpha x) + 1)) \big | _ {x = 0} ^ {x = + \infty} \tag {8} \\ = \frac {2}{\alpha} (\log (2) - \log (1)) = \frac {2 \log (2)}{\alpha}. \\ \end{array}
$$
Different from previous  $\operatorname{Clip}(x)$  and  $\operatorname{Quad}(x)$ , our  $\operatorname{Tanh}(\alpha x)$  can arbitrarily reduce the approximation error with  $\operatorname{Sign}(x)$  when  $\alpha$  in Eq. (8) is large enough. Besides, our  $\operatorname{Tanh}(\alpha x)$  is neither piecewise nor unchanged when  $x$  is outside the range of  $[-1, 1]$ . On the contrary, the weights and activations can still be updated when their absolute values are larger than 1. In addition, as depicted in the lower line of Fig. 3, the value ranges  $[0, 1]$  and fixed shapes of  $\frac{\partial\operatorname{Clip}(x)}{\partial x}$  and  $\frac{\partial\operatorname{Quad}(x)}{\partial x}$  are fundamentally different from those of  $\frac{\partial\operatorname{Sign}(x)}{\partial x} \in [0, +\infty)$ . In contrast, our  $\frac{\partial\operatorname{Tanh}(\alpha x)}{\partial x}$  can change its value range  $(0, \alpha)$  and shape by adapting the parameter  $\alpha$ . It is more flexible and can approximate  $\frac{\partial\operatorname{Sign}(x)}{\partial x}$  better.
In the binarized convolutional layer, the 32-bit weight  $\mathbf{W}_f$  is also binarized into 1-bit weight  $\mathbf{W}_b$  as
$$
w _ {b} = \mathbb {E} _ {w _ {f} \in \mathbf {W} _ {f}} (| w _ {f} |) \cdot \operatorname {S i g n} (w _ {f}), \tag {9}
$$
where  $\mathbb{E}$  represents computing the mean value. Multiplying the mean absolute value of 32-bit weight value  $w_{f} \in \mathbf{W}_{f}$  can narrow down the difference between binarized and full-precision weights. Subsequently, the computationally heavy operations of floating-point matrix multiplication in full-precision convolution can be replaced by pure logical XNOR and bit-count operations [49] as
$$
\mathbf {Y} _ {b} = \mathbf {X} _ {b} * \mathbf {W} _ {b} = \text {b i t - c o u n t} (\mathrm {X N O R} (\mathbf {X} _ {b}, \mathbf {W} _ {b})), \tag {10}
$$
where  $\mathbf{Y}_b$  represents the output and  $*$  denotes the convolution operation. Since the value range of full-precision activation  $\mathbf{X}_f$  largely varies from that of 1-bit convolution output  $\mathbf{Y}_b$ , directly employing an identity mapping to aggregate them may cover up the information of  $\mathbf{Y}_b$ . To cope with this problem, we first fed  $\mathbf{Y}_b$  into a RPReLU [32] activation function to change its value range and then add it with  $\mathbf{X}_f$  by a residual connection to propagate full-precision information as
$$
\mathbf {X} _ {o} = \mathbf {X} _ {f} + \operatorname {R P R e L U} \left(\mathbf {Y} _ {b}\right), \tag {11}
$$
where  $\mathbf{X}_o$  denotes the output feature and RPRLU is formulated for the  $i$ -th channel of  $\mathbf{Y}_b$  as
$$
\operatorname {R P R e L U} \left(y _ {i}\right) = \left\{ \begin{array}{l l} y _ {i} - \gamma_ {i} + \zeta_ {i}, & y _ {i} > \gamma_ {i} \\ \beta_ {i} \cdot \left(y _ {i} - \gamma_ {i}\right) + \zeta_ {i}, & y _ {i} \leq \gamma_ {i} \end{array} \right. \tag {12}
$$
(a) Binarized DownSample
(b) Binarized UpSample
(c) Binarized Fusion Down
(d) Binarized Fusion Up
Figure 4: Comparison between normal and our binarized convolutional modules, including (a) downsample, (b) upsample, (c) fusion down to half the channels, and (d) fusion up to double the channels. The red arrow  $\downarrow$  indicates the full-precision information flow, while the blue arrow  $\downarrow$  denotes the binarized signal flow.
where  $y_{i}\in \mathbb{R}$  indicates single pixel values belonging to the  $i$ -th channel of  $\mathbf{Y}_b$ .  $\beta_{i},\gamma_{i}$ , and  $\zeta_i\in \mathbb{R}$  represents learnable parameters. Please note that the full-precision information is not blocked by the binarized convolutional layer in the proposed BiSR-Conv. Instead, it is propagated by the bypass identity mapping, as shown in the red arrow  $\rightarrow$  in Fig. 2 (c). Based on this important property of BiSR-Conv, we design the four binarized convolutional modules, as illustrated in Fig. 2 (d) and (e).
3.3 Binarized Convolutional Modules
In model binarization, the identity mapping is critical to propagate full-precision information and ease the training procedure. The only source of full-precision information is the input end  $(\mathbf{X}_s$  in Fig. 2) of the binarized part. However, the dimension mismatch during the feature downsampling, upsampling, and aggregation processes blocks the residual connections, which degrades the HSI reconstruction performance. To tackle this problem, we use BiSR-Conv to build up four binarized convolutional modules including downsample, upsample, fusion up, and fusion down with unblocked identity mappings to make sure the full-precision information can flow through all binarized layers.