SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Specifically, Fig. 4 compares the normal and our binarized modules. The red arrow  $\downarrow$  indicates the full-precision information flow, while the blue arrow  $\downarrow$  denotes the binarized signal flow. The downsample modules in Fig. 4 (a) downscale the input feature maps and double the channels. The upsample modules in Fig. 4 (b) upscale the input spatial dimension and half the channels. The fusion down modules in Fig. 4 (c) maintain the spatial size of the input feature and half the channels. The fusion up modules in Fig. 4 (d) keep the spatial dimension of the input feature maps while doubling the channels. In the normal modules, the full-precision information is blocked by the Sign function and binarized into 1-bit signal. Meanwhile, the intermediate feature maps are directly reshaped by the binarized convolutional layers. For example, in the normal binarized downsample module, the intermediate feature is directly reshaped from  $\mathbb{R}^{H\times W\times C}$  to  $\mathbb{R}^{\frac{H}{2}\times\frac{W}{2}\times 2C}$  by a strided 1-bit conv4×4. The spatial and channel dimension mismatch of the input and output feature maps impede the identity mapping to propagate full-precision information from previous layers. In contrast, our binarized modules rely on the proposed BiSR-Conv that has a bypass identity connection for full-precision information flow, as shown in Fig. 2 (c). By using channel-wise concatenating and splitting operations, the intermediate feature maps at the input and output ends of BiSR-Conv are free from being reshaped. Therefore, the full-precision information can flow through our binarized modules. Finally, we derive our BiSRNet by using BiSR-Conv and the four modules to binarize  $\mathcal{E},\mathcal{B},$  and  $\mathcal{D}$  of the base model.
Algorithms	Bit	Category	Params (K)	OPs (G)	S1	S2	S3	S4	S5	S6	S7	S8	S9	S10	Avg
TwIST [59]	64	Model	-	-	25.16	23.02	21.40	30.19	21.41	20.95	22.20	21.82	22.42	22.67	23.12
0.700	0.604	0.711	0.851	0.635	0.644	0.643	0.650	0.690	0.569	0.669
GAP-TV [56]	64	Model	-	-	26.82	22.89	26.31	30.65	23.64	21.85	23.76	21.98	22.63	23.10	24.36
0.754	0.610	0.802	0.852	0.703	0.663	0.688	0.655	0.682	0.584	0.669
DeSCI [53]	64	Model	-	-	27.13	23.04	26.62	34.96	23.94	22.38	24.45	22.03	24.56	23.59	25.27
0.748	0.620	0.818	0.897	0.706	0.683	0.743	0.673	0.732	0.587	0.721
λ-Net [20]	32	CNN	62640	117.98	30.10	28.49	27.73	37.01	26.19	28.64	26.47	26.09	27.50	27.13	28.53
0.849	0.805	0.870	0.934	0.817	0.853	0.806	0.831	0.826	0.816	0.841
TSA-Net [18]	32	CNN	44250	110.06	32.03	31.00	32.25	39.19	29.39	31.44	30.32	29.35	30.01	29.59	31.46
0.892	0.858	0.915	0.953	0.884	0.908	0.878	0.888	0.890	0.874	0.894
BiConnect [36]	1	BNN	35	1.18	25.85	22.07	18.92	25.18	21.21	21.82	21.84	22.25	19.57	23.18	22.19
0.676	0.530	0.558	0.636	0.568	0.547	0.570	0.580	0.556	0.524	0.575
BNN [35]	1	BNN	35	1.18	26.69	23.98	20.58	28.53	22.96	24.12	23.20	23.29	21.65	23.86	23.88
0.661	0.551	0.566	0.679	0.584	0.599	0.568	0.590	0.588	0.547	0.593
Bi-Real [34]	1	BNN	35	1.18	28.06	26.05	24.92	31.04	25.32	26.54	25.09	25.47	24.69	25.41	26.26
0.701	0.644	0.654	0.733	0.664	0.671	0.631	0.678	0.644	0.622	0.664
IRNet [33]	1	BNN	35	1.18	27.91	25.84	25.27	31.77	25.12	26.31	25.29	25.14	25.07	25.20	26.30
0.700	0.620	0.661	0.723	0.663	0.685	0.665	0.662	0.668	0.603	0.665
ReActNet [32]	1	BNN	36	1.18	27.91	26.17	25.40	31.58	25.43	26.43	25.85	25.50	25.47	25.11	26.48
0.707	0.633	0.682	0.725	0.675	0.670	0.703	0.650	0.677	0.583	0.671
BBCU [31]	1	BNN	36	1.18	27.91	26.21	25.44	31.33	25.30	26.68	25.42	25.59	25.69	25.59	26.51
0.706	0.628	0.654	0.741	0.677	0.704	0.668	0.671	0.670	0.615	0.673
BTM [30]	1	BNN	36	1.18	28.75	26.91	26.14	32.74	25.87	27.37	26.26	26.20	26.10	25.73	27.21
0.739	0.674	0.708	0.794	0.692	0.739	0.707	0.718	0.717	0.671	0.716
BiSRNet (Ours)	1	BNN	36	1.18	30.95	29.21	29.11	35.91	28.19	30.22	27.85	28.82	29.46	27.88	29.76
0.847	0.791	0.828	0.903	0.827	0.863	0.800	0.843	0.832	0.800	0.837
Table 1: Quantitative results of BiSRNet, seven 1-bit BNN-based methods, two 32-bit CNN-based algorithms, and three 64-bit model-based methods on 10 scenes (S1~S10) of the KAIST [65] dataset. Params, OPs, PSNR (upper entry in each cell), and SSIM (lower entry in each cell) are reported.
Following [18, 26, 28, 29, 63], we select 28 wavelengths from  $450\mathrm{nm}$  to  $650\mathrm{nm}$  by using spectral interpolation manipulation to derive HSIs. We conduct experiments on simulation and real datasets.
Simulation Data. Two simulation datasets, CAVE [66] and KAIST [65], are adopted. The CAVE dataset provides 32 HSIs with a spatial size of  $512 \times 512$ . The KAIST dataset includes 30 HSIs at a spatial size of  $2704 \times 3376$ . We use CAVE for training and select 10 scenes from KAIST for testing.
Real Data. We adopt the five real HSIs captured by the CASSI system developed in [18] for testing.
Evaluation Metrics. The peak signal-to-noise ratio (PSNR) and structure similarity (SSIM) are adopted as metrics to evaluate the HSI reconstruction performance. Similar to [31, 32, 33, 34, 35], the operations per second of BNN  $(\mathrm{OPs}^b)$  is computed as  $\mathrm{OPs}^b = \mathrm{OPs}^f /64$ $(\mathrm{OPs}^{f} = \mathrm{FLOPs})$  to measure the computational complexity and the parameters of BNN is calculated as  $\mathrm{Params}^b = \mathrm{Params}^f /32$  where the superscript  $b$  and  $f$  refer to the binarized and full-precision models. The total computational and memory costs of a model are computed as  $\mathrm{OPs} = \mathrm{OPs}^b +\mathrm{OPs}^f$  and  $\mathrm{Params} = \mathrm{Params}^b +\mathrm{Params}^f$
Implementation Details. The proposed BiSRNet is implemented by PyTorch [67]. We use Adam [68] optimizer  $(\beta_{1} = 0.9$  and  $\beta_{2} = 0.999)$  and Cosine Annealing [69] scheduler to train BiSRNet for 300 epochs on a single RTX 2080 GPU. Training samples are patches with spatial sizes of  $256\times 256$  and  $96\times 96$  randomly cropped from 28-channel 3D HSI data cubes for simulation and real experiments. The shifting step  $d$  is 2. The batch size is 2. We set the basic channel  $C = N_{\lambda} = 28$  to store HSI information. We use random flipping and rotation for data augmentation. The training loss function is the root mean square error (RMSE) between reconstructed and ground-truth HSIs.
We compare our BiSRNet with 12 SOTA algorithms, including seven 1-bit BNN-based methods (BiConnect [36], BNN [35], Bi-Real [34], IRNet [33], ReActNet [32], BTM [30], and BBCU [31]), two 32-bit full-precision CNN-based methods ( $\lambda$ -Net [20] and TSA-Net [18]), and three 64-bit double-precision model-based methods (TwIST [59], GAP-TV [56], and DeSCI [53]). For a fair comparison, we set the Params and OPs of BNN-based methods to the same values.
Directly applying the seven SOTA BNN-based methods to HSI reconstruction task achieves unsatisfactory performance, ranging from 22.19 dB to 27.21 dB. Our BiSRNet surpasses these methods by large margins. More specifically, BiSRNet significantly outperforms BTM, BBCU, ReActNet, IRNet,
Figure 5: Reconstructed simulation HSIs of Scene 1 with 4 out of 28 spectral channels. Seven SOTA BNN-based algorithms and our proposed BiSRNet are compared. The spectral density curves (bottom-left) are corresponding to the selected region of the green box in the RGB image (Top-left). Please zoom in for a better view.
Bi-Real, BNN, and BiConnect by 2.55, 3.25, 3.39, 3.46, 3.50, 5.88, and 7.57 dB. This evidence suggests the significant effectiveness advantage of our BiSRNet in HSI restoration.
The proposed BiSRNet with extremely lower memory and computational complexity yields comparable results with 32-bit full-precision CNN-based methods. Surprisingly, our BiSRNet outperforms  $\lambda$ -Net by 1.23 dB while only costing  $0.06\%$  (36/62640) Params and  $1.0\%$  (1.18/117.98) OPs. In contrast, the previous best BNN-based method BTM is still 1.33 dB lower than  $\lambda$ -Net. When compared with TSA-Net, our BiSRNet only uses  $0.08\%$  Params and  $1.1\%$  OPs but achieves  $94.6\%$  (29.76/31.46) performance. These results demonstrate the efficiency superiority of the proposed method.
The three model-based algorithms are implemented by MATLAB, where the default type of variable is double-precision floating-point number. Although they use more accurate 64-bit data type, our 1-bit BiSRNet dramatically outperforms DeSCI, GAP-TV, and TwIST by 4.06, 5.40, and 6.64 dB.
4.3 Qualitative Results
Simulation HSI Restoration. Fig. 5 depicts the simulation HSIs on Scene 1 with 4 out of 28 spectral channels reconstructed by the 7 SOTA BNN-based algorithms and BiSRNet. Previous BNNs are less favorable to restore HSI details. They generate blurry HSIs while introducing undesirable artifacts. In contrast, BiSRNet reconstructs more visually pleasing HSIs with more structural contents and sharper edges. Additionally, we plot the spectral density curves (bottom-left) corresponding to the selected regions of the green box in the RGB image (Top-left). BiSRNet achieves the highest correlation score with the ground truth, suggesting the advantage of BiSRNet in spectral-wise consistency restoration.
Real HSI Restoration. Fig. 6 visualizes the reconstructed HSIs of the seven SOTA BNN-based algorithms and our BiSRNet. We follow the setting of [18, 26, 28, 29, 63] to re-train the models with all samples of the CAVE and KAIST datasets. To simulate the noise disturbance in real imaging scenes, we inject 11-bit shot noise into measurements during training. It can be observed that our BiSRNet is more effective in detailed content reconstruction and real imaging noise suppression.
Break-down Ablation. We adopt baseline-1 to conduct a break-down ablation towards higher performance. Baseline-1 is derived by using vanilla 1-bit convolution to replace BiSR-Conv and normal binarized convolutional modules (see Fig. 4) to replace our binarized convolutional modules. As shown in Tab. 2a, baseline-1 yields  $23.90\mathrm{dB}$  in PSNR and 0.594 in SSIM. When we apply BiSR-Conv, the model achieves  $3.90\mathrm{dB}$  improvement. Then we successively use our binarized downsample (BiDS), upsample (BiUS), fusion down (BiFD), and fusion up (BiFU) modules, the model gains by  $1.96\mathrm{dB}$  in total. These results verify the effectiveness of the proposed techniques.
BiSR-Conv. To study the effects of BiSR-Conv components, we adopt baseline-2 to conduct an ablation. Baseline-2 is obtained by removing Spectral-Redistribution (SR) operation and Sign approximation  $\mathrm{Tanh}(\alpha x)$  from BiSRNet. As reported in Tab. 2b, baseline-2 achieves 27.68 dB in PSNR and 0.723 in SSIM. When we respectively apply SR and  $\mathrm{Tanh}(\alpha x)$ , baseline-2 gains by 1.29 and 1.06 dB. When we exploit SR and  $\mathrm{Tanh}(\alpha x)$  jointly, the model achieves 2.08 dB improvement.
Figure 6: Reconstructed real HSIs of seven SOTA BNN-based algorithms and our BiSRNet on four scenes with 4 out of 28 wavelengths. BiSRNet is more effective in reconstructing detailed contents and suppressing noise.
Method	Baseline-1	+BiSR-Conv	+BiDS	+BiUS	+BiFD	+BiFU
PSNR	23.90	27.80	27.97	28.07	28.31	29.76
SSIM	0.594	0.737	0.729	0.758	0.776	0.837
OPs (M)	1176	1176	1176	1176	1176	1176
Params (K)	34.82	35.48	35.49	35.51	35.72	35.81
(a) Break-down ablation study towards higher performance
Baseline-2	SR	Tanh(αx)	PSNR	SSIM
✓	27.68	0.723
✓	✓	28.97	0.783
✓	✓	28.74	0.782
✓	✓	✓	29.76	0.837
(b) Ablation study of BiSR-Conv
Method	PSNR	SSIM
Clip(x)	28.97	0.783
Quad(x)	29.02	0.794
Tanh(αx)	29.76	0.837
(c) Study of approximation
Binarized Part	OPsf(M)	OPsb(M)	Paramsf	Paramsb	PSNRb	SSImb
Encoder E	3390	53	177878	5559	32.28	0.905
Bottleneck B	1096	17	278889	8715	33.80	0.932
Decoder D	5005	78	186562	5830	33.03	0.919
(d) Ablation study of binarizing different parts of the base model
Table 2: Ablations on the simulation datasets. In table (a), BiUS, BiDS, BiFU, and BiFD denote the binarized upsample, downsample, fusion up, and fusion down modules of Fig. 4. In table (b), SR refers to the Spectral-Redistribution of Eq. (11). In table (d), the full-precision model yields 34.11 dB in PSNR and 0.936 in SSIM.
Sign Approximation. We compare our scalable hyperbolic tangent function with previous Sign approximation functions. The experimental results are listed in Tab. 2c. Our  $\mathrm{Tanh}(\alpha x)$  dramatically surpasses the piecewise linear function  $\operatorname{Clip}(x)$  and quadratic function  $\operatorname{Quad}(x)$  by 0.79 and  $0.74\mathrm{dB}$ , suggesting the superiority of the proposed  $\mathrm{Tanh}(\alpha x)$ . This advantage can be explained by the analysis in Sec. 3.2 that our  $\mathrm{Tanh}(\alpha x)$  is more flexible and can adaptively reduce the difference with  $\mathrm{Sign}(x)$ .
Binarizing Different Parts. We binarize one part of the base model while keeping the other parts full-precision to study the binarization benefit. The experimental results are reported in Tab. 2d. The base model yields  $34.11\mathrm{dB}$  in PSNR and 0.936 in SSIM while costing  $10.52\mathrm{G}$  OPs and  $634\mathrm{K}$  Params. It can be observed from Tab. 2d: (i) Binarizing the bottleneck  $\mathcal{B}$  reduces the Params the most (270174) with the smallest performance drop (only  $0.31\mathrm{dB}$ ). (ii) Binarizing the decoder  $\mathcal{D}$  achieves the largest OPs reduction (4927 M) while the performance degrades by a moderate margin (1.08 dB).
In this paper, we propose a novel BNN-based method BiSRNet for binarized HSI restoration. To the best of our knowledge, this is the first work to study the binarized spectral compressive imaging reconstruction problem. We first redesign a compact and easy-to-deploy base model with simple computation operations. Then we customize the basic unit BiSR-Conv for model binarization. BiSR-Conv can adaptively adjust the density and distribution of HSI representations before binarizing the activation. Besides, BiSR-Conv employs a scalable hyperbolic tangent function to closer approach Sign by arbitrarily reducing the approximation error. Subsequently, we use BiSR-Conv to build up four binarized convolutional modules that can handle the dimension mismatch issue during feature reshaping and propagate full-precision information through all layers. Comprehensive quantitative and qualitative experiments demonstrate that our BiSRNet significantly outperforms SOTA BNNs and even achieves comparable performance with full-precision CNN-based HSI reconstruction algorithms.
This research was funded through National Key Research and Development Program of China (Project No. 2022YFB36066), in part by the Shenzhen Science and Technology Project under Grant (JCYJ20220818101001004, JSGG20210802153150005), National Natural Science Foundation of China (62271414), Science Fund for Distinguished Young Scholars of Zhejiang Province (LR23F010001), and Research Center for Industries of the Future at Westlake University.