Year: 2021

Abstract:
As the inverse process of snapshot compressive imaging, the hyperspectral image (HSI) reconstruction takes the 2D measurement as input and posteriorly retrieves the captured 3D spatial-spectral signal. Built upon several assumptions, numerous sophisticated neural networks have come to the fore in this task. Despite their prosperity under experimental settings, it’s still extremely challenging for existing networks to achieve high-fidelity reconstructive quality while maximizing the reconstructive efficiency (computational efficiency and power occupation), which prohibits their further deployment in practical applications. In this paper, we firstly conduct a retrospective analysis on aforementioned assumptions, through which we indicate the imminent aspiration for an authentically practicaloriented network in reconstructive community. By analysing the effectiveness and limitations of the widely-used reconstructive backbone U-Net, we propose a Simple Reconstruction Network, namely SRN, just based on some popular techniques, e.g., scale/spectral-invariant learning and identity connection. It turns out, under current conditions, such a pragmatic solution outperforms existing reconstructive methods by an obvious margin and maximize the reconstructive efficiency concretely. We hope the proposed SRN can further contribute to the cutting-edge reconstructive methods as a promising backbone, and also benefit the realistic tasks, i.e., real-time/high-resolution HSI reconstruction, solely as a baseline.

text: As the inverse process of snapshot compressive imaging, the hyperspectral image (HSI) reconstruction takes the 2D measurement as input and posteriorly retrieves the captured 3D spatial-spectral signal. Built upon several assumptions, numerous sophisticated neural networks have come to the fore in this task. Despite their prosperity under experimental settings, it’s still extremely challenging for existing networks to achieve high-fidelity reconstructive quality while maximizing the reconstructive efficiency (computational efficiency and power occupation), which prohibits their further deployment in practical applications. In this paper, we firstly conduct a retrospective analysis on aforementioned assumptions, through which we indicate the imminent aspiration for an authentically practicaloriented network in reconstructive community. By analysing the effectiveness and limitations of the widely-used reconstructive backbone U-Net, we propose a Simple Reconstruction Network, namely SRN, just based on some popular techniques, e.g., scale/spectral-invariant learning and identity connection. It turns out, under current conditions, such a pragmatic solution outperforms existing reconstructive methods by an obvious margin and maximize the reconstructive efficiency concretely. We hope the proposed SRN can further contribute to the cutting-edge reconstructive methods as a promising backbone, and also benefit the realistic tasks, i.e., real-time/high-resolution HSI reconstruction, solely as a baseline.

heading: 1 INTRODUCTION
text: Hyperspectral imaging (HSI) refers to multi-channel imaging where each channel stores information at a specific spectral wavelength for a fixed real-world scene (Plaza et al., 2009). By capturing spatial intensity in a spectral-wise manner, hyperspectral images empower richer information than traditional RGB image cubes and they have been applied in a wide range of scenarios, e.g., object detection (Kim et al., 2012; Xu et al., 2015), remote sensing (Borengasser et al., 2007; Melgani & Bruzzone, 2004; Yuan et al., 2017), medical image processing (Lu & Fei, 2014; Meng et al., 2020c) etc. HSI can be captured and measured by snapshot compressive imaging (SCI) systems, which tend to compress information of snapshots along the spectral axis into one single 2D measurement (Yuan et al., 2021). The coded aperture snapshot spectral imaging (CASSI) system (Wagadarikar et al., 2008; Meng et al., 2020b) forms one mainstream research direction among existing SCI systems due to its passive modulation property (Llull et al., 2013; Wagadarikar et al., 2008; 2009; Yuan et al., 2015).
The goal of HSI reconstruction is to transform the measurements into desired cubic hyperspectral images. As a result, a dimensional-expansion mapping function (2D to 3D) is required, for which reason such a mapping relationship approximation is deemed to be much harder than general image regression tasks. By introducing domain expertise, previous research efforts have proposed a quite number of reconstruction algorithms (Bioucas-Dias & Figueiredo, 2007; Liu et al., 2019; Miao et al., 2019; Meng et al., 2020b; Wang et al., 2020; 2017; 2019; Yuan, 2016), among which deep neural networks (Meng et al., 2020b;c; Miao et al., 2019; Wang et al., 2019; Wang et al., 2019; 2020; Zheng et al., 2021) enable an effective way to faithfully bridge between input and output compositional hierarchies (LeCun et al., 2015).

heading: 1.1 MOTIVATION
text: While neural networks become increasingly prevalent in solving the HSI reconstruction problem, their success is reliant on several idealistic hypotheses, which hardly holds in practical scenarios. This prohibits the further application and exploration of HSI.
Proper Dataset Volume. The feasibility of learning in a universal sense can be defined by satisfying a VC generalization bound Abu-Mostafa et al. (2012). From this perspective, deep neural networks are considered to be high potential due to their high VC bound (which can be described as the amount of free variables). Notably, the corresponding feasibility of neural networks is generally determined by their model complexity and the volume of datasets. In the recent HSI reconstruction methods, more increasingly complicated reconstructive networks have been proposed by assuming the underlying dataset is sufficiently large so that the learning is feasible. This neglects the fact that there’s limited data accessible. Under such circumstance, simpler model might be more promising.
Ideal reconstructive rate. Practically, any reconstructive algorithm appears with a substantial forward model. For temporal-insensitive situations, the inverse model processes the measurements at any reconstructive rate rrecon > 0, without considering cooperating with the forward model. However, the reconstructive rate should be higher than (or at least equal to) capture rate of the forward model, i.e., rrecon ≥ rcap > 0, for temporal-sensitive cases, i.e., real-time/high-speed HSI reconstruction. For example, λ-net proposed by Miao et al. (2019) can be applied in reality with rrecon > 30fps. More generally, reconstructive methods are always hypothetically deemed to be efficient enough, i.e., compatible with any forward systems, which however probably suffers from a deviation from current technical support. Considering the arbitrariness of the lower bound rcap, we cannot determine an “ideal” reconstructive rate but only to maximize it, i.e., minimize the reconstructive time. For neural network-based methods, eliminating the number of parameters and computational burdens is a sound solution.
Unconstrained computing power. The success of the deep neural networks can largely attribute to the overparameterization (Soltanolkotabi et al., 2018)—the parameters in neural networks are a lot more than the training samples for a good representation learning. Actually, just by investing more computing power P , the cornerstone which overparametrization is reliant upon, researchers have made great progress in diverse applications (Thompson et al., 2020). Latent performance boost can be further expected if more computing power is available, until one reaches the computing power bottleneck P ∗. It turns out it’s difficult to solve such a bottleneck in practical. For example, to conduct TSA-Net(Meng et al., 2020b) for high resolution (HR)-HSI restoration (1024×1024), the required GPU memory would be unrestrictedly over 18Gb, which indicates dilemma encountered by all high-performanced reconstructive methods with large model size.

heading: 1.2 CONTRIBUTIONS
text: In this study, bearing the limitations imposed by the scale of datasets, the capability of optical systems, and the computational platform, we posit the possibility of network construction with min-
imum inductive bias, based on an observation upon the reconstructive baseline U-net. We propose a simple yet quite promising CNN reconstructive network, namely Simple Recon Net (SRN), whose success is owing to revisiting and tailoring several practical techniques for HSI, i.e., spatial/spectral invariant learning and residual learning. We summarize the contribution of this work as follows.
• The proposed SRN provides a new state-of-the-art by improving the previous one (Huang et al., 2021) > 3dB in PSNR. Moreover, our approach presents clear perceptual improvement across different spectral channels (Fig. 1 Left).
• We significantly shrink the model size (see Fig. 1 Right) without sacrificing the performance, i.e., to most extent, we only use <1/3 parameters of GSM-based method (Huang et al., 2021) and to the maximum, reduce FLOPs by > 34times as shown in Tab. 3, both of which yield a higher temporal efficiency and computational efficiency. This makes realtime/higher-resolution images reconstruction practically executable under restricted GPU conditions.
• The proposed network can either be solely used as a baseline, or conjunctively referred to as a backbone, contributing to more complicated methods, i,e., more sophisticated E2E methods or unrolling methods.

heading: 2 RELATED WORKS
text: The basic idea of SCI is to modulate the high-dimensional signal using a higher frequency than the capture rate of the camera. In this manner, a compressed coded frame obtained will include the information in the high-dimensional signal and a high-performance algorithm can then be employed to recover the desired data. For compression, the novel implementation of SCI, CASSI, uses a coded aperture and a prism to conduct the (spatio-)spectral modulation.
Previously, iterative-based optimization algorithms predominate the field of HSI reconstruction by approximating the image priors through diverse regularization techniques, i.e., the total variance (TV) (Kittle et al., 2010; Wang et al., 2015; Wagadarikar et al., 2008), sparsity (Wang et al., 2017; 2015), non-local similarity (NLS) (Wang et al., 2016; Fu et al., 2016), Markov Random Field (MRF) (Tappen, 2007), Gaussian mixture model (Yang et al., 2015) etc, among which DeSCI (Liu et al., 2019) achieves best performance on both video and spectral compressive imaging. TwIST (Bioucas-Dias & Figueiredo, 2007) proposed a two-step Iterative shrinkage/thresholding algorithm by modeling the reconstructive problem as a a linear observation model with a nonquadratic regularizer (i.e., total variation). GPSR (M. A. T. Figueiredo et al., 2007) proposes to use gradient projection (GP) algorithms to solve the inverse problem that is formulated as bound-constrained quadratic programming (BCQP) process.
Inspired by the success of deep learning in other image translation problems, researchers have started using deep learning to reconstruct hyperspectral images from CASSI measurements (Meng et al., 2020b;c; Miao et al., 2019; Wang et al., 2019; Wang et al., 2019; 2020; Zheng et al., 2021), which can be substantially divided into two streams: end-to-end neural networks (E2E-NN) models and others. The former stream tends to directly learn a complete mapping function from measurements (always packaged with masks) to estimated HSIs. Other researchers turns to introduce NN models into conventional optimization algorithms, named deep unrolling/unfolding nad plug-and-play methods, leading to lightweight and interpretable methods.
Proposed in Ronneberger et al. (2015), U-Net sat atop leaderboards regarding medical image tracking and segmentation, followed by which many variants have been derived in recent years, i.e., 3D U-Net (Çiçek et al., 2016), Ternaus U-Net (Iglovikov & Shvets, 2018), MultiResUNet (Ibtehaz & Rahman, 2020) and Attention U-Net (Oktay et al., 2018) etc. Typical investigation correlates such a predominance with characteristics of medical images: 1) repeated tissue patterns and complicated gradient distribution; 2) locally anomalous region and atypical noise distribution. U-Net intelligently fits the above characteristics by firstly abstracts coarse information and then focus on localization through simultaneously expansion and skip connection.
As a classical network structure, U-Net serves as the first precast backbone in HSI reconstruction, among a series of famous architectures, and has been faithfully employed in both E2E and unrolling methods. For example, the λ-net (Miao et al., 2019) is a dual-stage generative model which employs a U-Net and residual learning strategy. The TSA-Net (Meng et al., 2020b) which combined
spatial-spectral self-attention with U-Net led to excellent results on both simulation and real data. As a deep unfolding method, GAP-net (Meng et al., 2020a) utilizes 15-layer U-Net as a trained denoiser in each stage. Recent Gaussian Scale Mixture Prior-based (GSM-based) baseline (Huang et al., 2021) employing U-Net for two objects: a lightweight U-Net for approximating the regularization parameters, another lightweight U-Net for estimating the local-mean of GSM prior. U-Net has exemplified the rationality of uncomplicated neural network in field of HSI reconstruction, even if considering the huge difference between the medical images and hyperspectral images. Actually, as reported in Tab. 1 and Tab. 2, U-net only achieves sub-optimal performance when solely referred to as a reconstructive baseline, which reveals both the validity and the limitations of such a generic solution. It turns out the performance of neural networks are sensitive to minor adjustments. Compared with the U-Net, it’s possible to construct a “substitute network” that enables a significant performance boost with mainstream techniques at hand.

heading: 3 METHODOLOGY

heading: 3.1 MATHEMATICAL MODEL OF CASSI
text: Recall that the key idea of SCI is to modulate the spatio-spectral signal with different modulation patterns, and CASSI implemented this in a low cost way by a fixed mask plus a disperser. Given the 3D spectral cube (Fig. 2 left), denoted by F ∈ RNx×Ny×Nλ , where Nx, Ny , and Nλ represent the spectral image’s height, width, and total number of wavelengths, respectively, we compute signal modulation (implemented by the mask) in a channel-wisely way as F ′(:, :, nλ) = F (:, :, nλ) M∗, where F ′ ∈ RNx×Ny×Nλ represents the modulated signals, M∗ ∈ RNx×Ny refers to a pre-defined physical mask governing on signal modulation, nλ ∈ [1, . . . , Nλ] indexes wavelengths, and is the Hadamard product. By passing F ′ to a disperser, the cube F ′ becomes tilted and could be sheared along the y-axis. Let F ′′ ∈ RNx×(Ny+Nλ−1)×Nλ be the tilted cube, and λc be a reference wavelength, i.e., F ′[:, :, nλc ] works like an anchor image that is not sheared along the y-axis, we have F ′′(u, v, nλ) = F
′(x, y+d(λn−λc), nλ), where (u, v) locates the coordinate system on the detector plane, λn denotes the nλ-th channel, λc refers to the anchored wavelength, and d(λn−λc) represents a spatial shift of the nλ-th channel in F ′. Notably, since the sensor integrates all the light within the wavelength range [λmin, λmax], we could model the compressed measurement at a detector y(u, v) with the following integral: y(u, v) = ∫ λmax λmin f ′′(u, v, nλ)dλ, where f ′′ is the analog (con-
tinuous) representation of F ′′. To discretize this model, we have Y = ∑Nλ nλ=1
F ′′(:, :, nλ) +G. Y ∈ RNx×(Ny+Nλ−1) represents a 2D measurement, which, in essence, is a compressed frame capturing the information, and G ∈ RNx×(Ny+Nλ−1) is a corresponding 2D measurement noise.
To simplify model notations, we define M ∈ RNx×(Ny+Nλ−1)×Nλ and F̃ ∈ RNx×(Ny+Nλ−1)×Nλ as the shifted masks and signal frames of different wavelengths as follows: M(u, v, nλ) = M∗(x, y + d(λn − λc)), F̃ (u, v, nλ) = F (x, y + d(λn − λc), nλ). By using M and F̃ , the measurement Y could be reformulated as
Y = ∑Nλ nλ=1 F̃ (:, :, nλ) M(:, :, nλ) +G. (1)
It is usually easier to use the vectorized notations. Let y = vec(Y ) and g = vec(G) ∈ Rn be the vectorization of matrices Y and G, where vec(·) concatenates all the columns of a matrix
as one single vector. Similarly, we have f̃ (nλ) = vec(F̃ (:, :, nλ)), resulting in the vector f = vec([f̃ (1) . . . f̃ (Nλ)]) ∈ RnNλ , where n = Nx(Ny + Nλ − 1). By defining the sensing matrix as Φ = [D1, . . . ,DNλ ] ∈ Rn×nNλ , where Dnλ = Diag(vec(M(:, :, nλ))) is a diagonal matrix expanded by vec(M(:, :, nλ)), rewrite equation 1 as
y = Φf + g. (2)
While obtaining the vectorized measure y is similar to compressive sensing (Donoho, 2006; Emmanuel et al., 2006), the sensing matrix Φ has a very special structure. The recent studies (Jalali & Yuan, 2019) have shown the signal can still be recovered even when Nλ > 1. Given the measurement y captured by the camera and Φ calibrated upon pre-design, one critical and practical problem of CASSI is to solve f used in equation 2, falling in the vein of inverse problem (Yuan et al., 2021), for which we will propose a simple yet promising deep convolutional neural network below.

heading: 3.2 NETWORK DESIGN
text: The reconstructive function generates the 3D hyperspectral estimations from measurements. Therefore, it would be better if one decompose the 2D measurements into 3D data cube beforehand. A typical operation is to firstly initialize network input by conducting dot productions between the measurement and each spectral channels of shifted variants of coded aperture, namely mask, with an intention to mimic the Φ−1y according to equation 2:
FY [:, :, nλ] := shift(Mnλ Y ), (3) where nλ-th channel of FY ∈ RNx×Ny×Nλ is computed through Hadamard product, i.e., , and shift operation (raw output will be shifted into the desired tensor shape). Overall, the proposed model aims to learn the mapping as G(·) : FY → F̂ , where F̂ denotes the reconstruction result. It has been previously proved that network depth is one of the determining factors for the success of estimation (Liang & Srikant, 2016; Telgarsky, 2016; Eldan & Shamir, 2016). We thereby preclude complicated modules while maintain sufficient depth. The specific network structure is quite simple and have the following function format:
gk =
{ g0, k = 0
g0 + ∑k i=1 Ri(gi−1), k = 1, ...,K,
(4)
where gk is intermediate embedding at level k, g0 is initialized by a single convolutional layer, i.e., g0 = CONV(FY ), where g0 ∈ RNx×Ny×Nγ . Note that Nγ > Nλ, based on which we intend to do a “learnable spectral interpolation”.
在此处键入公式。
d
This provides more spectral redundancy and thus enables the model to better learn the connections among different channels. For example, adjacent spectral channels tend to be more correlated to each other in general HSIs. Rk(·) in equation 4 is just a non-linear feature transformer that bridges between different levels of embedding. We simply set it as Rk(x) = CONV(ReLU(CONV(x))). As shown in Fig. 3, the main structure of the proposed network includes K different levels of embedding. Symmetric with the input initialization,
the final output is simply computed by a convolutional layer: F̂ = CONV(g∑) = CONV(gK + g0). Identity connections (Bishop et al., 1995) are widely used in the whole network. In Sec. 4, we show that such a simple network stands out among all existing reconstructive methods. Below we will introduce several guidelines for model establishment, followed by which the proposed network successfully learns the mapping function between cubic input-output data pairs.
3.2.1 SPATIAL/SPECTRAL INVARIANCE
As we mentioned in Sec. 2, U-net (Ronneberger et al., 2015) is barely satisfactory in this field, even being technically modified as in Miao et al. (2019). This is determined by twofold reasons: 1) Different characteristic between medical images and HSIs (natural scenes) as illustrated in Sec. 2; 2) Different utilities: medical image processing task is always more locally focused while the HSI reconstruction tends to be globally comprehensive.
It turns out the receptive field (RF) is important for image tasks. As demonstrated by Luo et al. (2016), for plenty of tasks, i.e., image classification, dense prediction etc., RF should be sufficiently large to capture the semantic information (highest RF should be at least encompass the raw input). In object detection, RF should be cautiously determined. We argue that in lowlevel regression tasks, its unnecessary for the highest RF to cover the whole image but is supposed to be large enough to capture the neigh-
boring for the estimated pixel. In Fig. 4, we compare the proposed model SRN with the U-Net utilized by Miao et al. (2019) in terms of the spatial/spectral dimension. The highest RF of the U-Net is much larger than the input (256 v.s. 532), which is oversized and thus improper. The highest RF of proposed SRN, in comparison, makes more sense (256 v.s. 69). In SRN, concurrent with such a linearly expanding behavior of RF, the spatial size intermediate embedding remains the same, which is thus called “Spatial Invariance”.
To approximate the 3D signal in a rank minimization manner, previous researchers attempt to reduce the spectral channels somewhere in the model, which turns out to be a lossy compression operation and is avoided in our work. While in the proposed model, we simply treat the data cube as a whole and thus keeps the spectral dimension constant through the main body (spectral invariant) after expanding from Nλ = 28 to Nγ = 64.

heading: 3.2.2 EFFICIENCY MAXIMIZATION: TOWARD PRACTICAL APPLICATIONS
text: In practical application scenarios, performance and efficiency of the proposed method consist of a paradox for all existing reconstructive methods, including our proposed network. For such a highperformanced (as demonstrated by considerable experiments in 4) and concise network, we seek to further maximize its efficiency at the expense of negligible performance descent. It turns out the metric, floating point operations (FLOPs, i.e., amount of additions and multiplications), provides a favourable perspective of view, which 1) is inversely proportional to the reconstructive rate mentioned in 1.1, 2) is positively related to the computing power. We minimize the FLOPs via introducing rescaling manipulation-pairs, which are consist of a downsampling operation, i.e., a convolutional layer with stride 2, and an upsampling operation (i.e., a convolutional layer followed by PixelShuffle operation (Shi et al., 2016)). Spatial size will be doubled/halved by a single upsampling/downsampling operation. As annotated by Fig. 3, rescaling manipulations can be flexibly inserted into model, i.e., right after the intermediate embeddings. Except for the initially proposed network SRN v1, we further try two more variants: SRN v2 contains one rescaling pair, which reduces the spatial size of g0 and recovers at g∑; SRN v3, in which second rescaling pair is inserted in the after g4 and g12. Interestingly, experimental results below proves both of the variants are still competitive to state-of-the-art performance (Tab. 1 and Tab. 2) in simulation.

heading: 4 EXPERIMENTS
text: Dataset. As in Meng et al. (2020b), we adopt the same 28 wavelengths distributed within the range of 450nm to 650nm obtained by spectral interpolation manipulation for the HSIs, and conduct experiments on the following two data type: 1) Simulation Data and 2) Real HSI data.
For simulation data, both CAVE (Park et al., 2007) synthetic dataset and KAIST (Choi et al., 2017) synthetic dataset are applied in our simulation experiment. Regarding the training set, we create 205 1024×1024×28 large image examplers from 30 256×256×28 images from CAVE dataset by randomly concatenating. Operations like rotation and rescaling are both used to magnify the robustness. Training samples of size 256×256×28 will be further randomly cropped from the examplers. We compare with other methods on the same ten different 256×256×28 HSIs abstracted from KAIST dataset. For real HSI data, we augment the above training set by additional 37 HSI images in KAIST (Choi et al., 2017) dataset, all of which will be cropped into 660×660×28 to match the real-world measurement, which are abstracted by CASSI system developed in Meng et al. (2020b).
Training and Testing Procedure. The model is trained to minimize a mean squared error (MSE) between the ground truth and output. For simulation experiment, reconstructive inputs can be computationally acquired by mimicking the actual CASSI reproduced in Meng et al. (2020b). Notably, the real-world captured measurements are disturbed by the noise introduced by the optical system. Therefore, we add Gaussian noise with standard deviation randomly generated from the range [0, 0.05] to mimic this scenario. For both simulation training/testing and real-wrold testing, the identical mask are employed for a fair comparison.
Compared Methods. We compare with seven state-of-the-art reconstruction algorithms, including U-Net (Ronneberger et al., 2015), HSSP (Wang et al., 2019), λ-net (Miao et al., 2019), TSA-Net (Meng et al., 2020b), GSM-based method (Huang et al., 2021), PnP-DIP-HSI (Meng et al., 2021) and GAP-net (Meng et al., 2020a), among which λ-net, TSA-Net and GSM-based methods yield plausible results. Peak Signal-to-Noise Racial (PSNR) and Structural SIMilarity (SSIM) (Wang et al., 2004) are used for a quantitative comparison. The PSNR is computed by
PSNRch = 10 log10( MAX2I MSEch
), where the channel-wise PSNR value will be gathered for average computing and MAX2I denotes the maximum pixel value in ground truth image I .
We implemented our model by PyTorch and trained it with the ADAM optimizer. There are K=16 network embeddings in main body. The learning rate was initialized as 4×10−4, decreased by half every 50 epochs, for both simulation/real settings. We set batch size = 4 for the best validation performance. The training is conducted on a NVIDIA TIATN RTX GPU on simulation data.

heading: 4.1 EXPERIMENTS ON SYNTHETIC DATA
text: As shown in both Tab. 1 and 2, the proposed method (i.e., our best model without rescaling, denoted as SRN v1) has achieved the highest averaged PSNR and SSIM of 10 synthetic testing samples as 33.45 dB and 0.932, respectively, which is an considerable breakthrough. Although the other two versions of our model (i.e., the proposed method with 1 or 2 rescaling pairs, termed as, SRN v2 and SRN v3) are mainly designed for reducing the computational cost, they also exhibit the competitive performance compared with existing methods. Validated by three variants of our model, the proposed method demonstrated great stability and effectiveness. Fig. 5 visualizes the reconstructed results between six compared methods and our model. Our reconstructed images contain the most details among four chosen spectral channels including 462.1nm, 498nm, 575.3nm and 625.1nm. We also plot the spectral density curves corresponding to two picked regions ( box “a” and “b” in Fig. 5). The highest correlations (i.e., 0.9792, 0.9975) and highly coincidence between our curves and reference proves the spectral-wise effectiveness of our model. To clearly check the reconstructive ability of the proposed model (i.e., (v1)), we put RGB reconstructions of eight 256×256×28 reconstructed simulation results in Fig. 7.

heading: 4.2 EXPERIMENTS ON REAL DATA
text: We validate the effectiveness of proposed method on hyperspectral images collected from real scenes. We show one example of legoman in Fig. 6 (a) by comparing our method with three best reconstructive methods, λ-net (Miao et al., 2019), TSA-Net (Meng et al., 2020b) and GSM-based method (Huang et al., 2021), on 8 spectral channels. Our reconstructive results are perceptually complete and demonstrate few artifact, which is obvious on the “face” of the legoman. We pick a small red region for spectral-wise performance verification and gets the highest correlation with the reference generated by a spectrometer.

heading: 4.3 EFFICIENCY ANALYSIS
text: Model size. Besides the breakthroughs in terms of performance, we further compare the model size and floating-point operations (FLOPs) between four E2E deep-learning-based methods and three variants of our proposed method. As shown in Tab. 3, the proposed SRN v1 (i.e., the one without rescaling pairs) contains only 2.82% trainable parameters compared with TSA-Net, less than 1/3 params as GSM-based method. We reduces 96% and 98% trainable parameters when compared with Unet (Ronneberger et al., 2015) and λ-net, respectively. The other two variants involve a little bit more trainable
parameters due to the additional covolutional layers introduced by the rescaling pairs. Our models requires much less computing power and thus can be flexibly deployed, which benefits the real-time reconstruction. We visualize the reconstructed high-resolution HSI, i.e.,1024×1024×28 in A.4. Floating-Point Operations (FLOPs). Given the input of spatial shape 256×256, we count the total amount of addition and multiplication for different deep learning-based model. As shown in Tab. 3, the FLOPs of the proposed model without rescaling pairs is 81.84G, which are slightly more than that of TSA-Net. The underlying intuition is that though the proposed model is rather small, the amount of operations conducted in the model is still formidable, which limits the capacity of processing large image patches. Similar problem shows on GSM-based method, which contains over 30× FLOPs than SRN v3. As shown in Tab. 3, two variants (i.e., the bottom two models, SRN v2 and SRN v3) mitigate this issue successfully. Typically, we reduce 69.4% FLOPs by using one rescaling pairs, which is less than 1/3 the FLOPs of TSA-Net. If we further add one more rescaling pair, the resulting FLOPs will be only 18.57G.

heading: 5 CONCLUSION AND FUTURE WORKS
text: We provided a simple yet promising method specially-designed for HSI reconstruction in this paper. The proposed method outperformed existing methods, by setting a new state-of-the-art performance with a negligible amount of model parameters. We alleviated the problem of inefficiency of neural network-based reconstructive methods and thus make a step forward to practical application of HSI reconstruction. Our method can not only be solely referenced as a baseline, also conjointly served as a qualified backbone for future research in the HSI community.

heading: A APPENDIX

heading: A.1 EXPERIMENTAL SETTINGS
text: CAVE dataset. We used CAVE dataset (Park et al., 2007) to train our model for both simulation and real data experiments. CAVE dataset contains 30 512×512 multispectral images corresponding to wavelength of 400nm∼700nm. We generate 175 examplers of 1024×1024 spatial size by random concatenation. To increase the robustness of the training set, images are randomly rotated before concatenation. Besides, another 30 examplers are generated via directly rescaling the original ones. For simulation, image patches of 256×256 spatial size are randomly cropped and then fed into the model. For real hyperspectral images reconstruction, 660×660 images patches are randomly cropped to be the training samples of our CNN model. For high-resolution image reconstruction, 1024×1024×28 examplers are directly fed into the model as training samples. Fig. 8 (a) shows a rescaled training exampler at the wavelength of 453.3nm and (b) shows a concatenated training exampler at the wavelength of 522.7nm.
KAIST dataset. KAIST (Choi et al., 2017) dataset contains 30 high-quality hyperspectral images of spatial size 2704×3376. In simulation experiment, ten image patches of 256×256×28 are used as the testing data. For real HSI reconstruction, 36 image patches of spatial size 1024×1024 are abstracted and combined with above 205 sample for training purpose. For high-resolution image reconstruction, 17 samples of spatial size 1024×1024 are used as the testing data. Fig. 8 (c) demonstrates an example of testing image patch at 498.0nm wavelength, which is used for the third task.
Evaluation metric. Two widely-used validation criteria, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) are used for quantitative comparison. SSIM is computed by
SSIM(x, y)= [ 2µxµy + C1 µ2x + µ 2 y + C1 ]α · [
2σxσy + C2 σ2x + σ 2 y + C2 ]β · [ σxy + C3 σxσy + C3 ]γ ,
(5)
where µx, µy are local means, σx, σy are standard deviations, σxy is cross-covariance for image x and y.
A.2 HYPER-PARAMETER TUNING
In this experiment, we perform a number of trials on simulation dataset, to find the optimal hyperparameter setting for the proposed reconstructive models. Specifically, ablation studies on batch normalization layer 4, initial learning rate of 0.0002, 0.0004, 0.0006 (see Table 5 for averaged PSNR and SSIM values), and batch size of 2,4,8 (see Table 6 for averaged PSNR and SSIM values) are involved in three versions of proposed model (SRN v1, SRN v2,SRN v3).
Besides, to further validate the superiority of Simplified ResBlcok (RB) over normal ResBlock, we compare the
Table 7: PSNR (dB) and SSIM values on 1024×1024×28 simulation testing data. Reconstruction is conducted by SRN v2model (i.e., the model with one rescaling pair), where optimal hyperparameter settings argued in Section A.2 are leveraged in the training phase.
scene 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Avg.
PSNR 28.78 29.97 30.40 31.74 33.31 30.96 34.56 34.46 34.49 27.28 34.95 33.07 31.88 33.36 29.20 29.85 30.03 31.66
SSIM 0.869 0.913 0.943 0.937 0.956 0.958 0.958 0.962 0.956 0.821 0.945 0.948 0.819 0.928 0.937 0.886 0.926 0.921
model size, FLOPs (FLoating-point OPerations) and quantitative evaluations (averaged PSNR and SSIM) on both settings by only adding the Batch Normalization (Ioffe & Szegedy, 2015) layer behind the first convolutional layer in each Simplified ResBlock (RB) (shown in Table 4). For comparison, TSA-Net (Meng et al., 2020b) achieves 30.15dB and 0.893 respectively, with 44.25M parameters and 80.08G FLOPs. Obviously, Batch Normalization layers not only result in more parameters and FLOPs, also undermine the performance. As demonstrated in the three tables above, we choose the optimal hyperparameter setting according to our best model’s performance.
learning rate = 0.0004,
batch size = 4. (6)
A.3 STATE-OF-THE-ART PERFORMANCE
batch size SRN v1 SRN v2 SRN v3
PSNR SSIM PSNR SSIM PSNR SSIM
2 32.24 0.909 31.44 0.894 31.02 0.888 4 33.45 0.932 31.77 0.900 31.36 0.900 8 32.05 0.909 31.68 0.902 31.51 0.905
Table 6: Averaged PSNR (dB) and SSIM values on different batch sizes in simulation dataset. Initial learning rate is kept as 0.0004 and reduced by half every 50 epochs.
In this section, we visualize more reconstructed HSIs produced by our best model, which is SRN v1 (i.e., the original model without rescaling pair). Firstly, we add one additional simulation and one additional real reconstructed results with spectral analysis (i.e., density curves) in Fig. 9 and Fig. 10. For comparison, the same sixteen spectral channels of reconstructed HSIs generated by λ-net (Miao et al., 2019), TSA-Net (Meng et al., 2020b) and GSM-based method (Huang et al., 2021) are also plotted, shown in Fig. 11∼Fig. 20. Better visualization performance can be achieved by zooming in.

heading: A.4 HIGH-RESOLUTION HSI RECONSTRUCTION
text: As compared in Fig. 1 right, our model outperforms the TSA-Net (Meng et al., 2020b) on simulation data of spatial size 256×256 with only 2.82% trainable parameters. As compared in Tab. ??, we outperforms GSM and concretely reduce FLOPs by > 34times, by comparison. Besides, the variants of original model, which are equipped with rescaling pairs, can still achieve promising performance with less than 1/3 FLOPs, as shown in Tab. 1 and Tab. 2. Both of the minimal model size and insignificant amount of computing enables our method a perfect candidate for higher-resolution hyperspectral image reconstruction. Therefore, we setup a simulation experiment—by directly using previous 1024×1024×28 examplers as training samples, we reconstruct high-resolution HSI accordingly through SRN v2 model (i.e., the model with one rescaling pair). The testing set is composed of 17 images patches of size 1024×1024×28 abstracted from KAIST dataset. We also generate a 1024×1024 noise map as mask for training and testing. As far as we know, this is the first time to reconstruct HSI at such a high resolution using deep learning-based method. Table 7 reports the PSNR and SSIM values on all 17 scenes. Fig. 21∼Fig. 26 show the reconstructed images on 16 selected spectral channels.
Figure 7: RGB reconstruction of eight reconstructed simulation results.
(a) Training Exampler: rescaling (b) Training Exampler: concatenation (c) High-resolution Ground Truth Image
Figure 8: 1024×1024 training/testing hyperspectral images for different tasks, shown on specific wavelengths.

heading: Ours

heading: Ours

heading: Ours

heading: Ours

heading: Ours

heading: Ours

heading: Ours

heading: Ours
text: GSM-based
!-net
TSA-Net

heading: Ours

Review summaries:
Different from the previous works about Unet structure for HSI reconstruction, this paper introduces a new backbone, which is efficient and lightweight. The experiments also demosntrate that the proposed model can significantly outperfom the previous works in quantitative evaluation results. From the above summary, this paper should be of significance to the application society of CASSI imaging reconstruction.

This paper introduces a hyperspectral image reconstruction method that aims to produce an image with multiple channels from a single-shot measurement with high reconstruction quality and efficiency. This is achieved through the development of a simple reconstruction network that adopts some tricks such as spatial and spectral invariant learning and identity connection. The proposed method was tested on CAVE and KAIST datasets, and was compared with several state-of-the-art methods. The results show clear advantages of the proposed method.

This work proposes a novel backbone network for hyperspectral image reconstruction for snapshot compressive imaging (CASSI). The key structure of the proposed method is illustrated in Figure 3, essentially using skip connections between Conv-ReLU-Conv blocks repeatedly. A number of techniques such as downsampling, upsampling, pixelshuffling were employed to maximize efficiency. The proposed method achieved SOTA performance in the given CASSI reconstruction tasks over recent works with small network size and fast computation.