A NEW BACKBONE FOR HYPERSPECTRAL IMAGE RECONSTRUCTION

As the inverse process of snapshot compressive imaging , the hyperspectral image ( HSI ) reconstruction takes the 2D measurement as input and posteriorly retrieves the captured 3D spatial-spectral signal . Built upon several assumptions , numerous sophisticated neural networks have come to the fore in this task . Despite their prosperity under experimental settings , it ’ s still extremely challenging for existing networks to achieve high-fidelity reconstructive quality while maximizing the reconstructive efficiency ( computational efficiency and power occupation ) , which prohibits their further deployment in practical applications . In this paper , we firstly conduct a retrospective analysis on aforementioned assumptions , through which we indicate the imminent aspiration for an authentically practicaloriented network in reconstructive community . By analysing the effectiveness and limitations of the widely-used reconstructive backbone U-Net , we propose a Simple Reconstruction Network , namely SRN , just based on some popular techniques , e.g. , scale/spectral-invariant learning and identity connection . It turns out , under current conditions , such a pragmatic solution outperforms existing reconstructive methods by an obvious margin and maximize the reconstructive efficiency concretely . We hope the proposed SRN can further contribute to the cutting-edge reconstructive methods as a promising backbone , and also benefit the realistic tasks , i.e. , real-time/high-resolution HSI reconstruction , solely as a baseline . 1 INTRODUCTION . Hyperspectral imaging ( HSI ) refers to multi-channel imaging where each channel stores information at a specific spectral wavelength for a fixed real-world scene ( Plaza et al. , 2009 ) . By capturing spatial intensity in a spectral-wise manner , hyperspectral images empower richer information than traditional RGB image cubes and they have been applied in a wide range of scenarios , e.g. , object detection ( Kim et al. , 2012 ; Xu et al. , 2015 ) , remote sensing ( Borengasser et al. , 2007 ; Melgani & Bruzzone , 2004 ; Yuan et al. , 2017 ) , medical image processing ( Lu & Fei , 2014 ; Meng et al. , 2020c ) etc . HSI can be captured and measured by snapshot compressive imaging ( SCI ) systems , which tend to compress information of snapshots along the spectral axis into one single 2D measurement ( Yuan et al. , 2021 ) . The coded aperture snapshot spectral imaging ( CASSI ) system ( Wagadarikar et al. , 2008 ; Meng et al. , 2020b ) forms one mainstream research direction among existing SCI systems due to its passive modulation property ( Llull et al. , 2013 ; Wagadarikar et al. , 2008 ; 2009 ; Yuan et al. , 2015 ) . The goal of HSI reconstruction is to transform the measurements into desired cubic hyperspectral images . As a result , a dimensional-expansion mapping function ( 2D to 3D ) is required , for which reason such a mapping relationship approximation is deemed to be much harder than general image regression tasks . By introducing domain expertise , previous research efforts have proposed a quite number of reconstruction algorithms ( Bioucas-Dias & Figueiredo , 2007 ; Liu et al. , 2019 ; Miao et al. , 2019 ; Meng et al. , 2020b ; Wang et al. , 2020 ; 2017 ; 2019 ; Yuan , 2016 ) , among which deep neural networks ( Meng et al. , 2020b ; c ; Miao et al. , 2019 ; Wang et al. , 2019 ; Wang et al. , 2019 ; 2020 ; Zheng et al. , 2021 ) enable an effective way to faithfully bridge between input and output compositional hierarchies ( LeCun et al. , 2015 ) . 1.1 MOTIVATION . While neural networks become increasingly prevalent in solving the HSI reconstruction problem , their success is reliant on several idealistic hypotheses , which hardly holds in practical scenarios . This prohibits the further application and exploration of HSI . Proper Dataset Volume . The feasibility of learning in a universal sense can be defined by satisfying a VC generalization bound Abu-Mostafa et al . ( 2012 ) . From this perspective , deep neural networks are considered to be high potential due to their high VC bound ( which can be described as the amount of free variables ) . Notably , the corresponding feasibility of neural networks is generally determined by their model complexity and the volume of datasets . In the recent HSI reconstruction methods , more increasingly complicated reconstructive networks have been proposed by assuming the underlying dataset is sufficiently large so that the learning is feasible . This neglects the fact that there ’ s limited data accessible . Under such circumstance , simpler model might be more promising . Ideal reconstructive rate . Practically , any reconstructive algorithm appears with a substantial forward model . For temporal-insensitive situations , the inverse model processes the measurements at any reconstructive rate rrecon > 0 , without considering cooperating with the forward model . However , the reconstructive rate should be higher than ( or at least equal to ) capture rate of the forward model , i.e. , rrecon ≥ rcap > 0 , for temporal-sensitive cases , i.e. , real-time/high-speed HSI reconstruction . For example , λ-net proposed by Miao et al . ( 2019 ) can be applied in reality with rrecon > 30fps . More generally , reconstructive methods are always hypothetically deemed to be efficient enough , i.e. , compatible with any forward systems , which however probably suffers from a deviation from current technical support . Considering the arbitrariness of the lower bound rcap , we can not determine an “ ideal ” reconstructive rate but only to maximize it , i.e. , minimize the reconstructive time . For neural network-based methods , eliminating the number of parameters and computational burdens is a sound solution . Unconstrained computing power . The success of the deep neural networks can largely attribute to the overparameterization ( Soltanolkotabi et al. , 2018 ) —the parameters in neural networks are a lot more than the training samples for a good representation learning . Actually , just by investing more computing power P , the cornerstone which overparametrization is reliant upon , researchers have made great progress in diverse applications ( Thompson et al. , 2020 ) . Latent performance boost can be further expected if more computing power is available , until one reaches the computing power bottleneck P ∗ . It turns out it ’ s difficult to solve such a bottleneck in practical . For example , to conduct TSA-Net ( Meng et al. , 2020b ) for high resolution ( HR ) -HSI restoration ( 1024×1024 ) , the required GPU memory would be unrestrictedly over 18Gb , which indicates dilemma encountered by all high-performanced reconstructive methods with large model size . 1.2 CONTRIBUTIONS . In this study , bearing the limitations imposed by the scale of datasets , the capability of optical systems , and the computational platform , we posit the possibility of network construction with min- imum inductive bias , based on an observation upon the reconstructive baseline U-net . We propose a simple yet quite promising CNN reconstructive network , namely Simple Recon Net ( SRN ) , whose success is owing to revisiting and tailoring several practical techniques for HSI , i.e. , spatial/spectral invariant learning and residual learning . We summarize the contribution of this work as follows . • The proposed SRN provides a new state-of-the-art by improving the previous one ( Huang et al. , 2021 ) > 3dB in PSNR . Moreover , our approach presents clear perceptual improvement across different spectral channels ( Fig . 1 Left ) . • We significantly shrink the model size ( see Fig . 1 Right ) without sacrificing the performance , i.e. , to most extent , we only use < 1/3 parameters of GSM-based method ( Huang et al. , 2021 ) and to the maximum , reduce FLOPs by > 34times as shown in Tab . 3 , both of which yield a higher temporal efficiency and computational efficiency . This makes realtime/higher-resolution images reconstruction practically executable under restricted GPU conditions . • The proposed network can either be solely used as a baseline , or conjunctively referred to as a backbone , contributing to more complicated methods , i , e. , more sophisticated E2E methods or unrolling methods . 2 RELATED WORKS . The basic idea of SCI is to modulate the high-dimensional signal using a higher frequency than the capture rate of the camera . In this manner , a compressed coded frame obtained will include the information in the high-dimensional signal and a high-performance algorithm can then be employed to recover the desired data . For compression , the novel implementation of SCI , CASSI , uses a coded aperture and a prism to conduct the ( spatio- ) spectral modulation . Previously , iterative-based optimization algorithms predominate the field of HSI reconstruction by approximating the image priors through diverse regularization techniques , i.e. , the total variance ( TV ) ( Kittle et al. , 2010 ; Wang et al. , 2015 ; Wagadarikar et al. , 2008 ) , sparsity ( Wang et al. , 2017 ; 2015 ) , non-local similarity ( NLS ) ( Wang et al. , 2016 ; Fu et al. , 2016 ) , Markov Random Field ( MRF ) ( Tappen , 2007 ) , Gaussian mixture model ( Yang et al. , 2015 ) etc , among which DeSCI ( Liu et al. , 2019 ) achieves best performance on both video and spectral compressive imaging . TwIST ( Bioucas-Dias & Figueiredo , 2007 ) proposed a two-step Iterative shrinkage/thresholding algorithm by modeling the reconstructive problem as a a linear observation model with a nonquadratic regularizer ( i.e. , total variation ) . GPSR ( M. A. T. Figueiredo et al. , 2007 ) proposes to use gradient projection ( GP ) algorithms to solve the inverse problem that is formulated as bound-constrained quadratic programming ( BCQP ) process . Inspired by the success of deep learning in other image translation problems , researchers have started using deep learning to reconstruct hyperspectral images from CASSI measurements ( Meng et al. , 2020b ; c ; Miao et al. , 2019 ; Wang et al. , 2019 ; Wang et al. , 2019 ; 2020 ; Zheng et al. , 2021 ) , which can be substantially divided into two streams : end-to-end neural networks ( E2E-NN ) models and others . The former stream tends to directly learn a complete mapping function from measurements ( always packaged with masks ) to estimated HSIs . Other researchers turns to introduce NN models into conventional optimization algorithms , named deep unrolling/unfolding nad plug-and-play methods , leading to lightweight and interpretable methods . Proposed in Ronneberger et al . ( 2015 ) , U-Net sat atop leaderboards regarding medical image tracking and segmentation , followed by which many variants have been derived in recent years , i.e. , 3D U-Net ( Çiçek et al. , 2016 ) , Ternaus U-Net ( Iglovikov & Shvets , 2018 ) , MultiResUNet ( Ibtehaz & Rahman , 2020 ) and Attention U-Net ( Oktay et al. , 2018 ) etc . Typical investigation correlates such a predominance with characteristics of medical images : 1 ) repeated tissue patterns and complicated gradient distribution ; 2 ) locally anomalous region and atypical noise distribution . U-Net intelligently fits the above characteristics by firstly abstracts coarse information and then focus on localization through simultaneously expansion and skip connection . As a classical network structure , U-Net serves as the first precast backbone in HSI reconstruction , among a series of famous architectures , and has been faithfully employed in both E2E and unrolling methods . For example , the λ-net ( Miao et al. , 2019 ) is a dual-stage generative model which employs a U-Net and residual learning strategy . The TSA-Net ( Meng et al. , 2020b ) which combined spatial-spectral self-attention with U-Net led to excellent results on both simulation and real data . As a deep unfolding method , GAP-net ( Meng et al. , 2020a ) utilizes 15-layer U-Net as a trained denoiser in each stage . Recent Gaussian Scale Mixture Prior-based ( GSM-based ) baseline ( Huang et al. , 2021 ) employing U-Net for two objects : a lightweight U-Net for approximating the regularization parameters , another lightweight U-Net for estimating the local-mean of GSM prior . U-Net has exemplified the rationality of uncomplicated neural network in field of HSI reconstruction , even if considering the huge difference between the medical images and hyperspectral images . Actually , as reported in Tab . 1 and Tab . 2 , U-net only achieves sub-optimal performance when solely referred to as a reconstructive baseline , which reveals both the validity and the limitations of such a generic solution . It turns out the performance of neural networks are sensitive to minor adjustments . Compared with the U-Net , it ’ s possible to construct a “ substitute network ” that enables a significant performance boost with mainstream techniques at hand .

Summary: This work proposes a novel backbone network for hyperspectral image reconstruction for snapshot compressive imaging (CASSI). The key structure of the proposed method is illustrated in Figure 3, essentially using skip connections between Conv-ReLU-Conv blocks repeatedly. A number of techniques such as downsampling, upsampling, pixelshuffling were employed to maximize efficiency. The proposed method achieved SOTA performance in the given CASSI reconstruction tasks over recent works with small network size and fast computation.