A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration

1 Introduction . Hyperspectral imaging ( HSI ) enables measurements of the electromagnetic spectrum of a scene on multiple bands ( typically about a hundred or more ) , which offers many perspectives over traditional color RGB imaging . For instance , the high-dimensional information present in a single pixel is sometimes sufficient to identify the signature of a particular material , which is of course infeasible in the RGB domain . Not surprisingly , hyperspectral imaging is then of utmost importance and has a huge number of scientific and technological applications such as remote sensing [ 8 , 22 , 46 ] , quality evaluation of food products [ 16 , 19 , 35 ] , medical imaging [ 2 , 18 , 37 ] , agriculture and forestry [ 1 , 36 , 40 ] , microscopy imaging in biology [ 25 , 56 ] , or exoplanet detection in astronomy [ 24 ] . Information contained in hyperspectral signals is much richer than in RGB images , but the price to pay is the need to deal with complex degradations that may arise from multiple sources , including sparse noise with specific patterns ( stripes ) , in addition to photon and thermal noise [ 29 , 51 ] . As a consequence , HSI denoising is a crucial pre-processing step to enhance the image quality before using data in downstream tasks such as semantic segmentation or spectral unmixing [ 30 ] . A second issue is the lack of large-scale collection of ground-truth high-quality signals and the large diversity of sensor types , which makes it particularly challenging to train machine learning models for restoration such as convolutional neural networks . To deal with the scarcity of ground-truth data , most successful approaches typically encode strong prior knowledge about data within the model architecture , which may be low-rank representations of input patches [ 17 , 23 , 53 , 60 , 70 ] , sparse coding [ 13 , 21 , 23 ] , or image self-similarities [ 39 , 49 , 71 ] , which have proven to be very powerful in the RGB domain [ 9 ] . ∗Equal contribution 1Code is available at https : //github.com/inria-thoth/T3SC . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we propose a fully interpretable machine learning model for hyperspectral images that may be seen as a hybrid approach between deep learning techniques , where parameters can be learned end to end with supervised data , and classical methods that essentially rely on image priors . Since designing an appropriate image prior by hand is very hard , our goal is to benefit from deep learning principles ( here , differentiable programming [ 6 ] ) while encoding domain knowledge and physical rules about hyperspectral data directly into the model architecture , which we believe is a key to develop robust approaches that do not require massive amounts of training data . More precisely , we introduce a novel trainable spectral-spatial sparse coding model with two layers , which performs the following operations : ( i ) The first layer decomposes the spectrum measured at each pixel as a sparse linear combination of a few elements from a learned dictionary , thus performing a form of linear spectral unmixing per pixel , where dictionary elements can be seen as basis elements for spectral responses of materials present in the scene . ( ii ) The second layer builds upon the output of the first one , which is represented as a two-dimensional feature map , and sparsely encodes patches on a dictionary in order to take into account spatial relationships between pixels within small receptive fields . To further reduce the number of parameters to learn and leverage classical prior knowledge about spectral signals [ 60 ] , we also assume that the dictionary elements admit a low-rank structure—that is , dictionary elements are near separable in the space and spectrum domains , as detailed later . Even though dictionary learning has been originally introduced for unsupervised learning [ 42 , 47 ] , we adopt an unrolled optimization procedure inspired by the LISTA algorithm [ 26 ] , which has been very successful in imaging problems for training sparse coding models from supervised data [ 33 , 34 , 55 , 63 ] . Our motivation for adopting a two-layer model is to provide a shared architecture for different HSI sensors , which often involve a different number of bands with different spectral responses . Our solution consists of learning sensor-specific dictionaries for the first layer , while the dictionary of second layer is shared across modalities . This allows training simultaneously on several HSI signals , the first layer mapping input data to a common space , before processing data by the second layer . We experimentally evaluate our HSI model on standard denoising benchmarks , showing a significant improvement over the state of the art ( including deep learning models and more traditional baselines ) , while being computationally very efficient at test time . Perhaps more important than pure quantitative results , we believe that our work also draws interesting conclusions for machine learning . First , by encoding prior knowledge within the model architecture directly , we obtain models achieving excellent results with a relatively small number of parameters to learn , a conclusion also shared by [ 33 , 34 ] for RGB imaging ; nevertheless , the effect is stronger in our work due to the scarcity of training data for HSI denoising and the difficulty to train deep learning models for this task . Second , we also show that interpretable architectures are useful : our model architecture can adapt to different noise levels per band and modify the encoding function at test time in a principled manner , making it well suited for solving blind denoising problems that are crucial for processing hyperspectral signals . 2 Related Work on Hyperspectral Image Denoising . Learning-free and low-rank approaches . Classical image denoising methods such as BM3D [ 12 ] may be applied independently to each spectral band of HSI signals , but such an approach fails to capture relations between channels ; Not surprisingly , multi-band techniques such as BM4D [ 39 ] have been shown to perform better for HSI , and other variants were subsequently proposed such as GLF [ 71 ] . Tensor-based methods such as LLRT [ 11 ] are able to exploit the underlying low-rank structure of HSI signals [ 17 , 53 , 67 ] and have shown particularly effective when combined with a non-local image prior as in NGMeet [ 27 ] . Finally , other approaches adapt traditional image processing priors such as total variation [ 65 , 59 ] , or wavelet sparsity [ 48 , 52 ] but they tend to perform worse than GLF , LLRT , or NGMeet , see [ 31 ] for a survey on denoising techniques for HSI . Sparse coding models . Dictionary learning [ 47 ] is an unsupervised learning technique consisting of representing a signal as a linear combination of a few elements from a learned dictionary , which has shown to be very effective for various image restoration tasks [ 15 , 43 ] . Several approaches have then combined dictionary learning and low-rank regularization . For instance , 3D patches are represented as tensors in [ 49 ] and are encoded by using spatial-spectral dictionaries [ 57 ] . In [ 70 ] , 2D patches are extracted from the band-vectorized representation of the 3D HSI data and sparsely encoded on a dictionary , while encouraging low-rank representations with a trace norm penalty on the reconstructed image . The low-rank constraint can also be enforced by designing the dictionary as the result of the matrix multiplication between spatial and spectral dictionaries learned by principal component analysis as in [ 21 ] . However , these methods typically compute sparse representations with an iterative optimization procedure , which may be computationally demanding at test time . Deep learning . Like BM3D above , convolutional neural networks for grayscale image denoising ( e.g. , DnCNN [ 68 ] ) may also be applied to each spectral band , which is of course suboptimal . Because deep neural networks have been highly successful for RGB images with often low computational inference cost , there have been many attempts to design deep neural networks dedicated to HSI denoising . For instance , to account for the large number of hyperspectral bands , several approaches based on convolutional neural networks are operating on sliding windows in the spectral domain , [ 38 , 54 , 66 ] , which allows training models on signals with different number of spectral bands , but the sliding window significantly increases the inference time . More precisely , attention layers are used in [ 54 ] , while more traditional CNNs are used in [ 38 ] , possibly with residual connections [ 66 ] . Recently , an approach based on recurrent architecture was proposed in [ 62 ] to process signals with an arbitrary number of bands , achieving impressive results for various denoising tasks . Hybrid approaches . SMDS-Net [ 63 ] adopts a hybrid approach between sparse coding and deep learning models by adapting the RGB image restoration method of [ 34 ] to HSI images . The resulting pipeline however lacks interpretability : SMDS-Net first denoises the input image with non-local means [ 9 ] , then performs subspace projection in the spectral domain using HySime [ 7 ] , before sparsely encoding 3D patches ( cubes ) with a trainable version of Tensor-based ISTA [ 50 ] . Although this method reduces considerably the number of parameters in comparison to vanilla deep learning models , the spectral sliding windows approach lacks interpretability since the same denoising procedure is applied across different bands , which may not suffer from the same level of noise . In contrast , we propose a much simpler sparse coding model , which is physically consistent with the nature of hyperspectral signals , by introducing a novel differentiable low-rank sparse coding layer . 3 A Trainable Spectral-Spatial Sparse Coding Model ( T3SC ) . In this section , we introduce our trainable spectral-spatial sparse coding model dedicated to hyperspectral imaging , and start by presenting some preliminaries on sparse coding . 3.1 Background on Sparse Coding . Image denoising with dictionary learning . A classical approach introduced by Elad and Aharon [ 15 ] for image denoising consists in considering the set of small overlapping image patches ( e.g. , 8× 8 pixels ) from a noisy image , and compute a sparse approximation of these patches onto a learned dictionary . The clean estimates for each patch are then recombined to produce the full image . Formally , let us consider a noisy image y in Rc×h×w with c channels and two spatial dimensions . We denote by y1 , y2 , · · ·yn the n overlapping patches from y of size c× s× s , which we represent as vectors in Rm with m = cs2 . Assuming that a dictionary D = [ d1 , · · · , dp ] in Rm×p is given—we will discuss later how to obtain a “ good ” dictionary— each patch yi is processed by computing a sparse approximation : min αi∈Rp 1 2 ‖yi −Dαi‖2 + λ‖αi‖1 , ( 1 ) where ‖ · ‖1 is the l1-norm , which is known to induce sparsity in the problem solution [ 42 ] , and αi is the sparse code representing the patch yi , while λ controls the amount of regularization . Note that the ` 0-penalty , which counts the number of non-zero elements , could also be used , leading to a combinatorial problem whose solution is typically approximated by a greedy algorithm . After solving the n problems ( 1 ) , each patch yi admits a “ clean ” estimate Dαi . Because each pixel belongs to several patches , the full denoised image x̂ is obtained by averaging these estimates . Finding a good dictionary can be achieved in various manners . In classical dictionary learning algorithms , D is optimized such that the sum of the loss functions ( 1 ) is as small as possible , see [ 42 ] for a review . Adapting the dictionary with supervision is also possible [ 41 ] , as discussed next . Differentiable programming for sparse coding . The proximal gradient descent method called ISTA [ 20 ] is a classical algorithm for solving the Lasso problem in Eq . ( 1 ) , which consists of the following iterations α ( t+1 ) i = Sλ [ α ( t ) i + ηD > ( yi −Dα ( t ) i ) ] , ( 2 ) where η > 0 is a step-size and Sλ [ u ] = sign ( u ) max ( |u| − λ , 0 ) is the soft-thresholding operator , which is applied pointwise to each entry of an input vector . By noting that the above iteration can be seen as a sequence of affine transformations interleaved with pointwise non-linearities Sλ , it is then tempting to interpret T iterations ( 2 ) as a multilayer feed-forward neural network with shared weights between the T layers . Following such an insight , Gregor and LeCun have proposed the LISTA algorithm [ 26 ] , where the parameters are learned such that the sequence approximates well the solution of the sparse coding problem ( 1 ) . Interestingly , the LISTA algorithm can also be used to train dictionaries for supervised learning tasks . This is the approach chosen in [ 34 , 55 ] for image restoration , which considers the following iterations : α ( t+1 ) i = Sλ [ α ( t ) i + C > ( yi −Dα ( t ) i ) ] , ( 3 ) which differs from ( 2 ) with the presence of a matrix C of the same size as D. Even if the choice C = ηD ( which recovers ISTA ) is perfectly reasonable , using a different dictionary C has empirically shown to provide improvements in results quality [ 34 ] , probably due to faster convergence of the LISTA iterations . Then , given some fixed parameters C , D , a clean estimate Wα ( T ) i for each patch yi is obtained by using a dictionary W , where T is the number of LISTA steps . The reason for allowing a different dictionary W than D is to correct the potential bias due to ` 1-minimization . Finally , the denoised image x̂ is reconstructed by averaging the patch estimates : x̂ = 1 m n∑ i=1 RiWα ( T ) i , ( 4 ) where Ri is the linear operator that places the patch x̂i at position i in the image , and we assume—by neglecting border effects for simplicity—that each pixel admits the same number m of estimates . In contrast to classical restoration techniques based on dictionary learning , the LISTA point of view enables us to learn the model parameters C , D , W in a supervised fashion . Given a training set of pairs of noisy/clean images , we remark that the estimate x̂ is obtained from a noisy image y by a sequence of operations that are differentiable almost everywhere , as typical neural networks with rectified linear unit activation functions . A typical loss , which we optimize by stochastic gradient descent , is then min C , D , W , λ Ex , y [ ‖x̂ ( y ) − x‖2 ] , where ( x , y ) is a pair of clean/noisy images drawn from some training distribution from which we can sample , and x̂ ( y ) is the clean estimate obtained from ( 4 ) , given the noisy image y .

Summary: The paper proposes a deep learning based algorithm for hyperspectral image restoration that is interpretable, in the sense that the learning process separates the structure of the spatial and spectral dimensions. This allows the method to leverage the significant amount of available data for the spatial dimensions (e.g, image denoising). The method is computationally efficient.