A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration

1 Introduction . Hyperspectral imaging ( HSI ) enables measurements of the electromagnetic spectrum of a scene on multiple bands ( typically about a hundred or more ) , which offers many perspectives over traditional color RGB imaging . For instance , the high-dimensional information present in a single pixel is sometimes sufficient to identify the signature of a particular material , which is of course infeasible in the RGB domain . Not surprisingly , hyperspectral imaging is then of utmost importance and has a huge number of scientific and technological applications such as remote sensing [ 8 , 22 , 46 ] , quality evaluation of food products [ 16 , 19 , 35 ] , medical imaging [ 2 , 18 , 37 ] , agriculture and forestry [ 1 , 36 , 40 ] , microscopy imaging in biology [ 25 , 56 ] , or exoplanet detection in astronomy [ 24 ] . Information contained in hyperspectral signals is much richer than in RGB images , but the price to pay is the need to deal with complex degradations that may arise from multiple sources , including sparse noise with specific patterns ( stripes ) , in addition to photon and thermal noise [ 29 , 51 ] . As a consequence , HSI denoising is a crucial pre-processing step to enhance the image quality before using data in downstream tasks such as semantic segmentation or spectral unmixing [ 30 ] . A second issue is the lack of large-scale collection of ground-truth high-quality signals and the large diversity of sensor types , which makes it particularly challenging to train machine learning models for restoration such as convolutional neural networks . To deal with the scarcity of ground-truth data , most successful approaches typically encode strong prior knowledge about data within the model architecture , which may be low-rank representations of input patches [ 17 , 23 , 53 , 60 , 70 ] , sparse coding [ 13 , 21 , 23 ] , or image self-similarities [ 39 , 49 , 71 ] , which have proven to be very powerful in the RGB domain [ 9 ] . ∗Equal contribution 1Code is available at https : //github.com/inria-thoth/T3SC . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . In this paper , we propose a fully interpretable machine learning model for hyperspectral images that may be seen as a hybrid approach between deep learning techniques , where parameters can be learned end to end with supervised data , and classical methods that essentially rely on image priors . Since designing an appropriate image prior by hand is very hard , our goal is to benefit from deep learning principles ( here , differentiable programming [ 6 ] ) while encoding domain knowledge and physical rules about hyperspectral data directly into the model architecture , which we believe is a key to develop robust approaches that do not require massive amounts of training data . More precisely , we introduce a novel trainable spectral-spatial sparse coding model with two layers , which performs the following operations : ( i ) The first layer decomposes the spectrum measured at each pixel as a sparse linear combination of a few elements from a learned dictionary , thus performing a form of linear spectral unmixing per pixel , where dictionary elements can be seen as basis elements for spectral responses of materials present in the scene . ( ii ) The second layer builds upon the output of the first one , which is represented as a two-dimensional feature map , and sparsely encodes patches on a dictionary in order to take into account spatial relationships between pixels within small receptive fields . To further reduce the number of parameters to learn and leverage classical prior knowledge about spectral signals [ 60 ] , we also assume that the dictionary elements admit a low-rank structure—that is , dictionary elements are near separable in the space and spectrum domains , as detailed later . Even though dictionary learning has been originally introduced for unsupervised learning [ 42 , 47 ] , we adopt an unrolled optimization procedure inspired by the LISTA algorithm [ 26 ] , which has been very successful in imaging problems for training sparse coding models from supervised data [ 33 , 34 , 55 , 63 ] . Our motivation for adopting a two-layer model is to provide a shared architecture for different HSI sensors , which often involve a different number of bands with different spectral responses . Our solution consists of learning sensor-specific dictionaries for the first layer , while the dictionary of second layer is shared across modalities . This allows training simultaneously on several HSI signals , the first layer mapping input data to a common space , before processing data by the second layer . We experimentally evaluate our HSI model on standard denoising benchmarks , showing a significant improvement over the state of the art ( including deep learning models and more traditional baselines ) , while being computationally very efficient at test time . Perhaps more important than pure quantitative results , we believe that our work also draws interesting conclusions for machine learning . First , by encoding prior knowledge within the model architecture directly , we obtain models achieving excellent results with a relatively small number of parameters to learn , a conclusion also shared by [ 33 , 34 ] for RGB imaging ; nevertheless , the effect is stronger in our work due to the scarcity of training data for HSI denoising and the difficulty to train deep learning models for this task . Second , we also show that interpretable architectures are useful : our model architecture can adapt to different noise levels per band and modify the encoding function at test time in a principled manner , making it well suited for solving blind denoising problems that are crucial for processing hyperspectral signals . 2 Related Work on Hyperspectral Image Denoising . Learning-free and low-rank approaches . Classical image denoising methods such as BM3D [ 12 ] may be applied independently to each spectral band of HSI signals , but such an approach fails to capture relations between channels ; Not surprisingly , multi-band techniques such as BM4D [ 39 ] have been shown to perform better for HSI , and other variants were subsequently proposed such as GLF [ 71 ] . Tensor-based methods such as LLRT [ 11 ] are able to exploit the underlying low-rank structure of HSI signals [ 17 , 53 , 67 ] and have shown particularly effective when combined with a non-local image prior as in NGMeet [ 27 ] . Finally , other approaches adapt traditional image processing priors such as total variation [ 65 , 59 ] , or wavelet sparsity [ 48 , 52 ] but they tend to perform worse than GLF , LLRT , or NGMeet , see [ 31 ] for a survey on denoising techniques for HSI . Sparse coding models . Dictionary learning [ 47 ] is an unsupervised learning technique consisting of representing a signal as a linear combination of a few elements from a learned dictionary , which has shown to be very effective for various image restoration tasks [ 15 , 43 ] . Several approaches have then combined dictionary learning and low-rank regularization . For instance , 3D patches are represented as tensors in [ 49 ] and are encoded by using spatial-spectral dictionaries [ 57 ] . In [ 70 ] , 2D patches are extracted from the band-vectorized representation of the 3D HSI data and sparsely encoded on a dictionary , while encouraging low-rank representations with a trace norm penalty on the reconstructed image . The low-rank constraint can also be enforced by designing the dictionary as the result of the matrix multiplication between spatial and spectral dictionaries learned by principal component analysis as in [ 21 ] . However , these methods typically compute sparse representations with an iterative optimization procedure , which may be computationally demanding at test time . Deep learning . Like BM3D above , convolutional neural networks for grayscale image denoising ( e.g. , DnCNN [ 68 ] ) may also be applied to each spectral band , which is of course suboptimal . Because deep neural networks have been highly successful for RGB images with often low computational inference cost , there have been many attempts to design deep neural networks dedicated to HSI denoising . For instance , to account for the large number of hyperspectral bands , several approaches based on convolutional neural networks are operating on sliding windows in the spectral domain , [ 38 , 54 , 66 ] , which allows training models on signals with different number of spectral bands , but the sliding window significantly increases the inference time . More precisely , attention layers are used in [ 54 ] , while more traditional CNNs are used in [ 38 ] , possibly with residual connections [ 66 ] . Recently , an approach based on recurrent architecture was proposed in [ 62 ] to process signals with an arbitrary number of bands , achieving impressive results for various denoising tasks . Hybrid approaches . SMDS-Net [ 63 ] adopts a hybrid approach between sparse coding and deep learning models by adapting the RGB image restoration method of [ 34 ] to HSI images . The resulting pipeline however lacks interpretability : SMDS-Net first denoises the input image with non-local means [ 9 ] , then performs subspace projection in the spectral domain using HySime [ 7 ] , before sparsely encoding 3D patches ( cubes ) with a trainable version of Tensor-based ISTA [ 50 ] . Although this method reduces considerably the number of parameters in comparison to vanilla deep learning models , the spectral sliding windows approach lacks interpretability since the same denoising procedure is applied across different bands , which may not suffer from the same level of noise . In contrast , we propose a much simpler sparse coding model , which is physically consistent with the nature of hyperspectral signals , by introducing a novel differentiable low-rank sparse coding layer . 3 A Trainable Spectral-Spatial Sparse Coding Model ( T3SC ) . In this section , we introduce our trainable spectral-spatial sparse coding model dedicated to hyperspectral imaging , and start by presenting some preliminaries on sparse coding . 3.1 Background on Sparse Coding . Image denoising with dictionary learning . A classical approach introduced by Elad and Aharon [ 15 ] for image denoising consists in considering the set of small overlapping image patches ( e.g. , 8× 8 pixels ) from a noisy image , and compute a sparse approximation of these patches onto a learned dictionary . The clean estimates for each patch are then recombined to produce the full image . Formally , let us consider a noisy image y in Rc×h×w with c channels and two spatial dimensions . We denote by y1 , y2 , · · ·yn the n overlapping patches from y of size c× s× s , which we represent as vectors in Rm with m = cs2 . Assuming that a dictionary D = [ d1 , · · · , dp ] in Rm×p is given—we will discuss later how to obtain a “ good ” dictionary— each patch yi is processed by computing a sparse approximation : min αi∈Rp 1 2 ‖yi −Dαi‖2 + λ‖αi‖1 , ( 1 ) where ‖ · ‖1 is the l1-norm , which is known to induce sparsity in the problem solution [ 42 ] , and αi is the sparse code representing the patch yi , while λ controls the amount of regularization . Note that the ` 0-penalty , which counts the number of non-zero elements , could also be used , leading to a combinatorial problem whose solution is typically approximated by a greedy algorithm . After solving the n problems ( 1 ) , each patch yi admits a “ clean ” estimate Dαi . Because each pixel belongs to several patches , the full denoised image x̂ is obtained by averaging these estimates . Finding a good dictionary can be achieved in various manners . In classical dictionary learning algorithms , D is optimized such that the sum of the loss functions ( 1 ) is as small as possible , see [ 42 ] for a review . Adapting the dictionary with supervision is also possible [ 41 ] , as discussed next . Differentiable programming for sparse coding . The proximal gradient descent method called ISTA [ 20 ] is a classical algorithm for solving the Lasso problem in Eq . ( 1 ) , which consists of the following iterations α ( t+1 ) i = Sλ [ α ( t ) i + ηD > ( yi −Dα ( t ) i ) ] , ( 2 ) where η > 0 is a step-size and Sλ [ u ] = sign ( u ) max ( |u| − λ , 0 ) is the soft-thresholding operator , which is applied pointwise to each entry of an input vector . By noting that the above iteration can be seen as a sequence of affine transformations interleaved with pointwise non-linearities Sλ , it is then tempting to interpret T iterations ( 2 ) as a multilayer feed-forward neural network with shared weights between the T layers . Following such an insight , Gregor and LeCun have proposed the LISTA algorithm [ 26 ] , where the parameters are learned such that the sequence approximates well the solution of the sparse coding problem ( 1 ) . Interestingly , the LISTA algorithm can also be used to train dictionaries for supervised learning tasks . This is the approach chosen in [ 34 , 55 ] for image restoration , which considers the following iterations : α ( t+1 ) i = Sλ [ α ( t ) i + C > ( yi −Dα ( t ) i ) ] , ( 3 ) which differs from ( 2 ) with the presence of a matrix C of the same size as D. Even if the choice C = ηD ( which recovers ISTA ) is perfectly reasonable , using a different dictionary C has empirically shown to provide improvements in results quality [ 34 ] , probably due to faster convergence of the LISTA iterations . Then , given some fixed parameters C , D , a clean estimate Wα ( T ) i for each patch yi is obtained by using a dictionary W , where T is the number of LISTA steps . The reason for allowing a different dictionary W than D is to correct the potential bias due to ` 1-minimization . Finally , the denoised image x̂ is reconstructed by averaging the patch estimates : x̂ = 1 m n∑ i=1 RiWα ( T ) i , ( 4 ) where Ri is the linear operator that places the patch x̂i at position i in the image , and we assume—by neglecting border effects for simplicity—that each pixel admits the same number m of estimates . In contrast to classical restoration techniques based on dictionary learning , the LISTA point of view enables us to learn the model parameters C , D , W in a supervised fashion . Given a training set of pairs of noisy/clean images , we remark that the estimate x̂ is obtained from a noisy image y by a sequence of operations that are differentiable almost everywhere , as typical neural networks with rectified linear unit activation functions . A typical loss , which we optimize by stochastic gradient descent , is then min C , D , W , λ Ex , y [ ‖x̂ ( y ) − x‖2 ] , where ( x , y ) is a pair of clean/noisy images drawn from some training distribution from which we can sample , and x̂ ( y ) is the clean estimate obtained from ( 4 ) , given the noisy image y .

Summary: This paper proposed a trainable spectral-spatial sparse coding model for hyperspectral image restoration. This paper proposed a hybrid approach based on sparse coding principles that retain the interpretability of classical techniques encoding domain knowledge with handcrafted image priors, while allowing to train model parameters end-to-end without massive amounts of data. The proposed model is claimed to be computationally efficient, and produce state-of-the-art restoration results.