A Trainable Spectral-Spatial Sparse Coding Model for Hyperspectral Image Restoration
Year: 2021

Abstract:
Hyperspectral imaging offers new perspectives for diverse applications, ranging from the monitoring of the environment using airborne or satellite remote sensing, precision farming, food safety, planetary exploration, or astrophysics. Unfortunately, the spectral diversity of information comes at the expense of various sources of degradation, and the lack of accurate ground-truth “clean” hyperspectral signals acquired on the spot makes restoration tasks challenging. In particular, training deep neural networks for restoration is difficult, in contrast to traditional RGB imaging problems where deep models tend to shine. In this paper, we advocate instead for a hybrid approach based on sparse coding principles that retains the interpretability of classical techniques encoding domain knowledge with handcrafted image priors, while allowing to train model parameters end-to-end without massive amounts of data. We show on various denoising benchmarks that our method is computationally efficient and significantly outperforms the state of the art. 1

heading: 1 Introduction
text: Hyperspectral imaging (HSI) enables measurements of the electromagnetic spectrum of a scene on multiple bands (typically about a hundred or more), which offers many perspectives over traditional color RGB imaging. For instance, the high-dimensional information present in a single pixel is sometimes sufficient to identify the signature of a particular material, which is of course infeasible in the RGB domain. Not surprisingly, hyperspectral imaging is then of utmost importance and has a huge number of scientific and technological applications such as remote sensing [8, 22, 46], quality evaluation of food products [16, 19, 35], medical imaging [2, 18, 37], agriculture and forestry [1, 36, 40], microscopy imaging in biology [25, 56], or exoplanet detection in astronomy [24].
Information contained in hyperspectral signals is much richer than in RGB images, but the price to pay is the need to deal with complex degradations that may arise from multiple sources, including sparse noise with specific patterns (stripes), in addition to photon and thermal noise [29, 51]. As a consequence, HSI denoising is a crucial pre-processing step to enhance the image quality before using data in downstream tasks such as semantic segmentation or spectral unmixing [30]. A second issue is the lack of large-scale collection of ground-truth high-quality signals and the large diversity of sensor types, which makes it particularly challenging to train machine learning models for restoration such as convolutional neural networks. To deal with the scarcity of ground-truth data, most successful approaches typically encode strong prior knowledge about data within the model architecture, which may be low-rank representations of input patches [17, 23, 53, 60, 70], sparse coding [13, 21, 23], or image self-similarities [39, 49, 71], which have proven to be very powerful in the RGB domain [9].
∗Equal contribution 1Code is available at https://github.com/inria-thoth/T3SC.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
In this paper, we propose a fully interpretable machine learning model for hyperspectral images that may be seen as a hybrid approach between deep learning techniques, where parameters can be learned end to end with supervised data, and classical methods that essentially rely on image priors. Since designing an appropriate image prior by hand is very hard, our goal is to benefit from deep learning principles (here, differentiable programming [6]) while encoding domain knowledge and physical rules about hyperspectral data directly into the model architecture, which we believe is a key to develop robust approaches that do not require massive amounts of training data.
More precisely, we introduce a novel trainable spectral-spatial sparse coding model with two layers, which performs the following operations: (i) The first layer decomposes the spectrum measured at each pixel as a sparse linear combination of a few elements from a learned dictionary, thus performing a form of linear spectral unmixing per pixel, where dictionary elements can be seen as basis elements for spectral responses of materials present in the scene. (ii) The second layer builds upon the output of the first one, which is represented as a two-dimensional feature map, and sparsely encodes patches on a dictionary in order to take into account spatial relationships between pixels within small receptive fields. To further reduce the number of parameters to learn and leverage classical prior knowledge about spectral signals [60], we also assume that the dictionary elements admit a low-rank structure—that is, dictionary elements are near separable in the space and spectrum domains, as detailed later. Even though dictionary learning has been originally introduced for unsupervised learning [42, 47], we adopt an unrolled optimization procedure inspired by the LISTA algorithm [26], which has been very successful in imaging problems for training sparse coding models from supervised data [33, 34, 55, 63].
Our motivation for adopting a two-layer model is to provide a shared architecture for different HSI sensors, which often involve a different number of bands with different spectral responses. Our solution consists of learning sensor-specific dictionaries for the first layer, while the dictionary of second layer is shared across modalities. This allows training simultaneously on several HSI signals, the first layer mapping input data to a common space, before processing data by the second layer.
We experimentally evaluate our HSI model on standard denoising benchmarks, showing a significant improvement over the state of the art (including deep learning models and more traditional baselines), while being computationally very efficient at test time. Perhaps more important than pure quantitative results, we believe that our work also draws interesting conclusions for machine learning. First, by encoding prior knowledge within the model architecture directly, we obtain models achieving excellent results with a relatively small number of parameters to learn, a conclusion also shared by [33, 34] for RGB imaging; nevertheless, the effect is stronger in our work due to the scarcity of training data for HSI denoising and the difficulty to train deep learning models for this task. Second, we also show that interpretable architectures are useful: our model architecture can adapt to different noise levels per band and modify the encoding function at test time in a principled manner, making it well suited for solving blind denoising problems that are crucial for processing hyperspectral signals.

heading: 2 Related Work on Hyperspectral Image Denoising
text: Learning-free and low-rank approaches. Classical image denoising methods such as BM3D [12] may be applied independently to each spectral band of HSI signals, but such an approach fails to capture relations between channels; Not surprisingly, multi-band techniques such as BM4D [39] have been shown to perform better for HSI, and other variants were subsequently proposed such as GLF [71]. Tensor-based methods such as LLRT [11] are able to exploit the underlying low-rank structure of HSI signals [17, 53, 67] and have shown particularly effective when combined with a non-local image prior as in NGMeet [27]. Finally, other approaches adapt traditional image processing priors such as total variation [65, 59], or wavelet sparsity [48, 52] but they tend to perform worse than GLF, LLRT, or NGMeet, see [31] for a survey on denoising techniques for HSI.
Sparse coding models. Dictionary learning [47] is an unsupervised learning technique consisting of representing a signal as a linear combination of a few elements from a learned dictionary, which has shown to be very effective for various image restoration tasks [15, 43]. Several approaches have then combined dictionary learning and low-rank regularization. For instance, 3D patches are represented as tensors in [49] and are encoded by using spatial-spectral dictionaries [57]. In [70], 2D patches are extracted from the band-vectorized representation of the 3D HSI data and sparsely encoded on a dictionary, while encouraging low-rank representations with a trace norm penalty on
the reconstructed image. The low-rank constraint can also be enforced by designing the dictionary as the result of the matrix multiplication between spatial and spectral dictionaries learned by principal component analysis as in [21]. However, these methods typically compute sparse representations with an iterative optimization procedure, which may be computationally demanding at test time.
Deep learning. Like BM3D above, convolutional neural networks for grayscale image denoising (e.g., DnCNN [68]) may also be applied to each spectral band, which is of course suboptimal. Because deep neural networks have been highly successful for RGB images with often low computational inference cost, there have been many attempts to design deep neural networks dedicated to HSI denoising. For instance, to account for the large number of hyperspectral bands, several approaches based on convolutional neural networks are operating on sliding windows in the spectral domain, [38, 54, 66], which allows training models on signals with different number of spectral bands, but the sliding window significantly increases the inference time. More precisely, attention layers are used in [54], while more traditional CNNs are used in [38], possibly with residual connections [66]. Recently, an approach based on recurrent architecture was proposed in [62] to process signals with an arbitrary number of bands, achieving impressive results for various denoising tasks.
Hybrid approaches. SMDS-Net [63] adopts a hybrid approach between sparse coding and deep learning models by adapting the RGB image restoration method of [34] to HSI images. The resulting pipeline however lacks interpretability: SMDS-Net first denoises the input image with non-local means [9], then performs subspace projection in the spectral domain using HySime[7], before sparsely encoding 3D patches (cubes) with a trainable version of Tensor-based ISTA [50]. Although this method reduces considerably the number of parameters in comparison to vanilla deep learning models, the spectral sliding windows approach lacks interpretability since the same denoising procedure is applied across different bands, which may not suffer from the same level of noise. In contrast, we propose a much simpler sparse coding model, which is physically consistent with the nature of hyperspectral signals, by introducing a novel differentiable low-rank sparse coding layer.

heading: 3 A Trainable Spectral-Spatial Sparse Coding Model (T3SC)
text: In this section, we introduce our trainable spectral-spatial sparse coding model dedicated to hyperspectral imaging, and start by presenting some preliminaries on sparse coding.

heading: 3.1 Background on Sparse Coding
text: Image denoising with dictionary learning. A classical approach introduced by Elad and Aharon [15] for image denoising consists in considering the set of small overlapping image patches (e.g., 8× 8 pixels) from a noisy image, and compute a sparse approximation of these patches onto a learned dictionary. The clean estimates for each patch are then recombined to produce the full image.
Formally, let us consider a noisy image y in Rc×h×w with c channels and two spatial dimensions. We denote by y1,y2, · · ·yn the n overlapping patches from y of size c× s× s, which we represent as vectors in Rm with m = cs2. Assuming that a dictionary D = [d1, · · · ,dp] in Rm×p is given—we will discuss later how to obtain a “good” dictionary— each patch yi is processed by computing a sparse approximation:
min αi∈Rp
1 2 ‖yi −Dαi‖2 + λ‖αi‖1, (1)
where ‖ · ‖1 is the l1-norm, which is known to induce sparsity in the problem solution [42], and αi is the sparse code representing the patch yi, while λ controls the amount of regularization. Note that the `0-penalty, which counts the number of non-zero elements, could also be used, leading to a combinatorial problem whose solution is typically approximated by a greedy algorithm. After solving the n problems (1), each patch yi admits a “clean” estimate Dαi. Because each pixel belongs to several patches, the full denoised image x̂ is obtained by averaging these estimates.
Finding a good dictionary can be achieved in various manners. In classical dictionary learning algorithms, D is optimized such that the sum of the loss functions (1) is as small as possible, see [42] for a review. Adapting the dictionary with supervision is also possible [41], as discussed next.
Differentiable programming for sparse coding. The proximal gradient descent method called ISTA [20] is a classical algorithm for solving the Lasso problem in Eq. (1), which consists of the following iterations
α (t+1) i = Sλ
[ α
(t) i + ηD > ( yi −Dα(t)i )] , (2)
where η > 0 is a step-size and Sλ[u] = sign(u) max(|u| − λ, 0) is the soft-thresholding operator, which is applied pointwise to each entry of an input vector.
By noting that the above iteration can be seen as a sequence of affine transformations interleaved with pointwise non-linearities Sλ, it is then tempting to interpret T iterations (2) as a multilayer feed-forward neural network with shared weights between the T layers. Following such an insight, Gregor and LeCun have proposed the LISTA algorithm [26], where the parameters are learned such that the sequence approximates well the solution of the sparse coding problem (1).
Interestingly, the LISTA algorithm can also be used to train dictionaries for supervised learning tasks. This is the approach chosen in [34, 55] for image restoration, which considers the following iterations:
α (t+1) i = Sλ
[ α
(t) i + C > ( yi −Dα(t)i )] , (3)
which differs from (2) with the presence of a matrix C of the same size as D. Even if the choice C = ηD (which recovers ISTA) is perfectly reasonable, using a different dictionary C has empirically shown to provide improvements in results quality [34], probably due to faster convergence of the LISTA iterations. Then, given some fixed parameters C,D, a clean estimate Wα(T )i for each patch yi is obtained by using a dictionary W, where T is the number of LISTA steps. The reason for allowing a different dictionary W than D is to correct the potential bias due to `1-minimization.
Finally, the denoised image x̂ is reconstructed by averaging the patch estimates:
x̂ = 1
m n∑ i=1 RiWα (T ) i , (4)
where Ri is the linear operator that places the patch x̂i at position i in the image, and we assume—by neglecting border effects for simplicity—that each pixel admits the same number m of estimates.
In contrast to classical restoration techniques based on dictionary learning, the LISTA point of view enables us to learn the model parameters C,D,W in a supervised fashion. Given a training set of pairs of noisy/clean images, we remark that the estimate x̂ is obtained from a noisy image y by a sequence of operations that are differentiable almost everywhere, as typical neural networks with rectified linear unit activation functions. A typical loss, which we optimize by stochastic gradient descent, is then
min C,D,W,λ
Ex,y [ ‖x̂(y)− x‖2 ] ,
where (x,y) is a pair of clean/noisy images drawn from some training distribution from which we can sample, and x̂(y) is the clean estimate obtained from (4), given the noisy image y.

heading: 3.2 A Trainable Low-Rank Sparse Coding Layer
text: We are now in shape to introduce a trainable layer encoding both sparsity and low-rank principles.
Spatial-Spectral Representation. As shown in [10, 21], HSI patches can be well reconstructed by using only a few basis elements obtained by principal component analysis. The authors further decompose these into a Cartesian product of separate spectral and spatial dictionaries. In this paper, we adopt a slightly different approach, where we consider a single dictionary D = [d1, . . . ,dp] in Rm×p as in the previous section with m = cs2, but each element may be seen as a matrix of size c× s2 with low-rank structure. More precisely, we enforce the following representation
∀j ∈ 1, . . . , p, dj = vec (Uj ×Vj) , (5)
where Uj is in Rs 2×r, Vj is in Rr×c, r is the desired rank of the dictionary elements, and vec(.) is the operator than flattens a matrix to a vector. The hyperparameter r is typically small with r = 1, 2 or 3. When r = 1, the dictionary elements are said to be separable in the spectral and spatial domains, which we found to be a too stringent condition to achieve good reconstruction in practice.
The low-rank assumption allows us to build model with a reduced number of parameters, while encoding natural assumption about the data directly in the model architecture. Indeed, whereas a classical full-rank dictionary D admits cs2p parameters, the decomposition (5) yields dictionaries with (s2 + c)rp parameters only. Matrices C and W are parametrized in a similar manner.
Convolutional variant and implementation tricks. Whereas traditional sparse coding reconstructs local signals (patches) independently according to the iterations (3), another variant called convolutional sparse coding (CSC) represents the whole image by a sparse linear combination of dictionary elements placed at every possible location in the image [55]. From a mathematical point of view, the reconstruction loss for computing the codes αi given an input image y becomes
min {αi∈Rp}i=1,...,n
1
2 ∥∥∥∥∥y − 1m n∑ i=1 RiDαi ∥∥∥∥∥ 2 + λ n∑ i=1 ‖αi‖1. (6)
An iterative approach for computing these codes can be obtained by a simple modification of (3) consisting of replacing the quantity Dα(t)i by the i-th patch of the reconstructed image 1 m ∑n i=1 RiDα (t) i . All of these operations can be efficiently implemented in standard deep learning frameworks, since the corresponding operations corresponds to a transposed convolution with D, followed by convolution with C, see [55] for more details. In this paper, we experimented with the CSC variant (6) and SC one (1), both with low-rank dictionaries, which were previously described. We observed that CSC was providing slightly better results and was thus adopted in our experiments. Following [34], another implementation trick we use is to consider a different λ parameter per dictionary element, which slightly increases the number of parameters, while allowing to learn with a weighted `1-norm in (6).

heading: 3.3 The Two-Layer Sparse Coding Model with Sensor-Specific Layer
text: One of the main challenge in hyperspectral imaging is to train a model that can generalize to several types of sensors, which typically admit different number of spectral bands. Whereas learning a model that is tuned to a specific sensor is perfectly acceptable in many contexts, it is often useful to learn a model that is able to generalize across different types of HSI signals. To alleviate this issue, several strategies have been adopted such as (i) projecting signals onto a linear subspace of fixed dimension, with no guarantee that representations within this subspace can be comparable between different signals, or (ii) processing input data using a sliding window across the spectral domain.
In this paper, we address this issue by learning a two-layer model, presented in Figure 1, where the first layer is tuned to a specific sensor, whereas the second layer could be generic. Note that the
second layer carries most of the model parameters (about 20× more than in the first layer in our experiments). Formally, let us denote by α in Rp×h×w the sparse encoding of an input tensor y in Rc×h×w as previously described. A sparse coding layer Φ naturally yields an encoder and a decoder such that:
Φenc : y 7→ α, and Φdec : α 7→ 1 n n∑ i=1 RiWαi. (7)
Given a noisy image y, the denoising procedure described in the previous section with one layer can be written as
x̂(y) = Φdec ◦ Φenc(y).
Then, a straightforward multilayer extension of the procedure may consist of stacking several sparse coding layers Φ1, . . . ,ΦL together to form a multilayer sparse coding denoising model:
x̂(y) = Φdec1 ◦ · · · ◦ ΦdecL ◦ ΦencL ◦ · · · ◦ Φenc1 (y).
The model we propose is composed of two layers, as shown in Figure 1. The first layer encodes spectrally the input HSI image, meaning that it operates on 1× 1 patches, whereas the second layer encodes both spectrally and spatially the output of the first layer.

heading: 3.4 Noise Adaptive Sparse Coding
text: An advantage of using a model based on a sparse coding objective (1) is to give the ability to encode domain knowledge within the model architecture. For instance, the Lasso problem (1) seen from a maximization a posteriori estimator implicitly assumes that the noise is i.i.d. If the noise variance is different on each spectral band, a natural modification to the model is to introduce weights and use a weighted-`2 data fitting term (which could be applied as well to the CSC model of (6)):
min αi∈Rp
1
2 c∑ j=1 βj‖Mj(yi −Dαi)‖2 + λ‖αi‖1, (8)
where Mj is a linear operator that extracts band j from a given HSI signal. From a probabilistic point of view, if σ2j denotes the variance of the noise for band j, we may choose the corresponding weight βj to be proportional to 1/σ2j . Yet, estimating accurately σ 2 j is not always easy, and we have found it more effective to simply learn a parametric function βj = g(Mjy)—here, a very simple CNN with three layers, see supplementary material for details—which is applied independently to each band. It is then easy to modify the LISTA iterations accordingly to take into account these weights, and learn the model parameters jointly with those of the parametric function g.

heading: 3.5 Self-Supervised Learning: Blind-Band Denoising with No Ground Truth Data
text: Even though acquiring limited ground truth data for a specific sensor is often feasible, it is also interesting to be able to train models with no ground truth at all, e.g., for processing images without physical access to the sensor. In such an unsupervised setting, deep neural networks are typically trained for RGB images by using blind-spot denoising techniques [32], consisting of predicting pixel values given their context. Here, we propose a much simpler approach exploiting the spectral redundancy between channels. More precisely, each time we draw an image for training, we randomly mask one band (ore more), and train the model to reconstruct the missing band from the available ones. Formally, the training objective becomes
min C,D,W,λ
Ex,y,S ∑ j /∈S ‖Mj(x̂S(y)− y)‖2  , (9)
where S is the set of bands that are visible for computing the sparse codes αi, leading to a reconstructed image that we denote by x̂S(y). Formally, it would mean considering the objective (8), but replacing the sum ∑c j=1 by ∑ j∈S . This is in spirit similar to blind-spot denoising, except that bands are masked instead of pixels, making the resulting implementation much simpler.

heading: 4 Experiments
text: We now present various experiments to demonstrate the effectiveness of our approach for HSI denoising, but first, we discuss the difficulty of defining the state of the art in this field. We believe indeed that it is not always easy to compare learning-free from approaches based on supervised learning. These two classes of approaches have very different requirements/characteristics, making one class more relevant than the other one in some scenarios, and less in others. Table 1 summarizes their characteristics, displaying advantages and drawbacks of both approaches.
Benchmarked models. Keeping in mind the previous dichotomy, we choose to compare our method to traditional methods such as bandwise BM3D [12] (implementation based on [44, 45]), BM4D [39], GLF [71], LLRT [11], NGMeet [27]. We also included deep learning models in our benchmark such as HSID-CNN [66], HSI-SDeCNN [38] 3D-ADNet [54], SMDS-Net [63] and QRNN3D [62]. Results of HSID-CNN, HSI-SDeCNN and 3D-ADNet on Washington DC Mall (available in the Appendix) are taken directly from the corresponding papers, as the train/test split is the same. Otherwise, the results were obtained by running the code obtained directly from the authors, except for SMDS-Net, where our implementation turned out to be slightly more effective. Note that the same architecture for our model was used in all our experiments (see Appendix).
Datasets. We evaluate our approach on two datasets with significantly different properties.
• ICVL [4] consists of 204 images of size 1392 × 1300 with 31 bands. We used 100 images for training and 50 for testing as in [62] but with a different train/test split ensuring that similar images—e.g., picture from the same scene—are not used twice. • Washington DC Mall is perhaps the most widely used dataset2 for HSI denoising and consists of a high-quality image of size 1280× 307 with 191 bands. Following [54], we split the image into two sub-images of size 600× 307 and 480× 307 for training and one sub-image of size 200× 200 for testing. Even though the test image does not overlap with train images, they nevertheless share common characteristics. Interestingly, the amount of training data is very limited here.
Specific experiments were also conducted with the datasets APEX [28], Pavia3, Urban[58] and CAVE [64], which appear in the supplementary material.
Normalization. Before denoising, HSI images are normalized to [0, 1]. For remote sensing datasets, we pre-compute the 2nd and 98th percentiles for each band, on the whole the training set. Then, normalization is performed on train and test images by clipping each band between those percentiles before applying bandwise min-max normalization, similar to [5, 38]. For the close-range dataset ICVL, we simply apply global min-max normalization as in [63, 62].
Noise patterns. We evaluate our model against different types of synthetic noise:
• i.i.d Gaussian noise with known variance σ2, which is the same on all bands. • Gaussian noise with unknown band-dependent variance: We consider Gaussian noise with different
standard deviation σj for each band, which is uniformly drawn in a fixed interval. These standard deviations change from an image to the other and are unknown at test time. • Noise with spectrally correlated variance: We consider Gaussian noise with standard deviation σj varying continuously across bands, following a Gaussian curve, see details in the appendix. • Stripes noise : similar to [62], we applied additive stripes noise to 33% of bands. In those bands, 10-15% of columns are affected, meaning a value uniformly sampled in the interval [−0.25, 0.25] is added to them. Moreover, all bands are disturbed by Gaussian noise with noise intensity σ = 25.
2https://engineering.purdue.edu/~biehl/MultiSpec/hyperspectral.html 3http://www.ehu.eus/ccwintco/index.php?title=Hyperspectral_Remote_Sensing_Scenes
Metrics. In order to assess the performances the previous methods, we used five different indexes widely used for HSI restoration, namely (i) Mean Peak Signal-to-Noise Ratio (MPSNR), which is the classical PSNR metric averaged across bands; (ii) Mean Structural Similarity Index Measurement (MSSIM), which is based on the SSIM metric [61]; (iii) Mean Feature Similarity Index Measurement (MFSIM) introduced in [69]; (iv) Mean ERGAS [14], and (v) Mean Spectral Angle Map (MSAM) [3]. We use MPSNR and MSSIM in the main paper and report the other metrics in the appendix.
Implementation details. We trained our network by minimizing the MSE between the groundtruth and restored images. For ICVL, we follow the training procedure described in [62]: we first center crop training images to size 1024× 1024, then we extract patches of size 64× 64 at scales 1:1, 1:2, and 1:4, with stride 64, 32 and 32 respectively. The number of extracted patches for ICVL amounts to 52962. For Washington DC Mall, we do not crop training images and the patches are extracted with stride 16, 8 and 8, for a total of 1650 patches. One epoch in Washington DC Mall corresponds to 10 iterations on the training dataset. Basic data augmentation schemes such as 90◦ rotations and vertical/horizontal flipping are performed. Code and additional details about optimization, implementation, computational resources, are provided in the supplementary material. As reported in Table 4, augmenting the number unrolled iterations improves the denoising performances at the expense of inference time. Since the Spectral-Spatial SC layer is the most time-consuming, the number of unrolled iterations chosen for the first and second layers are 12 and 5 respectively.
Quantitative results on synthetic noise. We present in Table 2 the results obtained on the ICVL dataset (results on DCMall are presented in the appendix). Our method uses the vanilla model of Section 3.3 for the experiments with constant σ or correlated noise. For the blind denoising experiment with band-dependent σ or for the stripe noise experiment, we use the variant of Section 3.4, which is designed to deal with unknown noise level per channel. The method “T3SC-SSL” implements the self-supervised learning approach of Section 3.5, which does not rely on ground-truth data.
• Our supervised approach achieves state-of-the-art results (or is close to the best performing baseline) on all settings. GLF performs remarkably well given that this baseline is learning-free. • Our self-supervised method achieves a relatively good performance under i.i.d. Gaussian noise, but does not perform as well under more complex noise. This is a limitation of the approach which is perhaps expected and overcoming this limitation would require designing a different self-supervised learning scheme; this is an interesting problem, which is beyond the scope of this paper.
A visual result on ICVL is shown in Figure 2 for stripes noise. Inference times are provided in Table 3, showing that our approach is computationally efficient.
Results on real noise. We also conducted a denoising experiment on the Urban dataset, reporting a visual result in Figure 3. Deep models were pre-trained on the APEX dataset, which has the same number of channels as Urban (even though the sensors are different), with band-dependent noise with σ ∈ [0−55]. Please note that for this experiment we did not use Noise Adaptive Sparse Coding3.4 for T3SC, as it is highly dependent on the type of sensor used for training. We show that learning-based models trained on synthetic noise are able to transfer to real data.
Comments on the additional results presented in the appendix. The appendix also contains (i) results on the DCMall dataset including additional baselines mentioned above; (ii) error bars for parts of our experimental results in order to assess their statistical significance; (iii) an experiment when learning simultaneously on several datasets with different types of sensors showing that the second layer can be generic and effective at the same time; (iv) additional visual results; (v) various ablation studies to illustrate the importance of different components of our method.
Broader Impact
Our paper addresses the problem of denoising the signal, which is a key pre-processing step before using hyperspectral signals in concrete applications. As such, it is necessarily subject to dual use. For instance, HSI may be used for environmental monitoring, forestry, yield estimation in agriculture, natural disaster management planning, astronomy, archaeology, and medicine. Yet, HSI is also used by the petroleum industry for finding new oil fields, and has obvious military applications for surveillance. We believe the potential benefits of HSI for society are large enough to outweigh the
potential harm. Nevertheless, we are planning to implement appropriate dissemination strategies to mitigate the risk of misuse for this work (notably with restrictive software licenses), while targeting a gold standard regarding the scientific reproducibility of our results.
Acknowledgments and Funding
This project was supported by the ERC grant number 714381 (SOLARIS project) and by ANR 3IA MIAI@Grenoble Alpes (ANR-19-P3IA-0003). This work was granted access to the HPC resources of IDRIS under the allocation 2021-[AD011012382] made by GENCI.

Review summaries:
The paper proposes a deep learning based algorithm for hyperspectral image restoration that is interpretable, in the sense that the learning process separates the structure of the spatial and spectral dimensions. This allows the method to leverage the significant amount of available data for the spatial dimensions (e.g, image denoising). The method is computationally efficient.

This paper proposed a trainable spectral-spatial sparse coding model for hyperspectral image restoration. This paper proposed a hybrid approach based on sparse coding principles that retain the interpretability of classical techniques encoding domain knowledge with handcrafted image priors, while allowing to train model parameters end-to-end without massive amounts of data. The proposed model is claimed to be computationally efficient, and produce state-of-the-art restoration results.

This paper proposes a denoising method for hyperspectral images in a supervised and unsupervised manner. The author proposed a trainable sparse coding model with two-steps dictionary learning. Their method can also process different HSI with different number of bands. The evaluation outperforms the baselines in the majority of scenes.