Augmented Deep Unrolling Networks for Snapshot Compressive Hyperspectral Imaging
Authors: Xinran Qin
Yuhui Quan
Hui Ji
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Hyperspectral Imaging
Snapshot Compressive Imaging
Image Reconstruction
Deep Unrolling Networks

Abstract: Snapshot compressive hyperspectral imaging requires reconstructing a hyperspectral image from its snapshot measurement. This paper proposes an augmented deep unrolling neural network for solving such a challenging reconstruction problem. The proposed network is based on the unrolling of a proximal gradient descent algorithm with two innovative modules for gradient update and proximal mapping. The gradient update is modeled by a memory-assistant descent module motivated by the momentum-based acceleration heuristics. The proximal mapping is modeled by a sub-network with a cross-stage self-attention which effectively exploits inherent self-similarities of a hyperspectral image along the spectral axis, as well as enhancing the feature flow through the network. Moreover, a spectral geometry consistency loss is proposed to encourage the model to concentrate more on the geometric layer of spectral curves for better reconstruction. Extensive experiments on several datasets showed the performance advantage of our approach over the latest methods.

Augmented Deep Unrolling Networks for Snapshot Compressive Hyperspectral Imaging
Anonymous Author(s)
Affiliation
Address
email
Abstract
Snapshot compressive hyperspectral imaging requires reconstructing hyperspec- tral image from its snapshot measurement, This paper proposes an augmented deep unrolling neural network for solving such challenging reconstruction problem. The proposed network is based on the unrolling of a proximal gradient descent algorithm with two innovative modules for gradient update and proximal mapping. The gradient update is modeled by a memory assitan-a descent module motivated by the momentum- baseet acceleration heuristics. The proximal mapping mod- eled by a sub-network with a cross- self-attention which effectively exploits inherent self-similarities of hyperspectral image along the spectral axis, as well as enhancing the feature flow through the network. Moreover, spectral geometry consistency loss is proposed to encourage the model to concentrate more on the geometrie layer of spectral curves for better reconstruction. Extensive experiments on several datasets showed the performance advantage of our approach over the latest methods.
11
13
Introduction
16 Hyperspectral imaging captures hyperspectral image (HSI) which is 3D cube of intensities
17 that represents the integrals the radiance of a real scene across a wide range of spectral bands.
18 As an HSI provides rich spectral characteristies of objects of a scene, hyperspectral imaging has
19 found wide applications in many areas, e.g., agriculture, industry, and science. Snapshot compres-
20 sive spectral imaging M. often known as coded aperture snapshot spectral imaging (CASSI,, is compressed-sensiee based technique for rapid and efficient acquisition of HSIs. In contrast to traditional hyperspectral imaging techniques which use sensor array for measuring the object at
21
22
23 many spectral bands, the CASSI only collects a single coded 2D snapshot, which measures the object modulated by physical mask and disperser at the mixture of different wavelengths. reconstruction algorithm is then called to reconstruct the 3D HSI from its 2D compressive snapshot.
24
26 Let X A X denote HSI with spatial indices m, and spectral index X. In general, the snapshot from CASSI device can be expressed as the following [
27
Y (m, n)) = Σ p(A)p(m
(m J(A), 2) + ((m, n),
(1)
A=1
28 where p(-) is the spectral response of the camera, ((. :) the coded aperture pattern, J(.) the dispersive function, and N the measurement noise. For convenience, re-express it in matrix-vector form:
t + n.
(2)
30 where t denotes the measurement matrix determined by p, v., and ær, y, the vectorized form of 31 X.Y Y. N, respectively. As Eq. (2 is an under-determined linear system with measurement noise,
32 HSI reconstruction needs to solve an ill-posed inverse problem.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

33 In recent years, deep learning has become a prominent approach for developing powerful solutions to HSI reconstruction; see e.g. 36 3 7 Most of them models the inverse mapping from 35 the 21) snapshot to its corresponding HSI by neural network (NN) trained over dataset. Among existing designs of NN architecture, deep unrolling is the most popular one for HSI reconstruction, as
34
36
37 it allows the inclusion of the physics of imaging. A typical deep unrolling network (DUN) unfolds
38 an iterative scheme for solving some regularized variational model of 2 where the regularization- concatenation of the steps that alternates between an updating step and a refinement step: x0)) Update, 2 Refine,
39
related parts are replaced by learnable NN modules. It can also be interpreted
40
(1) Updote, x1) Refine, xx(2) _
41
Despite extensive studies on HSI reconstruction, the practical need remains for the methods with better reconstruction accuracy.
42
43
The paper aims at developing a DUN for HSI reconstruction that brings noticeable performance improvement over existing deep NNs. The proposed DUN is based the proximal gradient descent
45
(PGD) algorithm [2], one often seen iterative scheme for solving inverse problems in imaging. The PGD algorithm alternatively iterates between the following two steps:
46
47
1. A gradient descent step for updating the estimate of the image
48
2. A proximal mapping for refining the estimate via fitting some regularization term.
49 In comparison to existing DUNs for HSI reconstructions, there are three innovations in the design and training of the proposed one:
50
51
1. Updating step: Modeling the gradient descent step using an NN block with a momentum- motivated memory- module which is implemented by long short-term memory.
52
53
2. Refinement step: Modeling the proximal mapping by a sub-NN with a across-stage self- attention module, for exploiting specific characteristics of HSIs and efficient feature flow.
54
3. Training loss: A spectral geometry consistency loss is proposed for regularizing the reconstruc- tion with better accuracy.
56
57
Learnable memory-ussstant module In most existing DUNs for HSI reconstruction, the updating step usually is some pre-defined non-learnable process, e-g. gradient-baset update. Gradient- bbase updates are in a zig-zag direction which slows down the movement to a minima. Also, the updates crawl near the minima or saddle points slowly the gradient magnitude vanishes rapidly over there.
58
59
60
61 A popular technique used for acceleration is the so-called momentum (e.g. RMSProp and Adam).
62
Instead of using only the current gradient, momentum accumulates the gradients of the past steps to determine the direction to go, which helps move more quickly towards the minima as it dampens the zig-zag oscillations and builds the speed quicken the convergence.
63
64
65
Motivated by the benefit brought by momentum in gradient- gradient-based update, we propose to learn an -N-based model for gradient- gradient-based update with the concept of momentum. As the effectiveness of momentum comes from its memory of the gradients of past steps, we propose an NN block with a memory-assistant mechanism such that it will leverage the gradient descents from previous stages, which is implemented using convolutional long short-term memory (ConvI.STM) units.
67
68
69
70
Cross-stage self-attention module An HSI has its specific physical characteristics. One is the self-similarity and strong correlation along the spectral axis, as the entries along the spectral axis measure the same object region but at different wavelengths. To exploit such specific physical property of HSIs, we propose a elff-attention module along the spectral axis. While not completely new in image reconstruction, our implementation different from existing ones by defining in a cross-stage manner.
71
72
73
is
74
75
One additional function for such cross-stage self-attention module is to exploit the similarity of the features learned over different stages by forming path between two different stages. Such similarities
77
78 among the featured learned at different stages come from the fact that the role of refinement step is supposed to the same across different stages. The benefit of utilizing such similarity is two-fold. One
79
80 for more efficient feature delivery across the full stages, and the other is for enabling interactions among the features at different stages during the training.
81
82 Loss on spectral geometry consistency In addition to the standard e, loss, a spectral geometry consistency loss is proposed for training the DUN for HSI reconstruction. Such loss encourages the

---

84
model to concentrate more on the profile of spectral changes during reconstruction, which helps to improve the reconstruction accuracy as empirically observed.
a5
2 Related Work
86
87
By imposing certain priors on HSIs, regularization is a widely-used approach to solving the problem of HSI reconstruction. The priors for natural images have been extended to HSIs, e.g, sparsity prior
88
89 in image gradients used in total variation [3 [4). sparsity prior under a learned dictionary 22 [3] and non-local self-similarity prior in the form of low-rankness for spatial-spectral patches 19].
91
These pre-defined priors are often insufficient for the HSIs with complex and diverse structures.
92
There is an increasing trend to use the implicit image prior encoded in pre-trained or untrained NN for regularization. Plug-and-play methods [4 20 21 employ the NNs pre-trained on the denoising tasks of HSIs or natural images to regularize the reconstruction process. However, pre- trained denoising NNs are usually not very effective to handle the noise and artifacts generated in the iterative reconstruction process. Self-supervised learning methods (22 2 use an untrained NN to re-parameterize the latent HSI and train it to match the observed snapshot. Such an online learning scheme is computationally expensive and cannot leverage the knowledge from external data.
93
94
95
96
97
98
99
has been a prominent approach that end-to-end train a DNN that maps a snapshot to the latent HSI; see e.g. [24 S 25 26
100
9 []. Many existing studies employ the DUN architecturee.g. [3 6 7 4 Recall that DUN often consists of pairs of steps: one step for updating the estimate of the latent HSI
101
102
and the other step for refining the estimate with a learnable prior. Most existing works focus on the latter, which can be viewed as a denoising NN that exploits different image priors, e-g., spatial-
103
prior
3 noon-local self-similarity prior 66) and patch-level Gaussian scale mixture prior 77.
105
Learning updating steps in DUNs Zhang et al. [4] replaced the operators &, $ appearing in the gradient descent step of PGD by convolutions and residual blocks, with channel attention block to estimate the step size in PGD from the estimate output by the previous stage. Different from that, we do not learn those operators but utilize them to have better update step. Working on natural image recovery rather than on HSI reconstruction, Mou et al. [27] used a residual block to estimate the LSTM to leverage the dependency between different
107
108
109
110
gradient descent step. In comparison,
111
stages for estimating the updating step.
112
Self-attention for HSI reconstruction Self-attention (SA) has been exploited in existing works for HSI reconstruction. Miao et al. [5] used generative adversarial network with SA for the initial stage SA modules to exploit the spatial-spectral in the NN. Meng el al. 28 used three spatial-spectral attention module with efficient feature
113
114
115
correlation of an HSI. Hu et al. [9] develops a fusion. comparison to these methods, ours treats spectral maps as tokens for SA and calculates the
116
SA along the spectral dimension. This shares similar idea with parallel work [8] which also treats spectral maps as tokens in a transformer-bat manner which enhances the feature flow at the same time.
118
model. Different from it, we use SA in a cross-stage
120
Training loss for HSI reconstruction Most existing NNs for HSI reconstruction are trained by the standard meaan-qqured-errror loss & loss. Hu et al. [9] introduced frequency-domain loss to narrow the frequency-domain discrepancy between network predictions and ground truths. In comparison, the loss we proposed narrows the discrepancy in terms of spectral geometric changes.
122
123
3 Proposed Approach
125
The proposed DUN for HSI reconstruction is based on the PGD algorithm 12] for the following optimization model regularized by the functional R:
128
min |
- + XR(x), E
(3)
127
The PGD algorithm for solving Eq. 3 alternately iterates between two steps: gradient-descent (GD) step for updating the estimate, and proximal mapping (PM) step for refining the estimate by fitting
128
129
the functional R with encoded image prior: For A = 1.... K.
[GD]:
(k) (k-1) + (A) x T (y - tx (1-)),
(4)
[PM]:
z (k) Proxæ (m (k)) e argmin || r 112 + (x) (()
(5)
3

---

where (()) denotes step size. Most existing DUNs focus on modeling the PM step by an NN for data-driven prior. The GD step usually is kept unchanged with the learnable parameter ((A).
130
We propose a Alemory-Asiistane Descent (MAD) block to model the GD step (4) and Cross-stage Attentive Proximal (CAP) sub-network to model the PM step (5). The former functions as gradient descent across different stages for momentum- moomentun-motivated acceleration, which leads more efficient update than that only using the gradient at current stage. The latter is to utilize the self-similarities existing in HSI with cross-stage manner, which enable to exploit special characteristics of
135
135
137
HSIs and fasten feature flow through the DNN. In short, the proposed NN, called MadcapNet, is the concatenation of K stages, each of which contains pair of MAD block and CAP sub-network; see Figure T for the diagram of MadcapNet.
138
139
c(0)
Stage 1 c(3)
Stage k-1 (k-1)
Stage k e.0
Stage K
MAD
MAD
MAD
MAD
CAP
CAP
CAP
CAP
Stage k
Cross-stage Self-Attention
CAP
e00
(k-)()
ConvLSTM
Self Self-Attention
Cross- Stage-Stage Stage
[(R)
rishape
restape
reshaoe
3x3
3x3
3x3
(k)
1x1
1x1
1x1
(K)
z(k
z(k-1
y
Skip connection
3x3 Conv
1x1 Conv
ReLU
Triplet Attention
Figure 1: Diagram of the proposed augmented deep unrolling neural network for HSI reconstruction.
3.1 Memory-Assistam Descent Blocks
The MAD blocks are set of ConvLSTTM units 29 placed each stage of the NN, which utilizes 142 the long-range dependencies among all cascading stages for momentum-assisstee gradient update. In each MAD block, the gradient map is defined by
hk) = Φ T (y- (y-$x (h-1) a
(6)
144 is taken as the input for the k-tth ConvLSTM unit, which introduces information on gradient descent. Let h (k), (kk) denote the hidden state and cell state in the ConvLSTM at the k-th stage respectively,
145
where h (8) of the same size as (A) The MAD block is defined as
148
(k)
(k) = ConvI_STM(ze (k) (*),))( (k--)) clle
147
for k = 1, 1,.... K. Different from original ConvLSTM units which use the previous hidden state (k-1) as input, we replace h (k-1) by (*-11, the output from the CAP sub-network of the previous stage. The motivation behind is to utilize the current gradient decent defined over (R-1)) Then, h (k)
148
149
used as the input of the CAP sub-network and c(k) is fed to the MAD block at the next stage as an accumulator of state information.
150
152 In the k-th stage, the CSMVLSTM unit calculates h&s Ck by the following rules
(k) fx ② (k- + (()) o (
(8)
h (8) (k) o (
(9)

---

153 where O denotes Hadamard product, and igs fx. Ok- Ih denote the input gate, forget gate, output
154 gate, and the intermediate result, respectively, which are calculated as follows:
{(k) = sigmoid W mi (k) + Wxic (k-1) + b;),
(10)
f(h) = sigmoid Wmru (R) + Wxrx (R-1) b,).
(11)
g(و W Wmg (k)
+ W xx + bgg
(12)
(k) sigmoid W ta W 2k (k-1) bo),
(13)
where W.. are implemented by 3 x 3 convolutional layers with bias terms b-,
156
3.2
Cross-stage Attentive Proximal Sub-networks
157
The CAP blocks function as a learnable PM step (5) which refines the estimate from the MAD block. It can be understood as a denoising NN by interpreting the estimation residual as noise. Given (k) (of the same size as 2) from the MAD block as input, we map it to feature tensor z(k) via
158
159
180
convolutional layer, which then processed by a stoss-stage SA module. Afterward, the results are
161
fed to a sequence of convolutional layers with rectified linear units (ReLUs) and triplet attention [30].
The output with the same size as æ is combined with the input (*) via a skip connection, yielding the reconstructed HSI x (R) at the current stage. See Figure for the details.
162
163
164
Recall that SA 31 relates input feature tokens to compute a refined feature representation. It first generates a key/query/value vector of length d from each token, and the key/query/value vectors are stored as K, Q, V respectively. Then, SA is calculated as follows:
165
- 1 LOKTV.
SA(Q, K, V) = softmax
(14)
187
We treat each feature channel as a token so as to exploit the self-iimilarities among feature channels. Such tokens are aligned due to natural aligmment of spectral slices of an HSI. In the kth stage, rather
188
than use the feature (A) at current stage to calculate (k(k) Q(4), V(A), we only use 2k) for Q(K)
189
while using the feature z( (R=-1) of previous stage for (()), (k), Concretely, we calculate
170
w(lk) w (k) Qd Qp
(x), K wi(h) (k) (k) Kp
wilk)
(k-1)) (A) w
(k) w (k-11, (15)
Vp
where W (<p) (-d) are 1 x 1 convolutions and 3 x 3 depth-wise convolutions respectively for better encoding spatial-chanme context.
171
172
The motivation of the cross-stage strategy is follows. The DUN architecture alternates between the update and the refinement. Since the CAP sub-networks at different stages play the same role of refinement, their extracted features should be highly correlated and the features extracted from the previous stage provide good initials for the corresponding ones at the next stage. However, the aforementioned pipeline does not utilize such correlations for more efficient training, which may result in a bottleneck for features flowing through the whole DUN. The proposed cross-stage SA scheme forms path between two stages, which allows efficient feature transmission during inference and enhances feature interactions during training.
174
176
177
178
179
180
The mutli-hean strategy [1 is adopted for the cross- cross-stage SA. First, split the key/query/value ma-
trices into H heads along channel dimension: ((k) (+) .... Q (k) (R)) |K| .... k (k)
182
and (A) [V (k)
(k), Then, the output is calculated
O = SA(O / (()))
(16)
184
which is reshaped for subsequent processing.
185
3.3
Loss function for Training
To better train NN for HSI reconstruction, we propose an additional loss called spectral geometry
187 consistency (SGC) loss. For an HSI X E IR MxNXA, we define the geometry map D(x) as follows.
D(X) V. (sign (Vx)) E E (-1,0, MxNXA.
(17)
5

---

188
where V. calculates the gradient along the spectral axis, and sign(-) denotes element- wise sign function. For spatial location (mo, D ((X) Im No: -] indicates the wavelengths where the monotony of spectral values changes, which is one geometrical property of the spectral curve. Based on D, the SGC loss emphasizes the geometrical layout consistency between the reconstructed HSI and ground truth.
189
190
191
192
193
Considering HSIs exhibit high spatial sparsity, the irrelevant dark regions are omitted for robustness. This is achieved by constructing a mask M that thresholds the max density along spectral dimension: Mx (m, n, A) = 1 if maxş X(m (m, n, A) 2 ag and Ootherwise. Let X, X. denote the reconstructed HSI and its ground truth respectively. The SGC loss is defined as
194
196
Lage
MM D(X)
Mx x D(X))l)
(18)
197
By minimizing Lage, the HSI predicted by the NN is biased to the one with the same wavelength-
198
density trends of ground truths, which helps to alleviate possible over-fitting. Then, the overall loss is
199
L - L1 7
|X- 111 + M @ ((X) M o D(X),,, 7 TeRt.
(19)
Experiments
200
201
We implement MadcapNet with PyTorch. The stage number is set to On all convolutional layers, the kernel sizes are all set to 3 x 3. and both the stride and padding number are set to 1. The head number H for the self-attention in CAP blocks is set 8. Regarding the training loss, we 5 for Mx and y = 0.5 for Eq. (19) The training is done via the Adam optimizer with fixed learning rate of 10 and a maximal epoch number of 200. The same data augmentation scheme as 7 is adopted, including rotation and flipping. All the models are trained and tested on an NVIDIA GeForce RTX 1080Ti GPU. Our code will be released on GitHub. upon paper's acceptance. Following 77. Peak- Soack-Sinnllto-WWee Noise-Ratio (PSNR) and Structured SIMilarity (SSIM) index are adopted the metrics to evaluate the reconstruction results quantitatively.
202
203
204
set a
255
205
206
207
208
210
4.1
Evaluation on Synthetic Data
211
CAVE and KAIST datasets Following 288 77. we use the CAVE dataset 332 containing 32 HSIs with 31 spectral bands for training, and 10 scenes with 31 spectral bands from the KAIST dataset [14]
212
213
for test. All these HSIs are cropped into patches with a spatial size of 256 x 256 and reduced to 28
214
wavelengths ranging from 450nm to 650nm via spectral interpolation. The snapshot measurements
215
are generated by the 256 x 256 mask of CASSI used in 22).
216
Ten existing methods are chosen for comparison, including (a) two conventional methods: GAP-
217
TV [33 and DeSCI 117: (b) one self-supervised learning-based method: PnP-DIP [22]; and (c)
seven supervised leasd methods: X-Net (5) HSSP (3) DNU (6, TSA-Net (28), DGSMP 77.
218
219
HDNet [9, and MST-L [8] The HSSP, DNU and DGSMP are based on DUNs. The HDNet and
220
MST-L are from two latest works accepted an upcoming conference.
221
The guantitative results are listed in Table I which are quoted from 8 9 whenever possible and otherwise obtained with released codes. It can be seen that our approach significantly outperforms the compared ones. Specifically, MadcapNet shows remarkable superior performance over other DUNs. It also surpasses MST-L and HDNet (i.e. two latest methods) with average PSNR gain of more than 1dB and 2dB respectively. Table allo compares the model complexity of different methods in terms of number of parameters and number of Giga Floating- point Operations Per Second (GFL.OS)). A Athoughh our model contains ConvI L.TM and self-attention blocks, it is still kept compact to maintain relatively-low model complexity. Among all compared methods, our MadcapNet has the smallest number of GLOPS, and it is smaller than all other models except DNU, These results show the practicability of MadcapNet for real applications. To conclude, our approach can achieve the best trade-off between performance and model complexity.
223
225
226
227
228
229
230
231
ICVL and Harvard datasets We also conduct experiments on the ICVL dataset 33 and the 232
Harvard dataset [34], respectively. The ICVL dataset consists of 201 HSIs of real-world objects, each
234
with 31 spectral bands collected from 400nm to 700 nm at a 10nm step. The Harvard dataset consists
of 50 outdoor scenes, each with 31 spectral bands collected from 420nm to 720mm at a 10nm step.

---

Table 1: Quantitative results in PSNR(dB) (even rows) and SSIM (odd rows) on KAIST dataset.
Method
#Param. #GFL.OPS Scene#1
#7
#8
#9
#10
Mean
26.82
22.89
26.31 30.65 23.64 21.85 23.76
21.98
22.63
23.10 24.36
GAP-TV
0.754
0.61
0,802 0.852 0.703 0.663
0,688 0.655 0,682 0,584 0.669
27.13
23.04 26.62 34.96 23.94 22.38 24.45 22.03 24.56 23,59 25.27
DeSCI
0.748
0.62 0.818 0.897 0.706 0.683 0.743 0.673 0.732 0.587 0.721
30.10
28.49 27.73 37.01 26.19 28.64 26.47 26.09 27.50 27.13 28.53
X-net
62.64M
117.98
0.849
0.805 0.870 0.934 0.817 0.853 0.806 0.831 0.826 0.816 0.841
31.48
31.09 28.96 34.56 28.53 30.83 28.71 30.09 30.43 28.78 30.35
HSSP
0.858
0.842 0.823 0.902 0.808 0.877 0.824 0.881 0.868 0.842 0.852
31.72
31.13 29.99 35.34 29.03 30.87 28.99 30.13 31.03 29.14 30.74
DNU
1.19M
163.48
0.863
0,846 0.845 0.908 0.833 0.887 0.839 0.885 0.876 0.849 0.863
32.68
27.26 31.30 40.54 29.79 30.39 28.18 29.44 34.51 28.51 31.26
PnP-DIP
33.85M
64.42
0.890
0.833 0,914 0,962 0.900 0.877 0.913 0.874 0.927 0.851 0.894
32.03
31.00 32.25 39.19 29.39 31,44 30.32 29.35 30.01 29.59 31.46
TSA-Net
44.25M
110.06
0.892
0.858 0.915 0.953 0.884 0.908 0.878 0.888 0.890 0.874 0.894
33.26
32.09 33.06 40.54 28.86 33.08 30.74 31.55 31.66 31.44 32.63
DGSMP
3.76M
646.65
0,915
0.898 0.925 0.964 0.882 0.937 0.886 0.923 0,911 0.925 0.917
34.95
32.52 34.52 43.00 32.49 35.96 29.18 34.00 34.56 32.22 34.34
HDNet
2.35M
154.00
0.948
0.953 0.957 0.981 0.957 0.965 0.937 0.961 0.958 0.950 0.957
35.40
35.87 36.51 42.27 32.77 34.80 33.66 32.67 35.39 32.50 35.18
MST-L
2.03M
28.15
0,941
0,944
0.953 0,973 0.947 0.955 0.925 0.948 0.949 0.941 0.948
36,24
37.49
37.07 42.85 34.09 35.61 35.37 33,96 36.67 33.12 36.32
MadcapNet
1.51M
24.24
0.951
0.961
0.963 0.981 0.962 0.966 0.949 0.962 0.960 0.948 0.961
236
Following the protocol of 35) 50 HSIs in the ICVL dataset and 9 HSIs in the Harvard dataset are used for test respectively, and the rest samples for training. All HSIs for training and test cropped into patches with a spatial size of 48 x 48, while keeping the band number unchanged. The snapshot measurements are generated by the 48 x 48 mask of CASSI used in
238
239
240
Six existing methods are selected for comparison, including (a) a conventional method: SSNR [6]] and (b) six supervised learning-basse methods: HSCNN [36]. A-eet [5). DNU (6). DTLP (37. and HDNet [9. The DNU and DTLP use DUNs, and the HDNet is a latest method.
242
243
See Table for the quantitative comparison. The results of the compared methods are cited from 337 The proposed one outperformed all other methods. with more than 0.85db PSNR improvement on both datasets. Such noticeable performance gains of MadcapNet over other DUNs again demonstrated the effectiveness of our network architecture.
244
245
246
Table 2: Quantitative results in PSNR(dB) and SSIM on ICVL. and Harvard datasets.
Dataset
Metric
SSNR
HSCNN
A-Net
DNU
DTLP
HDNet
MadcapNet
PSNR
30.40
28.45
29.01
32.61
34.53
36.38
37.60
ICVL
SSIM
0.943
0.934
0.946
0.966
0.977
0.981
0.985
PSNR
31.14
27.60
29.37
31.11
32.43
34.02
34.88
Harvard
SSIM
0.942
0.895
0.909
0.929
0.941
0.950
0.956
247
Visual inspection See Figure for the visualization of HSI reconstruction results on two samples from the KAIST and Harvard datasets respectively. The spectral curves (density versus wavelength) correspond to the points marked by green boxes in the RGB references. In the legends of both figures, we provide the curve correlation value between the result of compared method and the
248
249
250
251
ground truth. Those values show that the HSIs reconstructed by the proposed MadcapNet have
252
the highest correlation to the ground truths. We also visualize three spectral channels of an entire reconstructed HSI and zoom in the selected regions marked by yellow boxes. It can be seen that the
254
results of MadcapNet are more visually pleasing than that of other compared methods, with better reconstruction of structures.
7

---

RGB Ref.
Measurement
20
Tok 1aam 3.8271
Pehice Icam 2.647)
SCar
10c
S20
Wavelength
5,0
600
TSA-Net
PnP-DIP
DGMP
Ours
Ground Truth
RGB Ref. Mcasurement
10-
20
Densihy
BMU Iar à 9990
dee 20
X-Net
DNU
DTLP
Ours
Ground Truth
Figure 2: Visual comparison of HSI reconstruction on two samples from KAIST and Harvard datasets respectively. Left: spectra curves of the selected regions marked by green boxes. Right: reconstruction on the spectral channels.
256
4.2
Evaluation on Real Data
257
also conduct test on the real snapshots of spatial size 660 714 from [722). which are captured by real system with 28 wavelengths ranging from 450mm to 650mm and with 54-pixel dispersion in the column dimension. Following 7 28). we use the mask associated with that real system to generate snapshots both the CAVE and KAIST datasets, and then we inject 11-bit shot noise the snapshots for simulating real situations. The resulting data used to retrain our model. Due to the lack of ground truths in test data, we only compare the qualitative results of different methods. See Figure 3 for the reconstruction results on a real scene, and see more in the supplementary materials. The performance of MadcapNet is also good the real data. This indeed has demonstrated the good generalization performance of our model.
259
261
263
285
286
4.3
Ablation Studies
Ablation studies conducted the KAIST dataset. We form some baselines by removing
288
more main components of our approach. Concretely, we consider (a) replace the MAD blocks by
the GD steps 4 (b) replace the cross-stage SA in the CAP network with the inner-stage SA which
uses the features at current stage to calculate (k), (k), (k) in [5 (c) replace the cross-stage SA

---

Ref. Snapshot
DeSCI
TSA-Net
PnP-DIP
DGMP
Ours
Figure 3: Visual comparison of HSI reconstruction on real data, in terms of two spectral channels.
271
with same number of convolutional layers; (d) remove the SGC loss Lage- For fair comparison, each baseline is configured to have (nearly) the same number of parameters as the original model, by uniformly increasing the channel numbers of convolutional layers. The results are listed in Table
272
273
274
It can be seen that each main component in our approach plays an important role. Using the MAD blocks an alternate GD steps can improve PSNR by almost ldb. also brings improvement across all baseline settings. Benefiting from the power of SA, the cross-stage SA brings noticeable PSNR gain. In addition, the SA utilized in the cross-stage manner leads to around 0.36dB improvement in PSNR over that utilized in the inner-stage manner. The SGC loss also has a solid contribution to the performance. See Figure for an illustration of the effect of the SGC loss, where training with Cyge
276
277
278
279
makes the tendency of the predicted spectral curves closer to ground truths. See also supplementary materials for more results.
281
Table Results ablation studies on KAIST dataset.
Metric
w/o MAD
w/o CAP
Cross
w/o Lage
Original
PSNR(dB)
35.35
35.80 0.956
35.96 0.958
35.53 0.951
36.32
SSIM
0.947
0.961
w/o Ey:s GT
Eggs
W/o
13
Cox
Denety
Denalt
GT
GT
450 F7D 500 51h
455 475 520 525 Wiseelength (1mm)
456 415
455
450 475
575 589 455 650 Wacehengihh jum]
Wavelerglh wiahegghhhhc (mm)
Warehergth (rm)
Figure 4: Spectra of selected regions Scene#1 (first two) and Scene#5 (last two) of KAIST dataset.
5
Conclusion
283
In this paper, we proposed an augmented DUN for CASSI-based hyperspectral imaging. The proposed DUN is based on the unfolding of PGD, with three-fold augmentations: momentum mootui-ted module for improving the gradient descent steps, sub-network with cross-stae stage self-atte attentienio for exploiting self-similarities of an HSI and enhancing feature flow simultaneously. and loss to induce predictions biased spectral geometry consistency. The combination these augmentations leads to noticeable performance improvement in HSI reconstruction, which were demonstrated by extensive experiments. The proposed DUN also sees potential application to other compressive imaging problems. We will study it in the future.
285
ConvLSTMMassssteee
287
289

---

References
291
[1] Michael E Gehm, Renu John, David J Brady. Rebecca M Willett, and Timothy J Schulz. Single-shot compressive spectral imaging with dual-disperser architecture. Opt. Lett., 15(21):14013-14022 2007.
293
[2] Xing Lin, Yebin Liu, Jiamin Wu, and Qionghai Dai. Spatial-spectral encoded compressive hyperspectral imaging. ACM Trans. Graph., 1311--111, 2014.
295
296
[3] Lizhi Wang, Chen Sun, Ying Fu, Min H Kim, and Hua Huang. Hyperspectral image reconstruction using a deep spatial-spectral prior. In Proc. CVPR, pages 8032-8041, 2019.
297
298
[4] Xuanyu Zhang. Yongbing Zhang, Ruiqin Xiong, Qilin Sun, and Jian Zhang. Herosnet: Hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging. Proc. CVPR, 2022.
299
300
301
[5] Xin Miao, Xin Yuan, Yunchen Pu, and Vassilis Athitsos. X-net: Reconstruct hyperspectral images from a snapshot measurement. In Proc. CVPR, pages 4059-4069, 2019.
302
303
[6] Lizhi Wang, Chen Sun, Maoqing Zhang, Ying Fu, and Hua Huang. Dnu: deep non-local unrolling for computational spectral imaging. In Proc. CVPR, pages 1661-1671, 2020.
305
77] Tao Huang, Weisheng Dong, Xin Yuan, Jinjian Wu, and Guangming Shi. Deep gaussian scale mixture prior for spectral compressive imaging. In Proc. CVPR, pages 16216-16225, 2021.
307
[8] Yuanhao Cai, Jing Lin. Xiaowan Hu. Haoqian Wang. Xin Yuan, Yulun Zhang, Radu Timofte, and Luc Van Gool. Mask-guides spectral-wise transformer for efficient hyperspectral image reconstruction. Pruc. CVPR, 2022.
309
310
[9] Xiaowan Hu, Yuanhao Cai, Jing Lin, Haoqian Wang, Xin Yuan, Yulun Zhang, Radu Timofte, and Lue Van Gool. Hdnet: High-resolution dual-domain learning for spectral compressive imaging. Proc. CVPR, 2022.
311
312
313
[10] Ying Fu, Tao Zhang, Lizhi Wang, and Hua Huang. Coded hyperspectral image reconstruction using deep external and internal learning. IEEE Trans. Pattern Anal. Mach. Intell., 2021.
314
315
[1I] Patrick L Combettes and Jean- Christopte Pesquet. Proximal splitting methods in signal processing. In Fixed-point algoritims for inverse problems in science and engineering, pages 185-212. Springer, 2011.
316
317
[12] Atsushi Nitanda. Stochastic proximal gradient descent with acceleration techniques. Proc. NeurlPS, 27, 2014.
318
319
[13]
Xin Yuan. Generalized alternating projection based total variation minimization for compressive sensing. In Proc. ICIP, pages 2539-2543.. IEEE, 2016.
321
[14] Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. High hyperspectral reconstruction using spectral prior. ACM Trans. Graph., 36(6):218:1 13, 2017.
323
[15] Xin Yuan, Tsung-Han Tsai, Ruoyu Zhu, Patrick Llull, David Brady, and Lawrence Carin. Compressive hyperspectral imaging with side information. IEEE J. Sel. Top Signal Process, 96)9964-97,, 2015.
325
[16] Ying Fu, Yinqiang Zheng, Imari Sato, and Yoichi Sato. Exploiting spectral-spatial correlation for coded
326
hyperspectral image restoration. In Proc. CVPR, pages 3727-3736, 2016.
327
[17] Yang Liu, Xin Yuan, Jinli Suo, David J Brady, and Qionghai Dai. Rank minimization for snapshot compressive imaging. IEEE Trans. Pattern Anal. Mach. Intell., 41(12):29903300 -006, 2018.
328
[8]] Shipeng Zhang, Lizhi Wang, Ying Fu, Xiaoming Zhong, and Hua Huang. Computational hyperspectral imaging based on cimmension-dscciminne 2019.
330
low-rank tensor recovery. In Proc. CVPR, pages 10183-10192
331
332
[19] Yong Chen, Ting-Zhu Huang, Wei He, Naoto Yokoya, and Xi-Le Zhao. Hyperspectral image compressive sensing reconstruction using subspace-based nonlocal tensor ring decomposition. IEEE Transactions on Image Processing, 29:6813-6828, 2020.
333
334
335
[20] Xin Yuan, Yang Liu, Jinli Suo, and Qionghai Dai. Plug-and-play algorithms for large-scale snapshot compressive imaging. In Proc. CVPR, pages 1447-1457, 2020.
336
337
[21] Haiquan Qiu, Yao Wang. and Deyu Meng. Effective snapshot compressive-spenee -spettall imaging via deep denoising and total variation priors. In Proc. CVPR, pages 9127-9136, 2021.
10

---

339
[22] Ziyi Meng, Zhenming Yu, Kun Xu, and Xin Yuan. Self-supervised neural networks for spectral snapshot compressive imaging. Proc. CVPR, 2021.
[23] Yuhui Quan, Xinran Qin, Minggin Chen, and Yan Huang. High-quality self-supervised snapshot hyper- spectral imaging. In proc. ICASSP. pages 1526- 1530. IEEE, 2022.
343
[24]
Lizhi Wang. Tao Zhang. Ying Fu, and Hua Huang. Hyperreconnet: Joint coded aperture optimization and image reconstruction for compressive hyperspectral imaging. IEEE Trans. Image Process., 28(5):2257- 2270, 2018.
345
346
[25] Tao Zhang, Ying Fu, Lizhi Wang, and Hua Huang. Hyperspectral image reconstruction using deep extemal and internal learning. In Proc. CVPR, pages 8559-8568, 2019.
347
[26]
Kouhei Yorimoto and Xian-Hua Han. Hypermixnet: Hyperspectral image reconstruction with deep mixed network fryom a snapshot measurement. In Proc. ICCV Workshop, pages 1184-1193, 2021.
349
350
127| Chong Mou, Qian Wang, and Jian Zhang. Deep generalized unfolding networks for image restoration. Proc. CVPR, 2022.
351
[28] Ziyi Meng, Jiawei Ma, and Xin Yuan, End-to-end low cost compressive spectral imaging with spatial- spectral self-attention. In Proc. FCCV, pages 187-204. Springer, 2020.
353
354
[29] Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung. Wai-Kin Wong, and Wang-chun Woo. Con- volutional Istm network: A machine learning approach for precipitation nowcasting. Proc. NeurIPS, 28, 2015.
355
356
357
[30] Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai, and Qibin Hou. Rotate to attend: Convolutional triplct attention module. In Proc. WACV, pages 3139-3148, 2021.
358
359
[3] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proc. NeurlPS, 30, 2017.
360
361
[32] Fumihito Yasuma, Tomoo Mitsunaga, Daisuke Iso, and Shree K Nayar. Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process., 19(9):2241-22) 22552,331 2010.
382
384
[33] Boaz Arad and Ohad Ben Ben-Shahar. Sparse recovery of hyperspectral signal from natural rgb images. In Proc. ECCV, pages 19-34. Springer, 2016.
386
[34] Ayan Chakrabarti and Todd Zickler. Statistics of real-world hyperspectral images. In Proc. CVPR, pages 193 200. IEEE, 2011.
368
[35] Shipeng Zhang, Lizhi Wang, Lei Zhang, and Hua Huang. Learning tensor low-rank prior for hyperspectral image reconstruction. In Proc. CVPR, pages 12006-12015, 2021.
370
[36] Zhiwei Xiong. Zhan Shi, Huiqun Li, Lizhi Wang. Dong Liu, and Feng Wu. Hsenn: Can-based hyperspectral 2017.
image recovery from spectrally undersampled projections. In Proc. ICCV Workshop. pages
372
[37] Siming Zheng, Yang Liu, Ziyi Meng, Mu Qiao, Zhishen Tong, Xiaoyu Yang. Shensheng Han, and Xin Yuan. Deep plug-and-play priors for spectral snapshot compressive imaging. Photonics Res., 9(2:B11-BB2 2021.
374
Checklist
376
1. For all authors...
377
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
378
contributions and scope? [Yes]
379
(b) Did you describe the limitations of your work? [Yes] See supplemental material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplemental material.
381
382
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?[Yes
383
384
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
11

---

386
(b) Did you include complete proofs of all theoretical results? [N/A]
387
3. If you ran experiments..
388
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)? [No] But we promise
390
to release all our codes upon paper's acceptance, as stated in Section 4
381
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
392
395
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section +
396
397
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
398
(a) If your work uses existing assets, did you cite the creators? [Yes]
399
(b) Did you mention the license of the assets? [N/A]
400
(c) Did you include any new asset either in the supplemental material or as a URL? [N/A]
401
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
402
403
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
405
5. If you used crowdsourcing or conducted research with human subjects...
406
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
408
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
409
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
411
12


Reviews:
[{"review_id": "wp_Nu7QZ90B", "reviewer": "Reviewer_X9GH", "summary": "This paper is on spectral reconstruction from snapshot CASSI. It is based on the linear inverse model, and solved by merging model-based and learning-based methods in the framework of deep unrolling network. Compared with existing methods, memory-assistant descent blocks and cross-stage attentive proximal subnetworks are proposed, together with a new loss term. Experiments show that it outperforms exiting unrolling based methods and end-to-end methods. ", "questions": "Please refer to the weakpoints.", "limitations": "No potential negative social impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strong points: \n1.\tTo use deep unrolling for image/video restoration receives attention in recent years, and the authors proposed new add-ons to improve DUN for CASSI based spectral reconstruction. \n2.\tExperiment results have shown that the proposed method achieves favorable results than existing iterative methods, as well as the latest end-to-end learning based methods. \n\nWeak points: \n1.\tInitialization of the algorithm and its effect on convergence is not mentioned/discussed. \n2.\tModel/algorithm complexity and running time is not mentioned, especially the advantages/disadvantages when compared with end-to-end methods.\n3.\tMany results are directly quoted from other papers. Is there any reason for that? At least ,there are some randomness for data augmentation. To quote results from elsewhere tends to be improper. \n4.\tNo quantitative comparison is conducted on real data. Maybe a real dataset with GT should be prepared first.\n5.\tPersonally, the reviewer has severe concern about the real meaningfulness of trying to reconstruct spectra from CASSI. Given the spectral accuracy in Fig.2 and Fig4, most high-frequency details can not be recovered. To use several mosaiced filters can reach same level accuracy, yet has much better spatial resolution/details. Given this level of spectral details, reconstructed spectral curves are not meaningful for practical applications. One might directly use CASSI raw images for segmentation/recognition directly, without reconstruction. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "uoWdw1zEa6d", "reviewer": "Reviewer_MLWy", "summary": "This paper presents a method for reconstruction a hyperspectral image from its snapshot measurements. ", "questions": "Introduction should discuss the novelty and technical contributions of the proposed approach over previously proposed approaches.\nTheoretical background to support the proposed algorithms should be clearly described. ", "limitations": "The technical contribution of the proposed approach and theoretical background to support the proposed algorithm are not clear enough from the paper. \n", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper requires further improvements on paper organization. The technical contributions of the proposed approach and theoretical background to support the proposed algorithm are not clear enough from the paper. There are many places where the explanation is unclear. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "s3MJyMOSAV", "reviewer": "Reviewer_y9Ct", "summary": "The paper proposes a new unrolled neural network architecture for the problem of snapshot compressive hyperspectral imaging.\nThe architecture is inspired from proximal gradient descent, where both the gradient update and the proximal mapping steps are replaced by two recurrent modules.\nThe former (Memory-Assistant Descent) is based on ConvLSTM, and the latter (Cross-stage Attentive Proximal sub-network) uses a combination of convolutions, self-attention and triplet attention.\nThe authors compare against multiple recent approaches on several datasets.\n", "questions": "* Regarding generalization to real data:\n  * As previously stated, visual comparison on real data is not provided against the most performant methods (HDNet, MST-L). This would help assessing the performance of the method.\n  * A convincing argument would be to provide quantitative results on synthetic data **with noisy measurements**, similar to table 2 of the paper TSA-Net, where the authors argue that shot noise is realistic in this setting.\n* There is no evidence that the unrolled architecture is beneficial with this method. An ablation study with a few different number of iterations/stages would be more convincing (including the limit case of a single iteration).\n* Since both the Gradient Update and the Proximal Mapping modules benefits from using information across several stages/iterations, what is the justification for using different recurrent architectures (ConvLSTM and \"Cross-stage self-attention\") for tasks which are so similar in spirit ? If the authors disagree with this last statement, please explain why.\n* The use of triplet attention for the CAP module is mentioned, but it would be helpful to develop a bit more why it is used here.\n* It should be made clearer that the results presented in table 1 and table 2 are for noise-free synthetic data.\n* What is the batch size used ?\n* Minor:\n  * The same notation is used to denote convolutions (eq. 10-13 and 15) and matrix multiplication (eq. 14). It would be clearer to denote the convolution operation with a dedicated symbol (see for example the ConvLSTM paper), and more coherent as the Hadamard operator has its own explicit symbol.\n  * In eq 16, the symbol used for concatenation is ambiguous.\n  * l. 202: it is unclear what the authors refer to as \"padding number\". Another formulation would be clearer.\n  * l. 79: supposed to **be** (I guess)\n  * eq. 10-11: missing term with **C^(k-1)**\n  * l. 201: please provide citation for PyTorch\n  * Table 1: \n    * the table is difficult to read. It would benefit from enhancing the separation between methods (spacing or horizontal lines).\n    * the number of parameters of DNU should be in bold\n  * fig. 2-3 of main paper and fig. 3 of supplemental: DGMP -> DGMSP ?\n  * on visual comparison, it would be helpful to know which spectral channel are shown.\n  * on visual comparison on real data, it would be helpful to display the zoomed region on the reference and snapshot, as it is done with the results\n  * l. 149: decent -> descent\n\n", "limitations": "* The broader impact section is not included in the 9 pages limit. Instead it is addressed in the supplemental material.\n* The authors do not address the fact that their quantitative experiments are only valid on noise-free measurements only.\n* The authors adequately mention the limitations of the supervised framework, i.e. its dependency on the amount of data available.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* The method achieves very good performances on noise-free data\n* The performances are compared against several recent methods\n* Several datasets are used to show the superior performances of the proposed method\n* Ablation studies are provided for multiple parts of the method, allowing to validate the overall significance of the main contributions\n\n# Weaknesses\n* All quantitative results shown are valid on noise-free measurements only.\n  * No quantitative results are provided for noisy measurements, although the image formation model described in the introduction explicitly mentions the presence of noise on measurements. \n  * This is even more dissonant as the model is fine-tuned on synthetic **noisy** data (shot noise) before applying it to real data for visual assessment, which indicates that the noise-free setting is not very realistic.\n* The authors claim that the visual results on real data is sufficient evidence of the good generalization of the method on real measurement data (i.e. with noise). This is not a convincing argument as it is very difficult to see any improvement of the proposed method over its competitors on real data.\n* All quantitative results are provided without error bars.\n* No visual comparison is provided against the most competitive methods (HDNet, MST-L) on real data.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "aSOzpinN63K", "reviewer": "Reviewer_fWQz", "summary": "This paper proposes a proximal gradient descent (PGD) based deep unrolling network for snapshot compressive hyperspectral imaging, which employs a memory assistant descent module for gradient descent and a sub-network with cross-stage self-attention for proximal mapping. The experimental results show that the proposed method outperforms the compared methods.", "questions": "Please see strength and weakness", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n+ Deep unrolling network is an important technique for snapshot compressive sensing.\n+ Modeling correlation between different iteration stages in deep unrolling network is a good idea to improve the performance.\nWeakness\n- Novelty. Memory assistant deep unrolling network has been proposed in previous work, such as [1]. The relationship and the difference should be discussed. Besides, the comparison between these two methods is also necessary.\n- Interpretability. Deep unrolling network is famous for its interpretability, compared with conventional deep network. However, adding the connection between different iteration stages weaken the interpretability. Moreover, in this paper, employing ConvLSTM to model the correlation between gradient descent in different iteration stages, which does not follow the PGD iteration. Besides, for hyperspectral snapshot compressive sensing, employing other optimization methods, e.g, ADMM, always have a fast close-form solver for data-fidelity. Thus, gradient descent and further memory-assistant module may be unnecessary.\n- Experiment. PSNR and SSIM only evaluate the spatial fidelity between reconstructed HSI and groudtruth. Spectral fidelity evaluation metrics, e.g., SAM and ERGAS, are also necessary, especially the proposed method proposes a spectral loss.\n- Writing. The citation of RMSProp and Adam is lacking in line 61. There lacks blank between 0 and otherwise in line 195. Please checking the typos of this manuscript.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject