Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
Authors: Yuanhao Cai
Jing Lin
Haoqian Wang
Xin Yuan
Henghui Ding
Yulun Zhang
Radu Timofte
Luc Van Gool
Venue: NeurIPS 2022 Accept (2022)
Keywords: Applications
Computer Vision
Low-level Vision
Image Restoration
Snapshot Compressive Imaging
Hyperspectral Image Reconstruction

Abstract: In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyperspectral image (HSI) reconstruction methods are employed to recover the spatial-spectral signal from a compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill-posedness degree from CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, showing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-Aware Unfolding Framework (DAUF) that estimates parameters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-based deep unfolding method, Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST), for HSI reconstruction. Experiments show that DAUHST surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models are publicly available at https://github.com/caiyuanhao1998/MST

Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging
Yuanhao Cai 1.2.*, JJig Lin 1.2.+ Haoqian Wang 1,2.f, Xin Yuan" 3
Henghui Ding 4. Yulun Zhang 4, Radu Timofte 4.5 Luc Van Gool Shenzhen International Graduate School, Tsinghua University,
2 Shenzhen Institute of Future Media Technology.
3 Westlake University, ETH Zürich, University of Würzburg
Abstract
In coded aperture snapshot spectral compressive imaging (CASSI) systems, hyper- spectral image (HSI) reconstruction methods are employed recover the spatial- spectral signal from compressed measurement. Among these algorithms, deep unfolding methods demonstrate promising performance but suffer from two issues. Firstly, they do not estimate the degradation patterns and ill- ill-posedness degree from CASSI to guide the iterative learning. Secondly, they are mainly CNN-based, show- ing limitations in capturing long-range dependencies. In this paper, we propose a principled Degradation-AWw Aware Unfolding Framework (DAUF) that estimates param- eters from the compressed image and physical mask, and then uses these parameters to control each iteration. Moreover, we customize a novel Half-Shuffle Transformer (HST) that simultaneously captures local contents and non-local dependencies. By plugging HST into DAUF, we establish the first Transformer-be method, Degradation-Awart Unfolding HSI reconstruction. Experiments show that DAUHST surpasses state-of-the-art methods while requiring cheaper computational and memory costs. Code and models are publicly available at https: / //g thub. com/ caiyuanhao hathhaa19991 1998/MST
deep unfolding Transformer (DAUHST), for
1 Introduction
Hyperspectral images (HSIs) have more spectral bands than normal RGB images to store more detailed information. Thus, HSIs are widely applied in image recognition [1, 2. 3]. object detection [4, 5. 6], tracking [7, 8. 9], medical image processing [10. 11. 12], remote sensing [13, 14, 15, 16],
40
Ours
7ug 9elg
DAURST
ADMM--Ntt
38
GAP-Not
DGSMP
36
DNU
PSNR( (dB)
etc. To obtain HSIs, traditional imaging systems use spec- trometers to scan the scenes along the spectral or spatial dimensions, usually requiring a long time. These imaging systems fail to capture dynamic objects. Recently, snap- shot compressive imaging (SCI) systems [17, 18, 19] have been developed to capture HSIs at video rate. Among these SCI systems, coded aperture snapshot spectral imaging (CASSI) [17, 20, 21] stands out for its impressive perfor- mance. CASSI uses a coded aperture and a disperser to modulate the HSI signal at different wavelengths, and then mixes all modulated signal to generate a 2D compressed measurement. Subsequently, HSI restoration methods employed solve the CASSI inverse problem, ie.,, restore the HSis from the measurement. These methods are divided into four categories.
Unfolding
84
Reng
Untolding
82
50g
7x0g
28
25
50
100
200
400
700
FLOPS (G)
Figure 1: PSNR-FL FEORSFLOPS comparisons of DAUHST and SOTA unfolding methods.
(i) Model-based methods [17, 22, 23, 24, 25, 26, 27, 28] rely hand-crafted image priors, e.g., total variation [26], sparsity [17, 22), low-rank [23], etc. These methods have theoretically proven
Equal Contribution, i Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

properties and can be interpreted. Yet, these methods need manual parameter tweaking, which slows down reconstruction. Also, they suffer from limited representation capacity and generalization ability.
(ii) Plug-and-play (PnP) algorithms [29, 30, 31] plug pre-trained denoising networks into traditional model-based methods to solve the HSI reconstruction problem. Nonetheless, the pre-trained networks in PnP methods are fixed without re-training, therefore limiting the performance.
(iil) End-to-end (E2E) algorithms employ powerful model, usually convolutional neural network (CNN) [12, 20, 32, 33], to learn the E2E mapping function from a measurement to the desired HSIs. E2E methods enjoy the power of deep learning. However, they learn a brute-fore mapping from the compressed measurement to the underlying spectral images, thereby ignoring the working principles of CASSI systems. They come without theoretically proven properties, interpretability, and flexibility because the imaging models widely differ from each other for various hardware systems.
(iv) Deep unfolding methods [34, 35, 36, 37, 38, 39] adopt multi-stage network to map the measurement into the HSI cube. Each stage usually includes two phases, i.e., linear projection followed by passing the signal through a single-stage network that learns the underlying denoiser prior. In deep unfolding methods, the network architecture is intuitively interpretable by explicitly characterizing the image priors and the system imaging model. Besides, these methods also enjoy the power of deep learning and thus have great potential. Yet, this potential has not been fully explored.
Existing deep unfolding algorithms suffer from two issues. (a) The iterative learning is highly related to the CASSI system. However, current unfolding methods do not estimate CASSI degradation patterns and ill-posedsess degree to adjust the linear projection and denoising network in each iteration. (b) Existing deep unfolding methods are mainly CNN-based, therefore showing limitations in capturing non-local self- self-similarity and long-range dependencies, both critical for HSI reconstruction.
Recently, the emerging Transformer [40] has provided a solution tackle the drawbacks of CNN. Due to its strong capability in modeling the interactions of non-local spatial regions, Transformer has been widely applied in image classification [41, 42, 43]. object detection [44, 45, 46]. semantic segmentation [47, 48, 49]. human pose estimation [50, 51, 52]. image restoration [53, 54, 55], etc. Yet, the use of Transformer is confronted with two main issues. (a) The computational complexity of global Transformer [42] is quadratic to the spatial dimensions. This cost is sometimes unaffordable. (b) The receptive fields of local Transformer [41] are limited within position-specifie windows. Thus, some tokens with highly-related contents can not match each other when computing self-attention.
To address the above problems, in this paper, we firstly formulate a principled Degradation-Awart Unfolding Framework (DAUF) based on maximum a posteriori (MAP) theory for HSI reconstruction. Different from previous deep unfolding methods, our DAUF implicitly estimates informative parame- ters from the degraded compressed measurement and the physical mask used in the modulation. Then DAUF feeds the parameters, which capture key cues of CASSI degradation patterns and ill ill-posedness degree, into each iteration to udaptively scale the linear projection and provide the noise level informa- tion for the denoising network. Secondly, we design a novel Half-Shuffle Transformer (HST) as the denoiser prior in each iteration. Our HST can jointly extract local contextual information and model non-local dependencies, while requiring much cheaper computational costs than global Transformer. We achieve this by customizing a Half-Shuffle Multi-head Self Seff-Attention (HS-MSA) mechanism that composes the basic unit HST. More specifically, our HS- HSM has two branches, i.e., local branch and non-local branch. The local branch calculates the self-attention within the local window while the non-local branch shuffles the tokens and captures cross- woodsw interactions. We plug HST into DAUF to establish iterative architecture, Pegradation-Awart Unfolding Half-Shuffle Transformer (DAUHST). With the proposed techniques, DAUHST models dramatically outperform saate-of-the-ae (SOTA) deep unfolding methods with the same number of stages by over 4 dB, as shown in Fig. 1.
In a nutshell, our contributions can be summarized as follows:
(i) We formulate a principled MAP-based unfolding framework DAUF for HSI reconstruction.
(ii) We propose novel Transformer HST and plug it into DAUF to establish DAUHST. To the best of our knowledge, DAUHST is the first Transformer-based deep unfolding method for HSI restoration.
(iii) DAUHST outperforms SOTA methods by a large margin while requiring cheaper computational and memory costs. Besides, DAUHST yields more visually pleasant results in real HSI reconstruction.
2

---

Acerage
Rully
(a. (3)
02
8
0x
Layer
2/
7x
Xx
P
D
P
D
P
D
(y, bb)
Conn
(y, +))
(y, (b)
(y, 4))
Shift
Figure 2: The architecture of our DAUF with K stages (iterations). & estimates informative parameters from the compressed measurement, y and sensing matrix $. The estimated parameters and 3 fed into each stage of subsequent iterative learning. P and D denote the linear projection and denoising network in each stage.
Proposed Method
2.1 Degradation Model of CASSI
In CASSI, we denote the vectorized measurement as y E R", where n H(W +d(N)-1)) H. W. d. and N, denote the HSI's height, width, shifting step in dispersion, and total number of wavelengths. Given the vectorized shifted HSI signal x IR and the sensing matrix o E RaxnN, that is determined by the physical mask, the degradation model of CASSI can be formulated
(1) where n TR represents the vectorized imaging noise on the measurement. As analyzed in [56, 57, 58), + is a fat, sparse, and highly structured matrix that is hard to handle. Please refer to the supplementary material for details about the mathematical model of CASSI. Then the task of HSI reconstruction is given y (captured by the camera) and 4 (calibrated based on pre-design), solving
== 4x n.
2.2
Degradation-Aware Unfolding Framework
Previous unfolding frameworks [34, 35, 36, 37] do not estimate the CASSI degradation patterns to adjust the iterative learning. To alleviate this limitation, we formulate principled Degradation-Awan Unfolding Framework (DAUF) as depicted Fig. DAUF starts from the MAP theory. In particular, the original HSI signal could be estimated by minimizing the following energy function
X = arg min
fly
xx| 2 R(x)),
(2)
2
where 1 lly Φx|| is the data fidelity term, R(x) the image prior term, and r a hyperparameter balancing the importance. By introducing auxiliary variable Eq. (2) be reformulated as
X arg min
5 ly xx|| RZ(z). s.t. =.
This is constrained optimization problem. obtain an unfolding inference, we adopt half -ulf-quaaratic splitting (HQS) algorithm for its simplicity and fast convergence. Then Eq. (3) is solved by minimizing
Cx 2) = IIy 2x12 + TR(z) + "
x||
(4)
2
where μ is penalty parameter that forces and to approach the same fixed point. Subsequently, Eq. (4) can be solved by decoupling x and z into the following two iterative sub-problems as
X = arg min yyy - xx|12 /||x - 1=
Zk+1 = arg min 1 11 x + +R((2, (5)
where &o 0, 1, ... K - indexes the iteration. Note that the data fidelity term is associated with quadratic regularized least-squares problem, i.e., X in Eq. (5). has a closed-form solution as
X ④ T n1) -1 1 TTTT Ty pzz))
where is an identity matrix. Since & a fat matrix, ( T + uI) will be large and thus we simplify the computation of the inverse problem (T T Φ
+ pI) by the matrix inversion formula as
( -1 I Φ (I 2 0 - 44

---

By plugging Eq. (7) into Eq. (6), we can reformulate Eq. (6) as
T y + pizk
(I + 1 ①
T
11 1 $
T)
Xk+1
n22
(8)
In CASSI systems, Φ T is diagonal matrix which can be defined as 4.p def (i,,
By plugging T into (I + Du
) - and (I+ 1u
TT))
TT we obtain:
(I + by $1-
1
diag
1
r + th
1 + tn
(9)
1ml
[I By Φ - T = diag
r + th1
p thn
Let y def [yi...., "a.....,9ll! ynl and pz denotes the i-th element of Pzg. We plug Eq. (9) into Eq. (8)
y
91 1/2
+ /1[22e11
DaVa + 14 1(
X6+1 +
T
1 + 111
# + th
(10)
$ 11
yn 1
23 + T
41
1 + 2h
Note that {v: [222] can be directly updated by y $zz, and { I is pre-caleulated and stored in 4 Thus, by element-wise computation in Eq. (10), x can be updated very efficiently, According to Eq. (5), the penalty parameter μ should be large enough so that x and z can approach approximately the same fixed point. This indicates that controls the convergence and output of each iteration. Thus, instead of manually tweaking μi, we set μ as a series of iteration ieeratonnsppcciicc parameters to be automatically estimated from the CASSI system. We denote μ in the k-th iteration Px-
Returning to Eq. (5), we also set r as speciffe parameters and 26+1 can be reformulated as
Z = arg min
2(V TV+1///441) ||z - X 11 R(z).
(11)
From the perspective of Bayesian probability, Eq. (11) is equivalent to denoising image X&+1 with Gaussian noise at level / Th+ 1/441 Mk+1 [29]. To conveniently solve Eq. (11), we set
Mk41/ TT41 as parameters to be estimated from CASSL Let def pk. a def H4//TK, Tk, and B def [$1.... 3x]. Then we can formulate our DAUF as an iterative scheme:
= [a1,..., ax), Bx. def
(ax, B) Ey, b) X++1 Ply, 0k++1 4) Z++1 ((x++1, (k+1),
(12)
where E denotes the parameter estimator that takes the compressed measurement y and the sensing matrix & of the CASSI system as inputs, P equivalent to Eq. (10) denotes the linear projection, and 'D represents the Gaussian denoiser solving Eq. (11). Zo is initialized by passing the shifted y concatenated with & through convl x 1 (convolution with 1 1x1 kernel). Fig. 2 shows the architecture of E. It consists of conv1 x 1, a strided conv3 x 3, a global average pooling, and three fully connected layers. Through E, DAUF captures critical cues from CASSI by learning the degradation patterns and ill-posedness degree caused by the mask maak-mooulation and dispersion- integration. Parameters a and estimated by E direct the iterative learning by adaptively scaling the linear projection in Eq. (10) and providing noise level information for the denoiser prior in Eq. (11).
2.3 Half-Shuffle Transformer
When designing the denoiser prior, previous deep unfolding methods [34, 35, 36, 37] mainly adopt CNNs, showing limitations in capturing long-range dependencies. Directly applying local and global Transformers will encounter two problems, i.e., limited receptive fields and nontrivial computational costs. To address these challenges, we propose Half-Shuffle Transformer (HST) to play the role of D.
Network Architecture, As shown in Fig. (a). HST adopts three-level U-shaped structure built by the basic unit Half- Shuffle Attention Block (HSAB). Firstly, HST uses a conv3 x 3 to map reshaped X concatenated with stretched 3 into feature X, E IR #x where W W d(N,, 1). Secondly, X, passes through the encoder, bottleneck, and decoder to be embedded into deep feature X. IR H XxC Each level of the encoder decoder contains an HSAB and resizing module.

---

comidk3
comvEx3
(b) HSAB
FFN
Enxxsder
Deccader
HSAB
convixt
HSAB
Q.K.V
O. V:
Lucal
DownSomplo
UpSampp
Partiion
W
HSAB
HSAB
Xow
Quy Mag Ver
Branon
DownSammpl
UpSamppe
Partitin
Wet Unshuffo
Botteneck
Shufto
HS-MSA
HST
Diagram of HST. (a) HST adopts U-shaped structure. HSAB consists of FFN, HS-MSA, and two layer normalization. (c) Components FFN. (d) HS-MSA contains local branch and non-local branch.
Figure 3:
In Fig. 3 (b), HSAB consists of two layer normalization (LN), an HS-MSA, and a Network (FFN) that is detailed in Fig. (c). The downsampling and upsampling modules strided conv4 x 4 and deconv2 x 2. Finally, a come3 x 3 operates on X to generate a residual image R E R NNx, The output denoised image 2;; is obtained by the sum of X and reshaped R.
Half-Shuffle Multi-head Self-Attention. The most important element of HSAB is the proposed Half-Shuffle Multi-head Self- self-Attention (HS-MSA) module. Fig. 3 (d) depicts the HS-MSA used in the first level. The input tokens of HS-MSA denoted Xin E HR xC Subsequently, X;n. is linearly projected into query Q #x key K E R H XC and value V E #xWxxC as
Q Q=X,, K W V X WV
(13)
where Wº, WKK WV RCCC are learnable parameters and biases omitted for simplification. Our HS-MSA combines the advantages of global MSA [42] and local window- based MSA [41], i.e., HS-MSA can jointly capture local contextual information through the local branch and model long-range dependencies through the non-local branch, all while being computationally cheaper than global MSA. Specifically, Q, V split into two equal parts along the channel dimension as
Q QQi Qnd).
KK, Knd),
[Vi, Vnd),
(14)
where Q K₁, VL R xxixx & are fed into the local branch to capture local contents, while Qui Kn1, Vnn E & pass through the non-local branch to model non-local dependencies.
Local Branch. The local branch computes MSA within position-specife windows. As shown in the upper path Fig. 3 (d), Q Kr, Vi are partitioned into non-overlapping windows of size x M. Then they are reshaped into are x2x C Subsequently, Q K,, Vi are split along the channel wise into h heads: Q: = Q/ ..... Q J. K K K!.....Ke K l, and V. = [ ...... V ]. The dimension of each head is d, 24 Note that Fig. 3 (d) depicts the situation with h and some details are omitted for simplification. The local self-attention A is calculated inside each head
Qf K Vdn
A softmax(
+ +P/)
Vi,
1=
(15)
where P/- XM² are learnable parameters embedding the position information.
Non-local Branch. The non-local branch computes cross-window interactions through shuffle operations inspired by ShuffleNet [59]. In particular, Qnt, Kni, Val xxxxWx are firstly par- titioned into non-overlapping windows with size M x M. Then their shapes are transposed from #& xMxx x TR. M Hil x shuffle the positions of tokens and establish inter-window depen- Q% ], K K....... ..
dencies. Qnt Kuii V., split into h heads: Qni
5

---

Aposithans
Panans
GFLOPS
55
56
319
TwiST (60)
0.730
25.16
0.604
23,00
0.711
21.40
0.851
10.19
0.635
31.41
0.644
30.55
22.30
0.643
0.650
21.80
0.690
22.42
0.559
23.67
0,609
23.12
GAP-TV (26)
26.82
0.754
0.610
22.89
0.302
26.31
0.552
30.65
0.703
23.64
0.663
21.85
0.658
23.76
21.96
0.655
0.682
22.63
23.10
0.554
0.669
24.36
DeSCT (23)
27.13
0,248
23.04
0.6.0
0.318
34.96
23.04
0.706
0.085
0.943
24.45
22.00
0.733
24.56
0752
23.59
0.577
0,221
1331
62.64M
117.98
0.599
30.10
0.305
25,49
37.01
26.19
26,47
26,09
0.36
27.50
0.16
27.33
HSSPP [35]
51,48
03888
31,09
0.842
0.321
25.96
0.902
34,55
28.53
0.808
0.877
30.85
0.84
28.71
30.09
0.61
0.888
30.43
0.842
28.78
0.853
30.35
1.9M
163.48
51.72
0.853
0.846
31.13
29.95
03988
35.34
29.03
0.835
0.877
30.87
0899
28.59
30.13
0385
0816
31.00
29.14
0.49
0.861
30.74
DIP HSL
3385M
64.42
19.79
TSA Net
44225M
100.06
0.892
52,03
0.853
51.00
0.915
0,953
19.19
0.884
29.39
0.908
0.878
50.13
0.358
29.35
0.890
30,01
0.874
29.59
0.894
31,40
DOSMP (88)
3
646 65
0.915
33,26
0.598
32,09
0:925
33.06
0.964
40.54
28,86
0.682
0.937
33.08
0.856
30.74
31.55
0.23
0.011
31,56
31,44
0.925
0.917
32,63
GAP-NN (96)
427M
58
0.911
53.34
0.900
33.26
0.929
0.967
41.03
0.919
31.44
50.46
0.905
0.955
33.51
50.34
0.835
0.917
ADMMANN (37)
4.2TM
78.58
0.918
54.12
33.62
0.922
11.82
0.907
90.74
33.75
0.895
50.68
HDNA (52)
2.7MM
154.76
0.935
55.14
0.940
35.67
0.943
36,03
0.969
42.30
0.946
12.69
0.952
34.46
0.926
33.67
0.941
32.48
0.422
34,59
0.937
32.38
0.943
34,97
MSTE 161)
203M
28.15
33,40
0.41
35.87
0.944
0.951
36.51
0.925
42.27
0.947
32.77
0.955
34.50
0915
33.66
0948
32.67
0.49
35.39
0.41
32.50
0.948
35,18
MSTH+ (62)
133M
19.42
35.80
0965
0.967
0.953
33.36
095
0953
33.38
01455
CSTL (62)
30004
0.999
35.96
0.555
36.84
42.44
0.955
35.25
34.36
0961
34.54
0957
36,51
0.455
33.09
BERNAT [65]
4.408
2122.66
0951
36,79
0.957
37,59
0.971
40.61
0.965
46.94
0.964
35.42
0.959
35.30
0.55
36.58
33.96
0.56
0.700
39.47
0938
32,50
0.960
37,58
DAUBST-20g
1.40M
18.44
33.93
0.943
36,70
0.946
37.96
0:978
44.38
24:13
0:054
0.957
35.43
0.40
34.78
0.50
33.65
37.42
0.55
33.07
0.91
0.952
36,34
DAUHIST-312
205M
27.07
09499
DAURST-SMg
3.44
44.61
0.955
36.92
0.962
38.52
0.967
0,980
45.09
0.964
35
0.965
16.56
0.958
36.32
0.959
$4.94
35.71
0.933
0.952
34.27
0.962
37.75
DAURST-bU
65M
50
37.25
0.958
0.967
39.02
0.971
41.05
0.983
46.15
35,80
0,969
0,970
37,08
0.963
30,57
0.966
35.10
0,970
48.02
0.956
34,39
0.967
38.36
Table 1: Comparisons between DAUHST and SOTA methods on 10 simulation scenes (SI~S10).
Params, FLOPS, PSNR (upper entry in each cell), and SSIM (lower entry in each cell) are reported.
and Vnd = [V] Var...., VA J. Then the non-local self-attention A is computed in each head as
A softmax Qni K Vdr.
+P
V nls = h,
(16)
where P E R
x my 312 3²2 are learnable parameters representing the position embedding. Subsequently, g&
A ER-
RMMxx 2. x
Hi xda is unshuffled by being transposed shape R NT xM2xxd, Then the outputs of
NT
local branch in Eq. (15) and non-local branch in Eq. (16) are aggregated by a linear projection as
HS-MSA(X = >" AiWi+ + Σ"_ A W nl
(17)
==1
where Wi, W E Rd. XC refer to learnable parameters. We reshape the result of Eq. (17) to obtain the output Xout " Instead of globally sampling all tokens, HS-MSA builds inter- correlations by shuffle operations. The self-attention is calculated in the local window but with tokens from local regions. Therefore, HS-MSA is much computationally cheaper than global MSA.
nt
3 Experiment
3.1 Experiment Setup
Similar to [20, 32, 36, 38. 61], 28 wavelengths are selected from 450nm to 650nm and derived by spectral interpolation manipulation for the HSI data. Simulation and real experiments are conducted.
Simulation Dataset. We adopt two datasets, i.e., CAVE [64] and KAIST [65] for simulation experiments. The CAVE dataset consists of 32 HSIs with spatial size 512x512. The KAIST dataset contains 30 HSIs of spatial size 2704x Following the settings of [20, 32, 36, 38, 61]. the CAVE dataset is adopted as the training set while 10 scenes from the KAIST dataset are selected for testing.
Real Dataset. Five real HSIs collected by the CASSI system developed in [20] are used for testing.

---

b
Image
Weolingh aani
Spectrai Density
270. mm
530. Pm
00k. mm
040.
TWIST
GAP-TV
DeSCI
X-Net
HSSP
TSA-Net
D6SMP
GAP-Net
DAUHST-astE
Figure 4: Simulation HSI reconstruction comparisons of Scene 2 with (out of 28) spectral channels. The top-middle shows the spectral curves corresponding to the two green boxes of the RGB image. The top-right depicts the enlarged patches corresponding to the yellow boxes in the bottom HSIs. Zoom for a better view.
Implementation Details. We implement DAUHST by Pytorch. All DAUHST models are trained with Adam [66] optimizer (31 = 0.9 and 82 0.999) using Cosine Annealing scheme [67] for 300 epochs on an RTX 3090 GPU. The initial learning rate is 4x 10-4 Patches with spatial sizes 256 x 256 and 660 660 randomly cropped from the 3D HSI cubes with 28 channels as training samples for the simulation and real experiments. The shifting step d in the dispersion is set to 2. The batch size is 5. We set the basic channel C NA- 28 store HSI information. The weights of 'D in different stages are unshared. Data augmentation includes random rotation and flipping. The training objective minimize the Root Mean Square Error (RMSE) between reconstructed and ground HSIs.
3.2 Quantitative Comparisons with State-of-the-Art Methods
Tab. 1 compares the results of DAUHST and 16 SOTA methods including three model-based methods
(TwIST [60], GAP-TV [26), and DeSCI [23]), one PnP method (DIP-HSI [30]), seven E2E methods
(X-Net [33], TSA. TSA-Net [20], HDNet 3,2, MST [61]. MST++ [62], CST [68], and BIRNAT [63]). and
five deep unfolding methods (HSSP [35], DNU [34], DGSMP [38]. GAP-Net [36], and ADMM- Net [37]) on 10 simulation scenes. All algorithms are tested with the same settings as [38, 61].
(i) Our best model DAUHST-9stg (9-stage DAUHST) yields very impressive results, i.e., 38.36 dB in PSNR and 0.967 in SSIM. DAUHST-9stg significantly outperforms two recent SOTA methods BIRNAT [63] and MST-L [61] by 0.78 and 3.18 dB, suggesting the effectiveness of our method.
(ii) Our DAUHST models dramatically surpass SOTA methods while requiring cheaper computational E2E method MST, our DAUHST-2stg outperforms MST-L. by 1.16 dB but only costs 68.9% (1.40 / 2.03) Params and 65.5% (18.44/ / 28. FLOPS. When compared with CNN-based E2E methods, DAUHST-3stg surpasses HDNet, TSA-Net, and X-Net by 2.24, 5.75, and 8.68 dB while only requiring 87. 97.8%, 4.7%, 3.3% Params and 17.6%, 24.7%, 23.0% FLOPS. When compared with RNN-based E2E method BIRNAT, our DAUHST-5stg is 0.17 dB higher but only costs 2. 1% FLOPS and 78.2% Params. Fig. plots the PSNR-FLOPS comparisons of DAUHST and SOTA unfolding methods. DAUHST outperforms other competitors with the same number of stages by very large margins, over 4 dB.
and memory costs. For instance, when compared with the only one Transformer-
3.3 Qualitative Comparisons with State-of-the-Art Methods
Simulation HSI Reconstruction. Fig. depicts the simulation HSI reconstruction comparisons between our DAUHST and other SOTA methods on Scene 2 with 4 (out of 28) spectral channels. The top-right part shows the zoomed-in patches of the yellow boxes in the entire HSIs (bottom). can be observed that our DAUHST-9stg is more favorable to reconstruct visually pleasant HSIs with more

---

400
RGB Image
rmn
O
614..00
Messurement
TWIST
GAP-TV
DeSCI
N-Net
HSSP
TSA-Net
DGSMP
GAP-Net
MSTL
DAUHST-3stg
Figure 5: Real HSI reconstruction results of DAUHST-3stg and SOTA methods Scene 1 with 4 (out of 28) spectra. Only our method can clearly reconstruct the picked flower at all wavelengths. Zoom for a better view.
detailed contents, cleaner textures, and fewer artifacts while preserving the spatial smoothness of homogeneous regions. In contrast, previous methods either yield over-smooth results compromising fine- fine-grained structures, or introduce undesired chromatic artifacts and blotchy textures that are absent in the ground truth (GT). The top-middle part illustrates the bonsity-wavelenglee spectral curves corresponding to the green boxes identified as a and in the RGB image (top-left). The spectral curves of DAUHST-955 --9tg achieve the highest correlation and coincidence with the reference curves, showing the advantage of our proposed DAUHST in spectral-dimeenion consistency reconstruction.
Real HSI Reconstruction. We further evaluate the effectiveness of DAUHST in real HSI reconstruc- tion. Following the same settings as [20, 38, 61] for fair comparison, we re-train DAUHST-3s12 with the real mask on the CAVE and KAIST datasets jointly. To simulate the real imaging situations, the training samples are also injected with 11-bit shot noise. Fig. shows the visual comparisons between our DAUHST-3stg and nine SOTA methods. In the top three rows, only our DAUHST-3stt can reconstruct the flower patch corresponding to the yellow box at all wavelengths while other methods all fail to recover the entire patch. In the bottom row, DAUHST-3ste restores more HSI structural details and clearer contents with fewer artifacts. In contrast, other methods recover blurry images, generate incomplete responses, and are susceptible the noise corruption. This evidence suggests that DAUHST is more robust to the noise distortion and more effective in real HSI reconstruction.
3.4 Ablation Study
Break-down Ablation. We adopt baseline-1 I that is derived by removing HS-MSA and DAUF from DAUHST-3stg to conduct the break-down ablation. Our goal is to study the effect of each component towards higher performance. Baseline-1 is cascaded end to end by three single-stage networks. As shown in Tab. 2a, baseline-1 achieves 33.05 dB. When we respectively apply DAUF HS-MSA, the model achieves 2.32 and 2.44 dB improvements. When exploit DAUF and HS HS-MSA jointly, the model gains by 4.16 dB. These results demonstrate the effectiveness of our DAUF and HS-MSA
Self-Attention Mechanism. To compare HS-MSA with other MSAs, we adopt baseline-2 that is obtained by removing HS-MSA from DAUHST- stg to conduct the ablation Tab. 2b. We remove different position embedding schemes to avoid their impacts and only compare MSAs. For fairness, we keep the Params MSAs the same by fixing the number channels and heads. Baseline-2 yields 32.79 dB. We apply global MSA (G-MSA) [42], Swin MSA (SW-MSA) [41], Spectral-wise MSA (S-MSA) [61], and HS- HS.MSA Note that we downsample the input feature maps of G-MSA to avoid memory bottlenecks. As shown in Tab. 2b, HS-MSA yields the most significant improvement of 1.26 dB, which is 0.42, 0.30, and 0.23 dB higher than G-MSA, SW-MSA, and S-MSA. This superiority is mainly derived from HS- HS-M' ability jointly capture local contents and non-local dependencies.
Unfolding Framework. We compare our DAUF with previous unfolding frameworks including DNU [34), ADMM-Net [37], and GAP-Net [36]. For fair comparison, we replace each single-stage network of DNU, ADMM-Net, and GAP-Net by our HST. 3-stage architecture adopted conduct ablations. The results are shown in Tab. 2c. Our DAUF significantly outperforms DNU, ADMM, and GAP by 2.59. 1.69, and 1.63 while adding only 0.05M Params and 0.94G FLOPS. This is mainly because DAUF uses the parameters estimated from the compressed measurement and physical mask the CASSI system to direct the iterative learning. These parameters capture critical information of CASSI degradation patterns and ill- pl-posedsees degree, providing key cues for HSI reconstruction.

---

RGB Mossurement
10"
101
1044
10
16
Beration
20
21
X2
%
70
Xg
GT
Figure 6: Visualization of & and X with 4 (out of 28) spectral channels on Scene I in different iterations.
The bottom-left corner plots the curves of and changing with the iteration. Please zoom in for better view.
Bascline-1
DAUF
HS MSA
PSNK
SSIM
Params (M)
FLOPS (G)
Methed
Baseline-2
G-MSA
SW-MSA
S-MSA
HS-MSA
33.05
0.912
1.06
17.62
PSNR
32.79
33.63
33.75
33,82
34,85
35.37
0.938
1.11
18.55
SSIM
0.904
0.920
0.924
0.926
0,930
35.49
0.941
2.01
26.25
Params (M)
0:40
0.48
0.48
0.48
0.48
37.21
0.959
2008
27.17
FLOPS (G)
6.85
10.30
8.89
9.72
(a) Break- ablation towards higher performance.
(h) Ablation of various self-attention mechanisms.
Framework
DNU
ADMM [37]
GAP[36]
DAUF
Baseine-3
a
PSNR
SSIM
Params (M)
FLOPS (G)
PSNR
34,62
35.52
35.58
37.31
36.49
0.952
2.03
26.23
SSIM
0.942
0.957
Params (M)
2.03
2,03
2,03
2.08
36.83
0.956
2.08
27.17
FLOPS (G)
26.23
26.23
26.23
27.17
37.21
0.989
2.08
27.17
Ablation of different unfolding frameworks.
(d) Ablation to study the effect of parameters a and P.
Table Ablation studies on simulation datasets [64, 65]. PSNR, SSIM, Params, and FLOPS are reported.
To study the effect of the estimated parameters and B, we perform break-down ablation of DAUF. adopt DAUHST-3stg as baseline-3 but or is set as learnable parameters instead of being estimated by ε in Eq. (12) and B is not fed into D. The results are shown in Tab. 2d. Baseline-3 yields 36.49 dB. When a is set to be estimated by E, baseline-3 is improved by 0.45 dB. When 3 is fed into D, baseline-3 gains by 0.34 dB. When and [ are exploited jointly in the iterative learning, baseline-3 achieves a significant improvement of 0.72 dB. These results verify that the estimated parameters a and (3 are beneficial for the linear projection and denoising network of deep unfolding methods.
To further analyze the roles of the estimated parameters, we visualize 2₁ and x₁ of Eq. (12), and plot the curves of a and ,3 as they change with the iteration in Fig. 6. We observe: (i) Zo and x yield either blurry or noisy images. There
significant gap between them. Since ag = μi in Eq. (5) penalizes the differences between z and 0i is estimated to be a large value. From the linear projection of the second iteration (z) + X2) on, the gap between z and x decreases substantially. Therefore, ar estimated to be small values when k 2 2. This indicates that scale the linear projection P. (ii) The noise corruption is severe in the first iteration. Thus, 31 /4/T1 = 1/(1 (VTT////a /1 which is inversely proportional to the noise level, is estimated to be a small value. With further iterations, the noise level decreases, and thus the estimated B7 increases. These results demonstrate that B can provide the information about noise level for the denoising network D.
adaptively
Conclusion
In this paper, we remedy two issues of previous deep unfolding methods, i.e., they do not estimate informative parameters from the CASSI system direct the itcrative learning and they are mainly CNN-based showing limitations in capturing long-range dependencies. To cope with these challenges, we firstly formulate a principled MAP-based unfolding framework DAUF that estimates parameters from the compressed measurement and physical mask. Then the parameters, which capture critical cues of CASSI degradation patterns and ill posedness degree, are fed into each iteration to contextually scale the linear projection and provide noise level information for the denoising network. Secondly, we propose a novel Transformer HST that can jointly extract local contents and model non-local dependencies. By plugging HST into DAUF, we derive the first Transformer-bant DAUHST for HST reconstruction. Comprehensive experiments show that our DAUHST outperforms SOTA methods by large margin while requiring much cheaper memory and computational costs.
unfolding method

---

Limitation and Social Impact
The main limitation of our work is that the performance improvement of our method comes with lowering down the inference speed and increasing the model complexity. Until now, spectral snapshot compressive imaging reconstruction techniques have no negative social impact yet. Our proposed DAUHST does not present any negative foreseeable societal consequence, either.
Acknowledgement
This work is partially supported by the NSFC fund (61831014), the Shenzhen Science and Technology Project under Grant 1SGG220210001131150000
3GGIZD200000/7/0220000000 Xin Yuan acknowl-
edges the support of NSFC (62271418) 1414), Westlake Foundation (2021B1501-2) and the Research Center for Industries of the Future (RCIF) Westlake University.
References
[1] M. Fauvel, Y. Tarabalka, J. A. Benediktsson, J. Chamussot, and C. Tilton, "Advances in spectral-spatial classification of hyperspectral images," Proceedings of the IEEE, 2012.
[2] E. Maggiori, G. Charpiat, Y. Tarabalka, and P. Alliez, "Recurrent neural networks to correct satellite image classification maps," Transactions on Geoscience and Remote Sensing, 2017.
[3] Zhang, B. Du, and L. Zhang, "Scene classification via a gradient boosting random convolu- tional network framework," Transactions on Geoscience and Remote Sensing, 2015.
[4] M. H. Kim, T. A. Harvey, S. Kittle, H. Rushmeier, R. O. P. J. Dorsey, and D. J. Brady, "3d imaging spectroscopy for measuring hyperspectral patterns solid objects," ACM Transactions on on Graphics, 2012.
[5] Z. Pan, G. Healey, M. Prasad, and B. Tromberg, "Face recognition in hyperspectral images," TPAMI, 2003.
[6] H. V. Nguyen, A. Banerjee, and R. Chellappa, "Tracking via object reflectance using a hyper- spectral video camera," CVPRW, 2010.
[7] Y. Fu, Y. Zheng, Sato, and Sato, "Exploiting spectral-spatial correlation for coded hyper- spectral image restoration," in CVPR, 2016.
[8] B. Uzkent, M. J. Hoffman, and A. Vodacck, "Real-time vehicle tracking in acrial vidco using hyperspectral features," in CVPRW, 2016.
[9] B. Uzkent, A. Rangnekar, and M. Hoffman, "Aerial vehicle tracking by adaptive fusion of hyperspectral likelihood maps," in CVPRW, 2017.
[10] V. Backman, M. B. Wallace, L. Perelman, J, Arendt, Gurjar, M. Muller, Q. Zhang, G. Zonios,
E. Kline, and T. McGillican, "Detection of preinvasive cancer cells," Nature, 2000.
[11] G. Lu and B. Fei, "Medical hyperspectral imaging: a review," Journal of Biomedical Optics, 2014.
[12) Z. Meng, M. Qiao, J. Ma, Z. Yu, K. Xu, and X. Yuan, "Snapshot multispectral endomicroscopy." Optics Letters, 2020.
[13] M. Borengasser, W. Hungate, and Watkins, "Hyperspectral remote sensing: principles and applications," CRC press, 2007.
[14] F. Melgani and L.. Bruzzone, "Classification of hyperspectral remote sensing images with support vector machines," IEEE Transactions Geoscience and Remote Sensing, 2004.
[15] Y. Yuan, X. Zheng, and X. Lu, "Hyperspectral image superresolution by transfer learning." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2017.
[16] Solomon and Rock, "Imaging spectrometry for earth remote sensing," Science, 1985.
[17] A. Wagadarikar, R. John, R. Willett, and D. Brady, "Single disperser design for coded aperture snapshot spectral imaging," Applied Optics, 2008
[18] A. A. Wagadarikar, N. P. Pitsianis, Sun, and J. Brady, "Video rate spectral imaging using a coded aperture snapshot spectral imager," Optics Express, 2009.
10

---

[19] H. Du, X. Tong, X. Cao, and S. Lin, "A prism-based system for multispectral video acquisition." ICCV, 2009.
[20] Z. Meng, J. Ma, and X. Yuan, "End-to-end low cost compressive spectral imaging with spatial- spectral self-attention," in ECCV, 2020.
[21] M. E. Gehm, R. John, J. Brady, R. M. Willett, and T. J. Schulz, "Single-shee shot compressive spectral imaging with disl-disprrser architecture," Optics express, 2007.
[22] D. Kittle, K. Choi, A. Wagadarikar, and D. J. Brady, "Multiframe image estimation for coded aperture snapshot spectral imagers," Applied optics, 2010.
[23] Y. Liu, X. Yuan, Suo, D. Brady, and Q. Dai, "Rank minimization for snapshot compressive imaging," TPAMI, 2019.
[24] L. Wang, Z. Xiong, G. Shi, F. Wu, and W. Zeng, "Adaptive nonlocal sparse representation for dual-camera compressive hyperspectral imaging," TPAMI, 2016.
[25] S. Zhang. L. Wang. Y. Fu, X. Zhong, and H. Huang, "Computational hyperspectral imaging based on dimension-ddicciminaatve low-rank tensor recovery," in ICCV, 2019.
[26] X. Yuan, "Generalized alternating projection based total variation minimization for compressive sensing," in ICIP, 2016.
[27] J. Tan, Y. Ma, H. Rueda, D. Baron, and G. R. Arce, "Compressive hyperspectral imaging via approximate message passing." IEEE Journal of Selected Topics in Signal Processing, 2016.
[28] M. A. Figueiredo, R. D. Nowak, and S. J. Wright, "Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems." " IEEE Journal of selected topics in signal processing, 2007.
[29] S. H. Chan, X. Wang, and O. A. Elgendy, "Plug-and and-play admm for image restoration: Fixed- point convergence and applications," " Transactions on Computational Imaging, 2016.
[30] Z. Meng, Z. Yu, K. Xu, and X. Yuan, "Self-supervised neural networks for spectral snapshot compressive imaging." in ICCV, 2021.
[31] S. Zheng, Y. Liu, Z. Meng, M. Qiao, Z. Tong. X. Yang, S. Han, and X. Yuan, "Deep plug-and- play priors for spectral snapshot compressive imaging," Photonics Research, 2021.
[32] X. Hu, Y. Cai, J. Lin, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, "Hdnet: High-resolution dual-domain learning for spectral compressive imaging," in CVPR, 2022.
[33] X. Miao, X. Yuan, Y. Pu, and V. Athitsos, "I-net: Reconstruct hyperspectral images from snapshot measurement," in ICCV, 2019.
[34] L. Wang, C. Sun, M. Zhang, Y. Fu, and H. Huang, "Dnu: Deep non-local unrolling for computational spectral imaging," in CVPR, 2020.
[35] L. Wang, C. Sun, Y. Fu, M. H. Kim, and H. Huang, "Hyperspectral image reconstruction using a deep spatial-spectral prior," in CVPR, 2019.
[36] Z. Meng, Jalali, and X. Yuan, "Gap-net for snapshot compressive imaging," arXiv preprint arXiv:2012.08364 2020.
[37] J. Ma, X.-Y. Liu, Z. Shou, and X. Yuan, "Deep tensor admm-net for snapshot compressive imaging," in ICCV, 2019.
[38] T. Huang, W. Dong, X. Yuan, J. Wu, and G. Shi, "Deep gaussian scale mixture prior for spectral compressive imaging," " in CVPR, 2021.
[39] X. Zhang. Y. Zhang. R. Xiong, Q. Sun, and J. Zhang, "Herosnet: Hyperspectral explicable reconstruction and optimal sampling deep network for snapshot compressive imaging," in CVPR, 2022.
[40] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in NeurIPS, 2017.
[41] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, "Swin transformer: Hicrarchical vision transformer using shifted windows," in ICCV, 2021.
[42] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, "An image is worth 16x16 Transformers for image recognition at scale," in ICLR, 2021.
words:
11

---

[43] A. El-Nouby, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve. Verbeek, et al., "Xcit: Cross-covariance image transformers," arXiv preprint w:Xiv:2/06.09660 2021.
[44] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and Dai, "Deformable detr: Deformable transformers for end-to-end object detection," in ICLR, 2021.
[45] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, "End-to-end
object detection with transformers," in ECCV, 2020.
[46] X. Dai, Y. Chen, J. Yang, P. Zhang, L. Yuan, and L. Zhang, "Dynamic detr: detection with dynamic attention," in ICCV, 2021.
End-to-end object
[47] O. Petit, N. Thome, C. Rambour, L. Themyr. T. Collins, and Soler, "U-net transformer: Self and cross attention for medical image segmentation," " in International Workshop on Machine
Learning in Medical Imaging, 2021.
[48] E. Xie, W. Wang. Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo, "Segformer: Simple and efficient design for semantic segmentation with transformers," in NeurIPS, 2021.
[49] R. Strudel, R. Garcia. I. Laptev, and C. Schmid, "Segmenter: Transformer for semantic segmen-
tation," in ICCV, 2021.
[50] Y. Li, S. Zhang, 7.. Wang, S. Yang, W. Yang, S.-T. Xia, and E. Zhou, "Tokenpose: Learning keypoint tokens for human pose estimation," " in ICCV, 2021.
[51] S. Yang. Z. Quan, M. Nie, and W. Yang, "Transpose: Keypoint localization via transformer," in ICCV, 2021.
[52] K. Li, S. Wang. X. Zhang, Xu, W. Xu, and Z. Tu, "Pose recognition with cascade transformers," in CVPR, 2021.
[53] H. Chen, Y. Wang, T. Guo, C. Xu, Y. Deng, Z. Liu, S. Ma, C. Xu, C. Xu, and W. Gao, "Pre-trained image processing transformer," in CVPR, 2021.
[54] J. Liang, J. Cao, G. Sun, K. Zhang. L. Van Gool, and R. Timofte, "Swinir: Image restoration using swin transformer," " in ICCVW, 2021.
[55] Z. Wang, X. J.n, J. Bao, W. Zhou, J. Liu, and H. Li, "Uformer: A general u-shaped transformer for image restoration," in CVPR, 2022.
[56] A. Tropp and A. C. Gilbert, "Signal recovery from random measurements via orthogonal
matching pursuit," IEEE Transactions on Information Theory, 2007.
[57] D. L. Donoho, "Compressed sensing," " IEEE Transactions on Information Theory, 2006.
[58] S. Jalali and X. Yuan, "Snapshot compressed sensing: Performance bounds and algorithms,"
Trunsuctions on Information Theory, 2019.
[59] X. Zhang, X. Zhou, M. Lin, and J. Sun, "Shufflenet: An extremely efficient convolutional neural network for mobile devices," in CVPR, 2018.
[60] J. Mroucas-Dias and M. Figueiredo., "A new twist: Two-step iterative shrinkage/threshollingg algorithms for image restoration.," " TIP, 2007.
[61] Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. V. Gool, "Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction." in CVPR, 2022.
[62] Y. Cai, Lin, Z. Lin, H. Wang. Zhang, H. Pfister, R. Timofte. and L. V. Gool, "Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction," in CVPRW, 2022.
[63] Z. Cheng, B. Chen, R. Lu, Z. Wang, H. Zhang, Z. Meng, and X. Yuan, "Recurrent neural networks for snapshot compressive imaging." TPAMI, 2022.
[64] J. Park, M.-H. Lee, M. D. Grossberg, and S. K. Nayar, "Multispectral imaging using multi- plexed illumination." in ICCV, 2007.
[65] I. Choi, M. Kim, D. Gutierrez, D. Jeon, and G. Nam, "High-quality hyperspectral reconstruction using a spectral prior," in Technical report, 2017.
[66] D. P. Kingma and J. L. Ba, "Adam: A method for stochastic optimization," in ICLR, 2015.
[67] Loshchilov and Hutter, "Sgdr: Stochastic gradient descent with warm restarts," in ICLR, 2017.
[68] Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L.. V. Gool, "Coarse-to-fine sparse transformer for hyperspectral image reconstruction," in ECCV, 2022.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] We provide our code and models in the supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you're rou're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "gduagZVCegM", "reviewer": "Reviewer_BuNS", "summary": "This work proposes a novel unfolding Transformer-based algorithm for hyperspectral image reconstruction in CASSI system. There are two key components of the proposed method: \n\n(i) degradation-aware unfolding framework perceiving the CASSI degradation pattern and degree of ill-posedness, and\n\n(ii) half-shuffle Transformer capturing local contents and non-local dependencies jointly. \n\nHalf-shuffle Transformer is plugged into degradation-aware unfolding framework to establish the proposed degradation-aware unfolding half-shuffle Transformer.", "questions": "In Table 1, I am not sure why the FLOPS of BIRNAT is so large? Is it because BIRNAT employs an RNN architecture? ", "limitations": "The authors have discussed the limitations and social impacts in the supplementary material. \nThis work does not have any foreseeable negative social impacts.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(i) This work proposes the first Transformer-based unfolding method for HSI reconstruction in snapshot compressive imaging. \n\n(ii) The idea is novel and interesting. The degradation-aware unfolding framework changes the general unfolding paradigm of previous works (e.g., ADMM-Net, GAP-Net) by estimating parameters from CASSI system to adapt the iterative learning. Besides, the elaborately customized half-shuffle Transformer combines the advantages of local and global Transformers while using moderate Params and FLOPS.\n\n(iii) The performance is pretty impressive. DAUHST outperforms SOTA algorithms by large margins while using less Params and FLOPS. In particular, DAUHST achieves over 4dB improvements than previous unfolding methods. \n\n(iv) The experiments are comprehensive and the ablation study is well-designed. The comparisons in Table 1, Figure 4, and Figure 5 including 14 SOTA methods, which is very convincing. The ablation studies in Table 2 sufficiently evaluate the effectiveness of the proposed techniques. The in-depth visual analysis in Figure 6 reveals some important insights of unfolding frameworks for SCI.\n\n(v) The writing is good and easy to follow. Especially in Section 2.1. The mathematical notation and formulation are clear, rigorous, and neatly arranged. In addition, the presentation is well-dressed.\n\n(vi) The reproducibility can be guaranteed. I have check the submitted codes and models, and run the demo. The results in Table 1 of the main paper can be reproduced. The implementation details are described in Section 3.1. Meanwhile, the authors have claimed that they will release the code and models to the public. I believe this work can significantly benefit the community. \n\nWeakness:\n\n(i) The authors provide visual comparisons of DAUHST, HDNet, and BIRNAT in the supplementary but the main paper lacks these important comparisons. Why not move these results to the main paper?\n\n(ii) In Line 156 of the main paper, the mentioned reshape operation of R is unclear and should be explained.\n\n(iii) In Line 98 of the paper, the “vectorized” and “shifted” operations need more explanations.\n\n(iv) It would be better to evaluate the generality and extensibility of DAUF. For instance, replacing the HST by the same CNN of GAP-Net and ADMM-Net. This ablation can show whether the proposed DAUF fits CNN-based denoising networks or not. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6-F1huulKSq", "reviewer": "Reviewer_6Fb8", "summary": "this submission deals with designing unfolding based architecture for supervised learning of spectral compressive imaging (SCI). Compared with SOTA, it proposes to explicitly integrate the ill-posedness of the acquisition (degradation and mask) into each block (iteration) for data consistency and denoising. The denoising network also uses a U-shape transformer with half-shuffle multihead self-attention to capture both local and non-local contents. The experiments outperform SOTA in terms of both memory, computer, and reconstruction fidelity. ", "questions": "This work can be strengthened with more ablations that explains the design choices for the U-shaped transformer. For example, what if one simply uses a multi-head vision transformer instead of U-shaped architecture?\n", "limitations": "\nThe limitations are not included in the submission. The limitations need to be discussed. For example, it seems that the entire network needs to be trained from scratch each time the measurement model changes. The robustness of the model also needs to be discussed in terms of introducing hallucinations. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\npractical significance: the gains achieved are significant compared with the reported SOTA methods\npractical novelty: this is the first work deploying transformers as the denoiser for SCI.\nclear writing: the paper is well written and the idea is clear\n\nWeakness\nlimited technical novelty: The main novelty of this work is to propose a U-shaped transformer with half–shuffle multi-head self-attention for the denoising network in deep unfolding architectures for SCI. Deep unfolding is well established. Also, there are several works reporting superiority of transformers against CNNs for denoising and low level computer vision tasks, and it is not surprising to observe similar gains for SCI; see e.g., [Peit et al’21].\n\n[Peit et al’21]Petit O, Thome N, Rambour C, Themyr L, Collins T, Soler L. U-net transformer: Self and cross attention for medical image segmentation. InInternational Workshop on Machine Learning in Medical Imaging 2021 Sep 27 (pp. 267-276). Springer, Cham.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5sg1IBU24pn", "reviewer": "Reviewer_nC4p", "summary": "This paper introduces a hyperspectral image reconstruction method based on a deep unfolding method and a Half-shuffle Transformer (HST). The deep unfolding method estimates parameters from the image and uses them to direct the iterative learning. The HST captures both local contents and non-local dependencies with relatively low computation cost. The proposed method was evaluated on both synthetic and real image datasets and demonstrated excellent performance over a few other methods. Ablation studies were also provided to show the contribution of key components in the proposed method.", "questions": "Please address the comments in the Strengths And Weaknesses.", "limitations": "The spectral ranges covered by the images in the adopted synthetic datasets are narrow. Please consider experiments with more infrared spectrum. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is a solid paper. Each proposed component in the whole method structure seems to be working, and when putting together, they produce the best hyperspectral image reconstruction performance reported so far. However, I also have several concerns about this paper.\n\nFirst, the proposed DAUHST method has its novelty in combining transformer and deep unfolding for HSI restoration. However, the contribution can only be considered incremental since a similar idea (transformer as prior + unfolding) has been proposed for hyperspectral image super-resolution in November 2021, which is closely related to the reconstruction or restoration task: Learning A 3D-CNN and Transformer Prior for Hyperspectral Image Super-Resolution. Furthermore, both deep unfolding and transformer have been adopted separately for hyperspectral image processing. Take the self-attention HS-MSA module in the proposed Half-Shuffle Transformer (HST) as an example, it can be considered an extension of the S-MSA in MST-L [81]. Table 2 (b) also shows that the performance improvement of HS-MSA over S-MSA is marginal. \n\nSecond, the introduction section hasn't clearly described the motivation of the degradation-aware unfolding framework. It simply says previous methods do not estimate CASSI degradation patterns and ill-posedness degree. Please clearly define what are CASSI degradation patterns and ill-posedness degree and why they are important to the estimation process.\n\nThe method description also needs to be improved as well. In Figure 2, the parameter estimator takes the compressed measurement and the sensing of the CASSI system as inputs and produces \\alpha and \\beta. How is the estimation done? The details are missing. In lines 127-129, \\mu and \\tau are iteration-specific parameters. How their values are determined in each iteration? \n\nTable 1 shows that the 9 stages DAUHST produces the best results. An analysis of the influence of the number of stages on the final performance shall be given.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2SyiNX-aV5u", "reviewer": "Reviewer_Jov1", "summary": "This paper proposes a degradation-aware unfolding framework (DAUF) where the parameters are estimated by the input image and physical mask, which can serve as a guidance for each iteration stage towards the final output. Among each iteration stage, two parts, e.g., a linear projection and denoising network, perform successively based on maximum a posteriori (MAP). Specifically, the denoising network builds on Transformer-type framework replacing the vanilla self-attention mechanism with two branches, local and non-local branch, to extract the local and long-dependency representations. ", "questions": "1, For the input of denoising network, beta_k is stretched to concatenate with x_k following the Eq. 12, and in Tab 2(d) the learnable alpha is studied. I wonder if beta and alpha can be both set as learnable parameters and achieve a compatible result, i.e., learnable beta is also stretched as the original method do or just be added into x_k . In that case, the method would not be degradation-aware.\n2, In 3.4, HS-MSA is remove in the first two subparts. Is it substituted by identity operation? DAUHST-3stg is used to perform ablation studies except for self-attention mechanism (DAUHST-1stg). Is it because the memory bottlenecks of global self-attention?\n3, In Eq. 3, auxiliary variable z is introduced to replace x in the image prior term R(*), but this replacement occurs in the data term in Eq4. of [1]. What is the difference?\n4, It would be better to provide the comparison of complexity for attention alternatives.\n5, The DAUF works in an iterative manner. Is the same HST used or not for different stages? Is there any difference between training and test phase? For example, should the test input go though all stages?\n[1] Deep Unfolding Network for Image Super-Resolution, CVPR 2020.\n", "limitations": "The method works in an iterative manner where there is a Transformer-type HST in each stage, which may lead to a long training and inference time. In addition, the non-local branch captures relatively long-range dependencies but not globally. ", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is clearly written and the logic flow is easy to follow. The method sufficiently utilizes the potential prior knowledge in the input image and the mask to guide the reconstruction process. For the denoising sublayer, local window based and shuffle based self-attention are leveraged to capture local and long-range dependencies. Overall, the approach achieves over 4 dB higher performance than SOTA deep unfolding based methods, and 0.78 dB than SOTA.\n\nWeaknesses:\nIn my opinion, the authors combine three approaches to establish their method [1,2,3]. To be specific, the authors of [1] also explore a deep unfolding network based on MAP and HQS and then get two parameters, alpha and beta, to generate x_k and z_k iteratively. With respect to the two-branch attention paradigm, the local attention based on windows and long-range attention based on shuffle operation have already put forward in [2] but in successive manner instead of parallel. For the flow of transformer, this paper follows most parts of [3], e.g., FFN part and FPN-style connection. To sum up, I doubt the novelty of the method. If authors are inspired by these papers, it would be better to include them in the reference. If I am wrong, please correct me.\n\n[1] Deep Unfolding Network for Image Super-Resolution, CVPR 2020.\n[2] CAT: Cross Attention in Vision Transformer, arxiv 2021.\n[3] Mask-Guided Spectral-Wise Transformer for Efficient Hyperspectral Image Reconstruction, CVPR 2022.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept