Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a hyperspectral image reconstruction method based on a deep unfolding method and a Half-shuffle Transformer (HST). The deep unfolding method estimates parameters from the image and uses them to direct the iterative learning. The HST captures both local contents and non-local dependencies with relatively low computation cost. The proposed method was evaluated on both synthetic and real image datasets and demonstrated excellent performance over a few other methods. Ablation studies were also provided to show the contribution of key components in the proposed method.
strengths_and_weaknesses: This is a solid paper. Each proposed component in the whole method structure seems to be working, and when putting together, they produce the best hyperspectral image reconstruction performance reported so far. However, I also have several concerns about this paper.

First, the proposed DAUHST method has its novelty in combining transformer and deep unfolding for HSI restoration. However, the contribution can only be considered incremental since a similar idea (transformer as prior + unfolding) has been proposed for hyperspectral image super-resolution in November 2021, which is closely related to the reconstruction or restoration task: Learning A 3D-CNN and Transformer Prior for Hyperspectral Image Super-Resolution. Furthermore, both deep unfolding and transformer have been adopted separately for hyperspectral image processing. Take the self-attention HS-MSA module in the proposed Half-Shuffle Transformer (HST) as an example, it can be considered an extension of the S-MSA in MST-L [81]. Table 2 (b) also shows that the performance improvement of HS-MSA over S-MSA is marginal. 

Second, the introduction section hasn't clearly described the motivation of the degradation-aware unfolding framework. It simply says previous methods do not estimate CASSI degradation patterns and ill-posedness degree. Please clearly define what are CASSI degradation patterns and ill-posedness degree and why they are important to the estimation process.

The method description also needs to be improved as well. In Figure 2, the parameter estimator takes the compressed measurement and the sensing of the CASSI system as inputs and produces \alpha and \beta. How is the estimation done? The details are missing. In lines 127-129, \mu and \tau are iteration-specific parameters. How their values are determined in each iteration? 

Table 1 shows that the 9 stages DAUHST produces the best results. An analysis of the influence of the number of stages on the final performance shall be given.
questions: Please address the comments in the Strengths And Weaknesses.
limitations: The spectral ranges covered by the images in the adopted synthetic datasets are narrow. Please consider experiments with more infrared spectrum.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes a degradation-aware unfolding framework (DAUF) where the parameters are estimated by the input image and physical mask, which can serve as a guidance for each iteration stage towards the final output. Among each iteration stage, two parts, e.g., a linear projection and denoising network, perform successively based on maximum a posteriori (MAP). Specifically, the denoising network builds on Transformer-type framework replacing the vanilla self-attention mechanism with two branches, local and non-local branch, to extract the local and long-dependency representations. 
strengths_and_weaknesses: Strengths:
The paper is clearly written and the logic flow is easy to follow. The method sufficiently utilizes the potential prior knowledge in the input image and the mask to guide the reconstruction process. For the denoising sublayer, local window based and shuffle based self-attention are leveraged to capture local and long-range dependencies. Overall, the approach achieves over 4 dB higher performance than SOTA deep unfolding based methods, and 0.78 dB than SOTA.

Weaknesses:
In my opinion, the authors combine three approaches to establish their method [1,2,3]. To be specific, the authors of [1] also explore a deep unfolding network based on MAP and HQS and then get two parameters, alpha and beta, to generate x_k and z_k iteratively. With respect to the two-branch attention paradigm, the local attention based on windows and long-range attention based on shuffle operation have already put forward in [2] but in successive manner instead of parallel. For the flow of transformer, this paper follows most parts of [3], e.g., FFN part and FPN-style connection. To sum up, I doubt the novelty of the method. If authors are inspired by these papers, it would be better to include them in the reference. If I am wrong, please correct me.

[1] Deep Unfolding Network for Image Super-Resolution, CVPR 2020.
[2] CAT: Cross Attention in Vision Transformer, arxiv 2021.
[3] Mask-Guided Spectral-Wise Transformer for Efficient Hyperspectral Image Reconstruction, CVPR 2022.

questions: 1, For the input of denoising network, beta_k is stretched to concatenate with x_k following the Eq. 12, and in Tab 2(d) the learnable alpha is studied. I wonder if beta and alpha can be both set as learnable parameters and achieve a compatible result, i.e., learnable beta is also stretched as the original method do or just be added into x_k . In that case, the method would not be degradation-aware.
2, In 3.4, HS-MSA is remove in the first two subparts. Is it substituted by identity operation? DAUHST-3stg is used to perform ablation studies except for self-attention mechanism (DAUHST-1stg). Is it because the memory bottlenecks of global self-attention?
3, In Eq. 3, auxiliary variable z is introduced to replace x in the image prior term R(*), but this replacement occurs in the data term in Eq4. of [1]. What is the difference?
4, It would be better to provide the comparison of complexity for attention alternatives.
5, The DAUF works in an iterative manner. Is the same HST used or not for different stages? Is there any difference between training and test phase? For example, should the test input go though all stages?
[1] Deep Unfolding Network for Image Super-Resolution, CVPR 2020.

limitations: The method works in an iterative manner where there is a Transformer-type HST in each stage, which may lead to a long training and inference time. In addition, the non-local branch captures relatively long-range dependencies but not globally.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a novel unfolding Transformer-based algorithm for hyperspectral image reconstruction in CASSI system. There are two key components of the proposed method: 

(i) degradation-aware unfolding framework perceiving the CASSI degradation pattern and degree of ill-posedness, and

(ii) half-shuffle Transformer capturing local contents and non-local dependencies jointly. 

Half-shuffle Transformer is plugged into degradation-aware unfolding framework to establish the proposed degradation-aware unfolding half-shuffle Transformer.
strengths_and_weaknesses: Strengths:

(i) This work proposes the first Transformer-based unfolding method for HSI reconstruction in snapshot compressive imaging. 

(ii) The idea is novel and interesting. The degradation-aware unfolding framework changes the general unfolding paradigm of previous works (e.g., ADMM-Net, GAP-Net) by estimating parameters from CASSI system to adapt the iterative learning. Besides, the elaborately customized half-shuffle Transformer combines the advantages of local and global Transformers while using moderate Params and FLOPS.

(iii) The performance is pretty impressive. DAUHST outperforms SOTA algorithms by large margins while using less Params and FLOPS. In particular, DAUHST achieves over 4dB improvements than previous unfolding methods. 

(iv) The experiments are comprehensive and the ablation study is well-designed. The comparisons in Table 1, Figure 4, and Figure 5 including 14 SOTA methods, which is very convincing. The ablation studies in Table 2 sufficiently evaluate the effectiveness of the proposed techniques. The in-depth visual analysis in Figure 6 reveals some important insights of unfolding frameworks for SCI.

(v) The writing is good and easy to follow. Especially in Section 2.1. The mathematical notation and formulation are clear, rigorous, and neatly arranged. In addition, the presentation is well-dressed.

(vi) The reproducibility can be guaranteed. I have check the submitted codes and models, and run the demo. The results in Table 1 of the main paper can be reproduced. The implementation details are described in Section 3.1. Meanwhile, the authors have claimed that they will release the code and models to the public. I believe this work can significantly benefit the community. 

Weakness:

(i) The authors provide visual comparisons of DAUHST, HDNet, and BIRNAT in the supplementary but the main paper lacks these important comparisons. Why not move these results to the main paper?

(ii) In Line 156 of the main paper, the mentioned reshape operation of R is unclear and should be explained.

(iii) In Line 98 of the paper, the “vectorized” and “shifted” operations need more explanations.

(iv) It would be better to evaluate the generality and extensibility of DAUF. For instance, replacing the HST by the same CNN of GAP-Net and ADMM-Net. This ablation can show whether the proposed DAUF fits CNN-based denoising networks or not. 
questions: In Table 1, I am not sure why the FLOPS of BIRNAT is so large? Is it because BIRNAT employs an RNN architecture? 
limitations: The authors have discussed the limitations and social impacts in the supplementary material. 
This work does not have any foreseeable negative social impacts.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: this submission deals with designing unfolding based architecture for supervised learning of spectral compressive imaging (SCI). Compared with SOTA, it proposes to explicitly integrate the ill-posedness of the acquisition (degradation and mask) into each block (iteration) for data consistency and denoising. The denoising network also uses a U-shape transformer with half-shuffle multihead self-attention to capture both local and non-local contents. The experiments outperform SOTA in terms of both memory, computer, and reconstruction fidelity. 
strengths_and_weaknesses: Strength
practical significance: the gains achieved are significant compared with the reported SOTA methods
practical novelty: this is the first work deploying transformers as the denoiser for SCI.
clear writing: the paper is well written and the idea is clear

Weakness
limited technical novelty: The main novelty of this work is to propose a U-shaped transformer with half–shuffle multi-head self-attention for the denoising network in deep unfolding architectures for SCI. Deep unfolding is well established. Also, there are several works reporting superiority of transformers against CNNs for denoising and low level computer vision tasks, and it is not surprising to observe similar gains for SCI; see e.g., [Peit et al’21].

[Peit et al’21]Petit O, Thome N, Rambour C, Themyr L, Collins T, Soler L. U-net transformer: Self and cross attention for medical image segmentation. InInternational Workshop on Machine Learning in Medical Imaging 2021 Sep 27 (pp. 267-276). Springer, Cham.

questions: This work can be strengthened with more ablations that explains the design choices for the U-shaped transformer. For example, what if one simply uses a multi-head vision transformer instead of U-shaped architecture?

limitations: 
The limitations are not included in the submission. The limitations need to be discussed. For example, it seems that the entire network needs to be trained from scratch each time the measurement model changes. The robustness of the model also needs to be discussed in terms of introducing hallucinations.
review_title: null

Meta-review: metareview: This paper integrates a Half-shuffle Transformer (HST) into the deep unfolding framework, establishing an effective method for hyperspectral image (HSI) reconstruction. The reviewers generally agree that the paper is well-written and technically-solid. The majority of the reviews assert that the technical novelty is not so dramatic. This I concur, from the perspective of learning-based reconstruction. On the other hand, from the viewpoint of spectral compressive imaging, I also agree with the authors that the paper has certain novelty and significance. Thus, I would recommend accepting the paper if the space is enough. 
confidence: Less certain
Decision: Accept