Spectral Compressive Imaging via Unmixing-driven Subspace Diffusion Refinement

Authors: Haijin Zeng, Benteng Sun, Yongyong Chen, Jingyong Su, Yong Xu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: spectral compressive imaging, subspace, diffusion, fine-tune

Abstract:
Spectral Compressive Imaging (SCI) reconstruction is inherently ill-posed, offering multiple plausible solutions from a single observation. Traditional deterministic methods typically struggle to effectively recover high-frequency details. Although diffusion models offer promising solutions to this challenge, their application is constrained by the limited training data and high computational demands associated with multispectral images (MSIs), complicating direct training. To address these issues, we propose a novel Predict-and-unmixing-driven-Subspace-Refine framework (PSR-SCI). This framework begins with a cost-effective predictor that produces an initial, rough estimate of the MSI. Subsequently, we introduce a unmixing-driven reversible spectral embedding module that decomposes the MSI into subspace images and spectral coefficients. This decomposition facilitates the adaptation of pre-trained RGB diffusion models and focuses refinement processes on high-frequency details, thereby enabling efficient diffusion generation with minimal MSI data. Additionally, we design a high-dimensional guidance mechanism with imaging consistency to enhance the model's efficacy. The refined subspace image is then reconstructed back into an MSI using the reversible embedding, yielding the final MSI with full spectral resolution. Experimental results on the standard KAIST and zero-shot datasets NTIRE, ICVL, and Harvard show that PSR-SCI enhances visual quality and delivers PSNR and SSIM metrics comparable to existing diffusion, transformer, and deep unfolding techniques. This framework provides a robust alternative to traditional deterministic SCI reconstruction methods. Code and models are available at [https://github.com/SMARK2022/PSR-SCI](https://github.com/SMARK2022/PSR-SCI).

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1729164191356

thread_type:
official_review

<thread_object>
creator: reviewer_z3xv

<content>
summary:
The paper introduces a novel framework called Predict-and-Unmixing-driven Subspace Refinement (PSR-SCI) for Spectral Compressive Imaging (SCI) reconstruction. SCI is an inherently ill-posed problem where traditional methods often struggle to recover high-frequency details. The proposed PSR-SCI framework starts with a cost-effective predictor that provides an initial estimate of the multispectral image (MSI). It then introduces an unmixing-driven reversible spectral embedding module to decompose the MSI into subspace images and spectral coefficients. This decomposition allows the adaptation of pre-trained RGB diffusion models, focusing refinement on high-frequency details efficiently, even with minimal MSI data. Additionally, the authors design a high-dimensional guidance mechanism with imaging consistency to enhance the model's effectiveness. The refined subspace image is reconstructed back into the MSI using the reversible embedding, yielding the final MSI with full spectral resolution. Experimental results on standard datasets like KAIST and zero-shot datasets such as NTIRE, ICVL, and Harvard demonstrate that PSR-SCI enhances visual quality and achieves PSNR and SSIM metrics comparable to some existing diffusion, transformer, and deep unfolding techniques.

strengths:
1. The proposed high-dimensional guidance mechanism with imaging consistency is a noteworthy contribution that could enhance the model's performance.

2. The method is evaluated on both standard datasets and zero-shot datasets, showing competitive performance in PSNR and SSIM metrics compared to existing methods.

weaknesses:
1. The motivation for using the unmixing-driven spectral embedding decomposition is not entirely clear. The paper does not sufficiently justify why this specific decomposition method is suitable for the SCI problem or how it leads to superior performance in terms of convergence speed, recovery accuracy, or error reduction. While the approach is innovative, its superiority over existing methods is not well-established.

2. The claim that there is limited training data available for MSIs compared to RGB images is questionable. Recent works have utilized large-scale MSI datasets to train diffusion models effectively (e.g., "HSIGene: A Foundation Model For Hyperspectral Image Generation," https://arxiv.org/abs/2409.12470, https://github.com/LiPang/HSIGene). This undermines the premise that data scarcity necessitates their approach.

3. The proposed method appears to have a high number of parameters and computational demands. It is unclear whether the performance gains are due to the novel method or simply the result of using a much larger model compared to others. Methods like DAUHST and MST use models with millions of parameters, whereas the proposed method uses billions. This raises concerns about the practicality and efficiency of the approach, especially considering the longer inference times.

4. Despite the innovative approach, the performance of PSR-SCI on the KAIST dataset is reported as 38.14dB PSNR, which is lower than some recent works that achieve PSNR values exceeding 39dB (e.g., PADUT [ICCV 2023], RDLUF-MixS2 [CVPR 2023], and "Latent Diffusion Prior Enhanced Deep Unfolding for Snapshot Spectral Compressive Imaging" [ECCV 2024]). This suggests that the proposed method may not offer a significant improvement over existing techniques.

questions:
1. Could the authors elaborate on why the unmixing-driven spectral embedding decomposition is particularly suitable for SCI reconstruction? How does it contribute to faster convergence, higher recovery accuracy, or error reduction compared to other methods?

2. What is the total number of parameters in the proposed model, and how does it compare to other methods? Could the authors provide a detailed analysis of the computational complexity and inference times? How practical is the method for real-world applications given its computational demands?

3. Given that recent works have successfully trained diffusion models on large-scale MSI datasets, how do the authors address the claim of MSI data scarcity? Does this affect the necessity or uniqueness of the proposed approach?

4. To what extent does the high performance of PSR-SCI stem from the large number of parameters and depth of the model, as opposed to the proposed methodology itself? Have the authors conducted experiments to isolate these factors?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729997228788

thread_type:
official_review

<thread_object>
creator: reviewer_hqgd

<content>
summary:
The proposed Predict-and-Unmixing-driven Subspace-Refine framework (PSR-SCI) addresses the challenges in Spectral Compressive Imaging (SCI) reconstruction, particularly the difficulty of recovering high-frequency details from ill-posed inverse problems. By leveraging a cost-effective prediction module, unmixing-driven reversible spectral embedding, and pre-trained RGB diffusion models, PSR-SCI demonstrates superior visual quality and competitive performance compared to state-of-the-art diffusion, transformer, and deep unfolding approaches, providing an alternative to traditional deterministic methods.

strengths:
The problem this work aims to solve is crucial. Recovering high-frequency details from ill-posed inverse problems is always challenging for both traditional optimization-based methods and end-to-end deterministic networks. Leveraging the powerful generative capabilities of models pre-trained on large RGB datasets and making them transferable to spectral imaging is a promising approach to address this challenge.

In addition, I like the idea of using reversible spectral unmixing to bridge spectral image and RGB image. 

The novelty is enough, to address the high-frequency reconstruction challenge in spectral compressive imaging, this paper solved several problems and makes 4 key contributions: (1) the introduction of a spectral unmixing-driven predict-and-subspace refine strategy, offering improved perceptual quality and efficiency; (2) the inclusion of a reversible decomposition module, tackling the ill-posed nature of spectral unmixing; (3) focusing diffusion generation on high-frequency components, reducing training data requirements and accelerating fine-tuning; and (4) employing high-dimensional guidance with SCI imaging consistency to enhance robustness.

Moreover, this method shows nice results on the real SCI dataset, which is a hard case for existing methods.

weaknesses:
In lines 296-365 of the main paper and Section A.3 of the appendix, most of the mathematical derivations are easy to follow. However, some detailed notations are unclear in the context, which reduces readability.

The motivation for designing the reversible unmixing module is not sufficiently clear. For example, why not use an SVD-based method? What advantages does the proposed unmixing module offer?

questions:
(1) For the reversible unmixing module, although the experiments show that it is lightweight and efficient, I am curious how it compares with SVD, PCA, or band selection-based methods.

(2) It seems the hyperparameters, scale $s$and timestep $T$, shown in Fig. 12, are sensitive to the model's reconstruction accuracy. How do they influence the visual quality of the reconstructed image? The PSNR is not always a reliable metric for measuring image quality, especially for spectral images.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730224528041

thread_type:
official_review

<thread_object>
creator: reviewer_wq83

<content>
summary:
This paper proposes a new two-stage method, namely a predict-and-subspace refine (PSR-SCI) framework, using the diffusion model for snapshot compressive imaging reconstruction. In the first stage, an inexpensive predictor generates a rough MSI approximation. Then in the second stage, a spectral unmixing-driven subspace learning module is employed to reduce the dimensionality. Eventually, a diffusion model is fine-tuned to enhance the high-frequency details. There are three technical contributions:

(i) Given the complexities of spectral unmixing models, a reversible decomposition network is introduced to implement spectral subspace learning while maintaining high reversibility.

(ii) The diffusion generation is facilitated by low- and high-frequency decomposition.

(iii) A high-dimensional guidance strategy is introduced for fine-tuning subspace diffusion, enhancing the effectiveness of guidance within the subspace.

strengths:
(i) The novelty is good and interesting. The big idea of partitioning the reconstruction into two stages is cool. It seems like the latent space diffusion models. The first stage of prediction and decomposition is like the embedding process of VAE and this process is highly reversible. Besides, using the frequency domain decomposition (low- and high-frequency) to facilitate the generation process is also very fancy. It matches the nature of spectra. It is very exciting to have this work in the community of SCI reconstruction.

(ii) The performance is solid. As compared in Table 1, Figures 6, 8, and 9. The proposed method PSR-SCI not only outperforms the state-of-the-art end-to-end methods by large margins but also achieves better visual results. Compared to the SOTA diffusion-based method DiffSCI, the proposed PSR-SCI is 2.86 dB higher! It is great progress of the diffusion-based method in SCI. This work even contains the results of Pseudo-RGB on the RGB-to-HSI reconstruction task. 

(iii) The presentation is well-dressed. The figure of the pipeline looks clear. I can easily see the workflow of the proposed framework. The table of quantitative comparisons is also neat. The style of this table is from the series work of MST, MST++, CST, DAUHST, etc. I like it very much. The writing also looks good and easy to follow.

(iv) Code and the reconstruction results have been submitted,

weaknesses:
(i) Some modifications and explanations should be added. For example, why do you want to use an inexpensive predictor to produce an initial HSI instead of directly reconstructing the HSI results from the noisy measurements? Besides, since the process looks like the Stablediffusion - latent space diffusion. So I think it is better to discuss your work with it to highlight the differences and your contributions.

(ii) The experiments are insufficient now. The ablation study only has a visual study (Figure 10). This is far from satisfactory. I suggest you add a break-down ablation like MST, MST++, CST, and DAUHST series works to demonstrate the effectiveness of each technical contribution. Or your claim of your work - the three modules are not convincing since their effects are unknown.

(iii) The computational cost and memory usage of different methods are not reported in Table 1. I cannot judge the efficiency of the proposed method and compare it with other algorithms. I suggest you list the number of parameters and the floating-point operations per second (FLOPS).

questions:
Some parts may need more explanation. For example,

In Algorithm1 (Line 270 - 287), the return value $\mathcal{A}_{diff}^h$ is fed into the VAE decoder to obtain the final reconstructed hyperspectral images. However, for the latent space diffusion, the VAE is trained to decode normal RGB images. Can it directly work on hyperspectral images? Or did you fine-tune it or something like that?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625373412

thread_type:
official_review

<thread_object>
creator: reviewer_igvu

<content>
summary:
The authors present a novel framework named PSR-SCI aimed at enhancing CASSI reconstruction. The framework consists of a initial predictor(MST or DAUHST), a unmixing-driven reversible spectral embedding module to decomposes the MSI into subspace images and spectral coefficients, and a pre-trained RGB latent diffusion models (SD-2.1) refinement processes. Furthermore, the authors implement a high-dimensional guidance mechanism that ensures imaging consistency. Experiments on the KAIST, NTIRE, ICVL, and Harvard dataset demonstrate that PSR-SCI enhances visual quality and achieves PSNR and SSIM metrics.

---
After rebuttal: I read the authors response and have raised my score.

strengths:
1. The proposed method has higher performance than previous diffusion-based methods on CASSI reconstruction.
2. The method costs less time in both training and inference time than previous diffusion-based methods.

weaknesses:
1. The novelty of this approach is limited, as spectral decomposition and refinement using pre-trained RGB diffusion models have previously been introduced in the spectral image restoration field [1].
2. The method relies heavily on the initial prediction network. Although the authors claim robustness for the proposed framework, it may not perform well on unseen optical setups (such as different masks, noise, and number of channels). The framework depends on initial reconstruction results from a pre-trained CASSI reconstruction network. If the pre-trained network does not perform adequately on a new system with different optical settings (a common issue in CASSI reconstruction) the framework’s effectiveness diminishes. For example, PSR-SCI-T (initial predictor: MST) and PSR-SCI-D (initial predictor: DAUHST) demonstrate this, with weaker predictors leading to inferior reconstruction results.
3. The experiments presented are insufficient. The proposed method should be evaluated with recent state-of-the-art predictors, such as [2], DPU-9stg, and SSR-L. Additionally, comparisons with similar 'refinement' frameworks, such as [3], would further validate its performance.

[1] Pang, Li, et al. "HIR-Diff: Unsupervised Hyperspectral Image Restoration Via Improved Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[2] Yao, Zhiyang, et al. "SPECAT: SPatial-spEctral Cumulative-Attention Transformer for High-Resolution Hyperspectral Image Reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[3] He, Wei, et al. "An interpretable and flexible fusion prior to boost hyperspectral imaging reconstruction." Information Fusion (2024): 102528.

questions:
1. Why are the SSIM metrics omitted in Table 2?
2. The number of abundance maps A in spectral unmixing should correspond to the number of endmembers. Why is an up-sampled 3-channel feature map used instead? Is this solely for compatibility with the RGB pre-trained model? The URSe module seems more like a data dimension reduction or latent space encoding operation, with little relevance to spectral unmixing
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposed a Predict-and-unmixing-driven-Subspace-Refine framework (PSR-SCI), which achieves higher performance than previous diffusion-based methods. The motivation is clear and interesting.
Given that all four reviewers have provided high evaluations and the authors have responded in great detail to the reviewers’ comments, which has further improved the quality of the paper, I recommend accepting this paper.