SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening

Authors: yu  zhong
xiao  wu
zihan  cao
hong-xia  dou
liang-jian  deng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 18:44:29

Abstract: Pansharpening is a significant image fusion technique that merges the spatial content and spectral characteristics of remote sensing images to generate high-resolution multispectral images. Recently, denoising diffusion probabilistic models have been gradually applied to visual tasks, enhancing controllable image generation through low-rank adaptation (LoRA). In this paper, we introduce a spatial-spectral integrated diffusion model for the remote sensing pansharpening task, called SSDiff, which considers the pansharpening process as the fusion process of spatial and spectral components from the perspective of subspace decomposition. Specifically, SSDiff utilizes spatial and spectral branches to learn spatial details and spectral features separately, then employs a designed alternating projection fusion module (APFM) to accomplish the fusion. Furthermore, we propose a frequency modulation inter-branch module (FMIM) to modulate the frequency distribution between branches. The two components of SSDiff can perform favorably against the APFM when utilizing a LoRA-like branch-wise alternative fine-tuning method. It refines SSDiff to capture component-discriminating features more sufficiently. Finally, extensive experiments on four commonly used datasets, i.e., WorldView-3, WorldView-2, GaoFen-2, and QuickBird, demonstrate the superiority of SSDiff both visually and quantitatively. The code is available at https://github.com/Z-ypnos/SSdiff_main.

Main paper: 
 introduction: 
 Due to physical limitations, satellite sensors cannot directly acquire high-resolution multispectral images (HrMSI). Instead, they can obtain high-resolution panchromatic (PAN) images and lowresolution multispectral images (LrMSI). Pansharpening techniques can merge PAN images with LrMSI, generating HrMSI that possess both high spatial and spectral resolutions. Pansharpening, † Equal contribution. * Corresponding author.38th Conference on Neural Information Processing Systems (NeurIPS 2024). as a fundamental preprocessing method, has been widely utilized in various applications, including change detection [37] and image segmentation [45] .Pansharpening methods are roughly categorized into four types: component substitution (CS) methods, multi-resolution analysis (MRA) methods, variational optimization (VO) techniques, and deep learning (DL) methods, as shown in Fig. 1 (a) . The CS method [17, 22] involves projecting the LrMSI into a specific domain and replacing the spatial components of the LrMSI in that domain with the corresponding components from the PAN image. CS-based methods can generate fusion images with high spatial fidelity but spectral distortion with fast runtime. MRA-based methods [24, 32] extract spatial details from the PAN image through multiscale decomposition and inject them into the LrMSI. While MRA-based methods effectively preserve spectral information, they may sacrifice spatial details. Compared to CS-based and MRA-based methods, VO-based techniques [40, 41] gain more mathematical guarantees but require handling a higher computational burden and more adjustable parameters.In recent years, DL-based methods [12, 39, 42, 38, 8, 19] have increasingly been applied across various fields, including pansharpening tasks, yielding exciting results. Traditional DL-based methods typically utilize a single-scale model to process information from PAN images and LrMSI. In a single-scale network, PAN images and LrMSI are usually stacked together without distinguishing the information contained within them, serving as inputs to the network. They overlook the disparities in the deep-level information inherent in both, potentially leading to the omission of crucial discriminative features and subsequently influencing lower fusion performance. Then, dual-branch methods [6, 18] based on deep learning can differentiate and hierarchically learn information from PAN and LrMSI. Thanks to this design, it has shown outstanding performance in pansharpening tasks. However, the cumbersome structure of dual-branch networks makes it challenging to perform localized fine-tuning. Denoising diffusion probabilistic model (DDPM) [13] is attaining attention in remote sensing pansharpening [21, 4] . Unfortunately, existing DDPM-based methods have not yet designed models for the discriminative features required in the pansharpening task.Considering the characteristics of the pansharpening task, we propose a novel SSDiff method based on subspace decomposition, which leverages spatial and spectral branches to discriminatively capture global spatial information and spectral features, respectively. Additionally, we further construct an alternating projection fusion module (APFM) to fuse the captured spatial and spectral components. Besides, a frequency modulation inter-branch module (FMIM) is designed to overcome the problem of uneven distribution of frequency information between two branches in the denoising process. Finally, through the proposed LoRA-like branch-wise alternating fine-tuning (L-BAF), our SSDiff can further reveal spatial and spectral information not discovered in each branch. The main contributions of this work are as follows:• Our SSDiff is based on subspace decomposition to divide the network into spatial and spectral branches. In addition, for subspace decomposition, we give an illustration of vector projection and construct an alternating projection fusion module (APFM). APFM transforms the process of fusing HrMSI into the fusion process of spatial and spectral components. Moreover, our SSDiff is tested on four widely used pansharpening datasets and achieves state-of-the-art (SOTA) performance. • The frequency modulation inter-branch module is used at the junction of spectral and spatial branches to enrich extracted spatial information with more high-frequency information in the denoising process. • The proposed L-BAF method is used to fine-tune the network based on the proposed APFM, where the spatial and spectral branches are updated alternately. This design allows us to alternately fine-tune the two branches without increasing the parameter count, enabling the learning of more discriminative features. DL-based Methods. As a simple but effective method, the representative single-scale coupling model, namely PNN [20] , first proposes a simple and effective threelayer CNN architecture and achieves the best results at that time. Subsequently, other methods such as Fu-sionNet [5] , DCFNet [39] , and others adopt similar coupled input approaches and successfully design their networks. However, these methods still have significant room for improvement in spectral fidelity and generalization performance due to weak feature representation in their network structure designs. In multi-source image fusion tasks, images acquired from diverse sources exhibit varying characteristics. Coupling two information sources together may suffer from inadequate feature extraction. Then, spatial and spectral branches methods [6, 18, 25] based on deep learning can differentiate and hierarchically learn information from PAN images and LrMSI. These methods can better exploit the potential advantages of multi-scale information. 
related works: 
 Diffusion-based Model. DDPM, as a generative model, has been widely applied in various domains such as text-to-image generation [26] , image editing [16] and image classification [43] . In recent years, DDPM has shown its prominence in image processing tasks [9, 28, 44] . Among them, Song et al. [28] propose denoising diffusion implicit models (DDIM), where they design a non-Markov chain sampling process, accelerating the sampling of diffusion models. Then, through some simple modifications, IDDPM [23] achieves competitive log-likelihoods while preserving the high sample quality of DDPM. Currently, DDPM is attracting attention in the field of pansharpening [21, 4] . These DDPM-based methods treat PAN and LrMSI as model fusion conditions, unlike other pansharpening methods where they serve as fusion targets.LoRA: Low-rank Adaptation of Large Language Models. For the fine-tuning of parameters in large pre-trained models, Hu et al. [14] introduce the LoRA to freeze the pre-trained model weights and inject trainable low-rank decomposition matrices into each layer of the Transformer architecture. This significantly reduces the number of trainable parameters for downstream tasks. For H = W 0 X, the modified forward pass of the LoRA follows the formula:EQUATIONwhere issues such as insufficient feature learning and generalization capabilities, though they may employ low-rank adaptation (LoRA) to improve the performance of DDPM for the pansharpening task.W 0 ∈ R d×k is a pre-trained weight matrix, X ∈ R k×n , B ∈ R d×r , A ∈ R r×kTo alleviate these problems, we propose SSDiff, which transforms the problem of solving HrMSI into a fusion problem of spatial and spectral components. Significantly, we give an illustration of linear algebra to remove the gap between subspace decomposition and the self-attention mechanism. The SSDiff utilizes vector projection to discriminatively capture global spatial information and spectral features in spatial and spectral branches. By introducing subspace decomposition, we can further illustrate and generalize the vector projection to the matrix form. Based on this, we propose an APFM that naturally decouples spatial and spectral information and fuses the captured features. Unlike the LoRA method, the APFM can establish low-rank representations for the spatial-spectral branches more accurately, as shown in Fig. 1 . Furthermore, the spectral branch contains abundant low-frequency information. When low-and high-frequency information from the spatial branch is injected into the spectral branch, it may result in an overemphasis on low-frequency information, impacting the denoising performance of the model. Based on this, we propose FMIM to modulate the frequency information between different branches. The overall model is trained by L-BAF to uncover spatial and spectral information not discovered in each branch. 
ssdiff architecture: 
 Inspired by the LoRA approach, we view the pansharpening task as the fusion of spatial and spectral components, where the spatial and spectral elements can be considered as a matrix decomposition of a multi-spectral image. Based on these characteristics, our SSDiff employs a model comprising a spatial branch and a spectral branch, as shown in Fig. 3 . Both the spatial branch and the spectral branch comprise two encoder layers and two decoder layers. Down-sampling occurs between the encoder layers to decrease the spatial resolution while increasing the number of channels. Up-sampling operation between the two layers of the decoder increases the spatial resolution while decreasing the number of channels, and a down-sampling convolution layer connects the encoder and the decoder. The spatial branch employs ResNet [11] blocks to convert spatial images into features. These spatial features are transmitted to corresponding layers in the spectral branch via a frequency modulation inter-branch module. Additionally, fusing incoming spatial features and spectral information via an alternating projection fusion module.Eventually, it is delivered to the next stage via an MLP. In this work, we convert the objective from ϵ to x 0 , so the loss function L simple [4] takes the following form:EQUATIONwhere x θ denotes the prediction of the model and c is the conditions for injecting the model. Inspired by FusionNet [5] , our SSDiff changes the forward and backward denoising objects during the training process from HrMSI to the difference between HrMSI and up-sampled LrMSI. The detailed training process of SSDiff can be found in Appendix. D.This section starts with vector projection in linear algebra (See Lemma 1) and subspace decomposition (See Definition 1) to illustrate vector projection as a specific subspace decomposition. Then, the selfattention mechanism [30] is generalized into the proposed alternating projection fusion framework, i.e., APFM. In what follows, we rewrite vector projection as follows.Lemma 1 ( [29] ). Assuming that the existing two arbitrary vectors a ∈ domU ∈ R n and b ∈ domG ∈ R n , then Pb = λa = p, we have the following formula:EQUATION)where P is projection matrix, λ is the scaling factor, and p is the vector in the same domain as a.Proof. For any two vectors a and b, there exists a vector e = pb such that e is orthogonal to a.We have the following equation:EQUATIONthus, we haveEQUATIONTaking Eq. ( 5) into Pb = λa = p, we have the conclusion:EQUATIONDefinition 1 ( [7] ). Assume that D ∈ R S×L is the subspace and C ∈ R L×HW is the corresponding coefficients. We have:EQUATIONBased on subspace decomposition, we can take spatial and spectral components to accomplish the pansharpening of remote sensing images. According to Lemma 1, we can further determine a specific subspace decomposition in Definition 1. The subspace represents the projection relationship between vectors a and b. Interestingly, we find that we can generalize Eq. ( 3) and Eq. ( 7) to the matrix form of the self-attention mechanism, as shown in Fig. 2 . In other words, the self-attention mechanism is represented as the vector projection of Eq. ( 3), which is the low-rank subspace decomposition. Remark 1. Back to the pansharpening applications, the input PAN and LrMSI are mapped to a high-dimensional feature space. The features often exhibit significant correlations between frequency bands, while spectral vectors typically reside in a low-dimension subspace. These features can be represented as Eq. (7) . In this way, the characteristics in the image domain can be transformed into the subspace.Based on the above analysis, we can build an alternating projection framework, which is summarized in the following theorem. Theorem 1. Assuming that F spa ∈ R H×W ×S and F spe ∈ R H×W ×S from the spatial and spectral branches, they can be alternatively projected as follows: HW are constants related to the matrix size. Softmax(•) stands for the Softmax function.EQUATIONProof. According to Lemma 1, we can generalize the self-attention mechanism [30] , where Q and K are the features from domU, and V is the feature from domG, respectively. Thus, we have:EQUATIONwhere "⇒" suggests that the left and right parts of the equation are similar in form, and the left side is a special case of the right side. Through this equation, we can partially explain the principle of cross-attention using the vector projection theorem. Then we transform the projection relationship between the spatial domain (domU) and spectral domain (domG), where T a , T b ∈ domU and T c , T d ∈ domG. d is a self-attention constant. As a result, the alternating projection is complete from the spatial/spectral domain to the spectral/spatial domain, i.e., Eq. ( 8).In addition, we need to get fused outputs from T spa and T spe . Without loss of generality, we haveEQUATIONwhere ⊙ defines element-wise multiplication. Element-wise multiplication is used to fuse spatial and spectral information to obtain T f us ∈ R HW ×S ′ .Comparing Eq. ( 3) with Eq. ( 8), this subspace is built by vector projection and naturally decouples spatial and spectral information into the self-attention mechanism. This inspires us to apply a fine-tuning method similar to LoRA methods (See details in Sect. 3.4).Through the APFM, we build an effective fusion module from the characteristics of images. Interestingly, there are some differences between the spatial and spectral components. The spectral branch contains abundant low-frequency information. When low-and high-frequency information from the spatial branch is injected into the spectral branch, it may result in an overemphasis on low-frequency information, impacting the denoising performance of the model. We found that modulating the frequency distribution contributes to SSDiff obtaining better fusion results.As shown in Fig. 4 , the low-frequency components undergo a gradual modulation characterized by a slow and subtle rate of change in the denoising process. In contrast, the modulation process of the high-frequency components exhibits distinct dynamic variation. Considering the above phenomenon, we design a frequency modulation inter-branch module. Specifically, we utilize a Fourier filter to extract the high-frequency information of the feature map x spa obtained from the spatial branch, following:EQUATIONxEQUATIONwhere FFT and IFFT are Fourier transform and inverse Fourier transform. ⊙ denotes element-wise multiplication, and α is a Fourier mask [27] . Furthermore, we find in experiments that directly injecting high-frequency information into spectral branches will cause the frequency information imbalance. To solve this problem, half of the channels of feature x spec of the spectral branch are multiplied by a constant. For the three different channel numbers in the model, ranging from low to high, we set this constant to 1.2, 1.4, and 1.6, respectively.During the model training process, it is crucial to carefully maintain a balance between model underfitting and overfitting. Our SSDiff can hierarchically and discriminatively extract more features. However, achieving a simultaneous balance condition for both spatial and spectral branches during unified training is undoubtedly challenging. Therefore, this paper approaches the LoRA-like alternate fine-tuning of each branch as a feasible solution. As shown in Fig. 1 , LoRA methods fine-tune the output of the model by updating the parameters on the fully connected layer weights. Compared with LoRA, the proposed alternating projection method is also a low-rank matrix decomposition. In Fig. 1 (b), the difference is that we can have the backpropagation process and control the gradient of the projection process to achieve alternate fine-tuning in the proposed APFM. In practice, taking the fine-tuning of Eq. ( 8) (T spa ) as an example to update the spectral branch, we can detach the gradient propagation at T spa , preventing parameter updates in the spatial branch. In this case, gradients only propagate through the path shown in Eq. ( 8) (T spe ), which means only the parameters of the spectral branch are updated. Similarly, when fine-tuning the spatial branch, detaching the gradient propagation from the computation graph at T spe can prevent parameter updates in the spectral branch. A more intuitive fine-tuning process is provided in the supplementary material. [20] 3.6798±0.7625 2.6819±0.6475 0.8929±0.0923 0.9761±0.0075 0.0213±0.0080 0.0428±0.0147 0.9369±0.0212 DiCNN [12] 3.5929±0.7623 2.6733±0.6627 0.9004±0.0871 0.9763±0.0072 0.0362±0.0111 0.0462±0.0175 0.9195±0.0258 MSDCNN [36] 3.7773±0.8032 2.7608±0.6884 0.8900±0.0900 0.9741±0.0076 0.0230±0.0091 0.0467±0.0199 0.9316±0.0271 FusionNet [5] 3.3252±0.6978 2.4666±0.6446 0.9044±0.0904 0.9807±0.0069 0.0239±0.0090 0.0364±0.0137 0.9406±0.0197 CTINN [48] 3 [48] 0.8251±0.1386 0.6995±0.1068 0.9772±0.0117 0.9803±0.0015 0.0586±0.0260 0.1096±0.0149 0.8381±0.0237 LAGConv [15] 0.7859±0.1478 0.6869±0.1125 0.9804±0.0085 0.9906±0.0019 0.0324±0.0130 0.0792±0.0136 0.8910±0.0204 MMNet [49] 0.9929±0.1411 0.8117±0.1185 0.9690±0.0204 0.9859±0.0024 0.0428±0.0300 0.1033±0.0129 0.8583±0.0269 DCFNet [39] 0.8896±0.1577 0.8061±0.1369 0.9727±0.0100 0.9853±0.0024 0.0234±0.0116 0.0659±0.0096 0.9122±0.0119 PanDiff [21] 0.8881±0.1197 0.7461±0.1032 0.9792±0.0097 0.9887±0.0020 0.0265±0.0195 0.0729±0.0103 0.9025±0.0209 SSDiff (ours) 0.6694±0.1244 0.6038±0.1080 0.9836±0.0074 0.9915±0.0017 0.0164±0.0093 0.0267±0.0071 0.9573±0.0100Ideal value 0 0 1 1 0 0 14 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.  
experimental results: 
 Results on WorldView-3. On the WorldView-3 dataset, we evaluate the performance of SSDiff using 20 test images. The results for both reduced-resolution and full-resolution are presented in Table 1 .The running time of a single picture during the sampling process is 7.417 seconds under 10 timesteps. We compared our method with three traditional methods and some SOTA DL-based methods. To illustrate the performance of each method more clearly, we presented the fusion result images and error maps of some of these methods in Fig. 5 , and zoomed in on a specific location. On average, our method achieves SOTA performance on the reduced dataset, with our SSDiff reaching 2.84 (SAM) and 2.10 (ERGAS) metrics, outperforming all DL-based methods. The error map indicates that the images sampled by SSDiff are closer to the ground truth (GT). SSDiff achieves SOTA performance in obtaining full-resolution images on the WV3 dataset. The HQNR score close to 1 indicates better fusion quality of the full-resolution images. The obtained results demonstrate that our SSDiff can fuse HrMSI, reducing spatial and spectral distortions, thereby proving its excellent generalization ability at full resolution.Results on GaoFen-2. On the GaoFen-2 reduced dataset, we tested our SSDiff on 20 test images, as shown in Table 1 . Our SSDiff achieves SOTA performance. From the error maps in Fig. 5 , we can observe that there are still significant differences between traditional fusion methods and DL-based fusion methods. These experiments demonstrate that, compared to other DL-based methods used for comparison in the experiments, the proposed SSDiff exhibits superior spatial performance and effectively preserves spectral information.Effectiveness of Decoupling Branches. To investigate the effectiveness of the spatial and spectral branch design, we perform ablation experiments by training the diffusion model under the following conditions: V1) Coupling inputs, and training only with the spatial branch. V2) Coupling inputs and training only with the spectral branch. V3) Spatial and spectral branch structure and decoupling inputs, only the outputs of the two branches are concatenated without any inter-branch information interaction. V4) Spatial and spectral branch structure and decoupling inputs, replace each APFM with an additional operation, i.e., the way of LoRA. V5) Spatial and spectral branch structure and decoupling inputs, replace each APFM with a multiplication operation. V6) Our method. The results on the WV3 reduced dataset are reported in Table 2 . Training solely with a single branch significantly reduces the quality of the HrMSI. It can be seen that SSDiff is more suitable for pansharpening tasks than the LoRA way, multiplication, and concatenating way. Generalization. To test the generalization ability of DL-based methods, we evaluated models trained on the WV3 dataset using 20 reduced resolutions from the WorldView-2 dataset. The quantitative evaluation results, as reported in Table 4 , demonstrate that the SSDiff method achieves the best results across all four evaluation metrics. This indicates that our approach possesses a powerful generalization ability. Training SSDiff. To address the issue of insufficient local parameter training in the dual branches model, we design the L-BAF method to fine-tune the model. Taking the experiments on the reduced WV3 dataset, we first train the model without fine-tuning until convergence and perform branch-wise fine-tuning, which includes: 1) Full parameter fine-tuning without L-BAF. 2) Only fine-tuning the spatial branch parameters with the spectral branch parameters fixed. 3) Only fine-tuning the spectral branch parameters with the spatial parameters fixed. 4) Alternating fine-tuning spectral branch and spatial branch. The quantitative results are shown in Table 5 , and all three branch-wise fine-tuning methods lead to improved testing performance. The results of full fine-tuning show a decrease in some metrics, indicating that the method suffers from overfitting. The alternating fine-tuning approach showed the most significant performance improvement. This demonstrates the effectiveness of the L-BAF.  
conclusion: 
 In this paper, we propose a spatial-spectral integrated diffusion model for remote sensing pansharpening, named SSDiff. We design a spatial-spectral integrated model architecture, which utilizes spatial and spectral branches to learn spatial details and spectral features separately. By introducing vector projection, the spatial and spectral components in the subspace decomposition are further specified in the proposed APFM. Then, the self-attention mechanism is naturally generalized to the APFM. Furthermore, we propose an FMIM to modulate the frequency distribution between branches. Finally, the two branches of SSDiff can capture discriminating features. It is interesting that, when utilizing the proposed L-BAF method in the APFM, the two branches can be updated alternately, and then SSDiff produces more satisfactory results. We compare our SSDiff with several SOTA pansharpening methods on the WorldView-3, QuickBird, GaoFen-2, and WorldView-2 datasets. The results demonstrate the superiority of SSDiff both visually and quantitatively. 
acknowledgement: 
 This work is supported by the National Natural Science Foundation of China under Grants 12271083, 12171072 and Natural Science Foundation of Sichuan Province under Grants 2023NSFSC1341. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The models compared in the paper have either received author permission or are publicly available. The datasets are public, and original papers are cited meticulously. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The models compared in the paper have either received author permission or are publicly available. The datasets are public, and original papers are cited meticulously. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing or research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing or research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing or research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the supplementary material, we first introduce the background knowledge, i.e., denoising diffusion probabilistic models, and then illustrate the process of model training and LoRA-like branch-wise alternative fine-tuning, the limitations, and the broader impact of our method. After that, we provide more details for experiments on the pansharpening task, i.e., Implementation Details, Datasets, Benchmarks, and Quality Metrics. Finally, we display more qualitative evaluation results on the QuickBird and GaoFen-2 datasets.Denoising diffusion probabilistic models [13] are latent variable models, generating realistic target images progressively from a normal distribution by iterative denoising. The diffusion model contains two steps: forward and reverse processes.The forward process aims to make the prior data distribution x 0 noisy by a T step Markov chain that gradually transforms it into an approximate standard normal distribution x T ∼ N (0, I d ) and d denotes the dimension. One forward step is defined as follows:EQUATIONwhere N (•) is a Gaussian distribution with the mean of √ 1 -β t x t-1 and variance of β t I, β t is a pre-defined variance schedule in time step t ∈ [0, T ]. Through the reparameterization trick, we can derive x t directly from x 0 , The following equation gives this derivation:EQUATIONwhere ϵ ∼ N (0, I) and α t = 1 -β t , ᾱt = t i=0 α i . The reverse process aims to learn to remove the degradation brought from the forward process and sample the x 0 from x t . To accomplish this objective, we need to learn the distribution of p θ (x t-1 |x t ) using a neural network and perform iterative sampling as follows:EQUATIONwhere µ θ and Σ θ are the mean and variance of p θ (x t-1 |x t ), respectively, and θ is the parameters of model.According to Eq. ( 15), the mean and variance can be computed, following:EQUATIONEQUATION)For sampling from a standard Gaussian noise x T to get x T -1 , after performing T -step iterations of sampling as described above, we get the output x 0 from x T .C LoRA-like Branch-wise Alternative Fine-tuningIn Section 3.4 of the main text. The L-BAF method alternately fine-tunes the spatial and spectral branches based on the proposed APFM. As shown in Fig. 6 . When we fine-tune the spectral branch, we freeze the T a and T b parameters in the spatial branch and the T a parameter in the spectral branch. Block gradient propagation to prevent parameter updates for the entire spatial branch. Similarly, when fine-tuning the spatial branch, we freeze the T c and T d parameters in the spectral branch, as well as the T c parameter in the spatial branch. Block gradient propagation to prevent parameter updates for entire spectral branches.Frozen Trainable θ ← ∇ θ L simple (x 0 , x 0 ). 9 end4 x t ← √ ᾱt (x 0 -LrMSI) + √ 1 -ᾱt ϵ; x0 ← x θ (x t , cond) + LrMSI;The detailed training process of SSDiff can be found in Algorithm 1.First, we evaluate the effectiveness of the proposed SSDiff over the pansharpening task and we will extend our method to other multispectral fusion tasks, such as multispectral and hyperspectral image fusion. Second, the proposed FMIM can adjust the frequency information between the two branches, but it also introduces additional hyperparameters, increasing the difficulty of fine-tuning during training. In terms of cost, the average inference time for a single image is 7.416 seconds under 10 timesteps. The time cost for our approach is higher than other DL-based models, primarily due to the limitation imposed by the large number of sampling steps required in the diffusion model.Pansharpening, as a fundamental preprocessing method which is a key pre-processing technology overcoming the constraints of hardware before using high-resolution multispectral images, has been widely utilized in various applications, including change detection, environmental monitoring, and segmentation. At the same time, pansharpening, being a low-level task, the distortions that occur To better evaluate performance, we simulated reduced-resolution and full-resolution datasets. For reduced datasets, Pancollection follows Wald's protocol [35] to obtain simulated images with ground truth images. There are three steps involved: 1) Use a modulation transfer-based (MTF) filter to downsample the original PAN and MS images by a factor of 4. Downsampled PAN and MS are used as training PAN and MS images; 2) Treat the original MS image as a ground truth image, i.e. HRMSI; 3) Upsampling the training MS image using interpolation with polynomial kernels [2] and processing it into a LrMSI. When processing the full datasets, the original MS image is considered MS, the upsampled MS image is considered LrMSI, and the original PAN is considered PAN. Benchmark. To evaluate the performance of our SSDiff, we compared it with various state-of-the-art methods of Pansharpening (on WV3, QB, and GF2 datasets). Specifically, we choose three traditional methods: BDSD-PC [31] , MTF-GLP-FS [33] , BT-H [1] ; as well as nine machine learning-based methods: PNN [20] , DiCNN [12] , MSDCNN [36] , FusionNet [5] , CTINN [48] , LAGConv [15] , MMNet [49] , DCFNet [39] , and Pandiff [21] . To ensure fairness, we train DL-based methods using the same Nvidia GPU-3090 and PyTorch environment. Quality Metric. For the reduced data in Pansharpening tasks, we utilize four metrics to evaluate the results on reduced resolution datasets, including the spectral angle mapper (SAM) [46] , the erreur relative globale adimensionnelle de synthèse (ERGAS) [34] , the universal image quality index (Q2 n ) [10] , and the spatial correlation coefficient (SCC) [47] . As for full-resolution datasets, we apply D λ , D s , and hybrid quality with no reference (HQNR) indexes [3] for evaluation.We conduct experiments on the QuickBird reduced dataset and evaluate the performance of SSDiff. Similarly, the reference and non-reference metrics were obtained from 20 randomly selected test images from the QB dataset. The performance comparison is reported in Table 6 . Our SSDiff achieves SOTA performance. From the error maps in Fig. 7 , we can observe that there are still significant differences between traditional fusion methods and DL-based fusion methods.On the GaoFen-2 full-resolution dataset, we tested our SSDiff on 20 test images, Fig. 8 shows the results and HQNR maps, where an HQNR score close to 1 indicates better fusion quality of full-resolution images. The obtained results indicate that our SSDiff has a good generalization of the full-resolution dataset. The diffusion-based method generally has more running time than CNN-based methods due to the multiple timesteps of the diffusion mechanism. The comparison of inference running time shown in Table 7 ensures this point. However, for a fair comparison with another diffusion-based method for pansharpening, i.e., PanDiff, our method still gets a significant advantage.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://liangjiandeng.github.io/PanCollection.html. 


Decision: All reviewers found the proposed LoRA -based approach for pansharpening very interesting and novel. Authors presented very good responses to the concerns raised by the reviewers.

Reviews: review: paper_summary: paper_summary: This paper propose a novel spatial-spectral integrated diffusion model called SSDiff based on subspace decomposition, and its main contributions contain: 1) Considering existing DDPM-based methods have not yet designed models for the discriminative features required in the pansharpening task, SSDiff divide the network into spatial and branches based on subspace decomposition. Additionally, an alternating projection fusion module (APFM) is constructed for SSDiff to transform the process of fusing HrMSI into the fusion process of spatial and spectral components. Tests on four widely used pansharpening datasets demostrate SSDiff can achieve  state-of-the-art (SOTA) performance. 2) The frequency modulation inter-branch module is used at the junction of spectral and spatial branches to enrich extracted spatial information with more high-frequency information in the denoising process. 3) The proposed L-BAF method is used to fine-tune the network based on the proposed APFM, where the spatial and spectral branches are updated alternately. This design allows alternately fine-tuning the two branches without increasing the parameter count, enabling the learning of more discriminative features.
strength_weakness: strength_weakness: 1) Maybe expression should be pay more attention to, both grammatically and logically. For example, the sentences in line 135 and line 140 are incomplete. 

2) This paper claims the fusion result images and error maps of all these methods in Fig. 5, yet it's not the truth. And there lacks concerned descriptions about results on GaoFen-2 full dataset.

3) Paragraph from line 162 to line 167 seems not to be expressed clearly about how eq.(3) and eq.(7) can be generalized into matrix form of the self-attention mechanism as shown in Fig.3.
questions: questions: 1) Why introduce another two features from the features of spatial domain and spectral domain separately?

2) Why choose FFT as the specific way to extract the high-frequency information of the feature map obtained from the spatial branch rather than other methods?

3) The design of FMIM?
limitations: limitations: 1) This paper evaluate the effectiveness of the proposed SSDiff over pansharpening task and will extend our method to other multispectral fusion tasks.

2)  The proposed FMIM can adjust the frequency information between the two branches, but it also introduces additional hyperparameters, increasing the difficulty of fine-tuning during training.

3) The time cost for our approach is higher than other DL-based models, primarily due to the limitation imposed by the large number of sampling steps required in the diffusion model.
paper_summary: paper_summary: For the task of multispectral and panchromatic image fusion (Pansharpening), this paper proposes a spatial-spectral integrated diffusion model (SSDiff). The framework is novel, utilizing spatial and spectral branches to learn spatial details and spectral features separately. It introduces an alternating projection fusion strategy and a frequency modulation inter-branch module to enhance fusion quality. Furthermore, designing a LoRA-like fine-tuning strategy for the proposed dual-branch network to capture component-discriminating features more sufficiently. The study demonstrates promising results across multiple datasets.
strength_weakness: strength_weakness: 1. Figure 4 is not clear. It is hard to see how “ the low-frequency components undergo a gradual modulation characterized by a slow and subtle rate of change in the denoising process. In contrast, the modulation process of the high-frequency components exhibits distinct dynamic variation.”. Suggest enhancing the high-frequency image to make it more prominent.
2. To ensure the fairness of the experiment, it would be reasonable to add a control experiment. Continuing to train the 150k base model for 30k iterations using the fine-tuned model's configuration and then comparing the results to the fine-tuned model. This would provide a more comprehensive evaluation of the effectiveness of the fine-tuning strategy.
questions: questions: 1. What does the first row of data in Table 5 represent? The paper states that "the total training iterations of the WV3 datasets are set to 150k. ... and the total fine-tune training iterations are set to 30k." Is the first row's data the result after training for 150k iterations? If that is the case, please address the issue mentioned concern (see weakness 2).
2. Regarding the limitation mentioned about the efficiency disadvantage of diffusion models, can the authors provide more specific comparative results? The paper noted that diffusion models have an efficiency disadvantage compared to other super-resolution methods. To fully evaluate this limitation, it would be helpful if the authors could include more concrete performance comparisons.
3. Why do low-frequency components undergo progressive modulation while high-frequency components exhibit distinct dynamic variations? I don't quite understand this point.
4. What are the specific differences between this method and other DDPM methods in terms of training and sampling processes, e.g. Meng et al.2023.
limitations: limitations: The authors provide Limitations and Broader Impact, where limitations are influenced by the nature of the task itself.
paper_summary: paper_summary: This paper introduces SSDiff, a model that improves the resolution of satellite imagery by combining high-resolution panchromatic images and low-resolution multispectral images. SSDiff separates image details into spatial and spectral components, and processes them through different network branches, then merges them back together. It utilizes a denoising diffusion model and adopts a designed Alternating Projection Fusion Module (APFM) to complete the spatial-spectral fusion. Additionally, they propose a Frequency Modulation Inter-branch Module (FMIM) to modulate the frequency distribution between branches, as well as a LoRA-like branch-alternating fine-tuning method to further improve performance. The paper provides a solid theoretical foundation, and through experiments on various datasets, SSDiff demonstrates the ability to generate high-resolution multispectral images with minimal spatial and spectral distortion.
strength_weakness: strength_weakness: - Efficiency of the method: Evaluating and discussing the efficiency of the DDPM-based method used in SSDiff would help provide a more comprehensive understanding of the practical advantages and limitations of the proposed approach. This information could be valuable for readers to assess the real-world applicability and deployment feasibility of the SSDiff model.
- Experimental Section: The proposed method in this paper has been extensively evaluated for its generalization performance, but the experiments are limited to the Pansharpening task. It would be better if there were experiments on other datasets, but it's not essential.
questions: questions: 1. How does the element-wise multiplication operation help preserve the complementary information from the two domains?
2. Could the author elaborate on any challenges faced when integrating information from PAN and LrMSI into a diffusion model? This clarification would help in understanding your methodology.
limitations: limitations: The authors have provided reasonable Limitations and Broader Impact.

Comments: comment: Dear reviewers and AC,
﻿
We want to thank you for your insightful and inspiring feedback. We are very pleased that you consider our work interesting and innovative. Your feedback is incredibly valuable to us, and we are fortunate to have such diligent reviewers contributing to our work. We have addressed each reviewer's comments and questions one by one, and we have also updated some figures and tables. 
If we fail to address all the reviewers’ concerns, we sincerely request the reviewers to raise your concerns directly and we will respond as soon as possible.
﻿
Best regards,
Authors
Dear AC,
﻿
Thank you for your ongoing help and support. We have tried our best to address all concerns of the reviewers. If any new concerns arise, could you please ask the reviewers to inform us?
﻿
Best regards,
﻿
Authors
Dear Reviewers,
Authors have carefully prepared their rebuttal addressing the concerns you have raised. Please check the rebuttals and join the discussion about the paper. 
Thanks,
Your AC