SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening
Authors: Yu Zhong
Xiao Wu
Liang-Jian Deng
Zihan Cao
Hong-Xia Dou
Venue: NeurIPS 2024 poster (2024)
Keywords: denoising diffusion model
LoRA
Multi-source image fusion
Pansharpening
Primary area: machine_vision

Abstract: Pansharpening is a significant image fusion technique that merges the spatial content and spectral characteristics of remote sensing images to generate high-resolution multispectral images. Recently, denoising diffusion probabilistic models have been gradually applied to visual tasks, enhancing controllable image generation through low-rank adaptation (LoRA). In this paper, we introduce a spatial-spectral integrated diffusion model for the remote sensing pansharpening task, called SSDiff, which considers the pansharpening process as the fusion process of spatial and spectral components from the perspective of subspace decomposition. Specifically, SSDiff utilizes spatial and spectral branches to learn spatial details and spectral features separately, then employs a designed alternating projection fusion module (APFM) to accomplish the fusion. Furthermore, we propose a frequency modulation inter-branch module (FMIM) to modulate the frequency distribution between branches. The two components of SSDiff can perform favorably against the APFM when utilizing a LoRA-like branch-wise alternative fine-tuning method. It refines SSDiff to capture component-discriminating features more sufficiently. Finally, extensive experiments on four commonly used datasets, i.e., WorldView-3, WorldView-2, GaoFen-2, and QuickBird, demonstrate the superiority of SSDiff both visually and quantitatively. The code is available at https://github.com/Z-ypnos/SSdiff_main.

SSDiff: Spatial-spectral Integrated Diffusion Model for Remote Sensing Pansharpening
Yu Zhong?
Xiao Wu University of Electronic Science and Technology of China vxwsx19978gmait com
University of Electronic Science and Technology of China yuuzhong 1@gmai1..... com
Zihan Cao
Hong-Xia Dou Xihua University hongzladou11991012..co
University of Electronic Science and Technology of China i 1amzthan6660gmai. com
Liang-Jian Deng*
University of Electronic Science and Technology of China 1 liangj ian. deng@uestc. edu. cn
Abstract
Pansharpening a significant image fusion technique that merges the spatial content and spectral characteristies of remote sensing images to generate high- resolution multispectral images. Recently, denoising diffusion probabilistic models have been gradually applied to visual tasks, enhancing controllable image gener- ation through low-rank adaptation (LoRA). In this paper, we introduce a spatial- spectral integrated diffusion model for the remote sensing pansharpening task, called SSDiff, which considers the pansharpening process as the fusion process of spatial and spectral components from the perspective of subspace decomposition. Specifically, SSDiff utilizes spatial and spectral branches to learn spatial details and spectral features separately, then employs designed alternating projection fusion module (APFM) to accomplish the fusion. Furthermore, we propose a frequency modulation inter-branch module (FMIM) to modulate the frequency distribution between branches. The two components of SSDiff can perform fa- vorably against the APFM when utilizing a LoRA-like branch- wise alternative fine-tuning method. It refines SSDiff to capture component-diseee digppiienntddscrimiaainnn fea- tures more sufficiently. Finally, extensive experiments on four commonly used datasets, i.e., VorldView-3, World GaoFen-2, and QuickBird, demonstrate the superiority of SSDiff both visually and quantitatively. The code is available at htttss//gthub.com//yyypo//SSSS/t main
Introduction
Due to physical limitations, satellite sensors cannot directly acquire high-resolution multispectral images (HrMSI). Instead, they can obtain high-resolution panchromatic (PAN) images and low- resolution multispectral images (LrMSI). Pansharpening techniques can merge PAN images with LrMSI, generating HrMSI that possess both high spatial and spectral resolutions. Pansharpening,
'Equal contribution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
(b)
(c)
PAN
PAN
Network
Wo
AW
HrMSI
LAMSI
LrMSI 4x
Figure Schematic of (a) DL-based pansharpening approach in a supervised fashion, in which the "network" can be any deep module, e.g., denoising diffusion probabilistic models (DDPM), The comparison of (b) the LoRA based on DDPM and (c) the proposed APFM in our SSDiff. G and U represent the spectral and spatial domains, respectively. The LoRA can expand learnable weights Wo with AW (but without applications to pansharpening), and the given APFM can obtain pansharpened HrMSI from PAN image and LrMSI through alternating projections.
as fundamental preprocessing method, has been widely utilized in various applications, including change detection and image segmentation [45]
Pansharpening methods are roughly categorized into four types: component substitution (CS) methods, multi-resolution analysis (MRA) methods, variational optimization (VO) techniques, and deep learning (DL) methods, as shown in Fig. (a). The CS method 17 22 involves projecting the LiMSI into specific domain and replacing the spatial components of the LrMSI in that domain with the corresponding components from the PAN image. CS-based methods can generate fusion images with high spatial fidelity but spectral distortion with fast runtime. MRA-base methods [24 32 extract spatial details from the PAN image through multiscale decomposition and inject them into the LrMSI. While MRA-based methods effectively preserve spectral information, they may sacrifice spatial details. Compared to CS-based and MRA-based methods, VO-based techniques [40 41 gain more mathematical guarantees but require handling a higher computational burden and more adjustable parameters.
have increasingly been applied across various fields. including pansharpening tasks, yielding exciting results. Traditional DL-based methods typically utilize a single-scale model to process information from PAN images and LrMSI. single-scale network, PAN images and LrMSI are usually stacked together without distinguishing the information contained within them, serving as inputs to the network. They overlook the dispar- ities in the deep-level information inherent in both, potentially leading to the omission of crucial diseriminative features and subsequently influencing lower fusion performance. Then, branch methods 66 18 based on deep learning can differentiate and hierarchically learn information from PAN and LIMSI. Thanks to this design, it has shown outstanding performance in pansharpening tasks. However, the cumbersome structure of dual-branch networks makes it challenging to perform localized fine-tuning. Denoising diffusion probabilistic model (DDPM) [13 is attaining attention in remote sensing pansharpening [21 Unfortunately, existing DDPM-basee methods have not yet designed models for the discriminative features required in the pansharpening task.
In recent years, DL-based methods 12 39
38
Considering the characteristics of the pansharpening task, we propose novel SSDiff method based on subspace decomposition, which leverages spatial and spectral branches to discriminatively capture global spatial information and spectral features, respectively. Additionally, we further construct an alternating projection fusion module (APFM) fuse the captured spatial and spectral components. Besides, frequency modulation inter- branch module (FMIM) is designed to overcome the problem of uneven distribution of frequency information between two branches in denoising process. Finally, through the proposed LoRA-like branch-wiss alternating fine-tuning (L-BAF), our SSDiff can further reveal spatial and spectral information not discovered in each branch. The main contributions of this work as follows:

---

Our SSDiff is based on subspace decomposition to divide the network into spatial and spectral branches. In addition, for subspace decomposition, we give an illustration of vector projection and construct alternating projection fusion module (APFM). APFM transforms the process of fusing HrMSI into the fusion process of spatial and spectral components. Moreover, our SSDiff is tested on four widely used pansharpening datasets and achieves state-of-the-art (SOTA) performance.
The frequency modulation inter-brance module is used at the junction of spectral and spatial branches to enrich extracted spatial information with more high-frequency information in the denoising process.
The proposed L.-BFF method is used to fine-tune the network based on the proposed APFM, where the spatial and spectral branches are updated alternately. This design allows us to alternately fine-tune the two branches without increasing the parameter count, enabling the learning of more discriminative features.
Related Works
DL-based Methods. As a simple but effective method, the representative single-scale coupling model, namely PNN (20), first proposes simple and effective three- layer CNN architecture and achieves the best results at that time. Subsequently, other methods such as Fu- sionNet [5], DCFNet [3)], and others adopt similar coupled input approaches and successfully design their networks. However, these methods still have significant room for improvement in spectral fidelity and general- ization performance due to weak feature representation in their network structure designs. In multi-source im- age fusion tasks, images acquired from diverse sources exhibit varying characteristics. Coupling two infor- mation sources together may suffer from inadequate feature extraction. Then, spatial and spectral branches methods [6] [18 (25 based on deep learning can differ- entiate and hierarchically learn information from PAN images and LrMSI. These methods can better exploit the potential advantages of multi-scale information.
Subspace Decomposition
X
D
X
C
Projection
Self-attention Mechanism
X
K
X
V
f(
Figure 2: Schematic diagram of the rela- tionship between subspace decomposition and self-attention mechanism. f(Q, K) is the classic self-similarity equation in self- attention mechanism.
Diffusion-basse based Model. DDPM, as a generative model, has been widely applied in various domains such as text-to-image generation 266,, image editing [16] and image classification 43 In recent years, DDPM has shown its prominence in image processing tasks 99, 28 44 Among them, Song al. 28 propose denoising diffusion implicit models (DDIM), where they design non-Markov chain sampling process, accelerating the sampling of diffusion models. Then, through some simple modifications, IDDPM 23 achieves competitive log-lik likelihoodd while preserving the high sample quality of DDPM. Currently, DDPM is attracting attention in the field of pansharpening [1 These DDPM-based methods treat PAN and LrMSI model fusion conditions, unlike other pansharpening methods where they serve as fusion targets.
LoRA: Low-rank Adaptation of Large Language Models. For the fine-tuning of parameters in large pre-trained models, Hu et al. [4] introduce the LoRA to freeze the pre-trained model weights and inject trainable low-rank decomposition matrices into each layer of the Transformer architecture. This significantly reduces the number of trainable parameters for downstream tasks. For H = WoX, the modified forward pass of the LoRA follows the formula:
H W.X + AWX = W.X + BAX.
(1)
where W. E BBdk a pre-trained weight matrix, E Rkxn B Rdxr, E Br and the rank min(d, dd, k).. Actually, W. + AW W. BA represents a low-rank decomposition. Motivation. PAN images and LrMSI are obtained from different sensors and contain distinct feature information. PAN images exhibit richer spatial details, while LrMSI possesses more abundant spectral information. However, existing DDPM- methods have not yet designed models specifically for the discriminative features required in the pansharpening task. As result, these methods suffer from
3

---

Subtractio
Addition
Concal
Spectral
Mutipicationn
Layer
Fully Conneceed
Reshape
LrMSI
Resnet
FMIM
Down-sampling
Upsampinng
Xa
Spectral Flow
Spanal Flow
MLP
Xr
Spatial projection
T.T.
Softmax
HrMSI
C
Fap
v
Spectral projection
T
PAN
T.T.
Fape
Sollanx
Spatial Branch
201
APFM
Figure 3: Overall framework the proposed SSDiff. c = /1 āyc is Gaussian noise, where is the time step. Fepa is the ouiput of the spatial branch, and Fape is the output of the spectral branch. The process of APFM follows Theorem
issues such as insufficient feature learning and generalization capabilities, though they may employ low-rank adaptation (LoRA) to improve the performance of DDPM for the pansharpening task.
To alleviate these problems, we propose SSDiff, which transforms the problem of solving HrMSI into a fusion problem of spatial and spectral components. Significantly, we give an illustration of linear algebra to remove the gap between subspace decomposition and the self-attention mechanism. The SSDiff utilizes vector projection to discriminatively capture global spatial information and spectral features in spatial and spectral branches. By introducing subspace decomposition, we can further illustrate and generalize the vector projection to the matrix form. Based on this, we propose an APFM that naturally decouples spatial and spectral information and fuses the captured features. Unlike the LoRA method, the APFM can establish low-rank representations for the spatial-spectral branches more accurately, shown in Fig. Furthermore, the spectral branch contains abundant low-frequency information. When low- and high-frequened information from the spatial branch is injected into the spectral branch, it may result in an overemphasis on low-frequency information, impacting the denoising performance the model. Based on this, we propose FMIM to modulate the frequency information between different branches. The overall model is trained by L-BAF to uncover spatial and spectral information not discovered in each branch.
3 Methodology
3.1 SSDiff Architecture
Inspired by the LoRA approach, we view the pansharpening task as the fusion of spatial and spectral components, where the spatial and spectral elements can be considered as a matrix decomposition of multi-spectral image. Based on these characteristics, our SSDiff employs model comprising a spatial branch and spectral branch, as shown in Fig. 3 Both the spatial branch and the spectral branch comprise two encoder layers and two decoder layers. Down-sampling occurs between the encoder layers to decrease the spatial resolution while increasing the number of channels. Up-sampling operation between the two layers of the decoder increases the spatial resolution while decreasing the number of channels, and a down-sampling convolution layer connects the encoder and the decoder. The spatial branch employs ResNet [1] blocks to convert spatial images into features. These spatial features are transmitted to corresponding layers in the spectral branch via a frequency modulation inter-branch module. Additionally, fusing incoming spatial features and spectral information via an alternating projection fusion module.
Eventually, is delivered to the next stage via an MLP. In this work, we convert the objective from to To. so the loss function Leimple [4] takes the following form:
Csimple
E [||2
Te xx c, t)]l
(2)

---

where denotes the prediction of the model and is the conditions for injecting the model. Inspired by FusionNet [5], our SSDiff changes the forward and backward denoising objects during the training process from HrMSI to the difference between HrMSI and up-sampled LrMSI. The detailed training
process of SSDiff can be found in Appendix, [
Alternating Projection Fusion
This section starts with vector projection in linear algebra (See Lemma] 1 and subspace decomposition (See Definition [ to illustrate vector projection as a specific subspace decomposition, Then, the self-
attention mechanism [30] is generalized into the proposed alternating projection fusion framework, i.e. APFM. In what follows, we rewrite vector projection follows.
Lemma 1 (29). Assuming that the existing two arbitrary vectors a domU E R" and b domG E R", then Pb = Xa P. we have the following formula:
aa
b.
(3)
a a
where P projection matrix, X is the scaling factor, and p is the vector in the same domain
Proof. For any two vectors and b, there exists vector e = pp - such that is orthogonal to We have the following equation:
a² = a² a'(p - b) = a² (Aa - b) 0,
(4)
thus, we have
a =
(5)
a a
Taking Eq. 5
into Pb = λa = p. we have the conclusion:
aa² a²'a
p Pb
al
b.
(6)
Definition ((7). Assume that D E Rx5 is the subspace and C E LRLWHW is the corresponding
coefficients. We have:
Z = DC.
Based on subspace decomposition, we can take spatial and spectral components to accomplish the pansharpening of remote sensing images. According to Lemma [ we can further determine specific subspace decomposition in Definition T The subspace represents the projection relationship between vectors a and b. Interestingly, we find that we can generalize Eq. (3) and Eq. (7) to the matrix form of the self-attention mechanism, as shown in Fig. 2 In other words, the self-attention mechanism is represented as vector projection of Eq. (3). which is the low-rank subspace decomposition.
Remark 1. Back to the pansharpening applications, the input PAN and LrMSI are mapped high- dimenssinn feature space. The features often exhibit significant correlations between frequency bunds, while spectral vectors typically reside low-dimension subspace. These features can be represented as Eq. (7). In this way, the characteristics in the image domain can be transformed into the subspace.
Based the above analysis, can build an alternating projection framework, which is summarized in the following theorem.
Theorem 1, Assuming that Fspa x and Fspc E HRH from the spatial and spectral branches, they can be alternatively projected as follows:
T.
T.TZ d Vis HW
Tpp Soffmax
T
Tae Softmax
T
(8)
5

---

Generated image
Low frequency
High frequency
Denoising Process
Figure 4: The denoising process. The top consists of series of iteratively generated images from the gradual denoising process. The subsequent two rows represent the associated fryerreuenccy and figh-frequency spatial domain information obtained through inverse Fourier transform from the denoised image in the first row of each corresponding step.
where Tope E HWS' and Tope E R XHW denote the features of spatial domain and spectral domain separately. Ta E xs' and T, E RHH xS' are features in the spatial domain generated by Fepa- where S' is the channel self-attention Te EgE ERS'' &'W and Ta E RS xHW are features
the spectral domain generated by F Fspe- VS' and VSS Softmax(-) ) stands for the Softmax function.
HW are constants related to the matrix size.
Proof. According to Lemma [ we can generalize the self-attention mechanism 330 where Q and K are the features from d, and V is the feature from dom G, respectively. Thus, we have:
QK
aa
T
Softmax(
V b,
(9)
aaaa a
where "=" suggests that the left and right parts of the equation are similar in form, and the left side is a special case of the right side. Through this equation, we can partially explain the principle of cross-attention using the vector projection theorem. Then we transform the projection relationship between the spatial domain (domU) and spectral domain (domG), where Ta, T6 E domU and Te, Ta E domG. is self-attention constant. result, the alternating projection is complete from the spatial/spectral domain to the spectral/spatial domain, i.e., Eq. 8
In addition, we need to get fused outputs from Tapa and Tape Without loss of generality, we have
fus T Tee
(10)
where @ defines element wise multiplication. Element- wise multiplication is used to fuse spatial and spectral information to obtain Tus HW"'
Comparing Eq. (3) with Eq. (8) this subspace is built by vector projection and naturally decouples spatial and spectral information into the self-attention mechanism. This inspires us to apply a fine-tuning method similar to LoRA methods (See details in Sect. 3.4).
3.3 Frequency Modulation Inter-branch Module
Through the APFM, we build an effective fusion module from the characteristics images. Interest- ingly, there are some differences between the spatial and spectral components. The spectral branch contains abundant cow-frequency information. When low- and eigh-frequency information from the spatial branch is injected into the spectral branch, it may result in overemphasis frequeeyynccy information, impacting the denoising performance of the model. We found that modulating the frequency distribution contributes to SSDiff obtaining better fusion results.
As shown in Fig. the low-frequency components undergo a gradual modulation characterized by slow and subtle rate of change in the denoising process. In contrast, the modulation process of the

---

SON
05
4.6
2.5
SSCIf
LAGConV
Figure 5: Visual comparisons a reduced- resouuiiresolttion WorldView-3 and GaoFen-2 case. The first two rows are the results of WV3, and the last two rows are the results GF2. The first and third rows are the predicted HrMSI for each method, and the second and fourth rows are the eтror maps of the predicted HRMS versus ground truth (GT) for each one.
high-frequency components exhibits distinct dynamic variation. Considering the above phenomenon, we design a frequency modulation inter-branch module. Specifically, we utilize a Fourier filter to extract the high-frequency information of the feature map Xspa obtained from the spatial branch,
following:
F F'(xpa)) = FFT (pspa) 2 a Xapa IFFFFFFFFFF (F(xaa) (Xapa))
(11)
(12)
where FFT and IFFT are Fourier transform and inverse Fourier transform. O denotes element-wiss multiplication, and a is a Fourier mask injecting high-frequency information into spectral branches will cause the frequency information imbalance. To solve this problem, half of the channels of feature Xsee of the spectral branch are multiplied by a constant. the three different channel numbers in the model, ranging from low to high, we set this constant to 1.2, 1.4, and 1.6, respectively.
(27). Furthermore, we find in experiments that directly
3.4 LoRA-like Branch-wise Alternative Fine-tuning
During the model training process, is crucial to carefully maintain a balance between model underfitting and overfitting. Our SSDiff can hierarchically and discriminatively extract more features. However, achieving a simultaneous balance condition for both spatial and spectral branches during unified training undoubtedly challenging. Therefore, this paper approaches the LoRA-like alternate fine-tuning of each branch as a feasible solution. As shown in Fig. I LoRA methods fine-tune the output of the model by updating the parameters on the fully connected layer weights. Compared with LoRA, the proposed alternating projection method is also a low-rank matrix decomposition. In Fig. (b), the difference that we can have the backpropagation process and control the gradient of the projection process to achieve alternate fine-tuning in the proposed APFM. In practice, taking the fine-tuning of Eq. (8) (T*pa) as an example to update the spectral branch, we can detach the gradient propagation at T preventing parameter updates in the spatial branch. In this case, gradients only propagate through the path shown Eq. ( (T* which means only the parameters of the spectral branch are updated. Similarly, when fine-tuning the spatial branch, detaching the gradient propagation from the computation graph at Tope can prevent parameter updates in the spectral branch. more intuitive fine-tuning process is provided in the supplementary material.
7

---

Table 1: Result on the WV3 (the first thirteen rows) and GF2 (the last thirteen rows) reduced- resolution and full-resolution datasets. The best results are highlighted in bold and the second best results are underlined.
Method
Reduced resolution
Full resolution
SAM(+ std)
ERGASC+ std)
Q2"(+ std)
SCC(+ std)
PaC+ std)
D, std)
HQNR(+ std)
BDSD
31
5.4675
4.6549 1.4667
0811710.
1063 0.9049
0.0625
0.0730.10.0356
0.8698.10.057
MIF-OLP-PS 33
5.3233=1.6548 4.6452=14441
0.817720.1014
0.896410.0466
0.020611.0082
0.0630::0.0284
0.918020.0346
BT-H
4.8985-1.3028 4.5150=1.3315
0.8182+0. 1019
0.2200+0.0233
0.0574+0.0252
0.0810+0.0374
0.8670++0.050
PNN
3.6798=0:7625 2.6819=0.4455
0.8929=0.0023
0.976++0.007
0.0213++000000
0.0428::000147
0.9369=000212
DICNN 12
3.5929_00.6633 2.6733._0.6627
0.9004.00.0671
0.9763-10.007
0.03621.000111
0.0462110.015
0,9195110.0288
MSDCNN
36
3.7773-0.8022
0.8900=0000900
0.9741 +0.0066
0.0230++000091
0.0467++.0119
0.9316++0.0271
FusionNet
5
3.3252=0.6978 2.4666=0.6446
0.9044:0.0904
0.9807 00.9
0.0239:10.000)
0.0364:1.01137
0,9406:0.0197
CTINN 48
3.2523=0.6436 2.3936=0.5194
0,9056=0.0840
0.9826+000046
0.0550++0.0288
0.0679+0.0312
0.8815=+00488
LAGConY
3.1042=0.5585 2.2999=0.6128
0,9098=0.0007
0.9838+0.0008
0.0368++001148
0.0418+000152
0.9230+0.0247
MMNet 49]
3.0844-0.6398
2.3428
0.6266
0.9155:-0.0855
0.9829
0.0540++0.0232
0.0336-+0.01
0,9143-+0.0281
DCFNer 19
3.0264-0.7397
2.1588-0.4563
0.9051+000888
0.9861 +0.0088
0.0781
0.0508++0.0322 0,8771-00, 100S
PunDill 21
0.2960
2.4667
0.896010.0880
0.98001100006
0.0273:10.0123
0.0542.10.026 0.9203-10.0360
SSDif (ours)
2.8429 .0.5844
2,1059
1.1059_0.4560 _004560
0.915610.084
0.9867 10.003
0.0032.0.0049
0.0307.10.0029
0.9565.100055
BDSD-PC 31
1.7110=0.3210 1.7025=0:4056
0.9932::0.0088
0,94:8:200116
0.0759::0:030
0.1548::0.0280
0.7812:+00440
MTF-GIPP.FS 33
1.6757-0.3477 1.6023-0.3555
0.8914-0.0256
0.9390+0.0199
0.0336+0.0129
0.1404+0.0027
0.8309+0000334
BT-H
1.6110-0.3168 1.5524-0.3642
0.9089=000292
0.9506++00115
0.0602++000222
0.131340.0193
0.8165+000300
PNN
1.0477=0:2264 1.0572-002355
0.96001000110
0.9772.00.0054
0.02671100291
0.0943.10.0224
0.8726610.0373
DICNN
1.0525_0.2310 1.0812-.0.2500
0.959410.0100
0,9771 1000058
0.041311..0188
0.099211001131
0.8636.10.0165
MSDCNN
36
1,0472=0.2100 1.0413=0.2309
0.9612=000108
0,97
0.0269::001311
0.0730::0.0093
0.9020:10.0128
NusionNet
1
0.9735-022117 0.9878-0.2222
0.9641+000093
0.9806+0.0049
0.0400++0.0126
0.1013+0.0134
0.8628+0001884
CTINN 43
0.8251=0.1386 0.6995-0,100 1068
0.9772=0.0017
0.9803-+0.0015
0.0586+0.0260
0. 0.1096:t.0119
0.8381=000237
LAGComy
15
0.159900.1478
0.6869-0.1122
0.9804-0..088
0.9906.0.0019
0.002410.0130
0.0792.10.0136
0.8910.10.0204
MMNet 49
0,9929 0.1411 0.8117-0.1185
0.96901:0.0204
0.985910.002
0.0428
0.105311.0019 0.8583-0.0269
DCHNet 39
0.887=00.1577 0.8661=0.1369
0.9727=0.0100
0.9853-200002
0.023410.0011
0.0659:10.00%
0,9122:1.01199
PanDiff
21
1.881-001199 17461-0.1032
0,9792=0.0099 0097
0.9887+000022
0.0 0.02655+0.0195
0.0729+000100
0.9025+00.0209
SSDiff (ours)
0.669400.124
0.6038-0.1080
0.9836=0.0074
0.9915--0.017
0,0164+0.0093
0.0267=000071
0,9573++00100
Ideal value
1
4 Experiments
4.1
Experimental Results
Results on WorldView-3, On the WorldView-3 dataset, we evaluate the performance of SSDiff using 20 test images. The results for both reduced-resolution and full-resolution are presented in Table [ The running time of a single picture during the sampling process is 7.417 seconds under 10 timesteps. We compared our method with three traditional methods and some SOTA DL-based methods. To illustrate the performance of each method more clearly, we presented the fusion result images and error maps of some of these methods in Fig. [] and zoomed in on a specific location. On average, our method achieves SOTA performance on the reduced dataset, with our SSDiff reaching 2.84 (SAM) and 2.10 (ERGAS) metrics, outperforming all DL-based methods. The error map indicates that the images sampled by SSDiff are closer to the ground truth (GT). SSDiff achieves SOTA performance in obtaining full-resolution images on the WV3 dataset. The HQNR score close to indicates better fusion quality of the full-resolution images. The obtained results demonstrate that our SSDiff can fuse HrMSI, reducing spatial and spectral distortions, thereby proving its excellent generalization ability at full resolution.
Results on GaoFen-2. On the GaoFen-2 reduced dataset, we tested our SSDiff on 20 test images, as shown in Table [ observe that there are still significant differences between traditional fusion methods and DL-based fusion methods. These experiments demonstrate that, compared to other DL-based methods used for comparison in the experiments. the proposed SSDiff exhibits superior spatial performance and effectively preserves spectral information.
Our SSDiff achieves SOTA performance. From the error maps in Fig. 5 we can
4.2 Ablation Study
Effectiveness of Decoupling Branches. To investigate the effectiveness of the spatial and spectral branch design, we perform ablation experiments by training the diffusion model under the following
8

---

conditions: V1) Coupling inputs, and training only with the spatial branch. V2) Coupling inputs and training only with the spectral branch. V3) Spatial and spectral branch structure and decoupling inputs, only the outputs of the two branches are concatenated without any inter-branch information interaction. V4) Spatial and spectral branch structure and decoupling inputs, replace each APFM with an additional operation, i.e., the way of LoRA. V5) Spatial and spectral branch structure and decoupling imputs, replace each APFM with a multiplication operation. V6) Our method. The results on the WV3 reduced dataset are reported in Table Training solely with single branch significantly reduces the quality the HrMSI. It can be seen that SSDiff is more suitable for pansharpening tasks than the LoRA way, multiplication, and concatenating way.
Table 2: Ablation study on 20 reduced-ressoluiion samples from WV3 dataset without fine-tuning.
Method|
SAM(+ std)
ERGAS(+ std)
Q2" std)
SCC(+ std)
Params
V1
3.3612++0:697 2.5633++06629
0.8960+-.1033
0.9816+0000000
1100K
V2
3.0598==0.5560 2.2638=+0.6663
0.9097#0.0947 0.9847=0.0062
654K
V3
3.4040==066888 2.5564+1.6737 0.9023=000954
0.9796+000008
1420K
V4
3.3245-105342 2.4871=0.584]
0.90891000878 0.9830=10.0069
1420K
V5
3.1958=1057727 2.3962:10.5688 0,90351100962 0.983410.0067
1420K
V6
2.8646.10.5241 2.121710.4671 0.0125 0.986311..0400
1420K
Frequency Modulation Inter-branch Module. To validate the effectiveness of the FMIM, we remove FMIM from our model and train the diffusion model to converge the WV3 dataset. The results are shown in 3 Without using FMIM, the model's 's performance on the SAM/ERGAS/Q8 indicators decreased by approximately 3. 11%22% %, respectively. This demonstrates that utilizing FMIM for frequency transfer can effectively improve model performance.
Table 3: Ablation study on 20 reduced-resolltiion samples from WV3 dataset without fine-tuning.
FMIM SAM(L std)
ERGAS(1 std)
Q2" std)
SCC(1 std)
2.9798=0.608 2.1978=0.5399 0.9153+0.0868 0.9855+00.003 2.8646=0.5241 2.1217=0.4671 0.9125=100874 0.9863+100000
4.3 Discussion
Generalization. To test the generalization ability of DL-based methods, we evaluated models trained on the WV3 dataset using 20 reduced resolutions from the WorldView-2 dataset. The quantitative evaluation results, as reported in Table 4 demonstrate that the SSDiff method achieves the best results across all four evaluation metrics. This indicates that our approach possesses a powerful generalization ability.
Table 4: Generalization of DL-bbsed methods on WV2 dataset.
Method
SAM(+ std)
ERGAS(+ std)
Q2" std)
SCC(+ std)
PNN
7.1158+1.6811 5.6152-+0.9431
0.7619::00928 0.8782:t0.0175
DiCNN
6.9216=10.7998 6.2507+0.5745 0.7205=t0.0766 0.8552++0.089
MSDCNN
6.0064+0.6377 4.7438+t0.9939 0.824110.0799 0.8972+100109
FusionNet
6.4257 8602 5.1363:10.5151 0.00617 0.8746110.0134
CTINN
6.4103:10.5553 4.6435:0.3792 0.8172+40.083 0.9147=1.0102
LAGConv
6.9545-10:4739 5.3262-10.3185 0.8054110.8377 0.9125=10:0101
MMNet
6..309 5.2213++0.2133 0.8143++0.0700 0.9136+00.001
DCFNet
5.619440.6039 4.488710.3764 0.829240.0815 0.915410.0083
SSDiff (ours)
5.064710.5634 3.988510.4297 0.857710.0782 0.9335-0.0055
Training SSDiff. To address the issue of insufficient local parameter training in the dual branches model, we design the L-BAF method to fine-tune the model. Taking the experiments on the reduced WV3 dataset, we first train the model without fine-tuning until convergence and perform wranch-wise fine-tuning, which includes: 1) Full parameter fine-tuning without I.-BAF. 2) Only fine-tuning the spatial branch parameters with the spectral branch parameters fixed. 3) Only fine-tuning the spectral

---

branch parameters with the spatial parameters fixed. 4) Alternating fine-tuning spectral branch and spatial branch. The quantitative results are shown in Table[5] and all three branch-wise fine-tuning methods lead to improved testing performance. The results of full fine-tuning show decrease in some metrics, indicating that the method suffers from overfitting. The alternating fine-tuning approach showed the most significant performance improvement. This demonstrates the effectiveness of the L-BAF
Table 5: Fine-tune SSDiff. S and FF denote fine-tuning of spatial and spectral branches, respectively.
s
F
SAM(L std)
ERGAS(L std)
Q2" std)
SCC std)
2.8646++0.524)
2.1217=#04677
0.9125=000874
0.9863=000000 0.040
x
x
2.8681+0.588
2.1302==0.5355
0.9206=0.0850
0.9868+0.0048
2.8545+0.5244
2.1138=004658
0.9143=0.0857
0.9864
V
2.8460:10.5232
2.1132::0:4671
0.9152:10.0849
0.9864:10.0041
/
V
2.8429-10.5284 2.1559:104460
0.9156=+0.084
0.9867110.0038
Conclusion
In this paper, we propose spatial-spectral integrated diffusion model for remote sensing pansharp- ening, named SSDiff. We design spatial-spectral integrated model architecture, which utilizes spatial and spectral branches to learn spatial details and spectral features separately. By introducing vector projection, the spatial and spectral components in the subspace decomposition are further specified in the proposed APFM. Then. the self-attention mechanism is naturally generalized to the APFM. Furthermore, we propose an FMIM to modulate the frequency distribution between branches. Finally, the two branches of SSDiff capture discriminating features. It is interesting that, when utilizing the proposed L--AF method in the APFM, the two branches can be updated alternately, and then SSDiff produces more satisfactory results. We compare SSDiff with several SOTA pansharpening methods on the WorldView-3, QuickBird, GaoFen-2, and WorldView-2 datasets. The results demonstrate the superiority of SSDiff both visually and quantitatively.
6 Acknowledgement
This work is supported by the National Natural Science Foundation of China under Grants 12271083, 1217 1072 and Natural Science Foundation of Sichuan Province under Grants 2023NSFSC1341.
References
[1] Bruno Aiazzi, I. Alparone, Stefano Baronti, Andrea Garzelli, and Massimo Selva. Mtf-tailored multiscale fusion of high-resolution ms and pan imagery. Photogrammetric Engineering & Remote Sensing, 72(5):591-596, 2006.
[2] Bruno Aiazzi, Luciano Alparone, Stefano Baronti, and Andrea Garzelli. Context-driven fusion of high spatial and spectral resolution images based on oversampled multiresolution analysis. IEEE Transactions on Geoscience and Remole Sensing, 40(100)2300-23132 2002.
[3] Alberto Arienzo, Gemine Vivone, Andrea Garzelli, Luciano Alparone, and Jocelyn Chanussot. Full-resolution quality assessment of pansharpening: Theoretical and hands-on approaches. IEEE Geoscience and Remote Sensing Magazine, 10(3): 68-201, 2022.
[4] Zihan Cao, Shiqi Cao, Liang-Jinn Jian Deng, Xiao Wu, Junming Hou, and Gemine Vivone. Diffusion model with disentangled modulations for sharpening multispectral and hyperspectral images. Information Fusion, 104:102155 11 2023.
[5] Liang-Jian Deng, Gemine Vivone, Cheng Jin, and Jocelyn Chanussot, Detail injection-based deep convolutional neural networks for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, 5981)6695-700 2020.
[6] Shangqi Deng, Liang-Jian Deng, Xiao Wu, Ran Ran, and Rui Wen. Bidirectional dilation transformer for multispectral and hyperspectral image fusion. In Proc. 32nd Int. Joint Conf. Artif. Intell., pages 3633-3641,
10

---

[7] Renwei Dian and Shutao Li. Hyperspectral image super-resolution via subspace-based low tensor multi-rank regularization. IEEE Transactions on Image Processing, 28(10):515--514 2019.
[8] Junkai Fan, Jiangwei Weng, Kun Wang, Yijun Yang. Jianjun Qian, Jun Li, and Jian Yang. Driving-video dehazing with non-aligned regularization for safety assistance. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 26109-261 19, 2024.
[9] Sicheng Gao, Xuhui Liu, Bohan Zeng, Sheng Xu, Yanjing Li, Xiaoyan Luo, Jianzhuang Liu, Xiantong Zhen, and Baochang Zhang. Implicit diffusion models for continuous super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10021-10030, 2023.
[10] Andrea Garzelli and Filippo Nencini. Hypercomplex quality assessment of multi/hyperspectral images. IEEE Geoscience and Remole Sensing Letters, 6(4):662-665, 2009.
[1I] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pultern recognition, pages 770-778, 2016.
[12] Lin He, Yizhou Rao, Jun Li, Jocelyn Chanussot, Antonio Plaza, Jiawei Zhu, and Bo Li. Pansharpening via detail injection based convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, ((()):188-122) 2019.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[14] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09662 2021.
[15] Zi-Rong Jin, Tian-Jing Zhang, Tai-Xiang Jiang, Gemine Vivone, and Liang-Jian Deng. Lagconv: Local-context adaptive convolution kernels with global harmonic bias for pansharpening. In Proceedings of the AAAI Conference Artificial Intelligence, volume 36, pages 1113-1121, 2022.
[16] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6007-6017, 2023.
[17] P Kwarteng and A Chavez. Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis. Photogramm. Eng. Remote Sens, 55(1):333--345 1989.
[18] Yixun Liang, Ping Zhang, Yang Mei, and Tingqi Wang. Pmacnet: Parallel multiscale attention constraint network for pan-sharpening. IEEE Geoscience and Remote Sensing Letters, 19:1-5, 2022.
[19] Yu-Jie Liang, Zihan Cao, Liang-Jian Deng, and Xiao Wu. Fourier-enhanced implicit neural fu- sion network for multispectral and hyperspectral image fusion. arXiv preprint arXiv:2404.151/ 15174 2024.
[20] Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. Pansharpening by convolutional neural networks. Remote Sensing, 8(7):594, 2016.
[21] Qingyan Meng, Wenxu Shi, Sijia Li, and Linlin Zhang. Pandiff: A novel pansharpening method based on denoising diffusion probabilistic model. IEEE Transactions on Geoscience and Remote Sensing, 2023.
[22] Xiangchao Meng. Jie Li, Huanfeng Shen, Liangpei Zhang, and Hongyan Zhang. Pansharpening with guided filter based on three-layer decomposition. Sensors, 16(7):1068, 2016.

---

[23] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[24] Xavier Otazu, María González-Audicana, Octavi Fors, and Jorge Núñez. Introduction of sensor spectral response into image fusion methods. application wavelet-based methods. IEEE Transactions on Geoscience and Remote Sensing, 43(10):2336-2385 2005.
[25] Siran Peng, Chenhao Guo, Xiao Wu, and Liang-Jian Deng. U2net: general framework with ppatial-spetrall-ineerna double u-net for image fusion. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3219-3227, 2023.
[26] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning sext-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22500-225 22510, 10, 2023.
[27] Chenyang Si, Ziqi Huang. Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. arXiv preprint wxXiv2239.11499) 2023.
[28] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[29] Gilbert Strang. Introduction to linear algebra. SIAM, 2022.
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[31] Gemine Vivone. Robust band-dependent spatial-detail approaches for panchromatic sharpening. IEEE transactions on Geoscience and Remote Sensing, 57(9):6411-6433, 2019.
[32] Gemine Vivone, Rocco Restaino, and Jocelyn Chanussot. A regression-based high-pass mod- ulation pansharpening approach. IEEE Transactions on Geoscience and Remote Sensing, (6(2):984-996, 2017.
[33] Gemine Vivone, Rocco Restaino, and Jocelyn Chanussot. Full scale regression-based injection coefficients for panchromatic sharpening. IEEE Transactions on Image Processing, 27(7):3418 3431, 2018.
[34] Lucien Wald. Data fusioo: definitions and architectures: fusion of images of different spatial resolutions. Presses des MINES, 2002.
[35] Lucien Wald, Thierry Ranchin, and Mare Mangolini. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammetric Engineering and Remote Sensing, 63(6))691-699) 1997.
[36] Yancong Wei, Qiangqiang Yuan, Xiangchao Meng, Huanfeng Shen, Liangpei Zhang, and Michael Ng. Multi-scale-and and-dethh convolutional neural network for remote sensed imagery pan-sharpening. In 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pages 3413-3416 IEEE, 2017.
[37] Chen Wu, Bo Du, Xiaohui Cui, and Liangpei Zhang. A post-classification change detection method based on iterative slow feature analysis and bayesian soft fusion. Remote Sensing of Environment, 199:241-255, 2017.
[38] Hongtao Wu. Yijun Yang. Haoyu Chen, Jingjing Ren, and Lei Zhu. Mask-guides progressive network for joint raindrop and rain streak removal in videos. In Proceedings of the 3/ ACM International Conference on Multimedia, pages 7216-7225, 2023.
[39] Xiao WU, Ting-Zhu Huang, Liang-Jian Deng, and Tian-Jing Zhang. Dynamic cross feature fusion for remote sensing pansharpening. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 14687-14696, 2021.
12

---

[40] Shong-Cheng Wu, Ting Zhu Huang, Liang-Jian Deng, Jie Huang, Jocelyn Chanussot, and Gemine Vivone. Lrtefpan: Low-rank tensor completion based framework for pansharpening. 2023.
IEEF Transactions on Image Processing,
[41] Zhong-Cheng Wu, Ting-Zhu Huang, Liang-Jian Deng, and Gemine Vivone. A framelet sparse reconstruction method for pansharpening with guaranteed convergence. Inverse Problems and Imaging, pages 0-0, 2023.
[42] Yijun Yang. Angelica Aviles-Rivero Huazhu Fu, Ye Liu, Weiming Wang, and Lei Zhu. Video adverns-weaather-comppeee suppression network via weather messenger and adversarial backpropagation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 13200-13210, 2023.
[43] Yijun Yang. Huazhu Fu, Angelica I Aviles-Rivero, Carola-Bibiane Schönlieb, and Lei Zhu. Diffmic: Dual- guiddnce diffusion network for medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 105.
Springer, 2023.
[44] Yijun Yang, Hongtao Wu, Angelica I Aviles-Rivero, Yulun Zhang, Jing Qin, and Lei Zhu. Genuine knowledge from practice: Diffusion test-time adaptation for video adverse weather removal. arXiv preprint arXiv:2403.0766
2024.
[45] Xiaohui Yuan, Jianfang Shi, and Lichuan Gu. A review of deep learning methods for semantic segmentation of remote sensing imagery. Expert Systems with Applications, 169:1 14417, 2021.
[46] Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. In JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop, 1992.
[47] Jie Zhou, Daniel Civco, and John A Silander. wavelet transform method merge landsat tm and spot panchromatic data. International Journal of Remote Sensing, 19(4):743-757, 1998.
[48] Man Zhou, Jie Huang, Yanchi Fang, Xueyang Fu, and Aiping Liu. Pan-sharpening with customized transformer and invertible neural network. In Proceedings of the AAAÍ Conference on Artificial Intelligence, volume 36, pages 3553-3561, 2022.
[49] Man Zhou, Keyu Yan, Jinshan Pan, Wenqi Ren, Qi Xie, and Xiangyong Cao. Memory- augmented deep unfolding network for guided image super-resolution. International Journal of Computer Vision, 131(1):215-242, 2023.
13

---

A Appendix supplemental material
In the supplementary material, we first introduce the background knowledge, i.c., denoising diffusion probabilistic models, and then illustrate the process of model training and LoRA-like branch-wise alternative fine-tuning, the limitations, and the broader impact of our method. After that, we provide more details for experiments the pansharpening task, iee., Implementation Details, Datasets, Benchmarks, and Quality Metrics. Finally, we display more qualitative evaluation results on the QuickBird and GaoFen-2 datasets.
B Background
B.1 Denoising Diffusion Probabilistic Models
Denoising diffusion probabilistic models [ are latent variable models, generating realistic target images progressively from normal distribution by iterative denoising. The diffusion model contains two steps: forward and reverse processes.
The forward process aims make the prior data distribution Xo noisy by a I step Markov chain that gradually transforms it into approximate standard normal distribution Xy ~ (N, and d denotes the dimension. One forward step is defined as follows:
(xx//xx-1) -xxx; /1 1,
(13)
where (-)) is Gaussian distribution with the mean of vT- 3,x and variance of 3,I, Br is pre-defined variance schedule in time step E [0, 7). Through the reparameterization trick, can derive directly from Xo, The following equation gives this derivation:
<xx Xn)
VoxXo + v1 0ig,
(14)
where I), and a, 11 8r a. Ti
The reverse process aims to learn to remove the degradation brought from the forward process and sample the Xo from x,- To accomplish this objective, we need to learn the distribution of po (xx-1|x)) using a neural network and perform iterative sampling as follows:
p (x4) = N NN (x t), Ee(x, t)),
(15)
where μa and Eg are the mean and variance of p respectively, and is the parameters of model. According to Eq. 15). the mean and variance can be computed, following:
Be
14g
(xt
en ()),
(16)
Vat
VI ai
Ūt-1
Eo(x,,
1 - α 32.
(17)
For sampling from standard Gaussian noise XT get XT-1, after performing T-step iterations of sampling as described above, we get the output Xg from XT.
C LoRA-like Branch-wise Alternative Fine-tuning
In Section 3.4 of the main text. The L-BAF method alternately fine-tunes the spatial and spectral branches based on the proposed APFM. As shown in Fig. 6 When we fine-tune the spectral branch, we freeze the T, and T, parameters in the spatial branch and the T, parameter in the spectral branch. Block gradient propagation to prevent parameter updates for the entire spatial branch. Similarly, when fine-tuning the spatial branch, we freeze the T. and TĄ parameters in the spectral branch, as well as the T, parameter in the spatial branch. Block gradient propagation to prevent parameter updates for entire spectral branches.
14

---

Spectral branch fine-tuning
Spectral branch
Spatial branch
To
T
T2
Tb
Tg
Spatial branch fine-tuning
Spectral branch
Spatial branch
Tg T3
T
T
Frozen
Trainable
Figure 6: The sketch of the proposed LoRA-like branch-wise alternative fine-tuning process.
Algorithm I: Training stage proposed method.
Data: GT image Xo. diffusion model with its parameters 0, spectral and spatial branch parameter Sppe. Oepa. respectively, condition cond, timestep t, and denoised objective Xg-
Result: Optimized diffusion model xg.
cond + PAN, LrMSI, X₁:
while until convergence do t t Uniform T); NN I);
X; < Vā, - LrMSI) + VT 1y-mae O.G; 8o t cond) + LrMSI; if iteration > /50k then
fine-tune Oxpe or O Ospai // L-BAF
end
0 - V VaLeimpte (2o, Xo).
end
D Model Training
The detailed training process of SSDiff can be found in Algorithm
!
E
Limitation
First, we evaluate the effectiveness of the proposed SSDiff over the pansharpening task and we will extend our method to other multispectral fusion tasks, such multispectral and hyperspectral image fusion. Second, the proposed FMIM can adjust the frequency information between the two branches, but also introduces additional hyperparameters, increasing the difficulty of fine-tuning during training. In terms cost, the average inference time for single image is 7.416 seconds under 10 timesteps. The time cost for our approach is higher than other DL-based models, primarily due to the limitation imposed by the large number of sampling steps required in the diffusion model.
Broader Impact
Pansharpening, as fundamental preprocessing method which is key pre-processing technology overcoming the constraints of hardware before using high-resolution multispectral images, has been widely utilized in various applications, including change detection, environmental monitoring, and segmentation. At the same time, pansharpening, being a low-level task. the distortions that occur
15

---

2.0
1.5
GT
SSDIff
PanDiff
DCFNet
MMNst
LAGCony
CTINN
0.5
0.0
PNN
FusionNet
MSDCNN
DICNN
BT-H
MTF-GLP-FS
BDSD-PC
Figure Visual comparisons on a reduced-resolution QuickBird case. The first and third rows are the predicted HrMSI for each method, and the second and fourth rows are the error maps of the predicted HRMS versus ground truth (GT) for each one.
Table 6: Quantitative results on the QuickBird reduced-resolltion and full full-resolution datasets. Some conventional methods (the first three rows) and DL-based methods are compared. The best results are highlighted in bold and the second best results are underlined.
Method
SAM(+ skd)
BRGASC-
Reduced resolation Q2(+ std)
SCC(+ std)
DyC+ std)
Full resolution D2t+ snd)
HQNK(+
BDSD-PC A3
8.2620=2.0497
7.5420.20.8138
0.832310.101
0.197510.0334
0.163611000433
0.672210.0577
ATF-GLP-FS.
8.1131=1.9883
7.5102=007926
0.8206-00.005
0.8098=0..016
0.0499
0.1383++0.0238
0.8199:0.0144)
BTH
0
1.5523
7.4008:0.8378
0.8326:10.6300
0.9156:000012
0.2000
0.1648:10.0167
00.03411.0065
5,2054=009625
4.4722:10.7334
0.9180=20.0934
0.0011:0.0123
0.0569:0.0112
0.0624++0.0239
0.894420.0044
D&CNN
5.3795=1.0066
5.1354.10.1876
0.9012110.0942
0.9621
0.0920110.0143
0,10671000211
0.
MSDCNN
36
5.1471=20.9322
4.3828:0.34400
0.9188 09666
0.96892000121
0.06002::00010
0.0667110.029
0.877410.0388
FusionNes
4.9226=0.9077
4.1594.10.3212
0.9252:10.00 0902
0.97551100:04
0.058610.0189
0.0522-10.0088
0.8922:0.02.9
4.658310.7755
3.6969110228$$
0.9320110.007
0.98291000072
0.173811000322
0.0731
0.766310.0412
LAGCony
4.5473=008296
0.1959:0.:116
0.933540.0078
0.9007=000009
0.0844:50.0238
0.0676:+000134
0.8536::0.013 78
MMNNt
00.285
3.6669
0.913710.00441
0.982910.00000
0.08901:00012
0.097211.0032
0.8025
DCFNer
4..93-00.397
3.:155-022955
0.9325-00.990S
0.9011--0.010
0.0454-0.0117
0.1239+0.0026
0.8360-8.0188
PanDiff 21
3.7422:103309 7.5751
0.9818:10.0902 0.9045
0.0587:10.0223
0.0612:10.0522
0.881320000117
SSDII (ours)
0,,440
3.6320:10.2749
0.9346:10.9943
0.9829=000080
0.0314:000108
0.0360:10.0133
0.9338:0.0228
Ideal value
0
0
during the process of generating high-resolution multispectral images can significantly impact the success rate of subsequent high-level tasks.
G
Additional experiment
G.1 Setups
Implementation Details. Our SSDiff is implemented in PyTorch 1.7.0 and Python 3.8.5 using AdamW optimizer with an initial learning rate of 0.001 to minimize Lximple on Linux operating system with an Intel 12th Gen 17-12700K processor and two NVIDIA GeForce RTX3090 GPUs. For the diffusion denoising model, the initial number of model channels is 32, the diffusion time step used for training in the pansharpening is set to 1000, while the diffusion time step for sampling is set 10. The exponential moving average (EMA) ratio is set 0.9999. The total training iterations for the WV3, GF2, and QB datasets set to 150k, 100k, and 200k iterations, respectively. During the
16

---

1.0
0.95
0.90
SSDiff
PanDiff
DCFNet
MMNet
LAGConv
CTINN
0.85
0.80
0.75
PNN
FuslonNet
MSDCNN
DICNN
BT-H
MTF-GLP-FS
Figure 8: Fused GF2 full-resolution data and their corresponding HQNR map. The high value in the HQNR map means better full-resolution fusion performance.
model fine-tuning, the learning rate is set to 0.0001, and the total fine-tune training iterations are set 30k.
Datasets. To demonstrate the effectiveness of our SSDiff, we conducted experiments on widely used pansharpening datasets. The Pancollection dataset for pansharpening consists of data from four satellites: WorldView-3 (8 bands), World View-2 (8 bands), QuickBird (4 bands), and GaoFen-2 (4 bands).
To better evaluate performance, we simulated reduced-ressllttion and full-resolution datasets. For reduced datasets, Pancollection follows Wald's protocol 355 obtain simulated images with ground truth images. There are three steps involved: 1) Use a modulation transfer-based (MTF) filter to downsample the original PAN and MS images by factor of 4. Downsampled PAN and MS used as training PAN and MS images; 2) Treat the original MS image as ground truth image, i.e. HRMSI: 3) Upsampling the training MS image using interpolation with polynomial kernels (2) and processing it into a LrMSI. When processing the full datasets, the original MS image is considered MS, the upsampled MS image considered LrMSI, and the original PAN considered PAN.
Benchmark. To evaluate the performance of our SSDiff, we compared it with various state-of-the-ae methods of Pansharpening (on WV3, QB, and GF2 datasets). Specifically, we choose three traditional methods: BDSD-PC 31.. MTF-GL.P P- 33 BT-H 1 as well as nine machine learning- based methods: PNN 200, DiCNN 12]. MSDCNN 3d. FusionNet [3] CTINN 48, LAGConv [LS),
MMNet 49
DCFNet (39.. and Pandiff 21) To ensure fairness, we train based methods using
the same Nvidia GPU-3090 and PyTorch environment.
Quality Metric. For the reduced data in Pansharpening tasks, utilize four metrics to evaluate results on reduced resolution datasets, including the spectral angle mapper (SAM) [46), the erreur relative globale adimensionnelle de synthèse (ERGAS) (34], the universal image quality index (Q2") [10), and the spatial correlation coefficient (SCC) 47). As for full-resolution datasets, we apply Dx. D.. and hybrid quality with reference (HQNR) indexes (3) for evaluation.
17

---

G.2 Results on QuickBird:
We conduct experiments on the QuickBird reduced dataset and evaluate the performance of SSDiff. Similarly, the reference and non-reference metrics were obtained from 20 randomly selected test images from the QB dataset. The performance comparison reported in Table Our SSDiff achieves SOTA performance. From the error maps in Fig. 7 we can observe that there are still significant differences between traditional fusion methods and Di.-based fusion methods.
G.3 Results on GaoFen-2:
On the GaoFen-2 full-resolution dataset, we tested our SSDiff on 20 test images, Fig. g shows the results and HQNR maps. where an HQNR score close indicates better fusion quality of full-resolution images. The obtained results indicate that our SSDiff has good generalization of the full-resolution dataset.
Table Efficiency results on the WV3 reduced-resolltion datasets.
Method |SSDiff PanDiff DCFNet MMNet LAGConv
Runtime (s) 7.417 261.410 0.548
0.348
1.381
G.4 Efficiency analysis
The certuusion-bsedd method generally has more running time than CNN-based methods due to the multiple timesteps of the diffusion mechanism. The comparison of inference running time shown in Table this point. However, for fair comparison with another diffusion-basee method for pansharpening, i.e., PanDiff, our method still gets a significant advantage.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction clearly outline the main contributions and scope of the paper, ensuring that the claims made supported by the theoretical and experimental results presented.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations in Appendix D such as the inference overhead of the model and the additional hyperparameters introduced.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: For Theorem presented in the paper, a complete proof is given in the main text, along with detailed explanations and illustrations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed descriptions of the network architecture and experimental settings and provides links to the publicly available datasets used. The complete code will be published on GitHub upon acceptance for further research and discussion.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The datasets are publicly accessible, and the code will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
gublic/guidess htta://guiiee//deebbbiieeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips cc/ httpppullllllppieeee http://ooooooooosssooooooooeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Appendix G
present the datasets, implementation details, and the hyperparameters can be found in Sec. 3.3
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not involve error bars experiments concerning statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experiments were conducted on workstation equipped with an Intel 12th Gen 17-12700K processor, two NVIDIA RTX 3090 GPUs, and 128GB of memory. The detailed compute resources for the experiments can be found in the implementation details of Appendix. G.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //t:://nnuuee /tt///nurippp... https///../..lltttttcccccccaaaa.
Answer: [Yes]
Justification: The research adheres to the ethical guidelines set forth by NeurlPS, ensuring responsible conduct in all aspects of the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Both potential positive societal impacts and negative societal impacts of the work can be found in the implementation details of Appendix. F
22

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The models compared in the paper have either received author permission are publicly available. The datasets are public, and original papers are cited meticulously.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The models compared in the paper have either received author permission are publicly available. The datasets are public, and original papers are cited meticulously.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
23

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the palaaets.ppperssttt paperssii thcode. com con/datases package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing or research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing or research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "FEV24va8oF", "reviewer": "Reviewer_kLBw", "summary": "This paper propose a novel spatial-spectral integrated diffusion model called SSDiff based on subspace decomposition, and its main contributions contain: 1) Considering existing DDPM-based methods have not yet designed models for the discriminative features required in the pansharpening task, SSDiff divide the network into spatial and branches based on subspace decomposition. Additionally, an alternating projection fusion module (APFM) is constructed for SSDiff to transform the process of fusing HrMSI into the fusion process of spatial and spectral components. Tests on four widely used pansharpening datasets demostrate SSDiff can achieve  state-of-the-art (SOTA) performance. 2) The frequency modulation inter-branch module is used at the junction of spectral and spatial branches to enrich extracted spatial information with more high-frequency information in the denoising process. 3) The proposed L-BAF method is used to fine-tune the network based on the proposed APFM, where the spatial and spectral branches are updated alternately. This design allows alternately fine-tuning the two branches without increasing the parameter count, enabling the learning of more discriminative features.", "questions": "1) Why introduce another two features from the features of spatial domain and spectral domain separately?\n\n2) Why choose FFT as the specific way to extract the high-frequency information of the feature map obtained from the spatial branch rather than other methods?\n\n3) The design of FMIM?", "limitations": "1) This paper evaluate the effectiveness of the proposed SSDiff over pansharpening task and will extend our method to other multispectral fusion tasks.\n\n2)  The proposed FMIM can adjust the frequency information between the two branches, but it also introduces additional hyperparameters, increasing the difficulty of fine-tuning during training.\n\n3) The time cost for our approach is higher than other DL-based models, primarily due to the limitation imposed by the large number of sampling steps required in the diffusion model.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) Considering existing DDPM-based methods have not yet designed models for the discriminative features required in the pansharpening task,  this paper proposes a novel SSDiff method based on subspace decomposition, which leverages spatial and spectral branches to discriminatively capture global spatial information and spectral features, respectively. \n\n2) And from the perspective of subspace decomposition, an alternating projection fusion module (APFM) is constructed to fuse the captured spatial and spectral components, which is concerned with vector projection.\n\n3) To overcome the problem of uneven distribution of frequency information between two branches in the denoising process, a frequency modulation inter-branch module (FMIM) is designed.\n\n4) Through the proposed LoRA-like branch-wise alternating fine-tuning (L-BAF), SSDiff can further reveal spatial and spectral information not discovered in each branch. And L-BAF is inspired by the idea that subspace decomposition can be generalized into the self-attention mechanism.\n\n5) There are abundant experiments to support the effectiveness of SSDiff from various aspects.", "weaknesses": "1) Maybe expression should be pay more attention to, both grammatically and logically. For example, the sentences in line 135 and line 140 are incomplete. \n\n2) This paper claims the fusion result images and error maps of all these methods in Fig. 5, yet it's not the truth. And there lacks concerned descriptions about results on GaoFen-2 full dataset.\n\n3) Paragraph from line 162 to line 167 seems not to be expressed clearly about how eq.(3) and eq.(7) can be generalized into matrix form of the self-attention mechanism as shown in Fig.3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1.** Why introduce another two features from the features of spatial domain and spectral domain separately?\n\nThank you for your thorough review and feedback. Firstly, PAN images and LrMSI are obtained from different sensors and contain distinct feature information. PAN images exhibit richer spatial details, while LrMSI possesses more abundant spectral information. Considering the characteristics of the aforementioned pansharpening task, we leverage spatial and spectral branches to discriminatively capture global spatial information and spectral features, respectively. Additionally, we construct an APFM by incorporating the characteristics of the cross-attention mechanism and the task properties. Utilizing the alternate projection method to merge the captured spatial and spectral component features and generate HrMSI.\n\n>**Q2.**  Why choose FFT as the specific way to extract the high-frequency information of the feature map obtained from the spatial branch rather than other methods?\n\nThank you for your insightful question. The frequency domain representation provided by the FFT aligns well with the intuition that high-frequency information is crucial for capturing spatial details in the pansharpening task. While there may be other methods, such as wavelet-based approaches or high-pass filtering, the FFT-based approach is a common and effective choice due to its efficiency, shift-invariance, and widespread adoption in the field of image processing and fusion.\n\n\n>**Q3.** The design of FMIM?\n\nThanks for your review again. As shown in Fig.4 in the manuscript. During the denoising process, the low-frequency information (e.g., global structure) of images just has some slight changes in the artifacts. However, the high-frequency information (e.g., edges and textures) changes from noisy images to images with edges and textures. It can be considered that in the process of denoising and reconstructing the HRMSI, the majority of timesteps are mainly spent reconstructing the high-frequency information. Therefore, We designed FMIM to modulate the degree of high-frequency information.  Utilizing a Fourier filter to extract the high-frequency information of the feature map obtained from the spatial branch and inject enhanced high-frequency information into the spectral branch.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9O0EdbfzCZ", "reviewer": "Reviewer_Noom", "summary": "For the task of multispectral and panchromatic image fusion (Pansharpening), this paper proposes a spatial-spectral integrated diffusion model (SSDiff). The framework is novel, utilizing spatial and spectral branches to learn spatial details and spectral features separately. It introduces an alternating projection fusion strategy and a frequency modulation inter-branch module to enhance fusion quality. Furthermore, designing a LoRA-like fine-tuning strategy for the proposed dual-branch network to capture component-discriminating features more sufficiently. The study demonstrates promising results across multiple datasets.", "questions": "1. What does the first row of data in Table 5 represent? The paper states that \"the total training iterations of the WV3 datasets are set to 150k. ... and the total fine-tune training iterations are set to 30k.\" Is the first row's data the result after training for 150k iterations? If that is the case, please address the issue mentioned concern (see weakness 2).\n2. Regarding the limitation mentioned about the efficiency disadvantage of diffusion models, can the authors provide more specific comparative results? The paper noted that diffusion models have an efficiency disadvantage compared to other super-resolution methods. To fully evaluate this limitation, it would be helpful if the authors could include more concrete performance comparisons.\n3. Why do low-frequency components undergo progressive modulation while high-frequency components exhibit distinct dynamic variations? I don't quite understand this point.\n4. What are the specific differences between this method and other DDPM methods in terms of training and sampling processes, e.g. Meng et al.2023.", "limitations": "The authors provide Limitations and Broader Impact, where limitations are influenced by the nature of the task itself.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The writing of the article is quite clear.\n2. The experiments are conducted quite thoroughly.\n3. Subspace decomposition to divide the network into spatial and spectral branches has some degree of innovation. And it has reliable theoretical support\n4. The alternating projection fusion strategy and the accompanying LoRA-like fine-tuning method can provide insights for future work.", "weaknesses": "1. Figure 4 is not clear. It is hard to see how “ the low-frequency components undergo a gradual modulation characterized by a slow and subtle rate of change in the denoising process. In contrast, the modulation process of the high-frequency components exhibits distinct dynamic variation.”. Suggest enhancing the high-frequency image to make it more prominent.\n2. To ensure the fairness of the experiment, it would be reasonable to add a control experiment. Continuing to train the 150k base model for 30k iterations using the fine-tuned model's configuration and then comparing the results to the fine-tuned model. This would provide a more comprehensive evaluation of the effectiveness of the fine-tuning strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1.** Figure 4 is not clear.\n\nThank you for your valuable suggestion. We will fix this.\n\n\n>**W2&Q1.** To ensure the fairness of the experiment ..? \\& What does the first row of data in Table 5 represent? ..\n\nThank you for your detailed review. The first row's data is the result after training for 150k iterations. Io ensure the reliability of the L-BAF experiment, the modifications to Table 5 are as follows: the first row represents the evaluation results of the pre-training base model used for fine-tuning, and the second row displays the evaluation results of the experiment conducted with fine-tuning on the pre-training base model (lr=0.0001) for an additional 30k iterations without using the L-BAF method.\n\n|$\\mathcal{S}$/$\\mathcal{F}$|SAM ($\\downarrow$)|ERGAS ($\\downarrow$)|Q8 ($\\uparrow$)|SCC ($\\uparrow$)|\n|:-|:-:|:-:|:-:|:-:|\n|$-$/$-$|2.8646$\\pm$0.5241|2.1217$\\pm$0.4671|0.9125$\\pm$0.0874|0.9863$\\pm$0.0040|\n|✘/✘|2.8681$\\pm$0.5837|2.1302$\\pm$0.5235|**0.9206$\\pm$0.0850**|**0.9868$\\pm$0.0048**|\n|$\\checkmark$/✘|2.8545$\\pm$0.5244|2.1138$\\pm$0.4658|0.9143$\\pm$0.0857|0.9864$\\pm$0.0040|\n|✘/$\\checkmark$|2.8460$\\pm$0.5232|2.1132$\\pm$0.4671|0.9152$\\pm$0.0849|0.9864$\\pm$0.0041|\n|$\\checkmark$/$\\checkmark$|**2.8429$\\pm$0.5284**|**2.1059$\\pm$0.4560**|0.9156$\\pm$0.0841|0.9867$\\pm$0.0038|\n\n\n>**Q2.** Regarding the limitation mentioned about the efficiency disadvantage of diffusion models, can the authors provide more specific comparative results?\n\nThank you for pointing out this. The diffusion-based method generally has more running time than CNN-based methods due to the multiple timesteps of the diffusion mechanism. The comparison of running time shown in the table below ensures this point. However, $\\textbf{for a fair comparison}$ with another diffusion-based method for pansharpening, i.e., PanDiff, our method still gets a significant advantage.\n\n|Method|SSDiff|PanDiff|DCFNet|MMNet|LAGConv|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Runtime (s)|7.417|261.410|0.548|0.348|1.381|\n\n\n>**Q3.** Why do low-frequency components undergo progressive modulation while high-frequency components exhibit distinct dynamic variations?\n\nThanks for your review. For a spectral branch, the input LrMSI contains blurred spatial detail and abundant spectral information, which is low-frequency spatial information. Then, we further illustrate the changes in low-frequency and high-frequency information in Fig.~4. During the denoising process, the low-frequency information (e.g., global structure) of images just has some slight changes in the artifacts. However, the high-frequency information (e.g., edges and textures) changes from noisy images to images with edges and textures. Therefore, we designed FMIM to utilize the phenomenon to inject high-frequency information into the spectral branch to enhance changes in frequency information.\n\n>**Q4.** What are the specific differences between this method and other DDPM methods in terms of training and sampling processes, e.g. Meng et al.2023.\n\nThanks for your review again. Among the existing works based on DDPM, we specifically design a dual-branch structure for the special task, i.e., pansharpening, that essentially fuses two inputs to get one output. Besides, different from the LoRA in conventional DDPM, we propose a novel fine-tuning method (i.e., L-BAF strategy) based on our APFM, which is more suitable for the pansharpening task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kdvmrE1bWl", "reviewer": "Reviewer_qNHH", "summary": "This paper introduces SSDiff, a model that improves the resolution of satellite imagery by combining high-resolution panchromatic images and low-resolution multispectral images. SSDiff separates image details into spatial and spectral components, and processes them through different network branches, then merges them back together. It utilizes a denoising diffusion model and adopts a designed Alternating Projection Fusion Module (APFM) to complete the spatial-spectral fusion. Additionally, they propose a Frequency Modulation Inter-branch Module (FMIM) to modulate the frequency distribution between branches, as well as a LoRA-like branch-alternating fine-tuning method to further improve performance. The paper provides a solid theoretical foundation, and through experiments on various datasets, SSDiff demonstrates the ability to generate high-resolution multispectral images with minimal spatial and spectral distortion.", "questions": "1. How does the element-wise multiplication operation help preserve the complementary information from the two domains?\n2. Could the author elaborate on any challenges faced when integrating information from PAN and LrMSI into a diffusion model? This clarification would help in understanding your methodology.", "limitations": "The authors have provided reasonable Limitations and Broader Impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors propose an innovative spatial-spectral integrated diffusion model for pansharpening, which is a timely application of diffusion models to this domain.\n2. This paper provides a solid theoretical basis for the approach, grounded in concepts from linear algebra and subspace decomposition.\n3. The proposed methods in this paper achieve state-of-the-art performance on two datasets. The comprehensive ablation study further adds to the robustness and credibility of the results. \n4. This paper is well-organized and the content is clear and easy to read.", "weaknesses": "- Efficiency of the method: Evaluating and discussing the efficiency of the DDPM-based method used in SSDiff would help provide a more comprehensive understanding of the practical advantages and limitations of the proposed approach. This information could be valuable for readers to assess the real-world applicability and deployment feasibility of the SSDiff model.\n- Experimental Section: The proposed method in this paper has been extensively evaluated for its generalization performance, but the experiments are limited to the Pansharpening task. It would be better if there were experiments on other datasets, but it's not essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**W1.** Efficiency of the method: Evaluating and discussing the efficiency of the DDPM-based method used in SSDiff would help provide a more comprehensive understanding of the practical advantages and limitations of the proposed approach. This information could be valuable for readers to assess the real-world applicability and deployment feasibility of the SSDiff model.\n\nThank you for your thorough review and valuable suggestions regarding our manuscript. The diffusion-based method generally has more running time than CNN-based methods due to the multiple timesteps of the diffusion mechanism. The comparison of running time shown in the table below ensures this point. But compared to the Pandiff method, which is also based on a diffusion model, SSDiff still demonstrates a noticeable advantage in terms of efficiency.\n\n|Method|SSDiff|PanDiff|DCFNet|MMNet|LAGConv|\n|:-|:-:|:-:|:-:|:-:|:-:|\n|Runtime (s)|7.417|261.410|0.548|0.348|1.381|\n\n>**W2.** Experimental Section: The proposed method in this paper has been extensively evaluated for its generalization performance, but the experiments are limited to the Pansharpening task. It would be better if there were experiments on other datasets, but it's not essential.\n\nThank you for your suggestion. Despite conducting many generalization experiments on the pansharpening task, we plan to conduct experiments on datasets from various other tasks in the future to prove the generalization capability of SSDiff across tasks.\n\n\n>**Q1.** How does the element-wise multiplication operation help preserve the complementary information from the two domains?\n\nThanks for your review. Firstly, after the dot product, the information in $T_{spa}$ and $T_{spe}$ will not be lost. Secondly, in Table 2 of the manuscript, the experiment (i.e., V3, V4, and V5 of the “Effectiveness ..\") illustrated the differences when using dot product, concatenation, additional, and multiplication. The result proved that the dot product performed better.\n\n>**Q2.** Could the author elaborate on any challenges faced when integrating information from PAN and LrMSI into a diffusion model? This clarification would help in understanding your methodology.\n\nThanks for your careful review. We give detailed challenges. The PAN image contains rich high-frequency information and less spectral information, while the LrMSI mainly contains low-frequency information and rich spectral information. The pansharpening aims to fuse these two images to obtain the HrMSI. Besides,  LoRA in conventional DDPM is used to fine-tune with arbitrary domains, whereas our task requires fine-tuning in the spatial and spectral domains. A toy example is shown in Fig.~1. Therefore, it motivates us to propose SSDiff specifically for the pansharpening task. By introducing subspace decomposition into SSDiff, we generalized vector projection to matrix form and designed APFM, which further inspired us to propose the L-BAF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers found the proposed LoRA -based approach for pansharpening very interesting and novel. Authors presented very good responses to the concerns raised by the reviewers.
Decision: Accept (poster)