Panchromatic and Multispectral Image Fusion via Alternating Reverse Filtering Network
Authors: Keyu Yan
Man Zhou
Jie Huang
Feng Zhao
Chengjun Xie
Chongyi Li
Danfeng Hong
Venue: NeurIPS 2022 Accept (2022)
Keywords: multi-spectral image fusion
multi-spectral image restoration

Abstract: Panchromatic (PAN) and multi-spectral (MS) image fusion, named Pan-sharpening, refers to super-resolve the low-resolution (LR) multi-spectral (MS) images in the spatial domain to generate the expected high-resolution (HR) MS images, conditioning on the corresponding high-resolution PAN images. In this paper, we present a simple yet effective alternating reverse filtering network for pan-sharpening. Inspired by the classical reverse filtering that reverses images to the status before filtering, we formulate pan-sharpening as an alternately iterative reverse filtering process, which fuses LR MS and HR MS in an interpretable manner. Different from existing model-driven methods that require well-designed priors and degradation assumptions, the reverse filtering process avoids the dependency on pre-defined exact priors. To guarantee the stability and convergence of the iterative process via contraction mapping on a metric space, we develop the learnable multi-scale Gaussian kernel module, instead of using specific filters. We demonstrate the theoretical feasibility of such formulations. Extensive experiments on diverse scenes to thoroughly verify the performance of our method, significantly outperforming the state of the arts.

Panchromatic and Multispectral Image Fusion via Alternating Reverse Filtering Network
Keyu Yan 2, Man Zhou 12; Huang?, Feng Zhao?, Chengjun Xie Chongyi Li Danfeng Hong*t
Hefei Institute of Physical Science, Chinese Academy of Sciences, China
University Science and Technology of China, China
Nanyang Technological University, Singapore
Aerospace Information Research Institute, Chinese Academy of Sciences, China
Abstract
Panchromatic (PAN) and multi-spectral (MS) image fusion, named Pan-sharpening, refers super-resolve the low-resolution (LR) multi-spectral (MS) images in the spatial domain to generate the expected high-resolution (HR) MS images, condi- tioning on the corresponding high-resolution PAN images. this paper, we present simple yet effective alternating reverse filtering network for pan-sharpening. Inspired by the classical reverse filtering that reverses images to the status before filtering, formulate pan- sharpening process, which fuses LR MS and HR MS in an interpretable manner. Different from existing model-driven methods that require well-designed priors and degradation assumptions, the reverse filtering process avoids the dependency on pre-defined exact priors. To guarantee the stability and convergence of the iterative process via contraction mapping on a metric space, we develop the learnable multi-scale Gaussian kernel module, instead of using specific filters. We demonstrate the theo- retical feasibility of such formulations. Extensive experiments on diverse scenes to thoroughly verify the performance of our method, significantly outperforming the state of the arts.
alternately iterative reverse filtering
Introduction
Multispectral images are widely used in various fields such resource monitoring [ environmen-
tal protection (2) 3 and ecological monitoring [4]. However, due to the hardware limitations of
multispectral sensors, multispectral images usually lack high spatial resolution (5 6) In contrast, high-resolution panchromatic (PAN) images that contain rich spatial details of the same scene are easy to obtain. Therefore, pan-sharpening, generating high-resolution multispectral images by fusing panchromatic images with low-resolution multispectral images, has become important issue in the field of remote sensing
Many efforts have been made to solve the pan-sharpening problem, which can be generally divided into two large groups: traditional pan-sharpening methods and deep learning bassed methods 9 10). Traditional pan-sharpening methods usually require strict assumptions of multispectral image degradation via prior knowledge. Otherwise, the inexact assumptions may cause system model error. For example, both the assumptions established by component substitution methods [1] 12 and multi-resolution analysis methods [ 14 focus on the relationship between PAN and HR MS, which is destined to make these methods prone to spectral distortion. Unlike the approaches, variational optimization approaches consider the relationship among HR MS, LR MS
traditional
"Co-first authors contributed equally, corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

and PAN. and then construct the energy function based on well well-designed priors [13] 16]. However, the methods of this kind inflict high computational burden, restricting their practical applications.
Most recently, deep learning exhibited outstanding performance in the field of remote sensing images 17 18. Undoubtedly, the deep learning-based based (DL) methods become newly developed category to solve pan-sharpening [19-22]. Unfortunately, the pan-sharpening networks commonly lack the interpretability in theirs designs, which limits their performance. To solve this issue, the model- based deep learning methods build the network by unrolling the specific optimization algorithm However, the optimization algorithms of model-based deep learning methods still require desl-ne priors or assumptions. Additionally, the convergence of the optimization algorithms is not taken into account in the design of the unrolling networks.
23 24
To address these problems, we propose a novel pan-sharpening approach called alternating reverse filtering network, which combines classical reverse filtering (25 and deep learning. Unlike previous methods, we formulate pan-sharpening as a reverse filtering process, thus avoiding the dependency on pre-defined priors or assumption. In addition, we tailor the classical reverse filtering in an alternating iteration manner for the pan-sharpening problem. The process of alternating iterations is unrolled into a network. We demonstrate such formulations is theoretical feasibility. To guarantee the stability and convergence of the iterative process on the basis of contraction mapping in a metric space, we introduce the learnable multi- scale Gaussian kernel module in the network. Such a key issue is commonly neglected in previous unrolling-based deep learning methods.
The main contributions of this paper be summarized follows: 1) We introduce a new perspective for pan-sharpening by formulating it as a reverse filtering process. To the best of our knowledge, this is the first effort to solve multispectral image fusion problem using the method of this kind. 2) In contrast to existing model-driven methods, our iterative network can obtain HR MS without the need for pre-defined exact priors or assumptions. 3) Instead of using specific filters in reverse filtering, we constrain their formulation in a more compact learnable multi-scale Gaussian kernel module, which guarantees the stability and convergence of the iterative process. In addition, extensive experimental results on simulated and real-worke scenes show that the proposed network significantly outperforms the state of the arts.
2 Related Work
Pan-sharpening. Traditional pan-sharpening methods are classified into three types: component substitution (CS)-, multi-resolution analysis (MRA),, and variational optimization (VO)-based methods 126 16 27- The main idea of CS-based methods is to separate spatial and spectral information of the MS image in a suitable space and further fuse them with the PAN image. The representative CS-based methods include intensity hue-saturation (IHS) fusion (30, the principal component analysis (PCA) methods M 31). Brovey transforms [32], and Gram-Schmidt (GS) orthogonalization method (33]. MRA-based methods decompose MS and PAN images into multi- scale space via decimated wavelet transform (DWT) (34), high-pass filter fusion (HPF) [35). indusion method [13 and atrous wavelet transform (ATWT) (36]. Then, the decomposed version PAN images are injected into the corresponding MS images for information fusion. VO-based methods regard the pan mon-sharpening tasks as an ill-posed problem by minimizing loss function, including dynamic gradient sparsity property (SIRF) [7], local gradient constraint (LGC) 15] group low-rank constraint for texture similarity (ADMM) [66. However, the performance of these methods is limited due to the shallow non-linear expression in these models. Since then, deep learning-
pan-sharpening algorithms have dominated this field 9 38 39). Masi et al. [9] are the first to use CNN to deal with the issue of pan- nan-sharpening Although the structure is simple, the effect is much better than the traditional methods. Then, Yang et al. (38) designed deeper CNN by relying on resblock in [40]. Meanwhile, Yuan et al. [4] introduced multi-scale module into the basic CNN architecture.
attempt to combine domain knowledge with deep neural networks to propose deep unrolling networks which take advantages of the model-based based methods' interpretability and learning-bar -bsed methods' strong mapping ability. Specifically, the deep unrolling network firstly unrolls certain optimization algorithms [48] [5 23 51/57 and utilizes deep neural network to parameterize the unrolling model, then minimizes the loss function and optimizes the parameters in an end-to-end manner. For example, Xu et al. 24 developed two separate priors of PAN and MS modality to design the unrolling structure for
Unrolling-basse deep learning method. In recent years, many researchers 42
47
2

---

Alternate Iteration
H
Initial
SP2
multi-scale Gaussian kernel module
Dlmment-wie addition
multi--cale Gaussian kernels
(sr(ae) m]
Mulli-sakk Caussian kemol
Elemant-wise subtraction
BiCabii interolation
Figure 1: The overall architecture of alternating reverse filtering network.
The model-driven methods have better interpretability and clearer physical meaning. Cao al. [46] unrolled alternate optimization algorithm into CNN. However, the optimization of model-basee deep methods also require well-designed priors and exact degradation assumptions.
pan-sharpening.
3 Proposed Method
In this section, we provide a detailed introduction our proposed alternating reverse filtering network for pan-sharpening. For convenience, we first define some notations. Concretely, L E Ruxhx B denotes the low-resolution (L.R) multispectral image, E T W WwxHxB represents the corresponding high-resolution (HR) multispectral image, and P TR. x is the PAN image.
3.1 Problem Formulation
For multispectral image restoration, the degradation model is commonly formulated
L = (H k) 1s +e,
(1)
where *, k. 1s and e denote the convolution operation, blurring kernel, down-sampling operator and the measurement noise, respectively. To restore high quality H from L, high-resolution PAN image is introduced to help enhance structural detail. In CS-based models, the intensity component generalized IHS concept [58), is usually replaced by P but the discrepancy between P and I can cause spectral distortion in the fused image. The VO-based models solve the problem by explicitly constructing many well well-designed priors among H. L and P. However, hand-crafted priors don t work well in practical complex situations. Inspired by classical reverse filtering (25), we propose alternating reverse filtering method to estimate H by the more general multispectral image priors:
L = f(H)
p g(H))
(3)
where f(.) and g(-) denote the degradation processes, and H, is the intensity component of H.
3.2 Model Optimization
Definition 3.2 Suppose (H,,d) d) is metric space and T H. H is mapping function. For all v, E H, if there exists constant E [0, that makes the following formula
d(T()), S ed(æ, v).
(4)
mapping T: H -> H is called Contraction Mapping (59) In the image space, the metric space (H, d) can be expressed as
H x d(z, y) ||x vll. where h is the number of image pixels and d(x,, y) is the Euclidean distance.
(5)

---

Theorem 3.2 A variable x² is fixed point for given function & if = F.. When mapping : H. + H. is a contraction mapping, & admits unique fixed-poite x² in H. Further, x² can be found in the following way. Let the initial guess be and define sequence {In] In (((n-11 When the iterative process converges, lim In = x².
Reverse Filtering Without loss of generality, the degenerate processes f()) and g(-) can be consid- ered as broadly defined filters F(-) that smooth images, texture removal other properties. Filtering process can be described as
y F(x).
(6)
where a and are the input image and the filtering result. When F(-) is unknown, it's difficult to apply well-designed image priors to obtain the æ. However, reverse filtering can estimate æ without needing to compute F (-) and update restored images according to the filtering effect as
x k+1 = x* +y F(x*),
where x is the current estimate of in the k-th iteration. The iteration starts from x y and x gets closer and closer to æ with the increasing k. We make auxiliary function sp(+) as
<(xx) x + y F(x).
(8)
Therefore, the above iterative process can be regarded as a fixed point iteration
*++ pxx*)
With the above analysis, we take the filtering function F(.) in Equation as f(-) and g(-) in Equation 2 to obtain two reverse filtering:
(H,) = -+
Pi (H)
(10)
and hence our alternating reverse filtering method can be written by following
H+ H²
A = H
(11)
H++ = H PP-g H++1 = (H, (H²+ -H
where is the upsampled LR multispectral image L and H is an approximate estimate of H,, Equation 11 starts with initial state H - L. Note that the calculated H is used as input in the next iteration, the intensity component # is then sent into another iteration H H + P g (Ft) enhance the structural details with the help of P image. After that, the enhanced intensity
component replaces the original component H +++ the next iteration.
-H++ and the new H*+1 is used input in
3.3 Alternating Reverse Filtering Network
The end-to-end model we construct for GMIR, named ARFNet (Alternating Reverse Filtering Network), is based on two fixed point iterations in Equation TT with multi-scale Gaussian convolution module acting filters. See Figure for the overview of the proposed method. If reverse filtering satisfies the sufficient condition for definition 3.2 ARFNet will converge limg H² H" and
finally reaches f (H') = L. Take the (p (.) for instance, the sufficient condition that theorem 3.2 holds is that 40i (H) forms contraction mapping
l (H,) - 1 (H,)l = H, + L. - (H.) H, + L. f (H,)
(12)
H., - (H.)) H, - (H+)]] S e- H - Holl C (0,1)
For linear filters, the condition is further simplified as
H /()] S |||]| e (0,1)
(13)
In our case, the degenerate processes f(-) and g(-) are implemented by multi-scale Gaussian convolu- tion modules.

---

Multi-scale Gaussian Convolution Module Given an input image the output y of Gaussian convolution 60 can be expressed
g(r)) * æ.
(14)
where d denotes the variance of 2D Gaussian kernels. In particular, we define 01 = 0 and (
which is Dirac delta. Therefore,
H * (g(a)) g( = (1 - -( < c. H
(15)
where H denotes Fourier transform, denotes point-wised product and 1 denotes all-one matrix. Thus, the inequality holds true when g( is normalized Gaussian kernel that means these filters be strictly reversed using fixed-point iteration. In our model, normalized Gaussian kernel is used for initialization and its parameters can be further learned from the data in an end-to-end manner. Although the learned convolution kernels may not completely satisfy the condition, fixed point iteration can be split into two independent sequences:
H = H
1 + H = 91 (H*) + 91 (H)).
(16)
where (H.) is guaranteed to converge to the unique solution and {H could oscillate. Fortunately, after the first few epochs, the learned convolution kernels are close to the kernels we initialize, which makes {H } the majority and dominates the convergence of the whole process. In the implementation of the algorithm, we adopt multi-scale Gaussian convolution module to obtain better filtering effect. Compared to Gaussian convolution, the multi-scale Gaussian convolution module integrates Gaussian kernels with different kernel sizes, expressed as
+ (g(()) * x)) = - -.,
(17)
KEK
where ") is the learnable mixing coefficient and denotes different kernel sizes 1 x 1,3 x 3,5 x 5, M M. The multi-scale Gaussian convolution is a series simple group convolutional layers defined by initialized 2D Gaussian kernels. Clearly, the form of weighted summation still conforms the above analysis about sufficient condition.
Taken together, the forward process of alternating reverse filtering network can be described as Algorithm
Algorithm Proposed algorithm.
Input:
The upsampled low-resolution multispectral image É. panchromatic image P and maximum iteration number K.
initial H L;
for = 0, 2, 3, ......... K.....K do compute # = H
+ i f(H
fetch the intensity component H H, compute H = H + P g (H)));
replace the intensity component H, end for
H to get H
Output: fused high-resolution multispectral image H and estimated intensity component Hy
Loss Function We utilize two loss functions, there the reconstruction loss L, and the structure loss L,, as following
Leum C. + ACe.
(18)
where À is the hyperparameter which determines the balance between the overall performance and the structure texture details. Specifically, reconstruction loss L, a common pixel-wise L.2 loss and structure loss C. based on structural similarity (SSIM), Thus, the corresponding losses defined as follows:
C, H H|
(19)
= SSIM SSIM Hy).
(20)
where H. H,, H and H are the ground truth, intensity component H. the output of alternating reverse filtering network sp (-) and (p(()) respectively.
5

---

Table 1: Quantitative comparison with the state-of-the-ae methods. The best results are highlighted by bold. The f or +. indicates higher or lower values correspond to better results.
WordView II
GaoFen2
WordView III
Method
PSNRT
SSIMF
SAM,
ERGAS!
PSNR
SSIM
SAMI
ERGAS
PSNR
SSIMI
SAM,
ERGAS
SFIM
34.1297
0.8975
0.0439
2.3449
36,9060
0.8882
0.0318
1.7398
21.8212
0.5457
0.1208
8,9730
Brovey
35.8646
0.9216
0.0403
1,8238
37.7974
0.9026
0.0218
1,3720
22,506
0.5466
0.1159
8,2331
GS
35.6376
0.9176
0.0423
1,8774
37.2260
0.9034
0,0309
1.6736
22.5608
0,5470
0.1217
8,2433
THS
35.2962
0.9027
0.0461
2.0278
38.1754
0.9100
0.0243
1.5336
22.5579
0.5354
0.1266
8.3616
GFPCA
34.5581
0.9038
0.0488
2.1411
37.9443
0.9204
0.0314
1.5604
22.3344
0.4826
0.1294
8.3964
PNN
40,7550
0.9624
0.0259
1,0646
43.1208
0.9704
0.0172
0.8528
29.9418
0.9121
0.0824
3,3206
PANNet
40.8176
0.9626
0.0257
1.0557
43,0659
0.9685
0.0178
0.8577
29.684
0.9072
0.0851
3,4263
MSDCNN
41.3355
0.9664
0.0242
0.9940
45.6874
0.9827
0.0135
0.6389
30.3038
0.9184
0.0782
3.1884
SRPPNN
41.4538
0.9679
0.0233
0,9899
47.1998
0.9877
0.0106
0.5586
30.4346
0.9202
0.0770
3.1553
GPPNN
41.1622
0.9684
0.0244
1.0315
44.2145
0.9815
0.0137
0.7361
30.1785
0.9175
0.0776
3.2596
Ours
41.7587
0.9691
0.0229
0.9540
47.2238
0.9892
0.0102
0.5495
30,5425
0.9216
0.0768
3.1049
4
Experiments
In this section, the datasets and experimental settings firstly described. Then, we evaluate the effectiveness of our proposed alternating reverse filtering network (ARFNet) on simulated and real- world full-resolution scenes. Additionally, we conduct an ablation study to gain insight into the respective effect of different parameter configurations. More experimental results are included in the supplemental material.
4.1 Datasets and Experimental Settings
In order to verify the effectiveness of our models for GMIR, multispectral and panchromatic images obtained on three commercial satellites that widely are used, including WorldViews (WV2), World- ViewllI (WV3), and GaoFen2 (GF2), Each database contains thousands of image pairs, and they are divided into training, validation and testing sets that follow the prior works to generate the training set by employing the Wald protocol tool [1]. In the training set, each training pair contains one guided PAN image with the size of 128 x 128, one LR MS patch with the size of 32 x 32, and one ground truth HR MS patch with the size of 128 x 128.
Models are implemented via PyTorch and one NVIDIA GTX 3090 GPU is used for training. In the experiments, the SGD algorithm with momentum equals to 0.9 is adopted to train the models and the minibatch size is set to 4. The initial learning rate is set to x 10-2. When reaching 1000 and 1500 epochs, the learning rate is decayed by multiplying 0.5, and training ends after 2000 epochs. Through all experiments, We set the hyperparameter À in loss function 18 to 0.1, the number K of alternate iteration and the maximum size M of Gaussian kernels to 17. The sigma of the Gaussian function is set to one fourth of the kernel size.
4.2 Comparison with SOTAS
We conduct several experiments the benchmark datasets compared with several representative guided multispectral image restoration methods: five promising traditional methods, including smoothing filter-based intensity modulation ((SFIM) [62]. Brovey [32). GS [33], intensity hue-
saturation fusion (IHS) (63), and PCA guided filter (GFPCA) [4]; five commonly-recognized
sfate-of-the-an deep-learnng based methods, including PNN 19. PANNET [38), multiscale and
multidepth network (MSDCNN) [4T] super-resolution snpeerrrsoouiiigguuudee guided progressive network (SRPPNN) (65]
and deep gradient projection network (GPPNN) [66)
In our experiments, we select the widely-used image quality assessment (IQA) metrics for evaluation such as the peak signal-to-noise ratio (PSNR), the structural similarity (SSIM), the relative dimension-

---

10
MS
PAN
GFPCA
GS
SFIM
PNN
PANNet
MSDONN
SpPPNN
GPPNN
Ours
Ground Truth
GFPCA
GS
SFIM
PNN
PANNet
MSDCNN
SRPPNN
GPPNN
Figure 2: Qualitative visualization comparison of our method with other representative counterparts on a typical satellite image pair from the ViedIIe dataset. Images in the last row visualizes the MSE between the fused images and the ground truth.
Table 2: The average quantitative results the GaoFen2 datasets on the real-world full-resolution scene. The best results are highlighted by bold.
Metrics
SFIM
GS
Brovey
IHS
GPPCA
PNN
PANNET
MSDCNN
SRPPNN
GPPNN
Ours
Da 4.
0.0822
0.0696
0.1378
0.0770
0.0914
0.0746
0.0737
0.0734
0.0767
0.0782
0.0635
D. 4
0.1087
0.2456
0.2605
0.2985
0.1635
0.1164
0.1224
0.1151
0.1162
0.1253
0.1
QNR1
0.8214
0.7025
0.6390
0.6485
0.7615
0.8191
0.8143
0.8215
0.8173
0.8073
0.8237
less global error in synthesis (ERGAS) 67 the correlation coefficient (SCC), the four-band extension of Q. the spectral angle mapper (SAM) 68 the spectral distortion index Dx, the spatial distortion index Ds. the quality without reference (QNR) 69). The last three indicators are non-reference metrics.
Results on Simulated Scene To quantitatively compare the fused multispectral images with the paired reference ground truth images offered the simulated datasets, we conduct repeated experiments on three datasets. The average performance of representative GMIR methods tabled in T The higher values of PSNR and SSIM, the more similar structure between fused multispectral images and ground truth images. ERGAS takes into account the relative errors of all channels. SAM, Q and SCC focus on measuring spectral distortion. More experimental metric results the supplemental material. The qualitative comparison of the visual results over the representative sample from the WorldView-III dataset is in Figure 2 To highlight the differences in detail, show the error map between fused image and ground truth image in the last row. Similarly, more qualitative comparisons shown the supplemental material.
included in
Results on Real-world Full-resolution Scene To assess the generalization performance of models in real-world scene, we apply models trained on the GF2 dataset to additional 200 paired GaoFen2 satellite images which are constructed using the full-resolution setting as the real scene. Lacking available ground-truth at full-resolution scene, we employ three widely-used non-reference metrics for assessing the performance: Dx. Ds and QNR. The quantitative and qualitative results are summarized in T Figure that clearly demonstrate the higher generalization capacity of the proposed alternating reverse filtering network.
7

---

Real LR MS
Brovey
GFPCA
GS
SFIM
PNN
PANNet
Ours
PAN
Figure 3: Qualitative visualization comparison of our method with other representative methods real-world full-resolution scenes.
Table 3: The effect of hyperparameter À in loss function Soum.
0.001
0.01
0.5
2
PSNR
40.1274
40.8279
41.3267
41.7587
41.6932
41.2736
41.0746
SSIM
0.9598
0.9630
0.9673
0.9691
0,9694
0.9664
0.9653
SAMI 0,0264
0.0256
0.0242
0.0229
0.0232
0.247
0.0260
ERGAS!
1.0677
1.0561
0.9943
0.9540
0.9564
1.0037
1.0491
Table 4: Quantitative comparison of different initialization methods on the WorldView-I1 dataset.
Methods PSNR1 SSIM SAMI ERGAS} SCC1
Qt
Dx +
Ds QNR
(I) 40.3297 0.9601 0.0262
1.0672
0.9663 0.7310 0.0698 0.1277 0.8097
(II) 41.7587 0.9691 0.0229
0.9540
0.9749
0.7731 0.0631 0.1184 0.8285
(III) 40.4161 0.9612 0.0261
1.0668
0.9667 0.7316
0.0684 0.1275 0.8123
4.3 Ablation Study
Ablation studies are implemented on the WordView-Il dataset to explore the effect of different parameters and components the performance of models. We use the ARFNet in subsection 4.1 as the baseline for comparison by changing parameters and components, and all comparison models are trained in the same way. Firstly, to balance the overall performance and the structure details, fix the other components and change only the value of the hyperparameter A. The results in Table SableBsshon show that the larger the value of À is within certain range, the higher the structure similarity between the fused image and ground truth will be. Note that the reverse filtering network (P((-) lack the supervision C, that the performance of the entire network will be significant decrease.
Furthermore, we compare the results of different initialization methods. In Table 4 (I) represents the kernels that are randomly initialized by Kaiming (70), (II) and (III) represents the kernels that are initialized by Gaussian kernels, but the kernels in (III) are fixed. From Table 4 and Figure one could see that: 1) The kernels learned by the randomly initialized network cannot satisfy the sufficient condition, which leads to poor performance. 2) The learned kernels initialized by Gaussian kernels close to the kernels of the whole process. 3) Although the learned kernels are close to the initialized Gaussian kernels, a large number of multi-scale Gaussian kernels still bring performance improvements.
initialize, that makes {H } the majority and dominates the convergence
As can be seen from Table 5 the ablation studies about maximum kernel size M show that the multi-scale Gaussian convolution module can bring better performance improvement. To explore the impact the number of iterations K on the performance, we experiment with varying numbers of K. Table 6 shows the results of different K from to 7. It can be seen that the PSNR performance

---

Initialized random kernels
Learned kernels (initialized by random kernels)
X
13 13 Initialized Gaussian kemels
X
x
13 x 17 Learned kemelss (initialized by Gaussian kernels)
x
7
3 3
x
17 Figure 4: Visualization of initialized kernels and learned kernels.
13 13
17 x 17
3
7 x
13
increases the number of stages increases. We choose K 5 in our implementation to balance the performance and computational complexity.
Table 5: The comparison results of ablation study about maximum kernel size M.
M
15
21
PSNRT
39.7820
40.6804
41.0634
41.4371
41.5297
41.6445
41.6910
41.7252
41.7587
41.7126
41,6982
SSIMI
0.9584
0.9603
0.9657
0.9665
0.9679
0.9684
0.9688
0.9690
0.9691
0.9689
0.9683
4.4
Limitations and Discussions
First, we evaluate the effectiveness of the proposed framework over panchromatic and multispectral image fusion and we will extend the framework to other multispectral fusion tasks, such RGB and multispectral image fusion. Second, although we develop the multi-scale Gaussian kernel module ensure the convergence of the alternating reverse filtering, there is still large room explore learnalle filter function that can strictly satisfy sufficient conditions.
Conclusion
In this paper, we presented a simple yet effective alternating reverse filtering network for pan- sharpening. The proposed approach formulates pan-sharpening reverse filtering process and combines classical reverse filtering and deep learning. The classical reverse filtering is unrolled to network without pre-defined exact priors in alternating iteration manner. Besides, multi-scale Gaussian kernel module is developed to ensure the convergence of the iterative process. Furthermore, the ablation studies verified the effectiveness of the multi-scale Gaussian kernel module. Extensive experimental results on simulated and real-world scenes show that the proposed network significantly outperforms the state of the arts. In the future, will study the extension other image fusion problems such as RGB and multispectral image fusion.
Broader Impact
This research aims address the problem of panchromatic and multispectral image fusion, which is key pre- prepprocessing technology overcoming the constraints of hardware before using high-resolution multispectral image. The fused multispectral images are needed as a reference in the field of resource monitoring, such as land use planning, ocean development, and urban management, and in the field of ecological and environmental protection areas, such as pollution monitoring, vegetation biology and precision agriculture research. Despite the many benefits of fused multispectral images, negative consequences can still occur several special environments. When there case of algorithm failure, artifacts generated the fusion image may affect subsequent and lead to misjudgments.

---

Table 6: The comparison results of ablation study about the number of iterations K.
K
3
7
PSNRT
40.7130
41.056
41,3927
41.6869
41.7587
41.7614
41.7603
SSIMT
0.9611
0.9654
0.9667
0.9684
0.9691
0.9690
0.9688
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant 4227 1350 and the University Synergy Innovation Program of Anhui Province under Grant GXXT- 2019-025. We gratefully acknowledge the support of MindSpore, CANN, and Ascend Al Processor used for this research.
References
[1] Xiangchao Meng, Huanfeng Shen, Huifang Li, Liangpei Zhang, and Randi Fu. Review of the pansharpening methods for remote sensing images based on the idea of meta-analysis: Practical discussion and challenges. Information Fusion, 46:102-113, 2019.
[2] Farzaneh Dadrass Javan, Farhad Samadzadegan, Soroosh Mehravar, Ahmad Toosi, Reza Khatami, and Alfred Stein. review of image fusion techniques for pan-sharpening of high-resolution satellite imagery. ISPRS Journal of Photogrammetry and Remote Sensing, 171:101-117, 2021.
[3] Yong Yang, Hangyuan Lu, Shuying Huang, and Wei Tu. Remote sensing image fusion based on fuzzy logic and salience measure. IEEE Geoscience and Remote Sensing Letters, 17(11):1943- 1947, 2020.
[4] Yanfeng Gu, Tianzhu Liu, Guoming Gao, Guangbo Ren, Ma, Jocelyn Chanussot, and Xiuping Jia. Multimodal hyperspectral remote sensing: overview and perspective. Science China Information Sciences, 64, 02 2021.
[5] Gemine Vivone, Stefano Marano, and Jocelyn Chanussot. Pansharpening: Context-basse generalized laplacian pyramids by robust regression. IEEE Transactions on Geoscience and Remote Sensing, 58(9))6152-6167, 2020.
[6] Yuxuan Zheng, Jiaojiao Li, Yunsong Li, Jie Guo, Xianyun Wu, and Jocelyn Chanussot. Hyper- spectral pansharpening using deep prior and dual attention residual network. IEEE Transactions on Geoscience and Remote Sensing, 58(1 (811))8059-8076 2020.
[7] Lin He, Jiawei Zhu, Jun Li, Deyu Meng, Jocelyn Chanussot, and Antonio Plaza. Spectral-fidelity convolutional neural networks for hyperspectral pansharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remole Sensing, 13:5898-5914, 2020.
[8] Liang-Jian Deng, Gemine Vivone, Cheng Jin, and Jocelyn Chanussot. Detail injection-based deep convolutional neural networks for pansharpening. IEEE Transactions Geoscience and Remote Sensing, 5998))6995-7010, 2021.
[9] Giuseppe Masi, Davide Cozzolino, Luisa Verdoliva, and Giuseppe Scarpa. Pansharpening by convolutional neural networks. Remote Sensing, 8(7), 2016.
[10] Peng Wang, Lei Zhang, Gong Zhang, Hui Bi, Mauro Dalla Mura, and Jocelyn Chanussot. Superresolution land cover mapping based on pixel-, subpixel-, and superpixel-scale spatial dependence with pansharpening technique. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(10)4082-4400 40988 2019.
[11] P K Kwarteng and A Chavez. Extracting spectral contrast in landsat thematic mapper image data using selective principal component analysis. Photogrammetric Engineering and remote sensing, 3,,1,,, 1989.
10

---

[12] Sheida Rahmani, Melissa Strait. Daria Merkurjev, Michael Moeller, and Todd Wittman. An adaptive ihs pan-sharpening method. IEEE Geoscience and Remote Sensing Letters, 7(4):746 750, 2010.
[13] Muhammad Murtaza Khan, Jocelyn Chanussot, Laurent Condat, and Annick Montanvert. Indu- sion: Fusion of multispectral and panchromatic images using the induction scaling technique. IEEE Geoscience and Remote Sensing Letters, (())998-102, 2008.
[14) Gemine Vivone, Luciano Alparone, Jocelyn Chanussot, Mauro Dalla Mura, Andrea Garzelli, Giorgio A. Licciardi, Rocco Restaino, and Lucien Wald. A critical comparison among pansharp- ening algorithms. IEEE Transactions Geoscience and Remote Sensing, 53(5):255522866 2015.
[15] Xueyang Fu, Zihuang Lin, Yue Huang, and Xinghao Ding. variational pan-sharpening with local gradient constraints. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10265-10274, 2019.
[16] Xin Tian, Yuerong Chen, Changcai Yang, and Jiayi Ma. Variational pansharpening by exploiting cartoon-texture similarities. IEEE Transactions Geoscience and Remote Sensing, pages 1-16, 2021.
[17] Theo Bodrito, Alexandre Zouaoui, Jocelyn Chanussot, and Julien Mairal. A trainable spectral- spatial sparse coding model for hyperspectral image restoration. Adv. in Neural Information Processing Systems (NeurIPS), 2021.
[18] Keiller Nogueira, Mauro Dalla Mura, Jocelyn Chanussot, William Robson Schwartz, and Jefersson Alex dos Santos. Dynamic multicontext segmentation of remote sensing images based on convolutional networks. IEEE Transactions on Geoscience and Remote Sensing, 57(10):7503-7520 2019.
[19] Gemine Vivone, Paolo Addesso, Rocco Restaino, Mauro Dalla Mura, and Jocelyn Chanussot. Pansharpening based on deconvolution for multiband filter estimation. IEEE Transactions on Geoscience and Remole Sensing, 57(1):540-553, 2019.
[20] Lin He, Jiawei Zhu, Jun Li, Antonio Plaza, Jocelyn Chanussot, and Bo Li. Hyperpnn: Hyper- spectral pansharpening via spectrally predictive convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 120)):302,-3100, 2019.
[21] Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang, Effective pan- sharpening with transformer and invertible neural network. IEEE Transactions on Geoscience and Remote Sensing, 60:1-15: 2022.
[22] Man Zhou, Keyu Yan, Jie Huang. Zihe Yang, Xueyang Fu, and Feng Zhao. Mutual information- driven pan-sharpening. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1788-1798, 2022.
[23] Kai Zhang, Luc Van Gool, and Radu Timofte. Deep unfolding network for image super- resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3217-3226, 2020.
[24] Shuang Xu, Jiangshe Zhang, Zixiang Zhao, Kai Sun, Junmin Liu, and Chunxia Zhang. Deep gradient projection networks for pan-sharpening. In IEEE Conference on Computer Vision and Puttern Recognition, pages 1366-1375, June 2021.
[25] Xin Tao, Chao Zhou, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Zero-order reverse filtering. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 222-230, 2017.
[26] Xin Tian, Yuerong Chen, Changcai Yang, Xun Gao, and Jiayi Ma. A variational pansharpening method based on gradient sparse representation. IEEE Signal Processing Letters, 27:1 17:1180-1184 2020.
[27] Behnood Rasti, Pedram Ghamisi, and Richard Gloaguen. Hyperspectral and lidar fusion using extinction profiles and total variation component analysis. IEEE Transactions on Geoscience and Remote Sensing, 55(7))3997-4000 2017.

---

[28] Behnood Rasti, Magnus Om Ulfarsson, and Johannes R. Sveinsson. Hyperspectral feature extraction using total variation component analysis. IEEE Transactions on Geoscience and Remote Sensing, 54(12):6966-6885 2016.
[29] Behnood Rasti, Johannes R. Sveinsson, and Magnus O. Ulfarsson. Total variation based hyperspectral feature extraction. In 2014 IEEE Geoscience and Remote Sensing Symposium, pages 4644 4647, 2014.
[30] Wjoseph Carper, Thomasm Lillesand, and Ralphw Kiefer. The use of intensity-hue hue-saturation transformations for merging spot panchromatic and multispectral image data. Photogrammetric Engineering and remote sensing, 56(4))459-44) 467, 1990.
[31] Vijay Shah, Nicolas Younan, and Roger I.. King. An efficient pan-sharpening method via a combined adaptive pea approach and contourlets. IEEE Transactions Geoscience and Remote Sensing, 46(5): 2008.
[32] Alan R Gillespie, Anne B Kahle, and Richard E Walker. Color enhancement of highly correlated images. ii channel ratio and "chromaticity" transformation techniques. Remote Sensing of Environment, 22(3):343-36 1987.
[33] Craig A Laben and Bernard V Brower. Process for enhancing the spatial resolution of multi- spectral imagery using pan-sharpening, 2000. US Patent 6,011,875.
[34] SG Mallat. A theory for multiresolution signal decomposition: The wavelet representation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 11 693, 1989.
[35] Robert A Schowengerdt. Reconstruction of multispatial, multispectral image data using spatial frequency content. Photogrammetrie Engineering and Remote Sensing, (6(10):1325-1334 1980.
[36] Jorge Nunez, Xavier Otazu, Octavi Fors, Albert Prades, Vicene Pala, and Roman Arbiol. Multiresolution-bans image fusion with additive wavelet decomposition. IEEE Transactions on Geoscience and Remote sensing, 37(3): 1200::2222221 1999.
[37] Chen Chen, Yeqing Li, Wei Liu, and Junzhou Huang. Sirf: Simultaneous satellite image registration and fusion in unified framework. IEEE Transactions on Image Processing, (A(11):4213-422) 2015.
[38] Junfeng Yang, Xueyang Fu, Yuwen Hu, Yue Huang, Xinghao Ding, and John Paisley. Pannet: A deep network architecture for pan- pan-sharpening. In IEEE International Conference on Computer Vision, pages 5449-5457, 2017.
[39] Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. Effective pan- sharpening with transformer and invertible neural network. IEEE Transactions on Geoscience and Remote Sensing, 60: 1-5,, 2022.
[40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEÉE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[41] Qiangqiang Yuan, Yancong Wei, Xiangchao Meng. Huanfeng Shen, and Liangpei Zhang. A multiscale and multidepth convolutional neural network for remote sensing imagery pan- sharpening. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 11(3):778-99, 2018
[42] Weisheng Dong, Peiyao Wang, Wotao Yin, Guangming Shi, Fangfang Wu, and Xiaotong Lu. Denoising prior driven deep neural network for image restoration. IEEE transactions on pattern analysis and machine intelligence, 41(10):2305-233) 2018
[43] Risheng Liu, Zhiying Jiang, Xin Fan, and Zhongxuan Luo. Knowledge-driven deep unrolling for robust image layer separation. IEEE transactions on neural networks and learning systems, 31(5):1653-1666 2019.
12

---

[44] Risheng Liu, Long Ma, Yuxi Zhang, Xin Fan, and Zhongxuan Luo. Underexposed image correction via hybrid priors navigated deep propagation. IEEE Transactions on Neural Networks and Learning Systems, 2021.
[45] Kai Zhang, Wangmeng Zuo, Shuhang Gu. and Lei Zhang. Learning deep enn denoiser prior for image restoration. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3929-392 00383, 2017.
[46] Xiangyong Cao, Xueyang Fu, Danfeng Hong. Zongben Xu, and Deyu Meng. Pancsc-net: A model-driven deep unfolding method for pansharpening. IEEE Transactions on Geoscience and Remote Sensing, pages 1-13, 2021.
[47] Jian Zhang Jiechong Song, Bin Chen. Memory-augmented deep unfolding network for com- pressive sensing. In ACM International Conference on Multimedia (ACM MM), 2021.
[48] Manya V Afonso, José M Bioucas-Dias and Mário AT Figueiredo. Fast image recovery using variable splitting and constrained optimization. IEEE transactions on image processing, 19(9):2345-233) 2356, 2010.
[49] Xueyang Fu, Zheng-Jun Jun Zha, Feng Wu, Xinghao Ding, and John Paisley. Jpeg artifacts reduction via deep convolutional sparse coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2501-2510, 2019.
[50] Dror Simon and Michael Elad. Rethinking the CSC model for natural images. In NeurIPS, pages 2271-2281 2019.
[51] Liang Chen, Jiawei Zhang, Jinshan Pan, Songnan Lin, Faming Fang, and Jimmy S. Ren. Learning non-blind deblurring network for night blurry images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10542- 10550, June 2021.
[52] Shuang Xu, Ouafa Amira, Junmin Liu, Chun-Xia Zhang, Jiangshe Zhang, and Guanghai Li. Ham-mfn: Hyperspectral and multispectral image multiscale fusion network with rap loss. IEEE Transactions on Geoscience and Remote Sensing, 58(7):4618-4628, 2020.
[53] Shuang Xu, Lizhen Ji, Zhe Wang, Pengfei Li, Kai Sun, Chunxia Zhang, and Jiangshe Zhang. Towards reducing severe defocus spread effects for multi-foeus image fusion via an optimization based strategy. IEEE Transactions on Computational Imaging, 6: 5:1561-5570, 2020.
[54] Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongxuan Luo. Retinex inspired unrolling with cooperative prior architecture search for low-light image enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10561-10570, June 2021.
[55] Risheng Liu, Yuxi Zhang, Shichao Cheng, Zhongxuan Luo, and Xin Fan. A deep framework assembling principled modules for cs-mri: Unrolling perspective, convergence behaviors, and practical modeling. IEEE Transactions on Medical Imaging, 39(12):155-4413, 2020.
[56) Risheng Liu, Zhiying Jiang, Xin Fan, and Zhongxuan Luo. Knowledge-driven deep unrolling for robust image layer separation. IEEE Trans. Neural Networks Learn. Syst, (1(5): 2020.
[57] Yang Liu, Jinshan Pan, Jimmy S. J. Ren, and Zhixun Su. Learning deep priors for image dehazing. In 2019 IEEF/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27- November 2, 2019, pages 259-2 IEEE, 2019.
[58] Wen Dou, Yunhao Chen, Xiaobing Li, and Daniel Z. Sui. A general framework for component substitution image fusion: An implementation using the fast image fusion method. Computers & Geosciences, 33(2):219-223 2007.
[59] Banach. Surles operation dans ensembles abstraits etleur application aux equations integrals, fund, 1922.
13

---

[60] Yuhui Quan, Zicong Wu, and Hui Ji. Gaussian kernel mixture network for single image defocus deblurring. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan. editors, Advances in Neural Information Processing Systems, volume 34, pages 2081 20812-20824. Curran Associates, Inc., 2021.
[61] Lucien Wald, Thierry Ranchin, and Mare Mangolini. Fusion of satellite images of different spatial resolutions: Assessing the quality of resulting images. Photogrammelric Engineering and Remote Sensing, 63:691-699, 11 1997.
[62] J. G. Liu. Smoothing filter-based intensity modulation: A spectral preserve image fusion technique for improving spatial details. International Journal of Remote Sensing, 21(1))33461- 3472, 2000.
[63] R. Haydn, G. W. Dalke, J. Henkel, and J. E. Bare. Application of the ihs color transform the processing of multisensor data and image enhancement. National Academy of Sciences of the United States of America, 79(13):571-577, 1982.
[64] W. Liao, H. Xin, V. Coillie, G. Thoonen, and W. Philips. Two-stage fusion of thermal hyperspectral and visible rgb image by pea and guided filter. In Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing, 2017.
[65] Jiajun Cai and Bo Huang. Super-resolution-guieee progressive pansharpening based on a deep convolutional neural network. IEEE Transactions on Geoscience and Remote Sensing, (9(6):5206-5222 2021.
[66] Shuang Xu, Jiangshe Zhang, Zixiang Zhao, Kai Sun, Junmin Liu, and Chunxia Zhang. Deep gradient projection networks for pan-sharpening. In CVPR, pages 1366-1375, June 2021.
[67] L. Alparone, L. Wald, J. Chanussot, C. Thomas, P. Gamba, and L. M. Bruce. Comparison of pansharpening algorithms: Outcome of the 2006 grs-s data fusion contest. IEEE Transactions 2007.
on Geoscience and Remote Sensing, 35(10):3012-300
[68] Roberta H. Yuhas, Alexander F. H. Goetz, and Joseph W. Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. Proc. Summaries Annu. Airborme Geoscí. Workshop, pages 147-149, 1992.
[69] Luciano Alparone, Bruno Aiazzi, Stefano Baronti, Andrea Garzelli, Filippo Nencini, and Massimo Selva. Multispectral and panchromatic data fusion assessment without reference. Photogrammetric Engineering & Remote Sensing. 74(2):193 2000 2008
[70] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In 20/5 IEEE International Conference on Computer Vision (ICCV), pages 1026-1034, 2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes], see Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see
S
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes], See Section
3,2 and Section/3.3
(b) Did you include complete proofs of theoretical results? [Yes]. See Section 3.2 and
Section 3.3
14

---

3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material about experimental settings are in Section 4.1 and the URL of the project will be released after the paper is accepted.
URL)? [Yes], Instructions
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report enror bars (c.g., with respect to the random seed after running exper- iments multiple times)? [No]. the effect of random seed could almost be negligible since set the same initiation seed during experiments. Reproducibility can be guaranteed.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you using existing assets (e-g., code, data, models) curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes], the data creators claim that they allow and encourage the data for scientific research.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A], We train and test the framework using the datasets following the prior works.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "yNtiSijn2Fr", "reviewer": "Reviewer_X4rs", "summary": "The authors propose an alternating reverse filtering network for pan-sharpening. Specifically, a learnable multi-scale Gaussian kernel module is designed to ensure the convergence of the iterative process. The experiments on diverse scenes show that the proposed network achieves better results compared to other pan-sharpening methods.", "questions": "How about the parameter numbers and inference time for the proposed network and competing networks?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper is well-written. The experiments show that the method can perform well in terms of quantitative analysis and qualitative comparison.\n\nWeakness:\n1. Originality\nThe originality of this work is limited. The reverse filtering is derived from [23] and the fusion network is derived from [63]. The description in 3.2, 3.3, and 3.4 sections looks quite similar to [23]. \n2. Clarity\n-The multi-scale Gaussian convolution is a group of convolutional layers initialized by 2D Gaussian kernels, but the detailed architecture of these layers is missing. What are the exact architectures of g and f?\n-The contribution states that ‘In contrast to existing model-driven methods, our iterative network can obtain HR MS without the need for pre-defined exact priors or assumptions’. It is confusing and more explanations are needed. What does the pre-defined exact prior mean in existing model-driven methods? How does the proposed method obtain HR MS without the need for pre-defined exact prior, compared to prior methods? \n3. Quality\n-In Table2, the MSDCNN achieves 0.8251 in QNR and the result of proposed method is 0.8236. However, the result 0.8236 is in bold instead of 0.8251. The proposed method only achieves the best result in one metric.\n-Observed from Figure 3, the Brovey method achieves better fusion results in spatial texture compared to proposed method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cbl6gaPu18", "reviewer": "Reviewer_vNgg", "summary": "This paper proposes an alternating reverse filtering network for multi-spectral image fusion. Motivated by classical reverse filtering, the authors develop an unrolling network based on alternating fixed-point iterations and the learnable multi-scale Gaussian kernel module, which is interpretable. Unlike existing model-driven methods, the proposed iterative network avoids the dependency on pre-defined exact priors and its convergence can be guaranteed theoretically. In addition, this paper introduces a new perspective to solving the image fusion problems. Extensive experiments on diverse scenes to thoroughly verify the performance of the proposed method.", "questions": "1. The structure loss $L_s$ is based on structural similarity, what will happen if loss function $L_s$ uses the L2 loss function?\n2. If each alternate iteration output is supervised with losses, would it be better?\n3. In the ablation study about the number of iterations K, why does the performance seem to decrease slightly when K reaches 7 compared with K=5? What happens to the model's performance in the case of the larger K? \n4. As far as I know, the GPPNN network compared in the manuscript is also based on an unrolling method. What is the K used in GPPNN here? If the settings of K are different, how to ensure fair comparisons of the two unrolling methods?\n5. As mentioned in the Weaknesses, the authors are suggested to provide specific parameters for initializing the Gaussian kernel.", "limitations": "The limitations and potential negative social impact of their work have been addressed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The authors innovatively propose a new multi-spectral image fusion method without requiring well-designed priors. The exploration of tailoring classic methods for multi-spectral image fusion and combining them with deep networks is worth trying and encouraging.\n2. In the image fusion community, the idea that formulating pan-sharpening as an alternately iterative reverse filtering process is interesting. Other researchers could utilize it for further exploration in the field of image fusion.\n3. The learnable multi-scale Gaussian kernel module is well designed and the convergence of the iterative process is sufficiently supported by theoretical analysis. \n4. Quantitative and qualitative results clearly verify the effectiveness and scalability of the proposed method.\n\nWeaknesses:\n1. More ablation results about the loss functions and iterations should be provided. Please refer to the detailed Questions part.\n2. The parameters such as sigma for initializing the Gaussian kernel should be given in line 239.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9icFk68Lfdh", "reviewer": "Reviewer_T9TL", "summary": "The manuscript first formulate the pan-sharpening problem as a reverse filtering process and propose an alternating reverse filtering network (ARFNet) to solve it. Besides, the authors develop multi-scale Gaussian convolution modules to obtain better filtering effect. Experimental results on both simulated and real-world scenes show that the proposed ARFNet outperforms other methods.", "questions": "1.\tAs stated in [1], the optimal iteration number is different for variant filters and needs to be set empirically. Is the iteration number set equally in the ablation study of different initialization methods? \n\n2.\tThe learned kernels are close to the initialized Gaussian kernels as stated in Line 241, can the learned kernels gain large performance improvement compared with the initialized ones? \n\n3.\tThe authors cited the recent method[2], but not compared the experimental results with it.\n\n4.\tHow about the computation cost and efficiency compared with other methods?\n\n5.\tIt’s better to include related source citation for the definition, the theorem and reverse filtering formulation.\n\n6.\tPlease check the subtraction and addition symbols in Fig. 1, they are a bit confusing.\n\n[1] Xin Tao, Chao Zhou, Xiaoyong Shen, Jue Wang, and Jiaya Jia. Zero-order reverse filtering. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 222–230, 2017.\n[2] Man Zhou, Xueyang Fu, Jie Huang, Feng Zhao, Aiping Liu, and Rujing Wang. Effective pan-sharpening with transformer and invertible neural network. IEEE Transactions on Geoscience and Remote Sensing, 60:1–15, 2022.\n", "limitations": "There is no potential negative societal impact mentioned. Considering the paired dataset is expensive to obtain, the authors could also explore methods in unsupervised manner.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1.\tThis manuscript combines the deep unrolling model and the reverse filter process, and proposes a novel alternating update deep convolution network for pan-sharpening task.\n2.\tExperimental results show that this manuscript set up some new state-of-the-art results in both simulated and real-scene datasets.\n3.\tThe writing is clear and easy to understand.\nWeakness:\n1.\tThe reverse filter method has been already applied in super-resolution, deconvolution in [1], the Gaussian filter and its convergence condition has also been analyzed. Overall, the novelty is limited.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept