Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion
Authors: Yujie Liang
Zihan Cao
Shangqi Deng
Hong-Xia Dou
Liang-Jian Deng
Venue: NeurIPS 2024 poster (2024)
Keywords: Image fusion
Implicit neural representation
Fourier representation
Primary area: machine_vision

Abstract: Recently, implicit neural representations (INR) have made significant strides in various vision-related domains, providing a novel solution for Multispectral and Hyperspectral Image Fusion (MHIF) tasks. However, INR is prone to losing high-frequency information and is confined to the lack of global perceptual capabilities. To address these issues, this paper introduces a Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for MHIF task, targeting the following phenomena: The Fourier amplitudes of the HR-HSI latent code and LR-HSI are remarkably similar; however, their phases exhibit different patterns. In FeINFN, we innovatively propose a spatial and frequency implicit fusion function (Spa-Fre IFF), helping INR capture high-frequency information and expanding the receptive field. Besides, a new decoder employing a complex Gabor wavelet activation function, called Spatial-Frequency Interactive Decoder (SFID), is invented to enhance the interaction of INR features. Especially, we further theoretically prove that the Gabor wavelet activation possesses a time-frequency tightness property that favors learning the optimal bandwidths in the decoder. Experiments on two benchmark MHIF datasets verify the state-of-the-art (SOTA) performance of the proposed method, both visually and quantitatively. Also, ablation studies demonstrate the mentioned contributions.  The code can be available at https://github.com/294coder/Efficient-MIF.

Fourier-enhanced Implicit Neural Fusion Network for Multispectral and Hyperspectral Image Fusion
Yu-Jie Liang
Zihan Cao University of Electronic Science and Technology China wamzihan666622e com
University of Electronic Science and Technology of China rujieliangho 96 gma com
Shangqi Deng Xi'an Jiaotong University shanggideng00224egg 24@gma11.m1mmmmmmmmm com
Hong-Xia Dou Xihua University
hongxlladou1911112..0
Liang-Jian Deng"
University of Electronic Science and Technology of China angj an www.guestc.cceemmmmoo edu.cn
Abstract
Recently, implicit neural representations (INR) have made significant strides in various vision-related domains, providing a novel solution for Multispectral and Hyperspectral Image Fusion (MHIF) tasks. However, INR is prone losing high-frequency information and is confined to the lack of global perceptual capa- bilities. To address these issues, this paper introduces Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for MHIF task, targeting the Following phenomena: The Fourier amplitudes of the HR-HSI latent code and LR-HSI are remarkably similar; however, their phases exhibit different pat- terns. FeINEN, we innovatively propose spatial and frequency implicit fusion function (Spa-Fit Fre IFF), helping INR capture high-freeg freh-frreuency information and ex- panding the receptive field. Besides, a new decoder employing complex Gabor wavelet activation function, called Spatial-Frequeng: Interactive Decoder (SFID), is invented to enhance the interaction of INR features. Especially, we further theoretically prove that the Gabor wavelet activation possesses time-frequency tightness property that favors learning the optimal bandwidths in the decoder. Ex- periments two benchmark MHIF datasets verify the state-of-the-an (SOTA) performance the proposed method, both visually and quantitatively. Also, abla- tion studies demonstrate the mentioned contributions. The code can be available at https///ghhub..om///9ccdddfffffn MIF
Introduction
Hyperspectral imaging captures scenes across contiguous spectral bands, offering intricate details compared traditional single limited-band images, and improving computer vision application accuracy, such as target recognition, classification [47, tracking, and segmentation [12 38
39 37 44
48|49] However, practical optical sensors face challenges in balancing spatial resolution and spectral precision. Images with over 100 bands often exhibit lower spatial resolution, while those with fewer
'Equal contribution. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

49.5
Better Performnne
Better Performmnnce
49.0
Ours
Ours
Panerters
DSP
Paramters
MIMO-SST
PSNR(Hanvrrd
48.5
HERNE
(HN x8)
46
DSP
48.0
-SSRNet
PSRT
MogDCN
"SSRNet
Fusformer
PSRT Fustorme
MogDCN
47.5
DHIF
3DT-Net
47.0
MIMO-SST
HSRNet
DHIF
3DT-Net
46.0
ResTFNet
ResTFNet
45.5
45
43
49
51
PSNRE x4)
PSNR(CAVE x8)
Figure 1: Comparison of our method with other methods on the CAVE(X 4, 8) and Harvard( x 4, x 8) datasets. Closer to the top-right corner indicates better performance and the size of the circle indicates the number of parameters in the model.
bands display higher spatial resolution. Efforts for MHIF are underway to fuse high spatial-resolution multispectral images (HR-MSI) with low spatial-resolutinn hyperspectral images (LR-HSI) to finally obtain high spatial-resolution hyperspectral images (HR-HSI). Actually, MHIF technology could fuse hyperspectral images with multispectral images, extracting information not detectable by MSI to
enhance richness and precision. Recent MHIF literature explores model-based approaches g 9 45
and deep learning methods [17 TO 3 54). While model-based methods leverage image priors, challenges persist in obtaining high-fidelity, low-distortion HR-HSI due to the lack of large-scale training datasets. Among deep-learning approaches, CNN- bassed networks for HR-MSI and LR-HSI tend be limited and lack interpretability for MHIF tasks and Transformer frameworks [15] address the small receptive field of CNN but bring greater computational overhead.
In recent years, implicit representations of 3D scenes have garnered significant attention from researchers. For instance, Neural Radiance Field 433 models 3D static scenes by mapping coordinates to signals through a neural network. Inspired by this, researchers have revisited image representation for 2D tasks. Recent studies [5] replacing commonly used upsampling layers with local implicit image functions. Though these methods demonstrate superior performance in 2D tasks, they still have some drawbacks. Firstly, INR calculates the RGB values of a queried coordinate based on the relative distances to the surrounding four pixels, treating it as a local operation in space that lacks consideration for global information. Additionally, the MLP-ReLU structure used in traditional INR inherent high-frequency information bias 29 which is challenging be eliminated during training.
20 34 4 have achieved arbitrary-scale super-resolution (SR) by
To address these issues, we propose implicit fusion functions tailored for the MHIF task as a novel fusion paradigm. We first employ encoders to extract prior information from LR-HSI and HR-MSI, which is then fed into the implicit fusion functions in the form of latent codes. Unlike traditional INR, we transform latent codes into the Fourier domain and simultaneously perform spatial and frequency fusion in a unified network. This approach not only rectifies the high-frequency insensitivity induced by the MIP but also effectively extends the receptive field, encompassing scope of global information. To integrate spatial and frequency domain representations efficiently, we design decoder with time-frequency tightness, mapping features on both domains to pixel space. The contributions of this work are three folds:
comprehensive
We define a novel fusion framework based on INR, which innovatively extracts information from the spatial and Fourier domains, effectively enhances the representation ability of high-frequency information, and expands the receptive field.
We propose a new decoder employing Gabor wavelet activation function to enhance the interaction of INR features. Furthermore, we theoretically prove that the complex Gabor wavelet activation possesses time-frequency tightness property, which facilitates the decoder in learning the optimal bandwidths.
The proposed network reaches state-of-the-arr (SOTA) performance on the MHIF task across two widely used hyperspectral datasets at various fusion ratios. Fig. Tig.Tpm provides fair comparison with other SOTA methods.

---

FIR 715
COMEY
Amplitude
Amplitude
Phase
(a)
(b)
Figure 2: (a) The amplitude of latent code from the encoder fed by HR-HSI and LR-HSI (combined with HR-MSI) share a similarity, but the phases differ from each other. Exg- is trained encoder. (b) 3 x convolution would suffer from the issue of spectrum leakage, which can be alleviated by x convolution.
2 Related works
Implicit Neural Representation Unlike traditional discrete representations, neural implicit repre- sentation (INR) provides more elegant and continuous parameterized approach. Initially applied in 3D modeling tasks, NeRF 43 revolutionized 3D computer vision by representing intricate three-dimensional scenes with just 2D pose images. This line of work extends to the 2D imaging domain, where INR performs a weighted average on adjacent sub-codes to ensure output value continuity. LIIF (5] recently introduces local implicit image function for SR, leveraging MLP to sample pixel signals across the spatial domain. Several improvements focus on decoding networks; for example, UltraSR (46) incorporates residual networks, merging spatial coordinates and depth encoding. DINN (26] utilizes a dual-interactive implicit neural network to decouple content and position features, improving decoding capabilities. JIIF (36] proposes joint implicit image functions for multimodal learning, extracting priors from guided images. Regarding activation functions in the MLP, SIREN (34 recommends utilizing periodic activation functions for continuous INR fit complex signals. On the other hand, WIRE (32] further employs continuous complex Gabor wavelet activation functions to activate non-linearity, focusing more on spatial frequencies. However, there limited research dedicated to designing INR architectures specifically for the MHIF task. The unique characteristics of hyperspectral images pose challenges for INR networks, in their insensitivity to high-frrquency information.
Latent Enhancement by Fourier Transform Fourier transform a commonly used time- frequency analysis technique in signal processing, which converts signals from the time domain to the frequency domain. The Fourier domain has global statistical properties, and in recent years, many works use the Fourier transform enhance the representation ability of neural networks. For example, FDA 53 proposes exchanging amplitude and phase components in Fourier space between images to enhance and adjust frequency information. FFC 6 introduces novel convolution module that inter- nally fuses cross-scale information to capture global features in Fourier space. Similarly. GFNet [30] uses 2D discrete Fourier transform to extract features, implements learnable global filtering, and replaces the self-attention layer in Transformer. UHDFour [21] embeds Fourier transform into the image enhancement network to model global information. Together, these studies demonstrate utility of frequency domain information in improving performance on visual tasks. We exploit the architecture of FeINFN to transform latent codes into the frequency domain, implicitly integrating representations of amplitude and phase components, and enhancing high- figeffrryyency injection.
29 finds that most neural networks exhibit a phenomenon of spectral bias through Fourier analysis. This includes neural networks such MLP, which tend to learn low-frequency information during the early stages of training and are insensitive to high-frequency information. Moreover, we found this issue occurs in the MHIF task according to an experimental analysis as shown in Fig. 2 a). where HR-HSI and LR-HSI were concatenated with HR-MSI and fed into trained encoder obtain latent codes. These codes were transformed into the frequency domain to visualize the amplitude and phase. It can be observed that the amplitudes from HR-HSI and LR-HSI are very similar, while the phases differ significantly. The phase of HR-HSI should naturally contain more texture than LR-HSI, hypothesis validated by the visualized phase maps. Based on this finding, we transformed the latent codes into the Fourier domain to separately process amplitude and phase, to enhance the global learning of high- high-freeuency information in the images.
Motivation
3

---

Methodology
In this section, we first present the preliminary of INR and then provide the proposed framework tailored for MHIF task. Subsequently, elaborate the implementation details of the composited modules the proposed FeINEN.
Spatial Domlan
Kc
Zap RAmic
App
Fx
Zypo tronnd Axpe
Egg
Fourler Domian
Ie ge
Encoder
Imptict Noundl
SFID
Pixel by Pixel
Channel-wise Concal
Upsample
Element-mise Addition
Spa-Fre IFF
Positional Embadding
X
Figure 3: The flowchart of the FeINFN framework which is composed of spectral encoder Ex, spatial encoder Eg, MHII task-designed spatial and Fourier domains implicit fusion functions, and pixel space mapping decoder. Please note that ILR is the LR-HSI, I the HR-MSI, ILR is the bicubic interpolation LR-HSI, and X H is the HR normalized 2D coordinate map. Zapes Zepas Zhp. ōx correspond to individual pixel units, A and P represents amplitude and phase, respectively.
up
3.1 Preliminary: Implicit Neural Representation
Neural Radiance Fields 43 is represented by integral construction scenes. The value of pixel in a certain viewing angle image regarded as the integral of the characteristics of the sampling point from the proximal end to the far end of the ray. During actual training, the integral needs to discretized. Extended to 2D image representation (5), it is sampled pixel by pixel from vicinity of the query target. Taking the low-resolution (LR) image E 2xx=x3 upsampling the high-resolution (HR) image E R H xxxxxx xxx3 of the target coordinates X, E R can be regarded as interpolation form, expressed as:
example, the process of generating the RGB values
i(xx) = Σ Va.i
REN,
where Va.a E 1 is the interpolation pixel of interpolated by g's surrounding pixels N₁, R1 and W E R signifies the interpolation weight. In the implicit representation of local image features, the weights Wy, S.SS, where S, represents the area formed by q and in the diagonal region and S denotes the total area enclosed by the set Ng. The interpolation value Vq.á is effectively generated by a basis function:
V Vg.i (z X,, x:);
where de is typically an MLP, z, is the latent code generated by an encoder for the coordinates x,, and x, x, represents the relative coordinates. From the above equations, can be inferred that the interpolation features can be represented by a set of local feature vectors in the LR domain. Typically, interpolation-basee methods [28 18 achieve upsampling by querying X2 X; in the arbitrary SR
task. See more details in 5
3.2 Overview of the FeINEN Framework
In this work, we propose the FeINFN, which adopts novel framework for simultaneously performing neural implicit representation in both the spatial and frequency domains execute the MHIF task. Fig. provides an overview of the proposed framework, designed to fuse LR-HSI E R #xx 5 and HR-MSI HR HxW generate HR-HSI I R XS based on upsampling scale

---

Initially, the LR-HSI is fed into encoder Ex to extract spectral features Z E RAxxxC Simul- taneously, the concatenated bicubic interpolation LR-HSI upp E " x and I are fed into encoder Eg to extract spatial features Z spu E "x Additionally, the pixel's central position is represented as the coordinate point. The coordinate map is normalized into a two- troo-imensiooal grid [-1,1] x 1-,,1, obtaining HR normalized 2D coordinate map X R E " x2 The extracted Ssp and Spp along with the 2D coordinates of I are forwarded to Spatial-Frequency Implicit Fusion Function (Spa-Fre IFF), outputting spatial domain features & E R W x and frequency domain features E R H W xx The & and & inputs pixel space mapping decoder which generates the residual image I R x Finally, the residual image IlA is combined with the bicubicly upsampled image I via element- wise addition, yielding the ultimate fusion image I.
up
3.3 INR Encoder Networks
Analogous to local implicit representation functions 5
20 34 4. the initial step involves extracting
latent code representations. For the MHIF task, we address the challenges of both upsampling and fusion simultaneously, employing implicit neural representations as the solution.
The INR encoders try to extract spatial and spectral latent codes Zepa E B&H Zape E h one is extracted from IL8, serving as the carrier for spectral information; the other encoded from the concatenation of TLR and IR aiding in spatial information during the fusion process. This up
process can be denoted as:
Zspe Fi (TLR), Zspa Enp (Ta IHR)),
where Ex is the spectral encoder parameterized by x. Es is the spatial encoder parameterized by v,, and Cat IHR) denotes the concatenation along the channel dimension. In practice, we utilize EDSR 23 as INR encoder networks.
3.4 Spatial-Frequency Implicit Fusion Function
To address the mentioned issues 2] we propose Spatial-Frequency Implicit Fusion Function, dubbed Spa-Fre IFF which is a dual-branch fusion function and utilized for computing the fusion feature of Zope and Z spa in the spatial and frequency domains, respectively. Given a queried HR coordinate Xg E XHR of pixel unit g, Spa-Fre IFF estimates spatial feature vector e, RIx1xS (E, E.) and frequency feature vector E TR. R1xxxs
HR
(ef Eg) as follows:
Ex, E Spa-Fre IFF IFF Eapay 5x),
where Zape E IxIIxC represents the spectral latent code vector corresponding to X,, and Zepa E RAxxxC is the spatial latent code vector. 5x denotes the set of local relative coordinates, expressed by the following formula:
8x
{xg xg.
where Xy.s refers to the coordinates most proximate to the query coordinate Xg. representing the four corner pixels closest to in the HR space.
Spatial Implicit Fusion Function The Spatial Implicit Fusion Function aims to leverage the powerful representation capabilities of INR to achieve implicit fusion in the spatial domain, as shown Fig. 3 (see branch "Spatial Domain").. Specifically, we employ high-pass operators H filter the spectral latent codes, complement to the high-frequency information on the spectrum:
Zhp H (Zape)
(6)
where Zap 1xxxxCC represents the high-frequency latent code of LR Also, we suggest frequency encoding for relative positional coordinates follows:
7 (fx) [sin 6x), (2) fx),... sin(2"-1 cos(22-15xx))
where is hyperparameter, practice, we set T. to 10. Additionally, leveraging the graph attention mechanism [36]. we parameterize the solution for interpolation weights Wg, I and the
5

---

implicit fusion function simultaneously outputs fusion interpolation values Vg.á E R and interpolation weights Wg. W.. The implicit fusion function is specifically expressed as:
Wg.5 Vg.. = t0o (sspe Rspa Zhp: 7(6x),
(8)
where dh is MLP parameterized by 0. The weights used for interpolation need pass through softmax function, obtaining normalized weights Wq. The spatial implicit fusion interpolation. as shown in Eq. T yields the fused spatial feature E, E RIx1xs and can be described as follows:
E Wg.t Vggi-
IEN
Frequency Implicit Fusion Function From Fig. 2 a). we observed characteristies in the frequency features between LR-HSI and HR-HSI. Hence, we design a frequency implicit fusion function to express global features continuously the Fourier domain. Notably, directly applying static kernel convolution in the frequency domain would only enhance specific frequency range, which is inappropriate for the fusion task. However, by learning feature content generate weights, INR can be seen as dynamic interpolation method in continuous space, adaptively enhancing information in the frequency domain without overly altering the frequency distribution. Therefore, introducing INR into the Fourier domain is reasonable. Since amplitude and phase exhibit different forms, as shown in Fig. a), we handle them separately.
With the considerations mentioned above, as illustrated in Fig. (see branch "Fourier Domain"), "), we initially employ FFT transform latent codes Zape and Zspa from the spatial domain the frequency domain, obtaining Sape E IR 1xxxxxC and Espa 1I1x After the transformation, we further obtain amplitude components Afsspe and A(fspo), as well as phase components P P(fspe) and P(fspo).
For the amplitude, as shown in Fig. 2(b), the amplitude distribution of LR-H RSII and HR-HSI are very similar, and the non-point-wist convolution (e.g. Conv 3 x 3) causes an issue of spectrum leakage, confusing channel information. In contrast, point-wise convolution does not span multiple locations in the frequency domain and has no overlap allowing to capture information across channels effectively. Thus the fusion function for amplitude components is more suitable when applying point-wise convolution:
w v-A = (A((epe), 5x), 9,13 9.
(10)
where w-A. RIxs and v.A. E 1x4xx are the weights and interpolated values for the corresponding 9, amplitude component, and de is a simple network composed of two layers of point convolutions parameterized by a. Similar to operations in the spatial domain, implicit fusion interpolation is performed after obtaining interpolated values Vo.1 and the normalized weights w. g.3
q.2
A'
(11)
9.
iEN
where A' E RIIIXS is the integrated amplitude component.
For the phase, which encapsulates information such texture details, LR-HSI and HR-HSI often have different phase information. It is known that point convolutions fail to capture sufficient spatial representations. Therefore. we use a 3 convolution to learn phase information. Additionally, small changes in the frequency domain may result in significant variations in the spatial domain. We still consider using the form of INR interpolation for phase learning. The handling of the phase components P P(fspe) and P(f. are formally similar to Eqs. 10) and
w²
P P g:s: 9,6
(P((pe) P(fspa)) 8(x)),
(12)
2,
9.
RNN,
The simple network of consists of two layers of 3 convolutions parameterized by B. P/J [ x1x1xs represents the integrated phase component.
Finally, IFFT is applied map the frequency features A', and P' back to the image space, obtaining the frequency domain feature , E Ey. Since in frequency space, one frequency point may correspond to multiple pixels at different positions in the spatial domain, the receptive field INR the frequency domain is enlarged in the spatial domain.

---

3.5 Spatial-Frequency Interactive Decoder
After obtaining the spatial feature map and frequency domain feature map, it is essential to consider how to integrate them seamlessly. Firstly, our decoder needs to have dual input and interactive capabilities. Secondly, is nec- essary to focus on representing images in the spatial-frequency domain. With this mind, we introduce the complex Gabor wavelet acti- vation function with good time-frrend frequeryy tight- Interac- ness and propose the Spatial-Frequene, tive Decoder (SFID). Specifically, SFID consists of three layers, taking spatial and frequency domain features as inputs. The outputs I.R and I
comu
ix1
ixl
comn
cons
1s1
com3
Ixt
SFID
RELU
Channel-wise Concal
Gabor
Figure 4: Detailed composition of the proposed SFID.
contribute to the final fused image I. The decoding process is illustrated in Fig. 4 The complex Gabor wavelet function defined as:
((x) edwox eiwoxe-luox/?
(13)
where we is the center frequency in the frequency domain, Vo is constant that is consid- ered as the standard deviation of the Gaussian function, and x is a vector in the time (or spa- tial) domain. In what follows, we provide theorem below that this Gabor wavelet activa- tion has fime-frey tightness M1 which is helpful for the decoder's ss information interaction. Theorem 1. The complex Gabor wavelet activation in Eq. (13) has the freeureyyy tightness property (more preliminary can be found in (II). Moreover, from the perspective of signal spectrum analysis, this activation helps the decoder learn the optimal bandwidths.
Proof: First, for the dimed, the function [9(x)] in the time domain is primarily concen- trated around x = 0 due to the exponential de- cay term. The Gaussian term e | ensures that (xx) is bounded and rapidly decreases in
Compbex Gubor Perction
Touste Terrafor Cater Activatin
the time domain. Second, for the frequency- domain Tightness, the Fourier transform given by:
e-jxx dx. (14) (a) Time-domain
(b) Frequency-domain
F(g(x)) = ci
| |loxxl e jux
Frequency of
The Fourier transform of the Gaussian term -vox/| remains Gaussian function, and its bandwidth in the frequency domain is influenced by vo- Due to the characteristies of the Gaus- sian function in the frequency domain, G(x) is mainly concentrated around w wi. Com- bining the narrow-bandwidth properties in both time and frequency domains, we can apply the uncertainty principle to demonstrate the time- frequency tightness of the complex Gabor func- tion:
optimal
(c)
bandwith
500
Frequedcy mema feature in decoder using Gabor activation
1500
1000
Near-optimal
(d)
500
Frequency mean feature decoder using ReLU activation
750
Sub-optimal bandwith
500
250
|wol - vp
1
(15)
0.05
Angle frequency (w)
0.15
022
0.55
4T
where Up is the time-domain bandwidth, and (wo] is the frequency-doman practical training, we provide initial set of bandwidths and allow the network to learn the optimal bandwidths, which concludes the proof.
Figure 5: The complex Gabor wavelet function. (a) and (b) depict the visualization of the complex Gabor wavelet function. (c). (d), and (e) represent the frequency responses of GT and decoder's 's mean feature using Gabor and regular ReLU activations. respectively.
bandwidth. In
As depicted in Fig. 5) the frequency response of the decoder with Gabor wavelet activation closely ap- proximates the optimal bandwidth. Moreover, the decoder with Gabor activation achieves consistency with GT in frequency, demonstrating rapid frequency alignment.
7

---

Table 1: The average and standard deviation calculated for all the compared approaches on 11 CAVE examples and 10 Harvard examples simulating a scaling factor of 4. The best results are in bold, second-bee best in underline. "M" refers millions.
CAVE xd
Harvard x1
Methods
P)
SAM())
ERGAS()) SSIM(1) # #parums
PSNR(1)
SAM()
ERGAS(.) SSIM())
#params
Bicubic
34,33=3.88
4.451:662
7.21+4.90
0,94410.029
38.7114.33 0.6 4.4511881 0,948=0.027
CSTF-FUS
34.4614.28 14.37:5.300
8.29:5529 0.866.10.075
39.15:13.45 6.3 4.6611.81 0.914=0.049
LTTR
9
35.85.1..49 6.99:12.55
5.99 0.956
40.88:13.94 4.01 1.27 4.03 0.957-0.033
LIMR 8
36.54=3.30 6.71#2.19
5.39+2.53
0.963=20.021
42.06::3.563.51 0,59 3,59:2.03 0.970=0.020
IR- 1R-TenSR 45
35.61:3.44 12.30:14.68 4.68
5.90+1305 0.945:10.027
40.47:33044
4,36:1 1.52 5,57 0.963=0.014
Rese 24
45.58:3344 2.82+0.70
2.36+2..59 39 0,993+40.000 2.387M
45, 45.94:44.35
2.61=0.69
2.56+ 0.985=0.008 2.387M
SSRNet
S
48.62:3.92 2.54:0.84
1.63+1.21
0,995:10.002
0.027M
15.00::3..62.331:222.
2.30 42 0.987=0.007 0.027M
HSRNet
16
50.38:13338 2.23:1066
1.20:1.775
0,996:10.001
0.633M
48.29:13.03
2.26:10.56
1.8710..1 0.988=0.006 0.633M
MogDCN
10
51.63.14.10 2.03:10.62
L1110.8
0.997.10.002 6.840M
47.89.144.09
2.11:10.52
1.89:10.82 0.988-0.007 6.840M
Fusformer
[5]
49.98=8.10 2.20=#0.85
2.50=5,22 21 0.994+00011 0,504M
47, 47.87=5.113 13 2.2.11122222200000 2.07 2.04 0.986=0.010 0.467M
DHIF TV
$1.07=4.17 2.01:1063
1.22:00.97 0,997 002 22, 462M
47. 47.68:13.855
47.66::3..85..222000. 0.32=#0,53 1.95 002.9220.9900 0.988-0.007 22,462M
PSRT 7
50.47:16.19 2.19.10.4
2.06:13.71 71 0.996.10.003 0,247M
47.96.13.21 2.18:10.55 1.89 0.006 0.247M
Net
51138:14.18 2.16.10.70
1.14 1,00
0.906-100003 3.464M
47.78:14.42
2.04 0.51 1,98
0.989 0.006 3.464M
DSPNet 35
51.18+3.92 2.1540.64
1.13+0082
0.997+00.02 6.064M
48.29-+33. 2.30++0.55
1.93++0.93 0.006- 6.064M
MIMO-SST
|0]
50.98=3.39
2.23++0.70
1.18+0.73
0.997++0.002
4,983M
47.08+55.56
2.09=0.533
2.07+00.82 O 0.988=0.007
4,983M
FeINFN(Ours)
52.47 14.10 1.91
0.98
1..74 0.998 10,002
3.165M
49.06 1..15 10.53
1.78
L. 0.75 989-0.000
3.165M
4
Experiments
Datasets To evaluate the efficacy of our model, we conducted experiments using the CAVE and Harvard datasets. The CAVE dataset comprises 32 Hyperspectral Images (HSIs) with 31 spectral bands spanning from 400 nm to 700 nm at 10 nm intervals. We randomly selected 20 images for training and used the remaining 11 for testing. The Harvard dataset consists of 77 HSIs depicting indoor and outdoor scenes, covering the spectral range from 420 nm to 720 nm. We standardized the data by cropping the upper left sections of 20 Harvard images, with 10 for training and the rest for testing. The simulation of data can be found in Appendix.
Implementation Details We implement the proposed method FeINFN with Pytorch 227 on a workstation with an Intel 19 CPU and two 3090 GPUs. The optimizer is chosen as Adam W [9] and we use a Cosine anneal learning rate scheduler. The base channel number of the encoder is 1 128, that of the proposed implicit fusion function is 32 and in the decoder, the channel number is 31.
Results on CAVE Dataset In this section, we evaluate the effectiveness of FeINEN on the CAVE dataset and compare it with five traditional methods and some state-of-the-ar deep learning-basee approaches. As shown in Tab. [ on the left, our method achieves optimal performance in the tasks of x4 in all metrics. In the x4 experiment, compared to currently leading methods such as DSPNet (35,, 3DT-Net (25], and MogDCN [0], our approach demonstrates improvements in PSNR by 1. d ..9.B/0. 84dB, respectively. Notably, our method exhibits even more pronounced superiority in the x8 experiment, showcasing good generalization across various resolutions. To illustrate the advantages of our method, we provide visual comparisons in Fig. 6 including close-ups and error maps to highlight specific details. Our fusion results closely match the ground truth, achieving the best quality. In comparing error maps, the darker colors indicate closer proximity to the original image. In contrast to other excellent methods, the error maps of FeINFN distinctly exhibit superior restoration effects on details.
Results on Harvard Dataset In Tab. [ the right columns present the comparison results of our FeINFN with other methods on the Harvard dataset at scale factors 4. Our method performs excep- tionally well, with only SAM being slightly surpassed by 3DT-Net 255. FeINFN exhibits significant gains in PSNR/ERGAS metrics compared to the current -fate-of-the-an [66), with improvements of 0.77dB/0.06 respectively. The results with scale factor of 8 can be found in Appendix. As depicted in Fig. 0 our model outperforms others, highlighting the crucial role of FeINFN's continuous representation capability in high-scale factor scenarios. To better visualize the performance gap, Fig- Gilllustrate the fused images and error maps, confirming that our FeINEN maintains high fidelity in recovering the texture details of the images.
8

---

C019
CAVE
1000
Harvard
1100
0,000
(4)
(b) Ours
(e]MoGDUN
(c)
(6) DSPNet
(M) DHIF
(2) MIMO-SST
PSRT
Figure 6: The upper and lower parts respectively showcase the results of "Chart and Stuffed Toy" from the CAVE dataset and "Backpack" from the Harvard dataset using pseudo-cole representation. Green rectangles depict some close-up shots. The second and fourth rows show the residuals between the ground truth (GT) and the fusion products.
Table Quantitative comparisons with other up- sampling methods on the CAVE (x4) dataset.
Frequency
Methods
PSNR(()
SAMIJI
ERGASILI
SSSMIT
#params
1.95:061
0,99710.002
52.4754.00
181:0.9
0.8:00.44
0.998:4.0015
3,1654
PSWR (d8)
Table 3: Quantitative comparisons with reduced models on the CAVE (x4) dataset. & F mean the domain difference.
PSNRU))
SAMUL
SSIMLt)
#params
18510.59
104.0.52
3.921330
3.165M
40
Table Quantitative comparisons with different activation functions in SFID the CAVE (x4)
120k
160k
210k
Iterations
dataset.
Figure 7: Changes in PSNR on the CAVE dataset of our FeINFN over iterations with and without the "Fourier Domain". The Frequency IFF can help the network learn the high-frequency details and converge faster.
Nonlinear
PSNRIT)
SAM()
ERGASC.)
SSIM(1)
ReL.I GELU
52.03-3.84
2.00+0.59
1.02400.4
0.998-0.0013
$1.922388
200+00.0
10344075
0014
Leaky RcLU
$1.923992
20010:00
1003:0.6
0.98=000044
524474410
1.91-4159
0.98-0.74
0.998-00005
4.1 Ablation Studies
Upsampling Methods Implicit image representation can be seen as an advanced interpolation algorithm, offering additional spatial information and parameterized weight generation. In this section, we compare INR with other upsampling methods. We replace INR with pixel-shuffle and traditional CNN interpolation methods, presenting a comparative analysis. As seen in Tab. approach outperforms other methods in MHIF tasks.
2 our
To assess the dual-domain model's efficacy, performed model reduction, preserving spatial and Fourier domains independently. As shown Tab. 3 FeINEN excels by using both spatial and Fourier domains concurrently, underscoring the positive impact of Fourier domain integration on overall network performance.
Spatial Domain and Fourier Domain
Spectral deviation occurs during training, where the network tends to prioritize low-frequency information, capturing high-frequency details only in later stages. To validate our resolution of this issue, remove the "Fourier Domain" from Spa-Fre IFF, retain it, and the corresponding training

---

data is illustrated in Fig. 7 Our FeINFN, which incorporates Fourier domain fusion, leads to faster PSNR convergence and overall higher efficiency. The visual comparison of high-freet seeh-freuuency details in "chart and stuffed toy" from the cave dataset at 80k iterations further supports the significant improvement achieved with our results.
Decoder with Different Nonlinear In this section, we evaluate the impact of different activation functions in SFID, aiming to match SFIFF. Our dual-input decoder incorporates a complex Gabor wavelet activation function facilitate the fusion of spatial and frequency domain features.
Through experiments, we replaced the Gabor wavelet activation with other activations, presenting the results in Tab. 4 The findings distinctly demonstrate the enhanced fusion quality achieved with the complex Gabor wavelet activation. This emphasizes the critical role of wavelet activation in promoting robust and reliable learning in SFID.
5 Conclusion
Inspired by the distinct behaviors of LR-HSI and HR-HSI in the Fourier domain, we introduce novel Fourier-enhanced Implicit Neural Fusion Network (FeINFN) based on INR. Through Fourier transformation, latent features are converted into the frequency domain, allowing the modeling of frequency components to enrich seehffreyencc information images. Additionally, we propose a frreunnccy decoding module, achieving unified representation of both spatial and frequency domains using a femme-freuency--eee it outperforms state-of-the-ar methods in MHIF with appealing efficiency. We desire that our work will inspire future research on frequency fusion-basee MHIF methods.
activation function. Thanks to the unique design of our network,
6 Acknowledgement
This work is supported by the National Natural Science Foundation of China under Grants 12271083, 12171072 and Natural Science Foundation of Sichuan Province under Grants 2023NSFSC1341.
References
[1] Thierry Blu and Jérôme Lebrun. Linear Time- Frequency Analysis Wavelet-Type Representa- lions, chapter 4, pages 93-130. John Wiley Sons, Ltd, 2008
[2] Tamal Bose and Francois Meyer. Digital signal and image processing. John Wiley Sons, Inc., 2003.
[3] Zihan Cao, Shiqi Cao, Liang-Jian Deng, Xiao Wu, Junming Hou, and Gemine Vivone. Diffusion model with disentangled modulations for sharpening multispectral and hyperspectral images. Inf. Fusion 104:102-158, 2024.
[4] Hao- Wei Chen, Yu-Syuan Xu, Min-Fong Hong, Yi-Min Tsai, Hsien-Kai Kai Kuo, and Chun Chun-Yi Lee. Cascaded local implicit transformer rrbitrary-scale super-resolution. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 18257-18267, 2023.
[5] Yinbo Chen, Sifei Liu, Xiaolong Wang. Learning continuous image representation with local implicit image function. In IEEF/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 8628-8638, 2021.
[6] Lu Chi, Borui Jiang, and Yadong Mu. Fast fourier convolution. Advances in Neural Information Processing Systems (Neurips), 0004444944888 2020.
[7] Shang-Qi Deng, Liang-Jian Deng, Xiao Wu, Ran Ran, Danfeng Hong, and Gemine Vivone. PSRT: Pyramid shuffle-and-reshufffe transformer for multispectral and hyperspectral image fusion. IEEE Trans. Geosci. Remote Sens., 61:1-15, 2023.
[8] Renwei Dian and Shutao Li. Hyperspectral image super-resolution via subspace- low tensor multi-rank regularization. IEEE Trans. Image Process., 28(10):5135-11 5146611616,,,,,,, 2019.
10

---

[9] Renwei Dian, Shutao Li, and Leyuan Fang. Learning a low tensor- lensor-train rank representation for hyperspectral image super-resolution. IEEE Trans. Neural Netw. Learn. Syst. 30(9))267222688, 2019.
[10] Weisheng Dong, Chen Zhou, Fangfang Wu, Jinjian Wu, Guangming Shi, and Xin Li. Model- guided deep hyperspectral image super-resolution. IEEE Trans. Image Process., 30:55544-7 2021.
[11] Jian Fang, Jingxiang Yang, Abdolraheem Khader, and Liang Xiao. Mimo-sst: Multi-input multi-output spatial-spectral transformer for hyperspectral and multispectral image fusion. IEEE Trans. Geosci. Remote Sens., 62:1-20, 2024.
[12] Mathieu Fauvel, Yuliya Tarabalka, Jon Atli Benediktsson, Jocelyn Chanussot, and James C Tilton. Advances in spectral-spatial classification of hyperspectral images. IEEE, 101(3):652- 675, 2012.
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[14] Alain Horé and Djemel Ziou. Image quality metrics: Psnr vs. ssim. International Conference on Pattern Recognition (ICIP), pages 2366 2369, 2010.
[15] Jinfan Hu, Tingzhu Huang, Liangjian Deng, Hongxia Dou, Danfeng Hong, and Gemine Vivone. Fusformer: A transformer-basee fusion network for hyperspectral image super-resolution. IEEE Geosci. Remote Sens. Lett., 19:1-5, 2022.
[16] Jinfan Hu, Tingzhu Huang, Liangjian Deng, Taixiang Jiang, Gemine Vivone, and Jocelyn Chanussot. Hyperspectral image super-resolution via deep spatiospectral attention convolutional neural networks. IEEE Trans. Neural Netw: Learn. Syst., 2022.
[17] Tao Huang, Weisheng Dong, Jinjian Wu, Leida Li, Xin Li, and Guangming Shi. Deep hyper- spectral image fusion network with iterative spatio-spectral regularization. IEEE Trans. Comput Imaging. 8:201-214, 2022.
[18] Robert Keys. Cubic convolution interpolation for digital image processing. IEEE Trans. Acoust. Speech Signal Process.. 29(6): 153-1160, 1981.
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
[20] Jaewon Lee and Kyong Hwan Jin. Local texture estimator for implicit representation function. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1929-1938 2022.
[21] Chongyi Li, Chun-Le Guo, Man Zhou, Zhexin Liang, Shangehen Zhou, Ruicheng Feng, and Chen Change Loy. Embedding fourier for ultra- ultra-high- hww-hhighhhhneeee nigh-definition wight image enhancement. In International Conference on Learning Representations (ICLR), 2023.
[22] Shutao Li, Renwei Dian, Leyuan Fang, and José M Phoucas-Dias Fusing hyperspectral and multispectral images via coupled sparse tensor factorization. IEEE Trans. Image Process., 2788):41184433 2018.
[23] Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In JEEE/CVPR Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 136-144, 2017.
[24] Xiangyu Liu, Qingjie Liu, and Yunhong Wang. Remote sensing image fusion based on two- stream fusion network. Inf. Fusion., 1-51 2020.
[25] Qing Ma, Junjun Jiang, Xianming Liu, and Jiay Ma. Learning a 3d-cnn and transformer prior for hyperspectral image super-resolution. Inf. Fusion., page 101907, 2023.
11

---

[26] Quan H Nguyen and William J Beksi. Single image super-resolution via dual interactive implicit neural network. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1936-4945, 2023.
[27]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zemingg Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems (Neurips), pages 8024 8035. Curran Associates, Inc., 2019.
[28] William H Press. Numerical recipes 3rd edition: The art of scientific computing. Cambridge university press, 2007.
[29] Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In International Conference on Machine Learning (ICML), pages 5301-5310. PMLR, 2019.
[30] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances in Neural Information Processing Systems (Neurips), 34:980- 993, 2021.
[31] Dioline Sara, Ajay Kumar Mandava, Arun Kumar, Shiny Ducla, and Anitha Jude. Hyperspectral and multispectral image fusion techniques for high resolution applications: A review. Earth Science Informatics, 14(4):1685-1705 2021.
[32] Vishwanath Saragadam, Daniel LeJeune, Jasper Tan, Guha Balakrishnan, Ashok Veeraragha- van, and Richard G Baraniuk. Wire: Wavelet implicit neural representations. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18507-18516, 2023.
[33] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874-1883, 2016.
[34] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems (Neurips), 33:7462-7473, 2020.
[35] Yucheng Sun, Han Xu, Yong Ma, Minghui Wu, Xiaoguang Mei, Jun Huang, and Jiayi Ma. Dual spatial-spectral pyramid network with transformer for hyperspectral image fusion. IEEE Trans. Geosci. Remote Sensing, 2023.
[36] Jiaxiang Tang, Xiaokang Chen, and Gang Zeng. Joint implicit image function for guided depth super-resolution. In Proceedings of the 29th ACM International Conference on Multimedia (ACM MM), pages 4390-4399 2021.
[37] Yuliy Tarabalka, Jocelyn Chanussot, and Jón Atli Benediktsson. Segmentation and classification of hyperspectral images using minimum spanning forest grown from automatically selected markers. In IEEE Transactions on Systems, Man, and Cybernetics (TSMC), volume Part B (Cybernetics) 40, pages 1267-1279, 2009.
[38] Muhammad Uzair, Arif Mahmood, and Ajmal S Mian. Hyperspectral face recognition using 3d-dct and partial least squares. In The British Machine Vision Conference (BMVC), volume 1, page 10, 2013.
[39] Hien Van Nguyen, Amit Banerjee, and Rama Chellappa. Tracking via object reflectance using hyperspectral video camera. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition- Workshops( Recognition-Workkhopps
CVPR), pages 44-51. IEEE, 2010.
[40] Gemine Vivone. Multispectral and hyperspectral image fusion in remote sensing: survey. Information Fusion, 89:405-411 4114405-41,,00 2023.
12

---

[41] Lucien Wald. Data fusion: definitions and architectures: fusion of images of different spatial resolutions. Presses des MINES. 2002.
[42] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from emror visibility to structural similarity. IEEE Trans. Image Process., 13(4):600-61 -120006-6112220 2004.
[43] Zirui Wang. Shangzhe Wu, Weidi Xie, Min Chen, and Victor Adrian Prisacariu. NeRF-; Neural radiance fields without known camera parameters. arXiv preprint arXiv:2202.070000 2021.
[44] Zhaohu Xing, Tian Ye, Yijun Yang, Guang Liu, and Lei Zhu. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. arXiv preprint arXiv:2401.13560 2024.
[45] Ting Xu, Tingzhu Huang, Liangjian Deng, and Naoto Yokoya. An iterative regularization method based on tensor subspace representation for hyperspectral image super-resolution. IEEE Trans. Geosci. Remote Sens., 60:1-16, 2022.
[46] Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Ultrasr: Spatial encoding is a miss- ing key for implicit image function-based srbitrary-scale super-resolution. arXiv preprint MxXiv22003.12116 2021.
[47] Yijun Yang, Huazhu Fu, Angelica Rivero, Carola-Bibian Schönlieb, and Lei Zhu. Diffmic: Dual-guidance diffusion network for medical image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 95-105.
Springer, 2023.
[48] Yijun Yang, Shujun Wang, Lei Zhu, and Lequan Yu. Hedg: A hierarchical consistency framework for domain generalization on medical image segmentation. arXiv preprint arXiv:2109.05742. 2021.
[49] Yijun Yang, Zhaohu Xing, and Lei Zhu. Vivim: a video vision mamba for medical video object segmentation. arXiv preprint arXiv:2401.14168 2024.
[50] Yong Yang, Lei Wu, Shuying Huang, Weiguo Wan, Wei Tu, and Hangyuan Lu. Multiband remote sensing image pansharpening based on dual-injection model. IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens., 13:1888-118 2020.
[51] Roberta H Yuhas, Alexander FH Goetz, and Joe W Boardman. Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm. In AGW-3 Vol. 1: AVIRIS Workshop., 1992.
[52] Xueting Zhang, Wei Huang, Qi Wang, and Xuelong Li. SSR-NET: Spatial- reconstruc- tion network for hyperspectral and multispectral image fusion. IEFE Trans. Geosci. Remote Sens., (9(7))9953-59909 5965, 2020.
[53] Chen Zhao, Weiling Cai, Chenyu Dong, and Chengwei Hu. Wavelet- based fourier information interaction with frequency diffusion adjustment for underwater image restoration. arXiv preprint arX1v:2311.16845, 2023.
[54] Yu Zhong, Xiao Wu, Liang-Jian Deng, and Zihan Cao. Ssdiff:: Spatial-spectral integrated diffusion model for remote sensing pansharpening. arXiv preprint 13X1v:2404.11537, 2024.
13

---

A Appendix supplemental material
This supplementary material provides additional insights into the background, methodologies, and experimental details outlined in our paper. It includes limitations and broader impact, experiments compute resources, details on CNNs in MHIF, 2D Fourier transform, an elucidation of the global receptive field of convolution operations within the Fourier Domain, a description of data simulation, and quality metries. Furthermore, we present comprehensive comparison of all methods applied to the CAVE and Harvard datasets with a scale factor of 8. This includes an ablation study, affirming the efficacy incorporating the Fourier domain on the CAVE(x8) 8). The provided information aims to enhance the reader's 's understanding of the intricacies involved in research and its practical applications.
A.1 Limitations and Broader Impact
Limitations This study has certain limitations that should be acknowledged. One primary limitation the unavailability of ground truth (GT) data in real-world settings for the task of multispectral and hyperspectral image fusion. Due to this constraint, all datasets used in our experiments are simulated. The detailed steps for data simulation can be found in Appendix A.6 This reliance on simulated data may affect the gencralizability of our results to real-world scenarios. Consequently, while our methods show promising performance in experiments, their effectiveness in practical applications remains to be fully validated. However, this limitation is a challenge faced by the entire field, not unique to our work. Surveys 440 31 in the field of multispectral and hyperspectral image fusion highlight this common issue and discuss the need for improved data simulation methods and benchmarks.
Broader Impact This research addresses the task of multispectral and hyperspectral image fusion, which is crucial for enhancing the spatial resolution of hyperspectral images while preserving their spectral fidelity. The resulting high-resolution hyperspectral images (HR-HSI) invaluable for various applications, such as resource monitoring, environmental management, and urban planning. In the environmental domain, fused images aid in pollution tracking, vegetation analysis, and precision agriculture, contributing to sustainable practices and environmental protection. These fused images facilitate more accurate and detailed analysis in these fields, potentially leading to better-informed decisions and more effective resource management. Despite these benefits, there are potential negative consequences consider. Image fusion low-level task that significantly impacts subsequent steps. If the fusion process fails, resulting in distorted HR-HSI, it may adversely affect follow-up tasks and analyses, leading to incorrect conclusions or misguided decisions. Thus, ensuring the robustness and accuracy of the image fusion algorithm is critical to mitigating these risks.
A.2 Experiments Compute Resources
Our experiments were conducted on a workstation equipped with an Intel 12th Gen 17-12700K processor, two NVIDIA RTX 3090 GPUs, and 128GB of memory. This setup provided sufficient computational power to handle the intensive tasks involved in multispectral and hyperspectral image fusion.
A.3 Related Works: CNNs in MHIF
In recent years, CNN-based methods have exhibited significant success in the domain of multi- spectral and hyperspectral image fusion (MHIF). Their efficacy lies in their adeptness to extract high-level features from input data through end-to-end learning. SSRNet (52) leverages three distinct convolutional modules, w.r.t, a fusion module, spatial edge module, and spectral edge module, excelling in image reconstruction by associating spatial-spectral loss function, contributing to robust learning outcomes. Similarly, ResTFNet (24 adopts residual structures and a two-stream fusion network, drawing inspiration from the extensive application of ResNet [13] in super-resolution image processing. In contrast, the MHF network connects HR-HSI to HR-MSI and LR-HSI, facilitating ease of interpretation. MoG-DCN [10) em- ploys a dedicated subnet to approximate the decomposition matrix and conducts hyperspectral image super-resolution using DCN-based image regularization, leveraging prior knowledge of HSI. For the
[50 incorporates a well-explored linear mapping that
14

---

simultaneous extraction of spatial and spectral information and the acquisition of high-quality details, HSRNet [T6] integrates spatial and channel attention modules, enhancing the fusion performance. However, due to the model scaling and limited convolutional receptive field, the CNN-based models still struggle obtain satisfactory results for the MHIF task.
A.4 Preliminary: 2D Fourier Transform
Fourier transform is commonly employed in digital signal processing 2 aiming to convert signals from the time domain the frequency domain. Through this domain transformation, previously imperceptible features often become observable. For two-dimensional images, the Fourier transform converts the signal from the spatial domain to the frequency domain, enabling the transformation of images into spectrograms in the frequency domain. Given single-channel image X E &R the Fourier transform translates it into the Fourier space as the complex component Y CH*W This process can be represented as follows:
--IW-1 ΣΣ h=0 w=0
F(X)(u,)) = v( (((u,)) VHW
X(h, X(,,,, wee f2x (27
(16)
where (h, w) denotes the coordinates of in the spatial space, and (u, v) represent the coordinates of in the Fourier space. The Fourier space spanned by complex orthogonal basis functions, and each complex frequency component can be expressed amplitude A (Y (u, v)) and phase P(Y (u, v)) components:
A(Y(u,)) = R2 (Y, + 3 (u,,)),
(17)
3 33Y(u, e)) R R (u, v))
(u, o)) arctan
(18)
where R(Y) and 3YY) respectively represent the real and imaginary parts. For multi-channel images, in the utilization of the Fourier transform, we perform individual Fourier calculations for each channel. Additionally, the Fourier transform is reversible transformation, enabling bidirectional conversion between the original signal and the transformed signal, we denote FFl as the Fourier inverse transform.
A.5 The Receptive Field of INR in the Fourier Domain
Spatial Domain
Frequency Domain
Receptive Field
Figure 8: Convolving an image in the frequency domain is globally impactful in the spatial domain. To validate the global nature of implicit feature fusion in the Fourier domain, we conducted exper- iments as illustrated in Fig. g We transform the "chart and stuffed toy" sample from the CAVE dataset into the Fourier domain, performed convolutions only on specific frequency features, and then transform back the spatial domain. It can observed that Fourier domain convolution yields global response in the spatial domain. Performing INR in the frequency domain indeed expands the receptive field of INR, frecing it from local constraints.
A.6 Data Simulation
The proposed architecture takes LR-HSI and HR-MSI pairs [I RR as input, with the training ground-truth (GT) being HR-HSI. However, due to the unavailability of HR-HSI reference, simulation phase is necessary. In our experiments using the CAVE dataset, we cropped 20 training images, generating 3920 overlapping patches of size 64 x 64 x 31. These patches serve HR-HSI (ground truth) patches. To simulate appropriate LR-HSIS applied a 3 x 3 Gaussian blur kernel
15

---

(03
(b)
(c) (4) (c) 16
(-)
b
Figure 9: The testing images from the CAVE dataset: (a) balloons, (b) cd, (c) chart and stuffed toy, (d) clay, fake and real beers, fake and real lemon slices, (g) fake and real tomatoes, (h) feathers, (i) flowers, ( hairs, and (k) jelly beans. An RGB color representation is used to depict the images.
(8)
(b)
(c)
(c)
(7)
(g)
(h)
(
Figure 10: The 10 images tested on the Harvard dataset are (a) bikes, (b) safal, (c) window, (d) fence. (c) tree, (f) sofa2, (g) backpack, (h) wall, (i) door and (j) parcels.
with standard deviation of 0.5 to the original HR-HSK, Subsequently, we downsampled the blurred patches by a factor of 4. HR-MSI patches were generated using the spectral response function of Nikon D700 camera. Therefore, input pairs (I B) consist of 3920 LR-HSI patches of size 16 16 x 31 and RGB image patches of size 64 x x 3. Paired with their corresponding GTs, these pairs were randomly split into training data (80%) and validation data (20%). The testing set of the CAVE dataset is shown in Fig. 9 The same procedure was employed to simulate the input LR-HSI and HR-MSI pairs and GTs for the Harvard dataset. The Harvard test set is shown in Fig. 10
A.7 Quality Metrics
We compare our method with other methods using different image quality metrics to validate the image fusion capability of our model, including the Spectral Angle Mapper (SAM) [5I], the Erreur Relative Globale Adimensionnelle de Synthèse (ERGAS) [41). the Peak Signal-to-Noiss Ratio (PSNR) [14. and the Structural SIMilarity (SSIM) 42).
PSNR evaluates the spatial quality of each band in the reconstructed HR-HSI. It is calculated as follows:
B
PSNR (I.
Σ PT
(TT)),
(19)
B
1=1
Here, I' E HxW and I E H represent the i-th band of HxxB and i #xxBB respectively. The PSNR function is defined as:
max (I+)
PS . = 20 log10
(20)
MSE(T: T.)
where MSE (Mean Square Error) between I' and I and max(-) is the maximum value of I
16

---

SAM measures the spectral distortion of each hyperspectral pixel in the reconstructed HR-HSI. is given by:
HW
I.
SAM(I,T) HW
Σ cos
(21)
||
where cos" the arccosine function, I, E R nx1 and I, E nx1 are the spectra of the i-th pixel of I and L, respectively, || 2l2 is the &2 norm, and .x denotes the transpose.
ERGAS measures the global statistical quality of the reconstructed HR-HSI, taking into account the ratio the ground sample distances between HR-MSI and LR-HSI. It formulated as:
100
B
MSE(T, I+)
ERGAS(I,
(22)
B
where is the scaling factor, and p is the square the mean value of
SSIM is used assess the structural differences between GT and the reconstructed HR-HS, incorpo- rating both luminance and structural contrast functions. The SSIM function defined as:
(2 + C2) + C2) B P + + Ci) + o C2)
SSIM(L,
(23)
where, B is the number of bands, and I and are sets containing I' and I for = to B, respectively, pI and FT. represent the mean values of I and i while of and of denote their variances. The T.
term rT.. indicates the covariance between I' and i. Constants C₁ and C2 fixed values.
Higher PSNR values indicate better performance, while lower SAM and ERGAS values signify higher quality of the reconstructed HR-HSI. SSIM values range from -1 to 1, with values closer to 1 indicating better quality. Ideally, PSNR should be infinite, SAM and ERGAS should be zero, and SSIM should be one.
A.8
Benchmark
To evaluate FeINFN's performance, we compare it with MHIF methods on the CAVE and Harvard datasets. The bicubie-interpolated result of the upsampled LR-HSI in Tab. T serves as our base- line. Various model-based techniques, including the CSTF-FUS [22]. LTTR 9 LTMR ( and
IR-TenSR 45) approaches, are considered. Additionally, we compare our approach with various deep learning methods, such as SSRNet [52)] ResTFNet (24, HSRNet (16), MoGDCN [1O], Fus- former 113, and DHIF [17], PSRT 77 3DT-Net (25). DSPNet [35), MIMO-SST (. We compare
our method with other methods using different image quality metrics to validate the image fusion capability our model, including SAM [51, ERGAS 41), PSNR [4 and SSIM
A.9 More Comparisons with the Larger Scaling Factor on CAVE and Harvard Datasets
Due to space constraints in the main text, we present more detailed comparison of our FeINFN with other methods on four metrics for the CAVE dataset and the Harvard dataset in the supplementary material. As shown in Tab. 5 FeINFN demonstrates the best overall performance. While ranks second SAM on the CAVE Ca it maintains the optimal results for other metrics.
A.10 Ablation Study: The Effectiveness of Fourier Domain Incorporation on CAVE (x8)
In the main text, we described our ablation experiments to investigate the effectiveness of incor- porating the Fourier domain. Additionally, we were interested in assessing its efficacy at larger scale factors. Therefore, we applied pruning to the model on the CAVE (x8) dataset to validate its effectiveness. The experimental results, shown in Tab. 6 indicate that the model performs best when incorporating the Fourier domain operation, aligning with our hypothesis. This provides robust evidence supporting the enhancement of network performance through INR in the Fourier domain.
17

---

Table 5: The average and standard deviation calculated for all the compared approaches 11 CAVE examples and 10 Harvard examples simulating a scaling factor of 8. The best results are in bold, second-best best in underline. "M" refers millions.
CAVE
Harvard
Methods
PSNRR())
SAM(J)
ERGAS() SSIM(1) #params #Hlops
PSNR(()
SAM())
ERGAS())
SSIM(1) #purams #llops
Bicubic
29.96+33.54
13.18-26.85
10+0.00 3.83+11844 0.84-0000773
CST--US
22
88.44+4.225
7.00+2265
2.11:1115
0.95++0033
59,34:66511
4,49+1.52 L2 2.40++1.88 1.84 0.932-=0.D02
LTTR
9
37.9243.59 5.37
2.44
0.972:0.018
42.09:14.36 3.6-41136662211.44
1.80:00996 0,960::0.048
LTMR
8
38.41+3.57 5,04+11.70
2.24+0997
0.914+0.017
3.000 3.62:1..44
42.09::4.556
1.00+09222
0:959==0:000
IR-TenR
15
36.79133.64
12.87:44.98
2,68&1441
0.96110.031
00.04:33.99544411...
4.75
0.958:00.016
ResTFNNt
43.7715.34
3,49100.4
1.3811.25
0.99210.006
2.367M
1.75G
13.50:33.6...55211..1
1.7410,933
0.979:10.012 2.387M
L.75G
SSRNet
52
46.23:44.19 3.13=#0.97
1.05
0.993:0.004
0.466M
0.11G
15.76:::3.342999::::000
1.34::0.74
0,983=00.010
0.027M
0.11G
HSRNet
16
46.00-4-.48 2.91+00.66
0.93++D.33
0.992+00000
301M
2,00G
44.022+4.899 3.64+1.79
1,49-0881
0.980+00DD3
0.633M
2.00
MogDCN
10
49:21:4499
2.44:00.74
0.76::0.63
0.996:0.0033
6.840M
47.4:555.14::5.44
3.19::1445
1.75::1.66
0,980=0.0199
7.444M
47.48G
Pusfomer
47.96+77779
2,75+1130
1.45++2.09
0.990+00022
0.551M
9.83G
44.993-45565
3.63+-2444
1.49-0996
0:979+00.D17
0,467M
9.836
DHIF
7
48.46224.89
2,50::0,79 79
0/33:20667
0.996:0.003
22.462M
54.27G
3.700: .66
1.32=0061
0.983=00001
22.462M
54.27G
PSRT
47.86++7.53
2,73++080
1.52+33022
0.55
0.247M
1.14G
45.10--4.06
2,900-084
1:57+0084
0.985-00.009
0.247M
1.14G
3DT-No 23
49.41255.33
2.26:10.66
1.3=
0.996:0.003
3.464M
68.07G 44.44:
2.93:0.888
1.55::0.89
0.983:00.010
3,464M
68.076
DSPNet
19.18:14.54
0.99610.003
45.8411..62
2.97.0.755
MIMO-SST [1]
48. 48.31::5.04
2.88:0086
0.89:20.79 0.905:0 004 4.93M
1,580
46.59:33.34
2.9 2.91:
2.29:: 1.0 0.965=0.009
4,983M
1.580
FeNNEN(Ours
0,99610.003
36.9913.59 2.78.10.73
0.98620.009
3.165M
Table The four average QIs and the corresponding parameters on the testing images from the CAVE dataset simulating scaling factor of 8. S & JF means the domain difference.
$ Fl PSNR(1)
SAM(L) ERGAS(L) SSIM(T)
50,01
2.36:10.72
0.71=0.68 0.99610.0031
47.60=4.118
0.88=0.61 0.993+0000041 50.32-5.17 2.3310075
4.24#1.69
/
0.67=0.60 0.99610.0028
A.11
Ablation Study: The Effectiveness of Decoder with Complex Gabor Wavelet Activation
(a) GT
(b) Ours
(6) ReLU
(d) GELU
Figure 11: Error map for fusing an image with edges.
The Gabor wavelet activation demonstrates high representational power for visual signals, depicted in Fig. IT. Compared to other activation functions, we observe that models utilizing the Gabor wavelet function exhibit lower error and spatial compactness.
18

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction clearly outline the main contributions and scope of the paper, ensuring that the claims made supported by the theoretical and experimental results presented.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations in Appendix A. such as assumptions made in the model and potential areas where the results may not be generalized.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer: [Yes]
Justification: For Theorem presented in the paper, a complete proof is given in the main text, along with detailed explanations and illustrations.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper provides detailed descriptions of the network architecture and experimental settings, and the datasets used are publicly available. The complete code will be published on GitHub upon acceptance for further research and discussion.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The datasets are publicly accessible, and the code will be released soon.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https//niips.ccppbblei
BBiides/CodeSubbissiioooiii
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Sec. 4
present the datasets and implementation details, and the simula- tion details of the datasets can be found in Appendix AA.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not involve error bars experiments concerning statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources for the experiments can be found in the implementation details of Sec. (Experiments) and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpss//weuriis.cccpubulleeeehes
Answer: [Yes]
Justification: The research adheres the ethical guidelines set forth by NeurlPS, ensuring responsible conduct in all aspects of the study.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., .., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The broader impacts of this work can be found in Appendix A.I
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The models compared in the paper have either received author permission or are publicly available. The datasets are public, and original papers are cited meticulously.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The models compared in the paper have either received author permission or are publicly available. The datasets are public, and original papers are cited meticulously.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
23

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paaperswiteode.. https/ittthood..cocataaasttttt has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing or research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing or research with human subjects.
24

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "kjSrniY5HJ", "reviewer": "Reviewer_EUje", "summary": "This paper proposes a quite interesting hyperspectral and multispectral image fusion framework via implicit representation. Moreover, the authors introduce the Fourier transformation to decouple the amplitude and phase domain.", "questions": "Improvement of Fourier design.\nUnsupervised training manner. \nComputational cost. \nDifferent SR ratios.\nAs the weakness Sec.", "limitations": "Please refer to weakness sec.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tIntroducing implicit model into the task of hyperspectral multispectral fusion is quite interesting.\n\n2.\tA novel Gabor wavelet activation function is proposed. Theoratical analysis of Gabor activation is also given;\n\n3.\tThe decoupling process of spatial-spectral domain is reasoable for hyperspectral image;\n\n4.\tThe performance of the proposed method is competitative with SOTA methods; and\n\n5.\tPaper is easy to follow, figure is clear to vlidate the motivation of designing.", "weaknesses": "1.\tThe proposed Fourier decomposition seems to only result in very limited performance improvement. The results from table 3 and figure 7 seem not consistent.\n2.\tActually, such implicit representation is more plausible to work in a self-regularized overfitting manner than current train-test setting. Authors can make further design or give a unsupervised version.\n3.\tAuthors should also compare the computational cost of different methods.\n4.\tAuthors should also evaluate the performance with different SR ratios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.** Improvement of Fourier design. The results from table 3 and figure 7 seem not consistent.\n\n**A1.** Thank you for your insightful comments. We apologize for any confusion caused by our unclear presentation. To clarify:\n\n- **Inconsistencies Between Tab. 3 and Fig. 7:** The results in Tab. 3 are derived from the *test set*, while Fig. 7 displays the performance on the *validation set* (since we can not access test set when training and validating).\n- **Limited Performance Improvement:** While the proposed Fourier decomposition may offer limited performance improvement, **our primary motivation extends beyond simply enhancing the results. Our goal was to address inherent issues within Implicit Neural Representations (INRs) as state in Sect. 2(Motivation)**. Specifically, as illustrated in Fig. 7, incorporating the Fourier domain for implicit representation accelerates the network's convergence compared to using only the spatial domain. This demonstrates that the network can focus more on high-frequency information in the early stages of training, mitigating the spectral bias introduced by MLP-ReLU architectures. \n\n**Q2.**  Unsupervised training manner.\n\n**A2.**  Thank you for your valuable suggestion. Inspired by your feedback, we propose an unsupervised approach that leverages the self-regularized overfitting manner without relying on high-resolution ground-truth images to fuse an image.\n\n Specifically, **we select a LRHSI-HRMSI pair then downsample the pair by a fixed factor (e.g. 4). Then the downsampled pair is fed into the proposed INR network and produce the fused image. The self-regularized L1 loss is used to optimize the model, where the original LRHSI serves as the unsupervised GT.** This setup allows us to train in a more self-regularized, unsupervised manner. This training method does not conflict with our current training method. And our training diagram can enable our model to fuse images on different test pairs, not just on one single image. We appreciate your insightful suggestion and believe this approach could provide a promising direction for future work. \n\n**Q3.** Computational cost.\n\n**A3.** Thank you for your valuable suggestion. Comparing the computational cost of different methods is indeed important. To address this, we have added the FLOPs for each method and will update the final version of our paper to include a detailed comparison and analysis.\n\nWe provide both the time and spatial complexities of our proposed model below:\n- **The time complexity** of our model is $4D^2L\\approx \\mathcal O(L)$ (because usually $L\\gg D$), where $D$ represents the number of channels in the input image and $L$ represents the image dimensions (*i.e.,* $H\\times W$). In comparison, Transformer-based methods have a time complexity of $3D^2L+2DL^2\\approx \\mathcal O(L^2)$. Our model's time complexity is linear to sequence length, whereas other Transformer methods (*e.g.,* PSRT) scale with $\\mathcal O(L^2)$.\n- **The spatial complexity** (*a.k.a,* memory consumption) of the attention operator in Transformers is $\\mathcal O(L^2)$, which can lead to out-of-memory issues with high-resolution images. In contrast, the spatial complexity of the MLP in our INR-based model does not grow quadratically for the image size. This means that our model can handle larger images without encountering the same memory constraints as Transformer methods.\n\nThe table below shows the number of paramters and FLOPs among our method and other methods.\n\n|Methods|CAVE $\\times 8$ PSNR ($\\uparrow$)|CAVE $\\times 8$ SAM ($\\downarrow$)|CAVE $\\times 8$ ERGAS ($\\downarrow$)|CAVE $\\times 8$ SSIM ($\\uparrow$)|Harvard $\\times 8$ PSNR ($\\uparrow$)|Harvard $\\times 8$ SAM ($\\downarrow$)|Harvard $\\times 8$ ERGAS ($\\downarrow$)| Harvard $\\times 8$ SSIM ($\\uparrow$)|Params|FLOPs|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Bicubic|29.96|5.89|5.56|0.887|33.18|3.10|3.83|0.894|-|-|\n|CSTF-FUS|38.44|7.00|2.11|0.959|39.84|4.49|2.40|0.932|-|-|\n|LTTR|37.92|5.37|2.44|0.972|42.09|3.62|1.80|0.960|-|-|\n|LTMR|38.41|5.04|2.24|0.974|42.09|3.62|1.80|0.959|-|-|\n|IR-TenSR|36.79|12.87|2.68|0.944|40.04|5.40|4.75|0.958|-|-|\n|ResTFNet|43.77|3.49|1.38|0.992|43.50|3.53|1.74|0.979|2.387M|1.75G|\n|SSRNet|46.23|3.13|1.05|0.993|45.76|2.99|1.34|0.983|0.027M|0.11G|\n|HSRNet|46.69|2.91|0.93|0.994|44.02|3.64|1.49|0.980|1.09M|2.00G|\n|MogDCN|49.21|2.44|0.76|0.996|45.14|3.19|1.75|0.980|6.840M|47.48G|\n|Fusformer|47.96|2.75|1.45|0.990|44.93|3.63|1.49|0.979|0.504M|9.83G|\n|DHIF|48.46|2.50|0.83|0.996|45.00|3.70|1.32|0.983|22.462M|54.27G|\n|PSRT|47.86|2.73|1.52|0.994|45.10|2.90|1.37|0.985|0.247M|1.14G|\n|3DT-Net|49.41|**2.26**|0.83|0.996|44.41|2.93|1.55|0.983|3.464M|68.07G|\n|DSPNet|49.18|2.57|0.75|0.996|45.84|2.97|1.33|0.984|6.064M|6.81G|\n|MIMO-SST|48.31|2.88|0.89|0.995|46.59|2.91|2.29|0.985|4.983M|1.58G|\n|**Proposed**|**50.32**|2.33|**0.67**|**0.996**|**46.89**|**2.78**|**1.16**|**0.986**|3.165M|10.53G|\n\n**Q4.** Different SR ratios.\n\n**A4.** Thank you for your valuable comment. We have conducted additional fusion experiments on the CAVE and Harvard datasets with an SR ratio of 8. The results of these experiments can be found in supplementary A. 8. Furthermore, we have included a comparative visual illustration in Fig. 1, showcasing the performances of our method at SR ratios of x4 and x8 against other approaches. The results indicate that our model performs better under different fusion ratios, which is attributed to the powerful continuous representation capability of INR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5S6tT5mWXl", "reviewer": "Reviewer_8xGR", "summary": "The paper proposes a novel Fourier-enhanced Implicit Neural Fusion Network (FeINFN) specifically designed for the Multispectral and Hyperspectral Image Fusion task. The paper identifies the unique characteristics of the amplitude and phase of the latent codes in both HRHSI and LRHSI, and proposes to enhance high-frequency details and expand the receptive field from the Fourier domain. Additionally, the paper introduces a new decoder to achieve better interaction between spatial domain features and frequency domain features.", "questions": "See weaknesses", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. As mentioned in the Summary, the paper's observations on amplitude and phase are intriguing, and the proposed spatial-frequency domain fusion framework aligns with the paper's motivation.\n\n2. The paper presents clear theoretical proofs for its claims, such as the advantage of complex Gabor wavelet activation in finding the optimal bandwidths.\n\n3. The paper is well-written, with clear and organized content.\n\n4. The experimental observations are interesting and can be considered to have a positive impact.", "weaknesses": "1.\tWhat does 'current neural network-based methods are insensitive to high-frequency information' mean, and how does the proposed model demonstrate sensitivity to high-frequency information?\n2.\tAs mentioned in Section 3.4, the Spatial Implicit Fusion Function utilizes frequency encoding in the spatial domain. Does the Frequency Implicit Fusion Function also utilize frequency encoding in the frequency domain?\n3.\tShould the vertical axis scales of (c), (d), and (e) in Figure 5 be consistent? In my understanding, at least (c) and (d) should be consistent to demonstrate that the ground truth (GT) frequencies and the frequencies captured using Gabor activation are similar.\n4. This paper proposes a new fusion architecture based on INR. It would be even better if there were discussions and analyses regarding network complexity, such as comparisons in spatial complexity with Transformer methods mentioned in section I.\n\n5. There are some editing errors: the double quotes are displayed incorrectly in lines 307 and 310 as “Fourier Domain\" and “chart and stuffed toy\".\n\n6. In line 226, “I” should correspond to elsewhere in the article.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.** What does 'current neural network-based methods are insensitive to high-frequency information' mean, and how does the proposed model demonstrate sensitivity to high-frequency information?\n\n**A1.** Thank you for your careful review. **(1)** High-frequency insensitivity is a common issue with MLPs, which are the primary architecture of INR. Leveraging this insight, we propose a method to enhance high-frequency information in the frequency domain. **(2)** Ablation experiments (i.e., Sec. 4.1 in the manuscript) verify that our model alleviates the inherent shortcomings in INR, namely high-frequency insensitivity. Fig. 7 of the manuscript demonstrates that our model exhibits faster convergence and overall higher efficiency during training stages and exhibits more high-frequency details visually at the same iteration.\n\n**Q2.** As mentioned in Section 3.4, the Spatial Implicit Fusion Function utilizes frequency encoding in the spatial domain. Does the Frequency Implicit Fusion Function also utilize frequency encoding in the frequency domain?\n\n**A2.** Thank you for your question. In the frequency domain, we do not use frequency encoding. We tested the inclusion of frequency encoding in the frequency domain, but it did not yield any performance improvement in our experiments. *Consider the non-performance improvement and introduce another computational overhead*, therefore, we choose not to incorporate it to avoid unnecessary complexity.\n\n**Q3.** Should the vertical axis scales of (c), (d), and (e) in Figure 5 be consistent? In my understanding, at least (c) and (d) should be consistent to demonstrate that the ground truth (GT) frequencies and the frequencies captured using Gabor activation are similar.\n\n**A3.** Thank you for pointing this out. You are totally correct in your understanding. The vertical axis scales in subfigures (c) and (d) of Fig. 5 are indeed consistent; the scale of 1500 is simply not displayed in (c).\nSubfigures (c), (d), and (e) in Fig. 5 compares the frequency characteristics of fusion images generated by different activation functions with the ground truth (GT) images. The model using the complex wavelet activation function demonstrates better frequency fitting compared to the model using ReLU. This is evident in the comparison between (c) and (d), where the frequency distributions in the optimal bandwidth region are closer. In contrast, the frequency distributions in (c) and (e) show significant differences.\n\n**Q4.** This paper proposes a new fusion architecture based on INR. It would be even better if there were discussions and analyses regarding network complexity, such as comparisons in spatial complexity with Transformer methods mentioned in section I.\n\n**A4.** Thank you for your insightful suggestion. We have calculated both the computational and spatial complexities of our proposed model.\n**The time complexity** of our model is $4D^2L\\approx \\mathcal O(L)$ (because usually $L\\gg D$), where $D$ represents the number of channels in the input image and $L$ represents the image dimensions (*i.e.,* $H\\times W$). In comparison, Transformer-based methods have a time complexity of $3D^2L+2DL^2\\approx \\mathcal O(L^2)$. Our model's time complexity is linear to sequence length, whereas other Transformer methods (*e.g.,* PSRT) scale with $\\mathcal O(L^2)$.\nAdditionally, **the spatial complexity** (*a.k.a,* memory consumption) of the attention operator in Transformers is $\\mathcal O(L^2)$, which can lead to out-of-memory issues with high-resolution images. In contrast, the spatial complexity of the MLP in our INR-based model does not grow quadratically for the image size. This means that our model can handle larger images without encountering the same memory constraints as Transformer methods.\n\n**Q5.** There are some editing errors: the double quotes are displayed incorrectly in lines 307 and 310 as “Fourier Domain\" and “chart and stuffed toy\".\n\n**A5.** Thank you for pointing out the editing errors. We apologize for the oversight and have corrected the double quotes as indicated in lines 307 and 310.\n\n**Q6.** In line 226, “I” should correspond to elsewhere in the article.\n\n**A6.** Thank you for bringing this to our attention. We have corrected the reference to “I” in line 226 to ensure it corresponds appropriately with other parts of the article.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cVH3lrao8T", "reviewer": "Reviewer_EFSC", "summary": "The paper introduces a novel Fourier-enhanced Implicit Neural Fusion Network .The core innovation lies in the integration of Fourier transformations within an Implicit Neural Representation framework to address the loss of high-frequency information—a common limitation in existing INR approaches. The effectiveness of FeINFN is demonstrated through experiments on two benchmark MHIF datasets, showing superior performance in terms of PSNR and other metrics compared to state-of-the-art methods.", "questions": "1. Network Complexity: The introduction of Spa-Fre IFF and SFID seems to increase the model complexity. Could the authors comment on the computational efficiency of their proposed method compared to the baselines?\n\n2. In the explanation of the compound Gabor wavelet activation function in the article, Theorem 1 mentions optimal bandwidth. What does this refer to, and how does it affect the activation of features?\n\n3. The abstract of the paper claims that the proposed model achieved state-of-the-art performance on two datasets. For the Harvard x4 experiments, how can the fact that the SAM and SSIM metrics did not reach SOTA be explained?\n\n4. “frequency point may correspond to multiple pixels at different positions in the spatial domain”， Why does transforming into the Fourier domain amplify the frequency domain? I'm a bit unclear on this point.", "limitations": "The authors have provided reasonable Limitations and Broader Impact.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper introduces novel components to the INR framework, specifically tailored for MHIF tasks, a creative combination of existing ideas. The proposed modules are theoretically feasible. It provides clear and understandable graphical explanations for complex theories, such as Fig. 5. The paper is generally well-written, with clear explanations of complex concepts and a logical flow of ideas. The method proposed in the article performs optimally on two publicly available datasets, which should contribute significantly to the MHIF field.", "weaknesses": "1. Some sections of the paper are overly technical and dense, which might be challenging for readers unfamiliar with the specific domain of Fourier transforms and neural networks.\n\n2. In Table 3, the description of S and F in the last row is incorrect and should be indicated with two check marks. This seems to be an editing error.\n\n3. In the ablation experiments section of the paper, tables 2, 3, and 4 do not provide the corresponding parameter quantities. If available, could you please provide them? Ablation experiments need to compare models under similar parameter conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1.** Thank you for your comments. We have revisited the sections that were considered overly technical and dense and have made efforts to simplify the descriptions. For instance, **we have included preliminary on INR and Fourier Transforms directly in the main text. The background on these topics is now presented in Sections 3.1 and Supplementary A.4.** Although the theory involves signal processing concepts, the principles underlying the model's design are not overly complex, and its implementation is relatively straightforward.\n\n**A2.** We have corrected the description of S and F in the last row of Table 3 to include two check marks as intended.\n\n**A3.**  We have included the parameter quantities for Tab. 2 and Tab. 3 in the ablation experiments section. As Tab. 4 evaluates the effectiveness of different activation functions, changing the activation function alone does not affect the parameter quantities, so no modifications were made to Tab. 4. We will update the relevant sections of the manuscript to reflect these changes.\n|Methods|Params|\n|:-|-:|\n|Bilinear|3.119M|\n|Bicubic|3.119M|\n|Pixel Shuffle|3.057M|\n|Our|3.165M|\n\n|$\\mathcal{S}$/$\\mathcal{F}$|Params|\n|:-|-:|\n|$\\checkmark$/✘|2.869M|\n|✘/$\\checkmark$|2.940M|\n|$\\checkmark$/$\\checkmark$|3.165M|\n\n**A4.** Comparing the computational efficiency of different methods is indeed important. To address this, we have added the FLOPs for each method and will update the final version of our paper to include a detailed comparison and analysis. We provide both the time and spatial complexities of our proposed model. Please refer to the response to Reviewer EUje's Question 3 for specific details.\n\n**A5.** Thank you for your insightful question.\n\n**Explanation of Optimal Bandwidth in Theorem 1**: Theorem 1 addresses the concept of optimal bandwidth in the context of the compound Gabor wavelet activation function. The optimal bandwidth refers to the ideal range of frequencies that the Gabor wavelet function can effectively capture. This balance is crucial because it ensures that the Gabor wavelet activation can efficiently represent both low and high-frequency components of the signal.\n\n**How Optimal Bandwidth Affects Feature Activation**: (1) The Gabor wavelet activation is designed to have time-frequency tightness, meaning it maintains high precision in both the time and frequency domains. This is governed by the uncertainty principle, $|\\omega_0| \\cdot \\nu_0 \\geq \\frac{1}{4\\pi}$ , which ensures a balanced representation. (2)  The optimal bandwidth allows the activation function to capture detailed information, improving the network's ability to reconstruct and represent high-frequency details. This leads to more accurate feature activation, particularly for detailed textures and fine-grained information. (3) During training, the network automatically learns the optimal bandwidths, resulting in precise and effective feature activations. This property of the Gabor wavelet activation ensures that the features align closely with the ground truth, enhancing the overall performance of the model. (4) The time-frequency tightness and optimal bandwidth of the Gabor wavelet activation function lead to better performance metrics, compared to other activation functions like ReLU and GELU. This results in enhanced fusion quality and robust feature learning within the Spatial-Frequency Interactive Decoder.\n\n**A6.** For the Harvard x4 experiments, while the SAM and SSIM metrics did not reach SOTA, our method's performance is very close to that of 3DT-Net, with only a difference of 0.06 in SAM and 0.001 in SSIM. Additionally, our FeINFN model achieved SOTA performance in Harvard x8, demonstrating even more significant improvements.\n\nFurthermore, we have conducted experiments on remote sensing hyperspectral image datasets, including the Pavia Centre dataset and the Chikusei dataset. These experiments, compared with several leading traditional and deep learning methods, also show that our model achieves SOTA results. Detailed results are presented in the table below:\n|Methods|Chikusei $\\times 4$ (PSNR)|Chikusei $\\times 4$ (SSIM) |Pavia $\\times 4$ (PSNR)|Pavia $\\times 4$ (SSIM)|\n|-|-|-|-|-|\n|ResTFNet|42.33|0.950|33.56|0.885|\n|SSRNet|42.36|0.951|33.20|0.876|\n|HSRNet|42.01|0.947|32.17|0.867|\n|MogDCN|42.21|0.936|33.84|0.889|\n|Fusformer|43.37|0.959|35.31|0.924|\n|DHIF|43.69|0.960|35.30|0.924|\n|PSRT|43.48|0.961|34.86|0.916|\n|3DT-Net|43.53|**0.963**|35.10|**0.927**|\n|DSPNet|43.55|0.960|35.47|0.927|\n|MIMO-SST|43.36|0.958|35.37|0.922|\n|**Proposed**|**43.88**|**0.963**|**35.51**|**0.927**|\n\n> Due to the rebuttal length limitation, the full metrics will be provided in the discussion stage.\n\n**A7.**  In the spatial domain, a single frequency point may correspond to multiple pixels at different positions. This is because the Fourier transform decomposes an image into a sum of sinusoidal functions of varying frequencies. Each frequency component has a global influence across the entire image, meaning that a specific frequency can affect multiple spatial locations.\n\nThis amplification effect occurs due to the following reasons: (1) The Fourier transform separates the image into its constituent frequency components, making certain features more distinguishable in the frequency domain than in the spatial domain. This separation aids in identifying and manipulating specific frequency characteristics. (2) Convolution operations in the frequency domain may affect the entire spatial domain. A single point in the frequency domain corresponds to multiple points in the spatial domain, resulting in a global impact that makes certain features more prominent. (3) Operations in the Fourier domain can enlarge the receptive field of corresponding spatial domain operations. This means localized features in the frequency domain can be represented more globally in the spatial domain, leading to an apparent amplification of certain features.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0OD0a9IzdK", "reviewer": "Reviewer_2Qbn", "summary": "The paper provides a novel INR-based fusion framework tailored for the MHIF task, effectively capturing high-frequency details and global information through innovative architectural components. The key contributions are transforming the latent features into the frequency domain to enhance high-frequency information and introducing a complex Gabor wavelet activation function to strengthen feature representation and fusion.  Experiments on two MHIF datasets demonstrate the state-of-the-art performance achieved by the proposed method, outperforming existing approaches both visually and quantitatively.", "questions": "1. How does Fourier transformation within the INR framework specifically contribute to observed performance improvements? What are the practical benefits of frequency-domain features?\n2. After reviewing the proof of Theorem 1, I don't understand why finding the optimal bandwidth leads to better representation. Could you provide further explanation, especially regarding Fig. 5.\n3. For the PSNR curve in Figure 7, on which dataset was this experiment conducted? Was it on the training set, validation set, or test set? The paper does not explicitly specify this.\n4. Regarding Fig. 6, were the errors normalized? The more the error maps lean towards red, the greater the difference between the pixel values and the ground truth. Is my understanding correct? The differences between the close-up images seem very small to the naked eye, and I can't discern any differences between the images in the first and third rows.", "limitations": "The authors have included a discussion on the limitations of their work in Section A.1, covering the data simulation, and potential societal impacts.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This paper provides a novel solution to the problem of MLP-ReLU losing high-frequency information by leveraging domain transformation. The originality is high.\nQuality: The viewpoints presented in the paper are supported by theoretical proofs and corresponding experimental evidence.\nClarity: The paper is well-organized and the content is clear and easy to read.\nSignificance: The proposed solution could have a significant impact on the field of computer vision.", "weaknesses": "- Complexity: Some parts of the methodology, particularly the theoretical proofs and implementation details, are complex and may be challenging for readers to fully grasp without additional explanation.\n- Quality Metrics: The paper does not seem to provide the meaning and calculation methods of the four metrics (PSNR, SAM, ERGAS, SSIM).\n- Experimental Section: Besides the two benchmark MHIF datasets used in the experiments, how does the proposed method perform in other image fusion tasks or datasets? It would be better if there were experiments on other datasets, but it's not essential.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**A1.** Thank you for your detailed review. We have revisited the sections that were considered complex and have worked to simplify the descriptions. For example, we have included foundational knowledge on INR and Fourier Transforms directly in the manuscript. Although the theoretical aspects involve signal processing concepts, such as Fourier Transforms, the underlying principles of the model’s design are not overly complex, and the implementation is relatively straightforward.\n\n**A2.** We have added detailed descriptions and calculation methods for each metric in the revised manuscript. Specifically:\n> **PSNR** evaluates the spatial quality of each band in the reconstructed HR-HSI. It is calculated as follows:\n$\\text{PSNR}(\\mathbf{I}, \\tilde{\\mathbf{I}}) = \\frac{1}{B} \\sum_{i=1}^{B} \\text{PSNR}(\\mathbf{I}^i, \\tilde{\\mathbf{I}}^i), $\nhere, $\\mathbf{I}^i \\in \\mathbb{R}^{H \\times W}$ and $\\tilde{\\mathbf{I}}^i \\in \\mathbb{R}^{H \\times W}$ represent the $i$-th band. The PSNR function is defined as: $\\text{PSNR}(\\mathbf{I}^i, \\tilde{\\mathbf{I}}^i) = 20 \\cdot \\log_{10} \\left( \\frac{\\max(\\mathbf{I}^i)}{\\sqrt{\\text{MSE}(\\mathbf{I}^i, \\tilde{\\mathbf{I}}^i)}} \\right)$.\n\n> **SAM** measures the spectral distortion of each hyperspectral pixel in the reconstructed HR-HSI. It is given by: $\\text{SAM}(\\mathbf{I}, \\tilde{\\mathbf{I}}) = \\frac{1}{HW} \\sum_{i=1}^{HW} \\cos^{-1} \\left( \\frac{\\mathbf{I}_i^T \\tilde{\\mathbf{I}}_i}{||\\mathbf{I}_i||_2 ||\\tilde{\\mathbf{I}}_i||_2} \\right)$, $||\\cdot||_2$ is the $\\ell_2$ norm, and $\\cdot^T$ denotes the transpose.\n\n> **ERGAS** measures the global statistical quality of the reconstructed HR-HSI. It is formulated as: $\\text{ERGAS}(\\mathbf{I}, \\tilde{\\mathbf{I}}) = \\frac{100}{c} \\sqrt{\\frac{1}{B} \\sum_{i=1}^{B} \\frac{\\text{MSE}(\\mathbf{I}^i, \\tilde{\\mathbf{I}}^i)}{\\mu_{\\tilde{\\mathbf{I}}^i}^2}}, $ where $c$ is the scaling factor, and $\\mu_{\\tilde{\\mathbf{I}}^i}^2$ is the square of the mean value of $\\tilde{\\mathbf{I}}^i$.\n\n>**SSIM** is used to assess the structural differences between GT and the reconstructed HR-HSI. It is defined as: $\\text{SSIM}(\\mathbf{I}, \\tilde{\\mathbf{I}}) = \\frac{1}{B} \\sum_{i=1}^{B} \\frac{(2\\mu_{I^i}\\mu_{\\tilde{I}^i} + C_1)(2\\sigma_{I^i\\tilde{I}^i} + C_2)}{(\\mu_{I^i}^2 + \\mu_{\\tilde{I}^i}^2 + C_1)(\\sigma_{I^i}^2 + \\sigma_{\\tilde{I}^i}^2 + C_2)}$. $\\mu_{I^i}$ and $\\mu_{\\tilde{I}^i}$ represent the mean values of $I^i$ and $\\tilde{I}^i$, while $\\sigma_{I^i}^2$ and $\\sigma_{\\tilde{I}^i}^2$ denote their variances. \n\n**A3.**  In addition to the two benchmark MHIF datasets used in our experiments, we have also tested our method on outdoor remote sensing hyperspectral image datasets, specifically the Pavia Centre dataset and the Chikusei dataset. The detailed experimental results are presented in the response to Reviewer EFSC's Question 6 for specific details.\n\n**A4.**  **How:** (1) Neural networks, including MLPs, typically exhibit a bias toward learning low-frequency components more readily than high-frequency ones. This spectral bias can limit the network's ability to capture fine details. Transforming latent codes into the Fourier domain allows the framework to improve high-frequency injection and accurate image reconstruction. (2) In the frequency domain, a single frequency point corresponds to multiple pixels in the spatial domain. This relationship effectively enlarges the receptive field of the INR when operating in the frequency domain. Consequently, the network can capture more extensive contextual information and spatial correlations, which enhances its ability to model complex spatial dependencies and results in improved image quality and detail preservation.\n\n**What:** Frequency-domain features effectively preserve high-frequency details, crucial for tasks like super-resolution and texture synthesis. (1) The Fourier transform captures global patterns and structures efficiently, aiding tasks requiring holistic data understanding. (2) Integrating spatial and frequency features using an SFID and Gabor wavelet activation improves image reconstruction accuracy and robustness.\n\n**A5.** In the proof of Theorem 1, we primarily discuss the time-frequency tightness property of the Gabor wavelet activation function. \n\n**The Gabor wavelet activation**: Please refer to the response to Reviewer EFSC's Question 5 for specific details.\n\n**Regarding Fig. 5**, (c), (d), and (e) compare the frequency characteristics of fusion images generated by different activation functions with the GT images. The model using the complex wavelet activation function demonstrates better frequency fitting compared to the model using ReLU. This is evident in the comparison between (c) and (d), where the frequency distributions in the optimal bandwidth region are closer. In contrast, the frequency distributions in (c) and (e) show significant differences. By selecting the optimal bandwidth, we ensure that the Gabor wavelet achieves the best performance in processing time-frequency information, thereby enhancing the overall performance of the network.\n\n**A6.**  In Fig. 7 of our manuscript, the PSNR curve represents the performances **on the validation set**, plotted using the checkpoints saved per 5k iterations. As shown at the 80k-th iteration, the image exhibits high details, and loss on the testing set is lower, proving the effectiveness of the proposed IFF module.\n\n**A7.** You are correct.  **The error maps in Figure 6 are indeed normalized** to visually highlight the differences between the ground truth (GT) and predicted images. The performance of different methods is quite similar in visual observation since we **only select three bands from multiple bands to display as RGB images**, which cannot show the differences between all the bands very clearly. In addition, **the multispectral image has 16 bits of data**, thus, the different methods of visual observation are even small. Usually, we demonstrated the difference by numerical experiment and spectral vectors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a Fourier-enhanced implicit neural fusion network specifically designed for the multispectral and hyperspectral image fusion task. The paper identifies the unique characteristics of the amplitude and phase of the latent codes in the high-frequency and low- frequency components, and proposes to enhance high-frequency details and expand the receptive field from the Fourier domain.  Extensive experiments are theoretical analysis are provided in the submission and the rebuttal for validation. After the rebuttal, all the reviewers give positive recommendation for this submission.
Decision: Accept (poster)