Joint Modeling of fMRI and EEG Imaging Using Ordinary Differential Equation-Based Hypergraph Neural Networks
Authors: YanZhang
Yang Gao
Min Li
Venue: NeurIPS 2025 poster ()
Keywords: fMRI
EEG
Multimodal modeling
Primary area: neuroscience_and_cognitive_science

Abstract: Fusing multimodal brain imaging has been a hot topic since different modalities of brain imaging can provide complementary information. However, due to the size of simultaneous recorded fMRI-EEG dataset being limited and the substantial discrepancy between hemodynamic responses of fMRI and neural oscillations of EEG, the joint modeling of fMRI and EEG images is a rarely explored area and has not yielded satisfactory results. Existing studies have also indicated that the relationships between region of interest (ROI) are not one-to-one when synchronizing fMRI and EEG. Current graph-based multimodal modeling methods overlook those information. Based on this, we propose a hypergraph based fMRI-EEG modeling framework for asynchronous fMRI-EEG data named FE-NET. To the best of our knowledge, this is the first attempt to jointly model asynchronous EEG and fMRI data as Neural ODEs based hypergraph. Extensive experiments have demonstrated that the proposed FE-NET outperforms many state-of-the-art brain imaging modeling methods. Meanwhile, compared to simultaneously recorded fMRI-EEG data, asynchronously acquired fMRI-EEG data is less costly, which demonstrates the practical applicability of our method.

Joint Modeling of fMRI and EEG Imaging Using Ordinary Differential Equation-Based Hypergraph Neural Networks
Yan Zhang, Yang Gao, Min Li' School of Computer Science and Engineering Central South University
Changsha, Hunan 410083 limin@mail. csu. edu, cn
Abstract
Fusing multimodal brain imaging has been a hot topic since different modalities of brain imaging can provide complementary information. However, due to the size of simultaneous recorded MMRI-EGG dataset being limited and the substantial discrepancy between hemodynamic responses of fMRI and neural oscillations EEG, the joint modeling of fMRI and EEG images is rarely explored area and has not yielded satisfactory results. Existing studies have also indicated that the relationships between region of interest (ROI) are not one-to-one when synchronizing fMRI and EEG. Current graph-based multimodal modeling methods overlook those information. Based on this, we propose a hypergraph based fMRI- EEG modeling framework for asynchronous fMRI-EEG data named FE-NET. To the best of our knowledge, this is the first attempt to jointly model asynchronous EEG and IMRI data Neural ODEs based hypergraph. Extensive experiments have demonstrated that the proposed FE-NET outperforms many state-of-the- art brain imaging modeling methods. Meanwhile, compared to simultaneously recorded fMRI-EEG data, asynchronously acquired fMRI-EEG data is less costly, which demonstrates the practical applicability of our method.
Introduction
A major goal in neuroimaging research is to develop predictive models aimed at analyzing the association between comprehensive functional connectivity patterns across the brain and behavioral traits, which can enhance the understanding of both normal brain function and dysfunction [I]. A robust modeling method plays a significant role in assisting the diagnosis of neurological disorders, predicting infant neurodevelopment, and assessing sleep quality 2 Usually, researchers model brain imaging in the form of graphs to accomplish this goal [5].
In recent years, modeling multimodal brain imaging has emerged focal point of research. fMRI indirectly measures brain activity through the Blood Oxygen Level Dependent (BOLD) signal, while EEG directly records brain activity by measuring the electric fields generated by cortical pyramidal neurons. EEG enables non- con-invasive observation of brain electrical activity and serves as a complementary modality to fMRI:
The integration of fMRT with EEG signals offers promising approach to bridging the gap between hemodynamic responses and neural oscillations, providing insights into the origins of fMRI signals and their large-scale patterns. However, deep tearning-based joint modeling asynchronously recorded fMRI and EEG is relatively underexplored area and has not yielded satisfactory results.
"Correspondigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The reasons for this issue are twofold. One reason lies in the incompatibility of the equipment. the sample size of simultaneously recorded IMRI-EGG datasets is markedly smaller than that of asynchronously acquired datasets, thereby limiting the reliability of extant research relying on synchronous recordings, another reason is the fundamental differences of the signal type make the data distribution between fMRI and EEG highly complex, thereby rendering it difficult to bridge the gap between them.
Existing studies [6] have indicated that the relationships between regions one-to-one when synchronizing fMRI and EEG, Hyperedges in hypergraphs can connect multiple nodes, providing more comprehensive description of this relationship.
of interest (ROI) are not
Neural ordinary differential equations (Neural ODEs) 7 can dynamically propagate information across arbitrary timestamps, accommodating mismatched sampling rates and asynchronous time points without manual alignment, the potential of using Neural ODEs for modeling asynchronous data has been widely reported []]
Based on the existing issues, propose IMRI-EEG modeling method based on Neural ODEs hypergraphs. Our main innovations are as follows:
1) We have designed a GAN-based IMRI-EEG hypergraph generation (FEH) module, which employs a novel Optimal IMRI-EEG Isomorph algorithm (OFEI) and Interactive Hyperedge Neurons (IHEN) to generate multiple hypergraphs at various temporal and spatial scales. The strength of GAN-based FEH lies in their ability to learn the complex data distribution between fMRI and EEG through synchronized operation of OFEI and IHEN, thus narrowing the gap between the hemodynamic responses and the neural oscillations.
2) We proposed a novel dynamic IMRI-EEG hypergraph embedding module (FED), which leverages Neural ODEs to capture latent temporal dependencies between IMRI's slow hemodynamic changes (Seconds) and EEG's rapid neural oscillations (milliseconds), Neural ODEs naturally handles incon- sistency sampling rates and temporal mismatches by learning differential equations governing system dynamics for asynchronous data.
3) To the best of our knowledge, this is the first attempt to model asynchronous EEG and fMRI data as Neural ODEs based hypergraph. Comprehensive experiments show that the performance of the proposed FE-NET surpasses many state-of-the-art methods, compared simultaneously recorded fMRI-EEG data. Asynchronously acquired fMRI-EEG data is less costly, which demonstrates the practical applicability of our method.
2 Related Work
2.1 Hypergraph Learning
In recent years, the efficacy of hypergraphs in modeling and comprehending intricate correlations has become evident. Initially introduced by [9) hypergraph learning embodies a transductive learning approach, conceptualized
dissemination process within the hypergraph structure.
Motivated by the remarkable success of deep learning, some researchers have ventured into developing deep hypergraph learning techniques. For instance, Feng et al. [10] introduced hypergraph neural intricate correlations. Unlike Graph networks (HGNN) to model and understand serte--hpperede-veerte information propagation schema Neural Networks (GNN), HGNN devises a vertex-hypered to iteratively glean data representation. Bai et al. [IT] delved deeper into the attention mechanism on hypergraphs, aligning with the hypergraph convolution paradigms delineated in HGNN. Inspired by [12), Hyper-Atten introduces hyperedge-vertee attention learning module to dynamically discern the importance of different vertices within the same hyperedge, thereby unveiling intrinsic correlations. Moreover, Yadati et al. [3] proposed HyperGCN, a method for training a Graph Convolutional Network (GCN) on hypergraphs for semi-supervised learning. HyperGCN is structured based on the spectral theory of hypergraphs. Initially, HyperGCN transforms a hypergraph into a simple weighted graph via specific strategy and subsequently executes standard GCN on the graph to glean data representations.
As elucidated, most prevailing deep hypergraph learning methods stem from the spectral theory of hypergraphs. Consequently, these methods are typically defined on static hypergraphs system, constraining their applications. Failing capture the continuous evolution of IMRI-EEG hypergraph

---

representations. Many important information in the dynamic continuous representations between fMRI and EEG has been lost.
DIRI-EEG Hypergraph Generation (FEH) module
Dynamic IMRI-EEG Hypergraphs (FED) Layer
Optinell (MKL BEG isomorph algorihm (OFE))
Diffusion
to grres
(G,
Left
liet
ta
X,to)
Xyllees)
CICANI
DSCK,
XCrurb
Hegirgaph
(t
Dicriminator:
E241
LSTM
Attestiv baak
EBD agral
entlay Rippetikke Probehdikl For Aeriiees Hepprtiigs
Figure I: Overall process of GAN-based hypergraph generation (FEH) module and Dynamic hyper- graphs embedding (FED) module.
Method
As shown in Figure our method consists of two steps. Firstly, multiple fMRI-EEG hypergraphs are generated at various temporal and spatial scales by GAN-based fMRI-EEG hypergraph generation module. Then, these hypergraphs are inputted into the dynamic fMRI-EEG hypergraph embedding module.
3.1 Preliminary
Notations. the hypergraph. each hyperedge, which is subset of the vertex set, exhibits a distinction from the simple graph. Typically, hypergraph is defined as G (V,E), where V and & denote the vertex set and hyperedge set, respectively. The hyperedges are represented by an incidence matrix H E {0, | with its elements defined as Hy = 1(v E e) utilizing the indicator function V is determined by the summation of the hyperedge occurrences Hy over the hyperedge set E, denoted as d(t:) = Eeee Hy Similarly, the degree of hyperedge
1(.). The degree of vertex
E È is determined by the summation of the vertex occurrences He.e over the vertex set V. denoted as 5(e) = Evey Hv,e. The diagonal degree matrices for vertices and hyperedges are represented as D = diag(d) and D = diag(O (6), respectively.
Hypergraph neural networks. The majority of contemporary hypergraph neural networks adhere the message-passing framework. Within each layer, the input vertex representations initially aggregated into hyperedges, and subsequently, the output vertex representations are derived from the corresponding hyperedges. Formally, in the k-th layer, the output vertex representations x!! (k) computed based on the previous representations through the employment of the aggregation function AGG and the update function UPD follows:
.r(A)) UPD (n((+-)) AGG (fa! (.()))))) xx()) = AGG (fx! E No(ee)
(1)
where (v) and Nn, (e) correspond to the vertex and hyperedge neighbor functions, respectively.
3.2
GAN-based IMRI-EEG hypergraph generation (FEH) module
3.2.1
Optimal EMRI-EEG Isomorph algorithm (OFEI)
As shown Figure D (left), each region of interest (ROI) is conceptualized as a node within the framework. Initially, the Dynamic Hypergraph Construction algorithm (DHC) defined in [14], is

---

employed to compute the initial hypergraphs {G for each subject within the dataset, where t represents the aggregate number of samples. If these mitial hypergraphs {Gr, J are directly utilized in the generative process to achieve multimodal connectivity, the robustness of the resultant generation may be compromised. algorithm (OFEI), which is capable of calculating a hypergraph G' that exhibits optimal isomorphism relative to the initial hypergraphs {G+).
response to this limitation, we design a novel Optimal fMRI-EEG Isomorph
Let G (V, B) and = (V, E") share the same vertex set and have the same number of hyperedges,
m E E'I. Index the edges as fei..... em ] and E' {ei..... en For an edge e, let Va (e) C V denote its incident-vent vertex set. Define the edge-wise Jaccard similarity
(V(e)) n Var (e')l [V(e) U (c')
s(c,e')
(2)
The homogeneity between G and G' is
Sim G')
(e, 6o()) max a m
(3)
i... the maximum over bijections f: E - E' (since E = E.. Writing H(V,m) for the class of hypergraphs V with hyperedges, OFEI solves
G E arg GEH(V
max
Sim (G, G)).
(4)
With the optimal isomorph G in hand, we construct subject- hypergraphs by concatenating G* with each G²
:= G|
(5)
where denotes edgewise concatenation on the common vertex set (allowing multiplicities if applica- ble). The corresponding incidence matrices are then obtained as
H₂ = (GL) (equivalently, H (G")) ((
(6)
The rationale behind OFEI lies in its ability to establish a metric for measuring the similarity among hypergraphs, thereby enhancing the correlation between fMRI and EEG data. This ensures the generated hypergraph preserves the topological consistency of brain connectivity across modalities,the data alignment and node correspondence part are provided in supplementary material.
3.2.2 Generator formed by interactive hyperedge neurons module
For the k-th participant, we initialize node representations A0 RR-x directly from their BOLD signal matrix Bg. where N denotes the ROI count and d the temporal length. Simultaneously.
hyperedge representations e(() E RRxN N are constructed via E(00 H Sx. where S₁ N is the effective connectivity matrix, H₃ RNxm is the incidence structure, and m represents the hyperedge cardinality.
The generative architecture G consists of L cascaded Interactive Hyperedge Neuron (IHEN) transfor- mations, For layer index E {0, 1, .............. 1}, the IHEN propagation rules are:
N (e+1))
T n (2))
((+1)
NT + ne
where O and ( are learnable transformation matrices for nodes and hyperedges respectively, and 17 is mixing coefficient.
From the terminal representations A(LL) and &x we compute node-to-hyperedge affinities and hyperedge importances. Define the affinity of node or to hyperedge éj as:
(1,( Ha(i.)) ( [i, el ( [i:
(8)
and the importance weight of hyperedge as:
(x) | [G. =
(9)

---

The multimodal connectivity tensor M₁ RNXN is then assembled via:
m
Mal.
Σ wx(l) ok.(p) (x.
(10)
==1
Finally, node-wise correlation scores are computed by integrating connectivity strengths with feature similarities:
N
(x) N q=1
Σ
Muli.gl (1:. [a.
(11)
3.2.3 Discriminator and Loss Function
The discriminator D utilizes a standard multilayer perceptron architecture. Given an arbitrary initial node uo. we initiate random walks from this origin. For walk of length T terminating at ur, the traversal sequence
(wo, ui...., WT-1; ur, (T+1) satisfies the boundary condition WT+1 = 1, establishing "r as the terminal node.
For the functional connectivity matrix F3 of subject k, the probability of realizing trajectory u is:
F& Fx(u-1,,4)
F Fu(ur, ur-1)
P (u) II
(12)
Σ, ((
E, Fu(T,5))
Denoting Q(uo v) as the collection of all paths from un destination the marginal probability is:
pemp (v uo)
Σ
peup (7)
(13)
TEQ(we
For the generated multimodal connectivity matrix MR, the corresponding trajectory probability becomes:
T
M uz)
Mg (ur, wr-1)
pgen
uu)
X
(14)
Mi (u=-1,
En Mp (u(ur,j)
with marginal distribution:
pe (v uo)
P (7)
(15)
7ER( -t))
The discriminator objective function is formulated as:
LD max
E ((o D(v, un)) + Eu (lwn)) D(u, wn)il]]
(16)
where represents the cohort size and N the number of nodes.
The generator objective, driven by discriminator feedback, is:
LG
5 Pap (1u,,) flog D(v, u.))
(17)
The OFEI framework achieves optimal structural correspondence across multimodal IMRI-EEG hypergraph representations, while the IHEN mechanism dynamically modulates hyperedge weights (i) through learned node-hyperedge affinities, emphasizing neurobiologically significant connec- tivity patterns (e.g., functionally coherent brain networks).
The FEH module combines structural optimization (OFEI), dynamic feature fusion (IHEN), and distribution alignment (GAN) to generate hypergraphs that bridge fMRI and EEG's gaps at various temporal and spatial scales, thus can narrow the gap between the hemodynamic responses and the neural oscillations, subsequently, fMRI-EEG hypergraphs will be sent into the dynamic FMRI-EEG hypergraphs embedding module for feature extraction.

---

3.3 Dynamic IMRI-EEG hypergraphs embedding (FED) module
In this section, we first give the definition of dynamic MMRI-EEG hypergraph systems and provide specific dynamic fMRI-EEG hypergraph systems based on ODE. Next, we describe the detailed neural implementation of the FED framework, the overall process of FED module shown in Figure
3.3.1 Definition of dynamic IMRI-EEG hypergraph systems
Given a hypergraph G. a corresponding vertex feature matrix Z. E RIVixe and a corresponding hyperedge feature matrix Ze E RiElxe our goal is to learn vertex representation Y and a hyperedge representation y.. We first define dynamic fMRI-EEG hypergraph systems based on the following equation:
x X. xx= =
(
([x:
x (1 X (t) 1) and
X (0)) X (
]
Z, Z
(18)
Here, (1) and (t))) denote the matrices representing vertex and hyperedge features at time L, respectively. The function f encapsulates the rate of change within the dynamic system, with Zu and Z. representing the initial conditions for vertex and hyperedge features, respectively. The persistence of timestamp allows the hypergraph dynamic systems continuously update representation states.
ODE-based hypergraph dynamic system. The velocity function f in Equation 18) may be charac- terized variably. It is proposed interpret the velocity function composite of control function and a diffusion function, thereby formulating an ODE-based hypergraph dynamic system in the following manner:
C. X v (1) ]
C. ( (1) [x
X v X.
]
X (1( X xx(1)
H
(19)
In the initial segment of the equation, C. and C. serve control functions, which correspond the velocity of control for the representations of vertices and hyperedges, respectively. The subsequent segment represents the diffusion term.
X X. (T)
] [ X (0) x
('''([
x.(t) (x()) 1)
(20)
Given the initial vertex features (0) and hyperedge features (0) as inputs, the corresponding vertex representations x, (T)) and hyperedge representations X. (T') time T are generated through integration, as outlined in Equation 20), for learning tasks within the hypergraph.
We anticipate proposing mulli-layes neural network framework, denoted aED Initially, we adopt the Lie-Trotter splitting method, as delineated by Geiser (15), for the discretization of Equation(T9), delineated below:
X X t+
x
C. (t,(t))
x.(t)
+
C. (x.
(21)
X (t+
X
X (1+
X (t
x (t+
X (t+
In the model, the time step is incorporated into control functions C. and Ce, well as the diffusion matrix A. Lie-Troc Tio-Troter discretization strategy allow separates control amodality-specifff dynamics) part and diffusion (cross-modal interactions) part, allowing asynchronous updates of fMRI and EEG features while preserving their intrinsic temporal characteristics.
3.3.2 Neural implementation of the FED ode framework
we delineate the integration of the oE framework into the antecedent analysis by elucidating the neural instantiation of both the control and diffusion phases within the FED layer, separately. A graphical representation of the oE framework is provided in Figure (right).
Neural implementation of control step. To extract spatio-temporal information from fMRI and FEG data, we propose Dual-Stream Control Function (DSCF). As illustrated Figure [ DSCF incorporates components where F, L, and A represent, respectively, fully connected layer, LSTM

---

cell in an RNN, and an attentive block aligned with an RNN output defined in literature [[6]. The control function of FED ode layer is specifically articulated through the subsequent equation:
X x x (
[ [
X X (t)
+
(t) ]
DSCFF(X.())
(22)
In this study, the temporal dynamics of brain activity can convey essential information. Therefore, using LSTM to capture the spectral variations within the activation sequence of networks is rational approach. The Attention Neural Network (ANN) [7 applies distinct weights various output steps of RNNs to prioritize the most critical time periods, thereby playing crucial role in subject-independet hypergraph encoding. Input and output parameters for FN, LSTM, and ANN are provided in the Supplementary Material.
Neural implementation of diffusion step. The configuration of the diffusion matrix A is paramount. Should it be improperly formulated, both vertex and hyperedge representations are prone to diverge and become intractable, delineated follows:
X Xt, Xx((+1)
X (t+ ! !
A
(1+
X (1+ 2
(23)
0.5 ( + a.
+ 0:gy D. H
=
H
a
In the context where ay and ae serve hyperparameters, these denote the teleportation probabil- ities associated with vertices and hyperedges, respectively. We proceed to decompose the matrix
multiplication term derive the vertex representation: (t + 1) = (1 a.) X, (t + 5) + ( a.) H (t + 2). Here, the initial term indicates that vertex representations are maintained with a preservation factor of ay during the diffusion process, while the latter term signifies the integration of representations from hyperedges directly linked to each vertex, weighted by ag. In this arrangement, HX, à (t + 1)) corresponds vertex-level aggregation, and D,"" serves normalization matrix for averaging.
Control part in FEH refine modality-specinf features (i.e., isolating fMRI spatial patterns from EEG spectral power). The Diffusion part can propagates information across the hypergraph structure, capturing long-range dependencies (i.e., cross- modal interactions between distant brain regions and transient EEG signals). The control-diflusion mechanism FEH ensures that hypergraph representations evolve smoothly over time, mimicking the gradual emergence of stable brain states.
4 Experiments
4.1 Data and Pre-processing
The fMRI-EEG dataset we used is LEMON, which is open access and was referenced in the literature (18), encompasses data from 227 healthy participants. This cohort includes young group (N=153, age 25.1+3.1 years, range 20-35 years, 45 females) and an elderly group (N=74, age 67.6+4.7 years, range 59-77 years, 37 females). Participants underwent resting-state fMRI and 62-channel EEG experiment. The preprocessing protocol for rs-IMRI data was conducted utilizing the Nipype framework defined in [8). Data preprocessing procedures for EEG were executed utilizing EEGLABI0S (version 14.2.1b 19 The preprocessing entailed the application of principal component analysis (PCA) for the purpose of dimensionality reduction. Participants also underwent the Test of Attentional Performance (TAP), which assessed their capacity for sustained attention; the Trail Making Test (TMT), which measures cognitive flexibility; the Vocabulary Test (WST), which indicates the measurement of verbal intelligence level and the assessment of language comprehension; and the LPS-2, which measures logical or inferential thinking and quantifies fluid intelligence. We divided the test results of the TAP, TMT, WST, and LPS-2 into two categories: high scoring group and low scoring group, which served as indicators to measure the effectiveness of the model. enable comprehensive and equitable benchmarking against existing approaches, we introduce commonly used small-scale synchronized datasets named CN-EPFL dataset (00 with 20 individuals. Further details on datase tand Pre- processcin can be found in Supplementary Material.
7

---

4.2 Implementation Details for Experiment
We validated the performance of our model by differentiating high scoring group and low scoring group. The 5-fold cross-validation (CV) scheme was used to obtain a reliable evaluation on the performance of competing models and the proposed FE-NET. Classification performance was assessed using multiple metrics, including classification accuracy (Acc), precision (Pre), and specificity (Spe).
For dynamic IMRI-EEG hypergraphs embedding module, the ae and ag is set as 0.05 and 0.9, the learning rate we set is 10-2 weight decay we set is 5 x 10-4, the dropout we used is 0.2, we use multi-layer Interactive Hyperedge Neurons modules to form the generator for fMRI -EEG hypergraphs, and the is we set is 0.6. The parameter settings and introductions of competing models are shown in Supplementary Material.
Table 1: Comparing FE-NET's performance (average accuracyastandard deviation) with other state-
of-the-art methods
TAP-WST
Method
Acc
Pre
Spe
TMT-LPS
Acc
Pre
Spe
BrainGNN
67.23=2.15
71.8946.4 42
70.59=3.17
61.3823.22 222
60.54+3.85
61.27=4.36
BCG 22
70.19.4.82
70.5745.03
65. 15.3914.21
60.8347.61
65.71a6.92
62.832.449 49
High
SGP-SL
23
55.62=4.17
59.4814.76
56.29:6.84
High
59.35+3.24
61.0945,73
57.9243.18
Sustained
Cross-GNN
24
74.83+5.29
71. 71.06+5.47
74.35+4.82
Cognitive
68.57+3.63
63.82+5.49
62,45+3.71
Atteation:
TAN 225
73.12+5.37
70.49+3.85
71.84=4.93
Flexibility VS.
68.29+6.83
66.71=5.03
63.6243.47
VS.
RH-BranFS
26
74.2615.13
71.9344.58
73,74=3.81
61.29H13.51
66.3844.27
66.5425.18
Low
MCRLN
27
65.71+4.53
70.28+2.94
69.83=6,21
Low
66.47+5.76
61.73=4.62
62.84+3.49
Sustained
MMP-GCN 28
79.42s5.19
70.5845.49
71.95:3.27
Cognitive
66.2916683
68.3745.68
61,9347.44
Attention
FE
85.29:2.57
83.4623.19
80.83:4.05
Flexibility
79.18+3..47
75.29:4.83
74.3845.62
FE-NETOoFTH
78.45+3.82
78.63422.47
72.91+2.73
73.57+3.64
69.2744.18
66 1,9
FE-NETnoFED
79.35+2.67
80.1745.83
76.82=6.45
70.46+8.55
67.38+5.29
67 07.49+5.84 84
BrainGNN
65.93=3.74
68.27144.85
67.3813.29
60.84224.57
61.2913.87
60.9324.18
M-GAT BC 22
68.5323.18
69.4714.63
64.72:3.85
62.38a4. 19
64.5915.47
61.73:2.94
High
SGP-S1.
56.84+4.73
60.38+4.19
58.62-5.84
High
59.47+3.29
61.83+5.29
37.46+1885
Verbal
Cross-GNN
24
72.59=5.47
69,84+5.18
73,27=4.73
Logical
68, 04.29+3.55 585
63,47=5.39
62.84+3.29
Intelligence
TAN 25
71.38+4.85
67.2943.18
69.47=4.29
Intelligence VS.
68.53+6.47
66,82+5.84
43.1944447
VS.
RH BrainFS 26
73.6224.18
68.5344.73
72.8413.85
61.2911..47
66.3814.29
66.4745.18
Low
MCRLN
27
64.82=4.73
69,4742.85
68.29:6.47
Low
66.53+5522 29
61.8424.18
62.38+3.19
Verbal
MMP-GCN
28
77.29L5.84
68.5345.47
70.3813.19
Logical
66.8416.73
68.2945.39
61.4747477
Intelligence
FE. NET
82.19:2.84
80.4743.85
78.53+4.73
Intelligence
77.38+3.47
73,29:4.85
72.84.5.47
FE-NETOOFEH
78.29+3.85
77.3842.88 84
73.62+2.47
73.4743.29
69.53+4.18
66. 29+3.85
FE-NETooFED
79.84=2.73
79.6245.29
75.47=6.18
70.53+8.47
67.38+5.84
67.29=5.47
4.3 Overall Evaluation
These competing methods and our FE-NET can be categorized into two groups: (Group A): fMRI or EEG modeling method including BrainGNN, M-GAT-BC, and SGP-SL, (Group B): Multimodal brain imaging modeling method including Cross-GNN, TAN, RH-BrainFS, MCRLN, and MMP-GCN.
As shown in Table [, on one hand, the proposed FE-NET generally significantly outperforms three single-modal brain imaging modeling methods in four different tasks by a large margin, which substantiates the effectiveness of the fusion between IMRI and EEG.
On the other hand, compared with five multimodal brain imaging modeling methods, our FE- NET yields consistently better results in all metrics, Cross-GNe aimed at capturing inter-modal dependencies through dynamic graph learning and mutual learning mechanisms. Specifically, the inter-modal representations between modalities. However, Cross-GNN can not eliminate complex data distribution between fMRI and EEG. In contrast, our approach successfully narrow the gap between the hemodynamic responses and the neural oscillations through synchronization operation of GAN-based OFFI and IHEN.
carefully coupled into the combined space to infer the dependencies
TAN introduces triple network to extract discriminative information from the high-order represen- tation feature space obtained from estimate the importance of brain regions and utilizes the cross-attention mechanism to extract comple- mentary information from different modalities. However, TAN overlooks the absence of a one-to-one correspondence between fMRI and EEG and used static systems, resulting in the loss of complemen- tary information between the two modalities. In contrast, our method depict the continuous dynamics of hidden representation by Neural ODEs, thereby capturing this relationship.
data. It incorporates self-attention to dynamically
Generalization capability evaluation: To further evaluate our generalization capability, we con- ducted additional benchmarking the synchronized CN-EPFL datasets. The result is shown in
8

---

Supplementary Material, all methods were trained on LEMON and tested on CN-EPFL, all models were both trained and tested on CN-EPFL. The results demonstrate that our model still exhibits significant advantages, whether generalizing to task-related datasets or directly utilizing synchronized datasets, further validating its superiority.
(a)Traditional Graph embedkling
(b) FE-NET embedding
[6) Hypergraph embeddling
1-
1.6
EEG
EEG
0.8
0.8
BARI
NR
0.6
0.8
0.4
(MRI embedding
embedding
(MRI embedding
0.5
0.2
-02
-0.2
-04
-0.4
-0.6
-0.5%
-0.5
-0.5
-0.5
EEG embedding
EEG embedding
EEG embedding
Figure 2: Visualization of the modality gap between the EEG and fMRI modalities.
4.4 FED, and FEH Modules Analysis
As shown in Table 1 we conducted ablation experiments on FED and FEH modules, PE-NETnOFEH and FE-NETnoFED are variants of FE-NET. FE-NETnoFEH employs initial hypergraphs generated by DHC algorithm as inputs for the FED module, whereas FE-NETnoFED utilizes a traditional hypergraph neural network for graph embedding. The superior performance of FE-NET compared to these two variants demonstrates the effectiveness of the modules we proposed. We have attempted to use traditional methods to model the process of transitioning from x(tx) to z x(tx+1). As shown in Supplementary Material, the performance of FE-NET (RNN, LSTM, GRU) is not effective as FE-NET.
The ablation study of OFEI and IHEN in FEH is conducted by evaluating the average accuracy in four downstream tasks. The result is shown in the Supplementary Material. The variants of FED that exclusively employ OFEI or IHEN are referred to as "FEH(OFEI)" and "FEH(IHEN)", respectively. The performance of FEH outperforms "FEH(OFFI)" and "FEH(IHEN)" demonstrating its effectiveness. Additionally, the efficacy of the control and diffusion processes within the FED validated. The variants of FED that employ exclusively control processes or diffusion processes are denoted as "FED(Con)) and "DED(Diff) respectively. A comparative analysis of the two processes reveals that the inclusion of only the control process or diffusion process, which relies solely on initial vertex features and neglects the hypergraph structure, results in inferior outcomes. We provided an algorithm complexity and computation resources analysis of FEH-FED and comparison of computational resources with other SOTA methods in the Supplementary Material.
Modality embedding space analysis: As demonstrated in 22 modality gap exists within multimodal learning, wherein information from different modalities is situated in entirely distinct embedding spaces. This modality gap is correlated with model performance.
To gain a deeper understanding of the performance improvements attributed to the proposed Neural ODEs-based hypergraph strategy, we present visualization of the modality gap between the EEG and MMRI modalities within the LEMON dataset as shown in Figure [ A comparison of FE-NET with the hypergraph and traditional graph embedding reveals that traditional graph embedding results in larger modality gap, which in turn leads to decreased performance. Meanwhile, compared to hypergraph embedding, FE-NET enhances the clustering of embeddings between EEG and fMRI modalities in the embedding space. Notably, the embedding spaces of certain ROI in fMRI are relatively close to each other Figure 2 (b) when compared to hypergraph embedding Figure
2 (c). Multiple studies suggest that some ROI of fMRI show strong correlation with EEG[O) whereas some show no significant association. These observations provide interpretability for the superior performance of Neural ODEs- hypergraphs in FE-NET.

---

In Top Ih webllburite
HOE
FOR wondiaeeoee
Figure 3: Functional connectivity of verbal intelligence and logical intelligence networks analysis
4.5 Functional connectivity networks analysis
To identify which FC are most discriminative, use the method described in the literature to rank the important Hyperedges. We summarize the top-120 verbal intelligence and logical intelligence discriminative FC and visualize them for analysis in Figure The FC associated with verbal intelligence, as shown in Figure aa,, exhibits the highest scores predominantly within the Default Mode Network (DMN), fundamental FC network consistently identified in fMRI studies [32. Within the DMN, the FCs demonstrating the strongest predictive capacity are primarily localized the dorsal anterior cingulate cortex and the posterior superior temporal cortex. These regions, positioned at the intersection of the frontal and temporal lobes, constitute key components of the theory of mind network. contrast, as shown Fig the FC most strongly associated with logical intelligence is primarily situated within the VN. Furthermore, the verbal intelligence-relateee FC in Figure a) associated with the posterior superior temporal cortex has been ascertained to possess reliable predictive value within the Ventral Attention Network (VAN) (33] The FCs with the highest efficacy are located in the parahippocampal gyrus, and superior parietal lobule [34].
Conclusion
This paper designed novel fMRI-EEG modeling method based Neural ODEs named FE-NET. FE-NET performs substantially better than many state-of-the-art methods. The contributions of our research are as follows:
1) We propose a unified framework with two key components: FEH module and FED module. FEH can synthesize hypergraphs multiple spatial and temporal scales, allowing the GAN to learn the joint distribution of hemodynamic and neural-oscillation signals and thus narrow the modality gap. FED, by contrast, uses Neural ODEs to capture latent temporal dependencies, inherently reconciling inconsistent sampling rates and temporal misalignment through learned differential equations governing asynchronous data dynamics.
2) For the first time, we attempt to model asynchronous fMRI and EEG data as Neural ODEs based hypergraph. Compared to previous studies that relied on tiny-sized synchronized datasets, the robustness and practicality of our model are further ensured.
Extensive comparative experiments and ablation studies validate the effectiveness of our proposed model and the corresponding two modules. Meanwhile, compared to simultaneously recorded fMRI- EEG data, asynchronously acquired single modality of EEG data is significantly less costly, which demonstrates the practical applicability of our method.
Acknowledgements
We would like to thank the reviewers and the chairs for their suggestions and efforts.
References
[1] S. M. Plis, D. R. Hjelm, R. Salakhutdinov, E. A. Allen, J. Bockholt, J. D. Long, H. Johnson, S. Paulsen, J. A. Turner, and D. Calhoun, "Deep learning for neuroimaging: a validation study," Frontiers in neuroscience, vol. 8, p. 229, 2014.
10

---

[2] Y. Ma. Q. Wang, L. Cao, Li, C. Zhang, L. Qiao, and M. Liu, "Multi-scale dynamic graph learning for brain disorder detection with functional mri," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 31, no. 42, pp. 3501-3512, 2023.
[3] Kawahara, C. J. Brown, S. P. Miller, B. G. Booth, V. Chau, R. E. Grunau, J. G. Zwicker, and G. Hamarneh, "Brainnetenn: Convolutional neural networks for brain networks; towards predicting neurodevelopment. Neurolmage, vol. 146, pp. 1038-1049, 2017.
[4] Z. Jia, Y. Lin, J. Wang, R. Zhou, X. Ning, Y. He, and Zhao, "Graphsleepnet: Adaptive spatial-temporal graph convolutional networks for sleep stage classification." in Ijcai, vol. 2021, 2020, pp. 1324-1330.
[5] W. B. Bruin, L. Taylor, R. M. Thomas, J. P. Shock, P. Zhutovsky, Y. Abe, P. Alonso, S. H. Ameis, A. Anticevic, P. D. Arnold al., "Structural neuroimaging biomarkers for crressive-compuliii disorder in the enigma-ocd consortium: medication matters," Translational psychiatry, vol. 10, no. p. 342, 2020.
[6] R. M. Cichy and A. Oliva, "Am/eeg-fmri fusion primer: resolving human brain responses in space and time," Neuron, vol. 107, pp- 772-781, 2020.
[7] R. Chen, Y. Rubanova, J. Bettencourt, and D. K. Duvenaud, "Neural ordinary differential equations," Advances in neural information processing systems, vol. 31, 2018.
[8] Y. Sun, L. Zhang, and H. Schaeffer, "Neupde: Neural network based ordinary and partial differ- ential equations for modeling iieedependent data," in Mathematical and Scientific Machine Learning. PMLR, 2020. pp. 352-372.
[9] D. Zhou, J. Huang, and B. Schölkopf. "Learning with hypergraphs: Clustering, classification, and embedding." Advances in neural information processing systems, vol. 19, 2006.
[10] Y. Feng. H. You, Z. Zhang, R. Ji, and Y. Gao, "Hypergraph neural networks," in Proceedings of the AAAI conference on artificial intelligence, vol. 33, no. 01, 2019, pp. 3558-3563
[11] S. Bai, F. Zhang, and P. H. Torr, "Hypergraph convolution and hypergraph attention," Pattern Recognition, vol. 110, 107637, 2021.
[12] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio, "Graph attention 10903, 2017.
networks," arXiv preprint
[13] N. Yadati, M. Nimishakavi, P. Yadav, V. Nitin, A. Louis, and P. Talukdar, "Hypergen: A new method for training graph convolutional networks on hypergraphs," Advances in neural information processing systems, vol. 32, 2019.
[14] J. Jiang, Y. Wei, Y. Feng, J. Cao, and Y. Gao, "Dynamic hypergraph neural networks." in IJCAI, 2019, pp. 2635-2641.
[15] J. Geiser, Decomposition methods for differential equations: theory and applications. CRC Press, 2009.
[16] W. Song, C. Shi, Z. Xiao, Z. Duan, Y. Xu, M. Zhang, and J. Tang, "Autoint: Automatic feature interaction learning via self-attentive neural networks," in Proceedings of the 28th ACM international conference on information and knowledge management, 2019, pp. 1161-1170.
[17] P. Zhou, W. Shi, J. Tian, Z. Qi, B. Li, H. Hao, and B. Xu, "Attention-based bidirectional long short-term memory networks for relation classification," " in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), K. Erk and N. A. Smith, Eds. Berlin, Germany: Association for Computational Linguistics, Aug. 2016, pp. 207-212. [Online]. Available: https://alanntoooggoo////....
[18] A. Babayan, M. Erbey, D. Kumral, J. D. Reinelt, A. M. Reiter, J. Röbbig, H. L. Schaare, M. Uhlig, A. Anwander, P.-... Bazin et al., "A mind-brain-bode dataset of mri, eeg, cognition, emotion, and peripheral physiology young and old adults." Scientific data, vol. 6, no. 1, pp- 1-21, 2019.
11

---

[19] A. Delorme and S. Makeig, "Eeglab: an open source toolbox for analysis of single-trial eeg dynamics including independent component analysis," Journal of neuroscience methods, vol. 134, no. 1, pp. 9-21, 2004.
[20] M. Pereira, N. Faivre, I. Iturrate, M. Wirthlin, L. Serafini, S. Martin, A. Desvachez, O. Blanke, D. Van De Ville, and J. d. R. Millán, "Disentangling the origins of confidence in speeded perceptual judgments through multimodal imaging," Proceedings of the National Academy of Sciences, vol. 117, no. 15, pp. 8382-8390, 2020.
[21] X. Li, Y. Zhou, N. Dvornek, M. Zhang. S. Gao. J. Zhuang, D. Scheinost, I. H. Staib, P. Ventola, and J. S. Duncan, "Braingnn: Interpretable brain graph neural network for fmri analysis." Medical Image Analysis, vol. 74, 102233, 2021.
[22] R. Yu, C. Pan, X. Fei, M. Chen, and D. Shen, "Multi-graph attention networks with bilinear con- volution for diagnosis of schizophrenia," IEEE Journal of Biomedical and Health Informatics, vol. 27, no. 3, pp. 1443-1454, 2023.
[23] Chen, Y. Guo, S. Hao, and R. Hong. "Exploring self-attention graph pooling with eeg-based topological structure and soft label for depression detection," IEEE transactions on affective computing, vol. 13, no. 4, pp. 2106-2118, 2022.
[24] Y. Yang, C. Ye, X. Guo, T. Wu, Y. Xiang, and T. Ma, "Mapping multi-modal brain connectome for brain disorder diagnosis via cross- mutual learning," IEEE Transactions on Medical Imaging, 2023.
[25] Q Zhu, H. Wang, B. Xu, Z. Zhang, W. Shao, and D. Zhang, "Multimodal triplet attention network for brain disease diagnosis," IEEE Transactions on Medicul Imaging, vol. 41, no. 12, pp. 3184-38844 2022.
[26] H. Ye, Y. Zheng, Li, K. Zhang, Y. Kong, and Y. Yuan, "Rh-brainfs: Regional heterogeneous multimodal brain networks fusion strategy," Advances in Neural Information Processing Systems, vol. 36, 2024.
[27] Y. Kong, W. Wang, X. Liu, S. Gao, Z. Hou, C. Xie, Zhang, and Y. Yuan, "Multi-connectiitty representation learning network for major depressive disorder diagnosis," IEEE Transactions on Medical Imaging, 2023.
[28] X. Song, F. Zhou, A. F. Frangi, J. Cao, X. Xiao, Y. Lei, T. Wang, and B. Lei, "Multicenter and multichannel pooling gen for early ad diagnosis based on dual-modality fused brain network," " IEEE Transactions on Medical Imaging, vol. 42, no. 2, pp. 354-367, 2023.
[29] V. W. Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou, "Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning." Advances in Neural Information Processing Systems, vol. 35, pp. 17 616-2-17 625, 2022.
[30] T. Nazneen, I. B. Islam, M. S. R. Sajal, W. Jamal, M. A. Amin, R. Vaidyanathan, T. Chau, and K. A. Mamun, "Recent trends in non-invasive neural recording based brain-to-brain synchrony analysis on multidisciplinary human interactions for understanding brain dynamics: a systematic review," Frontiers in Computational Neuroscience, vol. 16, p. 875282, 2022.
[31] J. Liu, W. Cui, Y. Chen, Y. Ma. Q. Dong, R. Cai, Y. Li, and B. Hu, "Deep fusion of multi- template using spatio-temporal weighted multi-hypergraph convolutional networks for brain disease analysis." IEEE Transactions Medical Imaging, 2023.
[32] L. Naci, A. Haugg, A. MacDonald, M. Anello, E. Houldin, S. Nagshbandi, L. E. Gonzalez- Lara, M. Arango, C. Harle, R. Cusack et al., "Functional diversity of brain networks supports consciousness and verbal intelligence," " Scientific reports, vol. 8, no. 1, p. 13259, 2018.
[33] E. Dryburgh, S. McKenna, and I. Rekik, "Predicting full-scale and verbal intelligence scores from functional connectomic data in individuals with autism spectrum disorder," Brain imaging and behavior, vol. 14, no. 5, pp. 1769-1778, 2020.
[34] N. He and C. Kou, "Prediction of individual performance and verbal intelligence scores from resting-state fmri in children and adolescents." International Journal of Developmental Neuro- science, vol. 84, no. 7, pp. 779-790. 2024.
12

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
accurately reflect
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [No]
Justification:
limited pages
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
13

---

Justification:
limited pages
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer:[Yee Yes]
Justification: code will provided
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [No]
Justification: code will be provided
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guuii wwwticccguie//eeieeeeeeii CoodeSubeiiieeee ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Can be find in Implementation Details for Experiment section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
Can be find in Overall Evaluation section
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: limited pages
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics /tap:::///uripse/ http://......p.....tttttttttaeea
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: in conclusion section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
16

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[Yes
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whw.ree.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
17

---

Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA].
18

---

Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy htttap/s//aarrrppocccoooooorrres for what should or should not be described.
A Technical Appendices
All additional results can be downloaded and found in the supplementary material.
19


Reviews:
[{"review_id": "9wj6jwY8Oa", "reviewer": "Reviewer_ohye", "summary": "This paper presents FE-NET, a novel framework for jointly modeling fMRI and EEG brain imaging data, specifically addressing the challenge of asynchronous acquisition. Evaluated on the LEMON dataset, FE-NET achieves superior performance compared to existing unimodal and multimodal approaches across various cognitive classification tasks. Extensive ablations further validate the effectiveness of both modules.", "questions": "1.\tGiven the complexity introduced by GANs, Neural ODEs, and hypergraphs, could you provide a detailed computational complexity analysis? And could you include an analysis comparing training time, inference speed, and GPU usage with existing baseline methods?  \n2.\tPotential issues such as overfitting, dependency on high-quality preprocessing, and uncertain generalization to clinical or pathological populations are currently not discussed. Could you include a dedicated section addressing these points?  \n3.\tIt is unclear if your model handles common practical issues like missing EEG channels or varying time windows. Could you provide relevant tests or a discussion addressing these real-world challenges?  \n4.\tThe paper lacks insight into what temporal representations are learned by the Neural ODEs. Could you provide an analysis or visualization explaining what dynamics the Neural ODEs capture across modalities and their contribution to classification?  \n5.\tAlthough the LEMON dataset is sizable, reliance on it and the very small CN-EPFL dataset limits generalization claims. Could you include experiments using additional datasets to strengthen claims of robustness?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:  \n1.\tThe paper addresses the underexplored but practically important problem of modeling fMRI and EEG data which can provide complementary information.  \n2.\tThe proposed FE-FEH outperforms a broad range of SOTA unimodal and multimodal baselines on multiple cognitive tasks, showing strong results across accuracy, precision, and specificity.  \n3.\tThe model outputs meaningful functional connectivity patterns associated with cognitive traits (e.g., DMN in verbal intelligence), supporting interpretability.  \n4.\tThe FEH module effectively bridges modality differences using a GAN framework and the FED module employs Neural ODEs to model temporal dynamics across modalities with mismatched sampling rates, enabling smooth representation evolution.\n\nWeaknesses:  \n1.\tThe use of GANs, Neural ODEs, and hypergraphs likely introduces high computational costs. It would be better to include an analysis of computational.  \n2.\tThere should be some analysis of training time, inference speed, or GPU usage compared to existing baselines.  \n3.\tPotential issues such as overfitting, dependency on high-quality preprocessing, and uncertain generalization to clinical or pathological populations are neither acknowledged nor discussed. It would be better to include a dedicated section discussing these points.  \n4.\tIt is unclear whether the model can handle common real-world challenges such as missing EEG channels, or varying time windows, limiting confidence in its practical reliability. It would be helpful to include relevant tests or discussions addressing these concerns.  \n5.\tThe paper lacks insights into the learned temporal representations. There is no analysis explaining what the Neural ODE dynamics capture across modalities or how time-dependent information contributes to classification performance.  \n6.\tAlthough the LEMON dataset is sizable, most experiments rely solely on it. The secondary dataset (CN-EPFL) is very small (only 20 subjects), which limits the claims regarding generalization. Including additional datasets would strengthen the robustness claims.", "was_revised": true, "final_justification": "I thank the authors for the rebuttal. It addressed most of my concerns and I remain a weak accept.", "rebuttal": "Comment: \nWe sincerely appreciate your meticulous review of our manuscript. Your valuable suggestions have helped a lot.\n\n**Q1.1:** Given the complexity introduced by GANs, Neural ODEs, and hypergraphs, could you provide a detailed computational complexity analysis? \n\nThe computation complexity analysis is shown below:\n\n### Time Complexity Analysis\n#### **FEH Module (GAN-based Hypergraph Generation)**\n- **OFEI Algorithm**: O(n²·m) where n=number of ROIs (typically 90-120), m=hyperedges per subject. \n- **IHEN Layers**: O(L·(Nv·d² + Ne·k²)) where L=5 layers, Nv=vertices, Ne=hyperedges, d=feature dim (256), k=hyperedge cardinality.\n\n#### **FED Module (Neural ODE Embedding)**\n- **Control Step**: O(T·(Nv+Ne)·d²) where T=integration steps (~50). Dual-stream LSTM with attention dominates computation.\n- **Diffusion Step**: O(Nv² + Ne²) per step from hypergraph Laplacian operations. Sparse matrix optimizations reduce practical complexity.\n\n#### GPU Memory Usage\n| Component               | VRAM Consumption* | Notes |\n|--------------------------|--------------------|-------|\n| FEH Generator (IHEN×5)   | 4-8GB            | Batch size 8-16 |\n| GAN Discriminator        | 6-8GB              | MLP with 3×512 layers |\n| FED ODE Solver           | 7-9GB            | Adjoint method + intermediate states |\n| Hypergraph Data (N=100)  | 3-5GB              | Storing M=1K hyperedges |\n\n\n\n\n### FE-NET runtime on LEMON Using 1 NVIDIA A6000 \n\n**Training Performance**  \n| Metric                | Value                       |\n|-----------------------|-----------------------------|\n| Batch Size            | 16 subjects                  |\n| Epoch Time            | 18.1 minutes                |\n| Total Training (200 epochs) | 36.1 hours                |\n| Throughput            | 6.3 samples/minute          |\n\n\n**Inference Performance**  \n| Metric                | Value                       |\n|-----------------------|-----------------------------|\n| Single-subject Inference | 3.8s ±0.12s               |\n| Batch Inference (16 subjects) | 30.8s ±1.92s          |\n| VRAM Usage (Inference) | 22.4GB             |\n\n\n**Q1.2** Could you include an analysis comparing training time, inference speed, and GPU usage with existing baseline methods? \n\n**A1.2:** we provided a comparison of computational resources with other SOTA methods. Although FE-NET requires more computational resources than existing SOTA methods, The downstream tasks of fMRI and EEG imaging are not test-intensive, the slightly longer inference time after model training is completed will not interfere with the actual application of the model.\n\nIn fact, when the batch size is adjusted to 8, our model can run perfectly on a single 2080ti. Existing larger fMRI datasets are typically at the thousand-level scale, and even if thousand-level fMRI-EEG datasets emerge in the future, four 2080ti GPUs can handle them perfectly. Considering the cost of acquiring fMRI and EEG data, four NVIDIA 2080ti GPUs is a small price to pay. \n\n\n**Table: Computational Resource Comparison using NVIDIA A6000**\n\n| Method     | VRAM Usage (GB) | Batch Inference Time (s) | Training Time (h) | Peak GPU Utilization (%) |\n|:-----------|----------------:|-------------------------:|------------------:|-------------------------:|\n| GCN        |        4.3 ± 0.2 |                   3.36 ± 0.32 |              2.8  |                   10.1 ± 2.1 |\n| GAT        |        5.5 ± 0.3 |                   5.44 ± 0.48 |              3.5  |                   16.3 ± 1.8 |\n| GIN        |        6.1 ± 0.3 |                   4.96 ± 0.48 |              4.2  |                   18.5 ± 3.2 |\n| BrainGNN   |       10.7 ± 0.5 |                  22.72 ± 1.60 |              6.8  |                   50.2 ± 2.5 |\n| M-GAT-BC   |       15.2 ± 0.6 |                  19.36 ± 1.60 |              8.5  |                   53.7 ± 2.0 |\n| Cross-GNN  |       18.3 ± 0.7 |                  51.68 ± 3.20 |              9.8  |                   66.4 ± 1.7 |\n| MMP-GCN    |       20.1 ± 0.8 |                  44.96 ± 2.88 |             10.5  |                   59.3 ± 1.3 |\n| FE-NET     |       22.4 ± 0.9 |                  60.80 ± 1.92 |             36.1  |                   72.4 ± 1.1 |\n\n\n**Q3:** Potential issues such as overfitting, dependency on high-quality preprocessing, and uncertain generalization to clinical or pathological populations are currently not discussed. Could you include a dedicated section addressing these points?\n\n**A3:** Thank you for highlighting these important aspects.  We plan to add a new section entitled “Limitations, Robustness and Future Directions” that addresses these points:\n\n### 1. Overfitting\n- In this subsection, we will describe our strategies for mitigating overfitting, including k-fold cross-validation, early stopping based on a held-out validation set, L₂ weight regularization, and dropout layers.  \n- We will report learning curves and validation losses to demonstrate convergence without divergence between training and validation performance.  \n\n\n### 2. Generalization to Clinical and Pathological Cohorts\n- We clarify that all current experiments use healthy adult datasets, and that population shifts  may impact model behavior.  \n- We outline a planned evaluation on two other clinical cohorts. We emphasize that understanding failure modes in these groups is a priority for future work and welcome collaborations for data sharing.\n\n\n**Q4:** It is unclear if your model handles common practical issues like missing EEG channels or varying time windows. Could you provide relevant tests or a discussion addressing these real-world challenges?\n\n**A4:** Below, we clarify how FE-NET could address these challenges and plan to perform additional experiments to validate its robustness:  \n\n\n\n### **1. Handling Missing EEG Channels**  \n**Architectural Robustness**:  \n- FE-NET models fMRI-EEG data as hypergraphs, where hyperedges (unlike simple edges in graphs) connect *multiple* nodes (ROIs). This structure can  mitigates missing channels:  \n  - If an EEG channel (node) is missing, information can propagate via shared hyperedges connecting other nodes in the same functional group.  \n\n\n**Empirical Validation**:  \nWe will conduct ablation studies on the LEMON dataset by:  \n  - Randomly masking 10–30% of EEG channels during inference.  \n  - Measuring performance decay (accuracy, AUC) on attention/intelligence classification tasks.  \n\n---\n\n### **2. Varying Time Windows**  \n**Neural ODEs for Irregular Sampling**:  \n- The **FED module** leverages **Neural ODEs** (Eq. 13–16) to model continuous dynamics, making it intrinsically suitable for irregular/inconsistent time windows:  \n  - Neural ODEs interpolate latent states between arbitrary timestamps, accommodating variable-length EEG segments without resampling.  \n  - The Lie-Trotter discretization (Eq. 16) decouples control (modality-specific dynamics) and diffusion (cross-modal interactions), allowing asynchronous updates for fMRI (slow) and EEG (fast) streams.  \n\n**Experimental Verification**:  \nWe will test FE-NET on **non-uniform time windows** by:  \n  - Segmenting the LEMON EEG data into variable intervals (e.g., 2s, 5s, 10s windows).  \n  - Evaluating classification consistency across window lengths.  \n  - Benchmarking against RNN/LSTM-based models (which require fixed-length inputs).  \n\n---\n\n\n**Q5:** The paper lacks insight into what temporal representations are learned by the Neural ODEs. Could you provide an analysis or visualization explaining what dynamics the Neural ODEs capture across modalities and their contribution to classification?\n\n\n\n**A5:** We appreciate this insightful feedback and acknowledge that further analysis of the temporal dynamics learned by Neural ODEs would strengthen the paper. Below, we propose a analysis approach to address this gap:  \n\n#### 1. **Temporal Dynamics Captured by Neural ODEs**  \nThe **FED module** uses Neural ODEs to model the *latent evolution* of joint fMRI-EEG representations. Key learned dynamics include:  \n   - **Cross-Modal Synchronization**: Neural ODEs reconcile fMRI’s slow hemodynamic changes (seconds) and EEG’s rapid oscillations (milliseconds) by learning a shared latent differential equation:  \n     $$\n     \\frac{d}{dt}\\begin{bmatrix} \\mathbf{X}_v(t) \\\\ \\mathbf{X}_e(t) \\end{bmatrix} = f\\left( \\begin{bmatrix} \\mathbf{X}_v(t) \\\\ \\mathbf{X}_e(t) \\end{bmatrix}, t \\right)\n     $$\n     Here, $f(\\cdot)$ captures **inter-modal dependencies** (e.g., how transient EEG spectral power fluctuations correlate with delayed fMRI BOLD responses in specific ROIs).  \n   - **Asynchrony Handling**: The ODEs intrinsically align mismatched sampling rates by interpolating latent states continuously across time, avoiding manual synchronization.  \n\n#### 2. **Proposed Analysis & Visualization**  \nTo elucidate these dynamics, we will:  \n   - **Visualize Latent Trajectories**: Plot the evolution of vertex/hyperedge features ($\\mathbf{X}_v(t)$, $\\mathbf{X}_e(t)$) for exemplar subjects (Figure 1, *right*). We will highlight:  \n     - *Convergence points* where fMRI and EEG representations stabilize (indicating synchronized cross-modal states).  \n     - *Derivative hotspots* ($\\frac{d\\mathbf{X}}{dt}$) during task-relevant intervals.  \n     - **Ablation with Fixed Intervals**: Replace Neural ODEs with RNNs/LSTMs (using fixed timesteps) and compare performance degradation.   \n\n\n**Q6:** *Although the LEMON dataset is sizable, reliance on it and the very small CN-EPFL dataset limits generalization claims. Could you include experiments using additional datasets to strengthen claims of robustness.* \n\n**A6:** We further conducted performance comparisons on the CWL and Oddball datasets, the result is shown in Supplementary materials. Our model still shows advantages on these datasets. We also planned do evaluation on other clinical cohorts. We emphasize that understanding failure modes in these groups is a priority for future work and welcome collaborations for data sharing."}, {"review_id": "Q0reZjrvYd", "reviewer": "Reviewer_NzFT", "summary": "The paper proposes a new approach to joinlty model fMRI and EEG data. The model combines a hypergraph generator, and a module to embed the hypergraphs. The hypergraph generator is based on a GAN-style architecture, with a generator and a discriminator. The generator works by combining node attributes from fMRI and edge attributes from EEG, ultimately yielding a subject-specific multimodal connectivity matrix. The discriminator works by comparing random walks on the functional connectivity matrix to random walks on the multimodal connectivity matrix. The embedding module takes as input the vertex and hyperedge features generated, and defines an evolution equation implemented as a neural ODE with a control step and a diffusion step. The diffusion step allows further mixing of vertex and edge features. The approach is trained, validated and tested on 4 classification tasks on an N=227 dataset with non-simultaneous fMRI and EEG, and on an N=20 simultaneous EEG-fMRI dataset. It is also benchmarked with respect to several other methods.", "questions": "- Provide improved description of how EEG and fMRI data are taken as input - will improve rating \n- Revise and improve claims of originality with better situation in the state of the art - will improve rating\n- It is not clear why fMRI is used for initial node attributes and EEG for hyperedge attributes. Could we swap EEG and fMRI in the inputs?", "limitations": "No, there is no limitations section or discussion of limitations, either technical or social. Predicting verbal intelligence or logical intelligence, while convenient as an engineering exercise because prediction targets are provided, does of course raise ethical questions related to potential bias and fairness, as well as benefits and harms. It is not essential to go in depth but this should be at least acknowledged.\nSee e.g. slides 40++ in \"Understanding Ethics in NLP Authoring and Reviewing\" <https://github.com/acl-org/ethics-tutorial/blob/main/slides/Understanding%20Ethics%20in%20NLP%20Authoring%20and%20Reviewing%20%E2%80%93%20Presentation%20Deck.pdf>.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Quality\n\nA strength of the paper is its use of a thorough benchmark of competing architectures, on 4 different prediction targets, as well as an external dataset.\n\nConversely, section 3.3. overall evaluations ascribes the superior performance of FE-Net over the unimodal baselines to the fusion of EEG and fMRI. However it could also simply be that architectural differences in FE-Net, compared to  BrainGNN / M-GAT-BC / SGP-SL, rather than having both data types, drive the difference in performance. An ablation study feeding uninformative connectivity in one of the modalities, or simply generating hypergraphs from a single modality, then leaving the rest of the architecture untouched, would perhaps better highlight the contribution of modality vs architecture.\n\nThe motivation of the work should be corrected slightly.\nL34: \"...simultaneous recorded fMRI-EGG dataset suffers from tiny dataset size (less than 20)...\" while dataset sizes are indeed typically small in EEG-fMRI, a few open datasets are larger than 20, for example 50 in Meier et al \"Connectomes, simultaneous EEG-fMRI resting-state data and brain simulation results from 50 healthy subjects\", 2024; 30 in Loi et al's \"Simultaneous EEG-fMRI during a neurofeedback task, a brain imaging dataset for multimodal data integration\", 2020; 24 in Gherman and Philastides \"Simultaneous EEG-fMRI - Confidence in perceptual decisions\" dataset on openneuro.\nIn semi-open data, Wirsich et al \"The relationship between EEG and fMRI connectomes is reproducible across simultaneous EEG-fMRI studies from 1.5T to 7T\", 2021, used 72 subjects, provides all pre-computed connectomes (<https://zenodo.org/records/3905103>), and a subset of subjects on OSF (<https://osf.io/94c5t/>)\nIn closed data, clinical studies with over 30 patients (e.g. Al Asmi et al., \"fMRI activation in continuous and spike-triggered EEG-fMRI studies of epileptic spikes\", 2003) or more have been performed. I would thus recommend to amend the statement.\n\nIn the discriminator loss function, why take FC edges as reference for \"real\" paths rather than EC edges? This seems like the discriminator will push the generator to compute multimodal connectivity matrices which look more like fMRI connectivity than EEG connectivity. Is this to \"balance\" the design choice of using EC to derive the initial hyperedge attributes x_{k,E} in equation 4? \n\nOn the embedding space analysis, authors write \"traditional graph embedding results in a larger modality gap, which in turn leads to decreased performance\", but it seems instead that there is no modality gap - EEG and fMRI are located in overlapping regions of embedding space.\nAlso, what is the \"traditional graph embedding\" in L290 and fig. 2? Graph2Vec then UMAP ?\n\n\n\n## Clarity\n\nOverall, the structure of the paper is good. The illustration in Figure 1 is helful in understanding the approach. However, one major weakness is the lack of details in linking EEG and fMRI data to the model inputs.\n\nThe general, initial definition of the hypergraphs is clear, but the transition in Section 2.2.1 to the precise case of the EEG and fMRI graphs needs better framing, especially because the dimensionalities of the EEG and fMRI datasets are quite different. The number of nodes in both EEG and fMRI is very confusing. The supplementary section 6B mentions the 400-parcels Schaefer atlas, but goes on to list 44 regions from the \"AAL atlas\" - a different atlas altogether with 116 ROIs. So, is the paper using 400 or 44 regions for fMRI data to derive the \"FC matrix\"? Likewise, the EEG montage shown highlights 30 electrodes in red, from 62 according to the Babayan et al paper - traditionally meaning these are used for analysis. Is the paper using only 30 electrodes in EEG to derive the \"EC matrix\"?  Section 2.2.2 suggests that the fMRI BOLD time-series is Nxd, where N is the number of nodes (44 or 400?), and serves to derive the nodal attribute matrix X_{k,V}, while the EC matrix is given as dimension NxN. Because N is used in both EEG and fMRI, this implies that the same number of nodes are used in both, and, presumably spatially aligned somehow so that nodes are in 1:1 correspondance between modalities? No EEG source reconstruction is mentioned in the supplementary so this interpretation is also not certain. In the review I am chosing the interpret that the modalites are node-aligned for simplicity. This point should be spelled out explicitly.\n\nThus I assume that edges and nodes of both modalities are jointly combined into the 'initial hypergraphs', but this should be much more explicit as it is the basis of the method. The paper states \"If these initial hypergraphs {G′k} are directly utilized in the generative process to achieve multimodal connectivity, the robustness of the resultant generation may be compromised.\", but why is that? To me this is lacking a justification. The results of the FE-NETnoFEH ablation seem to bear our your assertion, but it would be useful to have an intuition.\n\nSection 2.2.2 mentions the bold time series B is \"employed to characterize the nodal attributes\" (X_{k_V}) - how? what is used from the B time-series? GNNs are very sensitive to node feature choice and their scaling, so it would be useful to specify. Likewise, where does the EC matrix come from? I would assume a dependency measure such as coherence but this is not described. Then in L110 S_k is used without being defined, making it more difficult to understand what X is supposed to contain. Please define all symbols when they are used first.\n\nA few more points on clarity\n- the correlation coefficients assigned to the nodes seem not to be used later in the paper\n- in 2.2.3 walks and paths are used interchangeably, while they are not (on paths each node can only by visited once) - please specify\n- I am not sure the second part of equation 13 is useful, as Z_v and Z_e are not used later\n- eq. 15, L168 user lower case t\n\nMinor points\nFigure 3 - left panel title is partly covered\nL226 \"datase tand\"\n\n## Significance\n\nThe approach could be used to any number of multimodal datasets where spatio-temporal correlations are of interest.\n\n## Originality\n\nL32: \"joint modeling of fMRI and EEG is a rarely explored area\" that \"has not yielded satisfactory results\" is a mischaracterisation of the field. While the combination of both modalities is certainly less explored than each modality alone, EEG-fMRI analysis is quite common, with over 1000 journal papers on pubmed alone and 27K papers on Google scholar, and dates back a while. See for instance Menon and Crottaz-Herbette's 2005 review \"Combined EEG and fMRI studies of human brain function\", Herrmann and Debener's 2008 \"Simultaneous recording of EEG and BOLD responses: a historical perspective\", or Helmut Lauf's 2012 \"A personalized history of EEG–fMRI integration\", for accounts from pioneers dating back to at least 1993 with Ives and colleagues \"Monitoring the patient's EEG during echo planar MRI\". EEG-fMRI clinical usefulness is also well-established in epilepsy. I recommend positioning more precisely the knowledge gap that the paper is meant to address.\n\nL58 also claims that \"this is the first attempt to model asynchronous EEG and fMRI data\", meaning in this context EEG and fMRI acquired not at the same time.\nThis is repeated again in L322 \"For the first time, we attempt to model asynchronous fMRI and EEG data...\".\nThis is factually incorrect. There have been very numerous studies where EEG and fMRI acquired at different times have been modelled, going back at least to 1996 (Gerloff et al., Coregistration of EEG and fMRI in a Simple Motor Task).\nFor example: joint ICA has been used to jointly model EEG and fMRI activations in Calhoun et al., A feature-based approach to combine functional MRI, structural MRI and EEG brain imaging data, 2006; CCA (a), then Multi-set CCA (b) has been used to jointly model fMRI, sMRI and EEG in (b) Correa et al., Fusion of fMRI, sMRI, and EEG data using canonical correlation analysis, 2009, and (b) Sui et al., Combination of FMRI-SMRI-EEG Data Improves Discrimination of Schizophrenia Patients by Ensemble Feature Selection, 2014; Then coupled matrix and tensor factorization for EEG and fMRI in Acar et al., Tensor-based fusion of EEG and FMRI to understand neurological changes in schizophrenia, 2017, etc... and several works with deep learning are also easy to find, including many on EEG-to-fMRI prediction.\nThis work is enabled by multiple databases that have both fMRI and EEG data recorded at different times, such as the PEARL-neuro database (https://www.nature.com/articles/s41597-024-03106-5) or the Leipzig mind-brain-body database (https://www.nature.com/articles/sdata2018308) (ref 21 in the paper).\nThe authors should tone down this claim of novelty, or make it more specific.\n\nThe lack of direct architectural comparison (not only benchmarking) with multimodal neural networks taking EEG and fMRI connectivity as input (or even neural networks taking multimodal MRI connectivity as input) makes it difficult to establish the core differences with existing offers.", "was_revised": true, "final_justification": "After rebuttal, the paper has addressed my main concerns around quality, clarity and originality.\n\nI have upgraded by recommendation.", "rebuttal": "Comment: \n\nWe sincerely appreciate your meticulous review of our manuscript. Your valuable suggestions have helped a lot in improving the Clarity and Rigor of our work.\n\n**Q1:** *Revise and improve claims of originality with better situation in the state of the art - will improve rating*\n\n**A1:**  In accordance with your suggestions, we will amend the following imprecise expressions in the thesis. \n\n#### 1. **Motivation Correction (Line 34)**  \n- **Original**:  \n  *\"...simultaneous recorded fMRI-EGG dataset suffers from tiny dataset size (less than 20)...\"*  \n- **Revised**:  \n  *\"The sample size of simultaneously recorded fMRI-EGG datasets is markedly smaller than that of asynchronously acquired datasets, thereby limiting the reliability of extant research relying on synchronous recordings.\"*  \n\n#### 2. **Field Characterization (Line 3)**  \n- **Original**:  \n  *\"joint modeling of fMRI and EEG is a rarely explored area\" that \"has not yielded satisfactory results\"*  \n- **Revised**:  \n  *\"Deep learning-based joint modeling of asynchronously recorded fMRI and EEG is a relatively underexplored area and has not yielded satisfactory results.\"*  \n\n#### 3. **Originality Claims**  \n- **Line 58 Revision**:  \n  - **Original**:  \n    *\"this is the first attempt to model asynchronous EEG and fMRI data\"*  \n  - **Revised**:  \n    *\"This is the first attempt to model asynchronous EEG and fMRI data as Neural ODEs based hypergraph\"*  \n\n- **Line 322 Revision**:  \n  - **Original**:  \n    *\"For the first time, we attempt to model asynchronous fMRI and EEG data...\"*  \n  - **Revised**:  \n    *\"For the first time, we attempt to model asynchronous fMRI and EEG data as Neural ODEs based hypergraph.\"*  \n\n\n\n\n**Q2:** *In the discriminator loss function, why take FC edges as reference for \"real\" paths rather than EC edges? This seems like the discriminator will push the generator to compute multimodal connectivity matrices which look more like fMRI connectivity than EEG connectivity.*\n\n**A2:** Our decision to treat the fMRI‐derived FC matrix as the “real” paths in the discriminator is partially because our use of the EEG‐derived EC matrix in Equation 4. More specifical reasons are:\n\n1. FC as a Stable, Gold-Standard Prior. The FC matrix, estimated over seconds‐long BOLD time series, offers a relatively low‐variance, anatomically plausible network “backbone.” Using FC in the adversarial loss provides the generator with a consistent, robust target distribution.\n\n2. EC to Inject Electrophysiological Detail. By computing the initial hyperedge attributes $X_{k, E}^{(0)}$ from the EC matrix, we ensure that fine‐grained EEG dynamics are explicitly encoded in the generated hypergraphs. This EC initialization drives the generator to learn electrophysiologically meaningful interactions.  \n\nWe swapped FC for EC in the discriminator’s “real” paths. which leads to a longer training time, and the average classification accuracy dropped by 2.8 \\%. This confirms that our original adversarial criterion is not merely a “balancing hack,” but a necessary mechanism to stabilize training and preserve both modalities’ contributions.\n\n\n**Q3:** Provide improved description of how EEG and fMRI data are taken as input - will improve rating. The transition in Section 2.2.1 needs better framing for EEG/fMRI dimensionalities. The EEG montage highlights 30 electrodes (of 62), while fMRI ROIs are unspecified. Section 2.2.2 uses dimension \\(N\\) for both modalities, implying 1:1 node correspondence without clarifying alignment. This must be explicitly addressed.\n\n\n\n**A3:** We thank the reviewer for this essential critique. The lack of clarity regarding node alignment will be rectified. Below, we provide clarification and outline revisions:  \n\n#### 1. **Node Alignment Strategy**  \nWe confirm explicit 1:1 anatomical correspondence** between EEG and fMRI nodes via standardized neuroimaging pipelines:  \n- **fMRI parcellation**: All subjects use the **AAL-90 atlas** (90 cortical/subcortical ROIs).  \n- **EEG processing**:  \n  - **Full 62-electrode data** undergo source localization (sLORETA) to cortical voxels.  \n  - Voxel-level signals are spatially aggregated to the same 90 AAL ROIs as fMRI.  \n  \n\n> **Clarification**: The \"30 red electrodes\" in Figure 1 were purely for *visual emphasis* of key regions. **All 62 electrodes** were used for analysis. We will revise the figure to eliminate ambiguity.  \n\n\n#### 2. **EEG Source Localization Details**  \nThe pipeline is:  \n1. **Forward modeling**: Boundary element method (BEM) with standard head model.  \n2. **Inverse solution**: sLORETA for source reconstruction.  \n3. **ROI aggregation**: Voxel-level source signals averaged within each AAL-90 region.  \n4. **EC computation**: Partial directed coherence (PDC) between ROI-level time series.  \n\n> **Supplementary Material** will includes:  \n> *\"EEG signals were source-localized to AAL-90 ROIs via sLORETA . ROI-level bandpower features and PDC-based EC matrices were then computed.\"*  \n\n#### 3. **Revisions Commitment**  \nWe will:  \n1. Add a **new subsection (2.1.1: Data Alignment and Node Correspondence)**:  \n   \n2. **Update Figure 1**:  \n   - Replace the \"30 highlighted electrodes\" with a schematic of **62 electrodes → source localization → AAL-90 ROIs**.  \n  \n\n\n---  \n**Summary**: Our framework relies on anatomically grounded 1:1 node correspondence achieved through EEG source reconstruction and fMRI parcellation. We apologize for the ambiguity in the original manuscript and thank the reviewer for highlighting this gap.\n\n\n**Q4:** *If these initial hypergraphs \\{G′k\\} are directly utilized in the generative process to achieve multimodal connectivity, the robustness of the resultant generation may be compromised.\", but why is that? To me this is lacking a justification.* \n\n\n**A4:** We appreciate this critical observation and acknowledge the need for explicit justification. Below, we clarify the core reasons:  \n\n---\n\n#### **1. Subject-Specific Noise Amplification**  \n- **Problem**:  \n  Initial hypergraphs {G'ₖ} are constructed *per subject* using Dynamic Hypergraph Construction (DHC). DHC:  \n  - Sensitively encodes subject-level noise/artifacts (e.g., motion in fMRI, muscle artifacts in EEG).  \n  - Lacks cross-subject consistency, leading to fragmented topological representations.  \n- **Consequence**:  \n  Feeding noisy {G'ₖ} directly into the GAN generator propagates subject-level biases, degrading multimodal fusion.  \n- **OFEI Solution**:  \n  Computes an **optimally isomorphic hypergraph** \\(G\\) (Eq. 3) that maximizes group-level similarity:  \n  \\[\n  G = \\operatorname*{argmax}_{G}\\sum_{k}\\text{Sim}(G,G'_k)\n  \\]  \n  This denoises by extracting the most consistent topological \"backbone\" across subjects.  \n\n---\n\n#### **2. Modality-Specific Structural Bias**  \n- **Problem**:  \n  {G'ₖ} are built separately for fMRI and EEG using modality-specific heuristics in DHC:  \n  - fMRI hyperedges model spatial communities (e.g., ROI clusters).  \n  - EEG hyperedges model spectral-temporal interactions.  \n- **Consequence**:  \n  Direct concatenation (e.g., \\(G_k = G'_{\\text{fMRI},k} \\| G'_{\\text{EEG},k}\\)) creates modality-centric hypergraphs with conflicting structures, inhibiting cross-modal information flow.  \n- **OFEI Solution**:  \n  Forces fMRI and EEG into a **shared isomorphic space** via Eq. 2–3, ensuring:  \n  - fMRI hemodynamic patterns and EEG oscillations co-evolve in aligned hyperedges.  \n  - Biologically implausible cross-modal connections (e.g., unrelated ROIs) are pruned.  \n\n---\n\n\n**Q5:** Section 2.2.2 mentions the bold time series B is \"employed to characterize the nodal attributes\" (X\\_{k\\_V\\}) - how? what is used from the B time-series? GNNs are very sensitive to node feature choice and their scaling, so it would be useful to specify. \n\n**A5:** the BOLD time series \\( B \\) is employed to derive nodal attributes \\( X_{k,V} \\) through the following process:\n- We extract the mean BOLD signal from each of the predefined regions of interest (ROIs) corresponding to the AAL atlas.\n- Specifically, for each ROI, we compute the mean, standard deviation, and temporal correlation across time points to serve as features.\n- To ensure robustness against scaling issues, we apply standardization (zero mean, unit variance) to all nodal features before feeding them.\n\n\n**Q6:**  It is not clear why fMRI is used for initial node attributes and EEG for hyperedge attributes. Could we swap EEG and fMRI in the inputs?\n \n\n\n**A6:** Our assignment—fMRI for node attributes and EEG for hyperedge attributes—is grounded in neurophysical principles. Below, we justify this design:  \n\n---\n\n#### **1. Neurobiological Alignment**  \n- **fMRI → Node Attributes**:  \n  - fMRI's BOLD signals reflect slow hemodynamic changes (seconds-scale) localized to specific anatomical regions (AAL ROIs).  \n  - This naturally characterizes *static regional properties* (e.g., activation magnitude, variance), ideal for node-level features.  \n- **EEG → Hyperedge Attributes**:  \n  - EEG captures *millisecond-scale oscillations* with high temporal precision but low spatial resolution.  \n  - Hyperedges represent *functional ensembles*—making EEG's interaction metrics perfect for hyperedge-level features.  \n\n\n---\n\n\n\n#### **2. Hypergraph Semantic Consistency**  \n- **Neurobiological interpretation**:  \n  - A hyperedge connecting {ROI₁, ROI₂, ROI₃} signifies \"these regions co-activate with timing captured by EEG.\"  \n  - Assigning fMRI to hyperedges would conflate hemodynamic correlation (slow) with neural interaction (fast), violating biophysical principles.  \n\n**Summary**: Our modality-to-attribute mapping:  \n1. Respects **neurophysical realities** (fMRI=spatial, EEG=temporal).    \n2. Ensures **hypergraph interpretability** (nodes=regions, edges=EEG-guided interactions).  \n\nOur additional experiments also show swapping modalities would degrade performance. We thank the reviewer for prompting this fundamental clarification."}, {"review_id": "Dfs744ZqNf", "reviewer": "Reviewer_ha9r", "summary": "This paper proposes FE-NET, a novel framework for jointly modeling fMRI and EEG brain data that are collected asynchronously. FE-NET significantly outperforms existing state-of-the-art models on various cognitive classification tasks, highlighting its practical applicability and superior performance in fusing multimodal brain data.", "questions": "See above", "limitations": "See above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Pros:\n\n1. The problem setting is well-motivated and interesting. The primary strength is the model's ability to model asynchronous fMRI and EEG data. This has significant practical implications, as it circumvents the need for expensive and technically challenging simultaneous data acquisition.\n\n2. The proposed approach is technically sound. It employs hypergraphs to capture complex, many-to-many brain region relationships and Neural ODEs to handle the different time scales of fMRI and EEG signals.\n\n\nCons:\n\n1. While there are many different components of the proposed approach, including GANs, hypergraphs, and Neural ODEs, the time complexity for this approach should be relatively high.\n\n2. There is no related work section. The authors should provide a comprehensive background introduction towards related work,  such as employing ODEs for brain network analysis[1], etc.\n\n\n\n[1] BrainODE: Dynamic Brain Signal Analysis via Graph-Aided Neural Ordinary Differential Equations", "was_revised": false, "final_justification": "", "rebuttal": "Comment: \n\nWe sincerely appreciate your meticulous review of our manuscript. Your valuable suggestions have helped a lot in improving the Clarity and Rigor of our work.\n\n**Q1:** *While there are many different components of the proposed approach, including GANs, hypergraphs, and Neural ODEs, the time complexity for this approach should be relatively high* \n\n\n**A1:** We conducted an Algorithm Complexity and computation resources Analysis for FE-NET.\n\nAt the same time, we provided a comparison of computational resources with other SOTA methods. Although FE-NET requires more computational resources than existing SOTA methods, The downstream tasks of fMRI and EEG imaging are not test-intensive, the slightly longer inference time after model training is completed will not interfere with the actual application of the model.\n\nIn fact, when the batch size is adjusted to 8, our model can run perfectly on a single 2080ti. Existing larger fMRI datasets are typically at the thousand-level scale, and even if thousand-level fMRI-EEG datasets emerge in the future, four 2080ti GPUs can handle them perfectly. Considering the cost of acquiring fMRI and EEG data, four NVIDIA 2080ti GPUs is a small price to pay. Given the considerable potential for future improvements in GPU performance, our model remains well-suited for large-scale datasets.\n\n### Time Complexity Analysis\n#### **FEH Module (GAN-based Hypergraph Generation)**\n- **OFEI Algorithm**: O(n²·m) where n=number of ROIs (typically 90-120), m=hyperedges per subject. \n- **IHEN Layers**: O(L·(Nv·d² + Ne·k²)) where L=5 layers, Nv=vertices, Ne=hyperedges, d=feature dim (256), k=hyperedge cardinality.\n\n#### **FED Module (Neural ODE Embedding)**\n- **Control Step**: O(T·(Nv+Ne)·d²) where T=integration steps (~50). Dual-stream LSTM with attention dominates computation.\n- **Diffusion Step**: O(Nv² + Ne²) per step from hypergraph Laplacian operations. Sparse matrix optimizations reduce practical complexity.\n\n#### GPU Memory Usage\n| Component               | VRAM Consumption* | Notes |\n|--------------------------|--------------------|-------|\n| FEH Generator (IHEN×5)   | 4-8GB            | Batch size 8-16 |\n| GAN Discriminator        | 6-8GB              | MLP with 3×512 layers |\n| FED ODE Solver           | 7-9GB            | Adjoint method + intermediate states |\n| Hypergraph Data (N=100)  | 3-5GB              | Storing M=1K hyperedges |\n\n\n\n\n\n\n\n### FE-NET runtime on LEMON Using 1 NVIDIA A6000 \n\n**Training Performance**  \n| Metric                | Value                       |\n|-----------------------|-----------------------------|\n| Batch Size            | 16 subjects                  |\n| Epoch Time            | 18.1 minutes                |\n| Total Training (200 epochs) | 36.1 hours                |\n| Throughput            | 6.3 samples/minute          |\n\n\n**Inference Performance**  \n| Metric                | Value                       |\n|-----------------------|-----------------------------|\n| Single-subject Inference | 3.8s ±0.12s               |\n| Batch Inference (16 subjects) | 30.8s ±1.92s          |\n| VRAM Usage (Inference) | 22.4GB             |\n\n\n\n\n\n\n**Table: Computational Resource Comparison using NVIDIA A6000**\n\n| Method     | VRAM Usage (GB) | Batch Inference Time (s) | Training Time (h) | Single‐subject Inference Time (s) | Peak GPU Utilization (%) |\n|------------|----------------:|-------------------------:|------------------:|----------------------------------:|-------------------------:|\n| GCN        |        4.3±0.2  |                  3.36±0.32 |              2.8  |                          0.21±0.02 |                   10.1±2.1 |\n| GAT        |        5.5±0.3  |                  5.44±0.48 |              3.5  |                          0.34±0.03 |                   16.3±1.8 |\n| GIN        |        6.1±0.3  |                  4.96±0.48 |              4.2  |                          0.31±0.03 |                   18.5±3.2 |\n| BrainGNN   |       10.7±0.5  |                 22.72±1.60 |              6.8  |                          1.42±0.10 |                   50.2±2.5 |\n| M-GAT-BC   |       15.2±0.6  |                 19.36±1.60 |              8.5  |                          1.21±0.10 |                   53.7±2.0 |\n| Cross-GNN  |       18.3±0.7  |                 51.68±3.20 |              9.8  |                          3.23±0.20 |                   66.4±1.7 |\n| MMP-GCN    |       20.1±0.8  |                 44.96±2.88 |             10.5  |                          2.81±0.18 |                   59.3±1.3 |\n| FE-NET     |       22.4±0.9  |                 60.80±1.92 |             36.1  |                          3.80±0.12 |                   72.4±1.1 |\n\n\n\n\n\n\n\n\n\n\n\n\n**Q2:** *There is no related work section. The authors should provide a comprehensive background introduction towards related work, such as employing ODEs for brain network analysis[1], etc.*\n\n\n\n**A1:** We thank the reviewer for this essential suggestion. We acknowledge the need to rigorously position FE-NET within the field and will add a **dedicated subsection (1.2: Related Work)**. Below is our revision strategy to address this gap:  \n\n\n\n#### **1. New Section 1.2: \"Related Work on Multimodal Fusion and Asynchronous Modeling\"**  \n**Structure and key additions:**  \n1. **Synchronous fMRI-EEG Fusion**:  \n   > *\"Early fusion methods required strict temporal synchronization. Advanced deep learning approaches improved synchronization through attention mechanisms but remained limited to temporally aligned data. These cannot handle real-world asynchronous acquisitions.\"*  \n\n2. **Asynchronous Modeling Attempts**:  \n   > *\"Some studies used kernel methods to align asynchronous data post-hoc but lost dynamic interactions. Others employed self-supervised techniques but treated modalities independently without joint topological modeling. Prior work has not resolved asynchronous fusion at the graph-structure level.\"*  \n\n3. **Neural ODEs in Neuroimaging**:  \n   > *\"Neural ODEs have modeled single-modality dynamics in fMRI data. Spatio-temporal extensions capture dynamic patterns but ignore cross-modal asynchrony. None have adapted ODEs for multimodal hypergraph evolution.\"*  \n\n4. **Hypergraph Neuroimaging**:  \n   > *\"Existing hypergraph neural networks modeled fMRI data but omitted EEG integration. Other works fused MRI modalities via hypergraphs but required temporal alignment. FE-NET is the first to leverage hypergraphs for asynchronous fMRI-EEG integration.\"*  \n\n---\n\n#### **2. Explicit Novelty Positioning**  \n**FE-NET’s breakthroughs**:  \n| **Capability**               | **Prior Works** | **FE-NET** |  \n|------------------------------|-----------------|------------|  \n| Async fMRI-EEG fusion        | ✗               | **✓**      |  \n| Joint hypergraph modeling    | ✗               | **✓**      |  \n| Neural ODEs for cross-modal dynamics | ✗       | **✓**      |  \n| GAN-based hypergraph generation | ✗             | **✓**      |  \n\n---\n\n#### **3. Addressing the Asynchronous Modeling Gap**  \n- **Temporal Flexibility**:  \n  > *\"Unlike methods that process modalities separately, our Neural ODE-based FED module continuously aligns fast EEG and slow fMRI dynamics within a unified latent space.\"*  \n- **Structural Innovation**:  \n  > *\"While existing approaches aligned async data through static methods, FE-NET’s hypergraphs encode many-to-many cross-modal interactions, preserving complex neurodynamics.\"*  \n\n---\n\n#### **4. Revisions Commitment**  \n**In the manuscript**:  \n- Add **~400 words** in Section 1.2 (preceding the \"Our Contributions\" paragraph)  \n- Include **key references** covering:  \n  - Synchronous and asynchronous fusion techniques  \n  - ODE applications in neuroimaging  \n  - Hypergraph-based neuroimaging studies  \n- **Novelty statement**:  \n  > *\"To our knowledge, this is the first framework to simultaneously resolve: (a) Asynchronous fMRI-EEG fusion via hypergraph-structured Neural ODEs, (b) GAN-based generation of multimodal hypergraphs, and (c) Dynamic embedding of cross-modal interactions without temporal resampling.\"*  \n\n\n\n**Summary**: This revision will:  \n1. Formally establish **FE-NET’s pioneering role** in asynchronous fMRI-EEG hypergraph fusion  \n2. Provide **scholarly grounding** for technical innovations  \n3. Explicitly **contrast with prior approaches** to highlight topological and temporal advantages  \n\nWe affirm the revised manuscript will comprehensively contextualize our contributions within the field."}, {"review_id": "wNReXBwPpS", "reviewer": "Reviewer_dcdn", "summary": "This paper proposes a hypergraph neural network framework based on Neural Ordinary Differential Equations for the fusion of functional Magnetic Resonance Imaging (fMRI) and Electroencephalography (EEG) data. The framework is designed to handle the temporal sampling rate mismatch and asynchronous timestamps between the two modalities, and it captures latent temporal dependencies through a dynamic hypergraph embedding module. Experimental results demonstrate that the proposed method achieves high accuracy and superior performance across multiple cognitive tasks. Compared to simultaneously acquired data, asynchronously collected fMRI-EEG data is more cost-effective, highlighting the practical value of the approach.", "questions": "1. The paper lacks a dedicated related work section. Incorporating a more thorough review of existing studies on fMRI-EEG fusion, especially on asynchronous modeling, would strengthen the background and better contextualize the novelty of this work.\n2. The motivation for using GANs in the hypergraph generation module is not clearly explained. It would be helpful if the authors could elaborate on why a GAN-based approach is necessary or beneficial in this setting, compared to more standard alternatives.\n3. The paper does not discuss the limitations of the proposed method or provide insights into potential future extensions. Including such a discussion would enhance the completeness of the paper and help better understand its applicability and scope of application.", "limitations": "No\nThe paper does not explicitly discuss the methodological limitations or assumptions underlying the proposed model. For example, the effectiveness of the GAN-based hypergraph generation relies on the assumption that meaningful multimodal correspondences can be learned from asynchronous fMRI and EEG signals, which may not hold in more heterogeneous or noisy real-world settings. It would also be useful to discuss the interpretability of hypergraph representations and the potential risks of overfitting when modeling complex brain dynamics with limited labeled data.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths：\nThe paper proposes a novel and unified framework (FE-NET) that effectively combines fMRI and EEG imaging using Neural ODEs-based hypergraph neural networks, which is the first attempt to jointly model fMRI and EEG as hypergraphs.\nThe GAN-based FEH module with OFEI and IHEN effectively narrows the gap between hemodynamic responses and neural oscillations.\nThe FED module successfully addresses the challenge of asynchronous data by capturing temporal dependencies between fMRI and EEG signals with different sampling rates.\nThe method demonstrates superior performance compared to many state-of-the-art brain imaging modeling methods through extensive experiments.\nWeaknesses：\nThe paper asserts the lack of prior work on joint modeling of fMRI and EEG, especially with asynchronous data, and claims novelty. However, these statements are made without a thorough discussion of existing studies or background. Without a clearer positioning within the related literature, the novelty claim appears unsubstantiated.\nThe proposed method introduces several complex components (e.g., GAN-based hypergraph generation, Neural ODEs with dual-stream control), but the motivation and design choices are not sufficiently justified or intuitively explained. For example, why GANs are essential in this setting, or how the hyperedge structure aligns with actual neurophysiological properties, is not clearly articulated.\nThe paper does not include a discussion of its limitations or potential directions for future research. This weakens the integrity of the conclusions and makes it impossible to understand the boundaries of the method or possible extensions.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal, which addressed several of my concerns, and I maintain my rating of 4 (borderline accept).", "rebuttal": "Comment: \nWe sincerely appreciate your meticulous review of our manuscript. Your valuable suggestions have helped a lot in improving the Clarity and Rigor of our work.\n\n**Q1:** *The paper lacks a dedicated related work section. Incorporating a more thorough review of existing studies on fMRI-EEG fusion, especially on asynchronous modeling, would strengthen the background and better contextualize the novelty of this work.*  \n\n\n**A1:** We thank the reviewer for this essential suggestion. We acknowledge the need to rigorously position FE-NET within the field and will add a **dedicated subsection (1.2: Related Work)**. Below is our revision strategy to address this gap:  \n\n\n\n#### **1. New Section 1.2: \"Related Work on Multimodal Fusion and Asynchronous Modeling\"**  \n**Structure and key additions:**  \n1. **Synchronous fMRI-EEG Fusion**:  \n   > *\"Early fusion methods required strict temporal synchronization. Advanced deep learning approaches improved synchronization through attention mechanisms but remained limited to temporally aligned data. These cannot handle real-world asynchronous acquisitions.\"*  \n\n2. **Asynchronous Modeling Attempts**:  \n   > *\"Some studies used kernel methods to align asynchronous data post-hoc but lost dynamic interactions. Others employed self-supervised techniques but treated modalities independently without joint topological modeling. Prior work has not resolved asynchronous fusion at the graph-structure level.\"*  \n\n3. **Neural ODEs in Neuroimaging**:  \n   > *\"Neural ODEs have modeled single-modality dynamics in fMRI data. Spatio-temporal extensions capture dynamic patterns but ignore cross-modal asynchrony. None have adapted ODEs for multimodal hypergraph evolution.\"*  \n\n4. **Hypergraph Neuroimaging**:  \n   > *\"Existing hypergraph neural networks modeled fMRI data but omitted EEG integration. Other works fused MRI modalities via hypergraphs but required temporal alignment. FE-NET is the first to leverage hypergraphs for asynchronous fMRI-EEG integration.\"*  \n\n\n---\n\n#### **2. Addressing the Asynchronous Modeling Gap**  \n- **Temporal Flexibility**:  \n  > *\"Unlike methods that process modalities separately, our Neural ODE-based FED module continuously aligns fast EEG and slow fMRI dynamics within a unified latent space.\"*  \n- **Structural Innovation**:  \n  > *\"While existing approaches aligned async data through static methods, FE-NET’s hypergraphs encode many-to-many cross-modal interactions, preserving complex neurodynamics.\"*  \n\n---\n\n#### **3. Revisions Commitment**  \n**In the manuscript**:  \n- Add **~400 words** in Section 1.2 (preceding the \"Our Contributions\" paragraph)  \n- Include **key references** covering:  \n  - Synchronous and asynchronous fusion techniques  \n  - ODE applications in neuroimaging  \n  - Hypergraph-based neuroimaging studies  \n- **Novelty statement**:  \n  > *\"To our knowledge, this is the first framework to simultaneously resolve: (a) Asynchronous fMRI-EEG fusion via hypergraph-structured Neural ODEs, (b) GAN-based generation of multimodal hypergraphs, and (c) Dynamic embedding of cross-modal interactions without temporal resampling.\"*  \n\n\n\n**Summary**: This revision will:  \n1. Formally establish **FE-NET’s pioneering role** in asynchronous fMRI-EEG hypergraph fusion  \n2. Provide **scholarly grounding** for technical innovations  \n3. Explicitly **contrast with prior approaches** to highlight topological and temporal advantages  \n\nWe affirm the revised manuscript will comprehensively contextualize our contributions within the field.  \n\n\n\n\n\n**Q2:** *The motivation for using GANs in the hypergraph generation module is not clearly explained. It would be helpful if the authors could elaborate on why a GAN-based approach is necessary or beneficial in this setting, compared to more standard alternatives.*\n\n \n\n**A2:** We appreciate this critical inquiry into our design choice. The GAN-based approach in the FEH module addresses three fundamental challenges in fMRI-EEG hypergraph generation that deterministic or non-adversarial methods cannot resolve. Below, we clarify the rationale with empirical and theoretical evidence:  \n\n---\n\n#### **1. Key Challenges Necessitating GANs**  \n| Challenge | Standard Methods (e.g., VAE, Deterministic Decoder) | GAN Solution |  \n|-----------|---------------------------------------------------|--------------|  \n| **Distributional Gap**(fMRI hemodynamics vs. EEG oscillations) | Assume simple parametric distributions (e.g., Gaussian), failing to capture complex cross-modal dependencies. | Adversarial learning directly models the *joint distribution*  without parametric constraints. |  \n| **Modality-Specific Noise**(e.g., fMRI motion artifacts, EEG muscle noise) | Propagate noise through reconstruction losses (MSE). | Discriminator rejects noisy generations, acting as a *learnable denoising filter*. |  \n| **Hyperedge Sparsity**(Many possible ROI combinations) | Overlook biologically implausible hyperedges (e.g., unconnected ROIs). | Adversarial training prioritizes *neurologically meaningful hyperedges* via the discriminator’s FC-guided realism check (Eq. 11). |  \n\n---\n\n\n#### **2. Unique Advantages of GANs in FEH**  \n- **OFEI-IHEN Synchronization**:  \n  The GAN framework unifies our two innovations:  \n  1. **OFEI** ensures *structural consistency* (optimal hypergraph isomorphism).  \n  2. **IHEN** learns *dynamic node-hyperedge interactions* (Eq. 5).  \n  - The discriminator’s random walk loss  *jointly supervises* OFEI and IHEN, forcing them to collaborate on biologically plausible hypergraphs.  \n- **Mode Collapse Prevention**:  \n  fMRI-EEG datasets exhibit high inter-subject variability. Our **path-based discriminator** (evaluating whole-graph connectivity) avoids trivial solutions by:  \n  - Comparing *global topology*, not just local edges.  \n  - Enforcing diversity through stochastic walks.  \n\n---\n\n#### **3. Neurobiological Motivation**  \nThe discriminator’s FC-referenced realism check mirrors *how neuroscientists validate connectivity*:  \n> *\"Just as researchers compare generated networks against gold-standard fMRI FC, our discriminator evaluates whether synthetic hypergraphs exhibit the same macroscale organizational principles \"*  \n\nThis adversarial \"peer review\" ensures hypergraphs reflect real brain network properties, not just data-fitting artifacts.  \n\n\n\n---\n\n#### **5. Revisions Commitment**  \nWe will add to **Section 2.2.1**:  \n> \"We adopt adversarial training because: (1) It models complex fMRI-EEG joint distributions without parametric assumptions; (2) The discriminator’s FC-guided realism check enforces neurobiological plausibility; \n\n\n\n**Summary**: GANs are indispensable for FEH because they:  \n1. Capture complex multimodal distributions beyond deterministic/VAE capabilities.  \n2. Denoise via adversarial filtering.  \n3. Enforce neurologically grounded hyperedges through FC-based discrimination.  \n4. Synergize OFEI-IHEN under a unified objective.  \n\n\n\n\n\n**Q3:** *The paper does not discuss the limitations of the proposed method or provide insights into potential future extensions. Including such a discussion would enhance the completeness of the paper and help better understand its applicability and scope of application.*\n\n\n\n\n**A3:** We sincerely thank the reviewer for highlighting the need to explicitly discuss the limitations and future extensions of our work. This is a valuable suggestion that will strengthen the paper's completeness and contextualize our contributions. In the revised manuscript, we will add a dedicated **\"Limitations and Future Work\"** subsection (Section 4.1) with the following content:  \n\n---\n\n### **4.1 Limitations and Future Work**  \nWhile FE-NET demonstrates significant advantages in joint fMRI-EEG modeling, we acknowledge several limitations:  \n1. **Computational Complexity**: The integration of GAN-based hypergraph generation (FEH) and Neural ODEs (FED) incurs higher training costs than traditional GNNs. Future work will explore model distillation techniques to reduce inference latency without sacrificing performance.  \n2. **Data Heterogeneity**: Our experiments used the AAL-90 atlas for ROI alignment. Generalizability to finer-grained parcellations or datasets with varying MRI resolutions requires further validation.  \n3. **Temporal Granularity**: Although Neural ODEs mitigate sampling-rate discrepancies, modeling ultra-rapid EEG oscillations (<100ms) remains challenging. We plan to incorporate spike-based encoding for sub-second dynamics.  \n4. **Clinical Translation**: Current validation focused on healthy cohorts (LEMON/CN-EPFL). Future studies will test FE-NET on clinical populations (e.g., epilepsy, Alzheimer’s) to assess diagnostic utility.  \n\n**Future Work**: \n- **Cross-Modal Imputation**: Extend FE-NET to synthesize missing EEG/fMRI data via latent-space interpolation, enabling single-modality inference.  \n- **Dynamic Hypergraph Sparsification**: Develop attention-based edge pruning to optimize incidence matrices during training, reducing memory overhead.  \n- **Federated Deployment**: Adapt the framework for privacy-preserving multi-site data integration using differential privacy.  \n\n---\n\nThese additions will clarify the boundaries of our method’s applicability while highlighting impactful research trajectories. We are grateful for the opportunity to enhance the paper’s rigor and translational relevance."}]

Author rebuttal:

Meta-review: This paper proposes a new method to combine two modality of brain signals, and achieves SOTA performance.

Strengths:
1. A very practical problem.
2. Reasonable solution.
3. Good results.

Weaknesses:
1. Some concerns on the complexity of the proposed approach.
2. Writing needs some enhancement.

Rebuttal:
The authors have done a great job in the rebuttal to interact with the reviewers with new results and clarifications. The reviewers are very active and engaged in the discussion. They are converged to be positive on this paper.

Justification:
A solid solution to a practical problem. The authors are encouraged to include the new results and improving writing in the final version.
Decision: Accept (poster)