Title: Joint Modeling of fMRI and EEG Imaging Using Ordinary Differential Equation-Based Hypergraph Neural Networks
Venue: NeurIPS 2025
Accepted: yes
Keywords: fMRI

EEG

Multimodal modeling
Abstract: Fusing multimodal brain imaging has been a hot topic since different modalities of brain imaging can provide complementary information. However, due to the size of simultaneous recorded fMRI-EEG dataset being limited and the substantial discrepancy between hemodynamic responses of fMRI and neural oscillations of EEG, the joint modeling of fMRI and EEG images is a rarely explored area and has not yielded satisfactory results. Existing studies have also indicated that the relationships between region of interest (ROI) are not one-to-one when synchronizing fMRI and EEG. Current graph-based multimodal modeling methods overlook those information. Based on this, we propose a hypergraph based fMRI-EEG modeling framework for asynchronous fMRI-EEG data named FE-NET. To the best of our knowledge, this is the first attempt to jointly model asynchronous EEG and fMRI data as Neural ODEs based hypergraph. Extensive experiments have demonstrated that the proposed FE-NET outperforms many state-of-the-art brain imaging modeling methods. Meanwhile, compared to simultaneously recorded fMRI-EEG data, asynchronously acquired fMRI-EEG data is less costly, which demonstrates the practical applicability of our method.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hypergraph neural network framework based on Neural Ordinary Differential Equations for the fusion of functional Magnetic Resonance Imaging (fMRI) and Electroencephalography (EEG) data. The framework is designed to handle the temporal sampling rate mismatch and asynchronous timestamps between the two modalities, and it captures latent temporal dependencies through a dynamic hypergraph embedding module. Experimental results demonstrate that the proposed method achieves high accuracy and superior performance across multiple cognitive tasks. Compared to simultaneously acquired data, asynchronously collected fMRI-EEG data is more cost-effective, highlighting the practical value of the approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\nThe paper proposes a novel and unified framework (FE-NET) that effectively combines fMRI and EEG imaging using Neural ODEs-based hypergraph neural networks, which is the first attempt to jointly model fMRI and EEG as hypergraphs.\nThe GAN-based FEH module with OFEI and IHEN effectively narrows the gap between hemodynamic responses and neural oscillations.\nThe FED module successfully addresses the challenge of asynchronous data by capturing temporal dependencies between fMRI and EEG signals with different sampling rates.\nThe method demonstrates superior performance compared to many state-of-the-art brain imaging modeling methods through extensive experiments.\nWeaknesses：\nThe paper asserts the lack of prior work on joint modeling of fMRI and EEG, especially with asynchronous data, and claims novelty. However, these statements are made without a thorough discussion of existing studies or background. Without a clearer positioning within the related literature, the novelty claim appears unsubstantiated.\nThe proposed method introduces several complex components (e.g., GAN-based hypergraph generation, Neural ODEs with dual-stream control), but the motivation and design choices are not sufficiently justified or intuitively explained. For example, why GANs are essential in this setting, or how the hyperedge structure aligns with actual neurophysiological properties, is not clearly articulated.\nThe paper does not include a discussion of its limitations or potential directions for future research. This weakens the integrity of the conclusions and makes it impossible to understand the boundaries of the method or possible extensions.\n\nQUESTIONS:\n1. The paper lacks a dedicated related work section. Incorporating a more thorough review of existing studies on fMRI-EEG fusion, especially on asynchronous modeling, would strengthen the background and better contextualize the novelty of this work.\n2. The motivation for using GANs in the hypergraph generation module is not clearly explained. It would be helpful if the authors could elaborate on why a GAN-based approach is necessary or beneficial in this setting, compared to more standard alternatives.\n3. The paper does not discuss the limitations of the proposed method or provide insights into potential future extensions. Including such a discussion would enhance the completeness of the paper and help better understand its applicability and scope of application.\n\nLIMITATIONS:\nNo\nThe paper does not explicitly discuss the methodological limitations or assumptions underlying the proposed model. For example, the effectiveness of the GAN-based hypergraph generation relies on the assumption that meaningful multimodal correspondences can be learned from asynchronous fMRI and EEG signals, which may not hold in more heterogeneous or noisy real-world settings. It would also be useful to discuss the interpretability of hypergraph representations and the potential risks of overfitting when modeling complex brain dynamics with limited labeled data.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-05"}

comments: SUMMARY:
This paper proposes FE-NET, a novel framework for jointly modeling fMRI and EEG brain data that are collected asynchronously. FE-NET significantly outperforms existing state-of-the-art models on various cognitive classification tasks, highlighting its practical applicability and superior performance in fusing multimodal brain data.

STRENGTHS AND WEAKNESSES:
Pros:

1. The problem setting is well-motivated and interesting. The primary strength is the model's ability to model asynchronous fMRI and EEG data. This has significant practical implications, as it circumvents the need for expensive and technically challenging simultaneous data acquisition.

2. The proposed approach is technically sound. It employs hypergraphs to capture complex, many-to-many brain region relationships and Neural ODEs to handle the different time scales of fMRI and EEG signals.


Cons:

1. While there are many different components of the proposed approach, including GANs, hypergraphs, and Neural ODEs, the time complexity for this approach should be relatively high.

2. There is no related work section. The authors should provide a comprehensive background introduction towards related work,  such as employing ODEs for brain network analysis[1], etc.



[1] BrainODE: Dynamic Brain Signal Analysis via Graph-Aided Neural Ordinary Differential Equations

QUESTIONS:
See above

LIMITATIONS:
See above
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new approach to joinlty model fMRI and EEG data. The model combines a hypergraph generator, and a module to embed the hypergraphs. The hypergraph generator is based on a GAN-style architecture, with a generator and a discriminator. The generator works by combining node attributes from fMRI and edge attributes from EEG, ultimately yielding a subject-specific multimodal connectivity matrix. The discriminator works by comparing random walks on the functional connectivity matrix to random walks on the multimodal connectivity matrix. The embedding module takes as input the vertex and hyperedge features generated, and defines an evolution equation implemented as a neural ODE with a control step and a diffusion step. The diffusion step allows further mixing of vertex and edge features. The approach is trained, validated and tested on 4 classification tasks on an N=227 dataset with non-simultaneous fMRI and EEG, and on an N=20 simultaneous EEG-fMRI dataset. It is also benchmarked with respect to several other methods.\n\nSTRENGTHS AND WEAKNESSES:\n## Quality\n\nA strength of the paper is its use of a thorough benchmark of competing architectures, on 4 different prediction targets, as well as an external dataset.\n\nConversely, section 3.3. overall evaluations ascribes the superior performance of FE-Net over the unimodal baselines to the fusion of EEG and fMRI. However it could also simply be that architectural differences in FE-Net, compared to  BrainGNN / M-GAT-BC / SGP-SL, rather than having both data types, drive the difference in performance. An ablation study feeding uninformative connectivity in one of the modalities, or simply generating hypergraphs from a single modality, then leaving the rest of the architecture untouched, would perhaps better highlight the contribution of modality vs architecture.\n\nThe motivation of the work should be corrected slightly.\nL34: \"...simultaneous recorded fMRI-EGG dataset suffers from tiny dataset size (less than 20)...\" while dataset sizes are indeed typically small in EEG-fMRI, a few open datasets are larger than 20, for example 50 in Meier et al \"Connectomes, simultaneous EEG-fMRI resting-state data and brain simulation results from 50 healthy subjects\", 2024; 30 in Loi et al's \"Simultaneous EEG-fMRI during a neurofeedback task, a brain imaging dataset for multimodal data integration\", 2020; 24 in Gherman and Philastides \"Simultaneous EEG-fMRI - Confidence in perceptual decisions\" dataset on openneuro.\nIn semi-open data, Wirsich et al \"The relationship between EEG and fMRI connectomes is reproducible across simultaneous EEG-fMRI studies from 1.5T to 7T\", 2021, used 72 subjects, provides all pre-computed connectomes (<https://zenodo.org/records/3905103>), and a subset of subjects on OSF (<https://osf.io/94c5t/>)\nIn closed data, clinical studies with over 30 patients (e.g. Al Asmi et al., \"fMRI activation in continuous and spike-triggered EEG-fMRI studies of epileptic spikes\", 2003) or more have been performed. I would thus recommend to amend the statement.\n\nIn the discriminator loss function, why take FC edges as reference for \"real\" paths rather than EC edges? This seems like the discriminator will push the generator to compute multimodal connectivity matrices which look more like fMRI connectivity than EEG connectivity. Is this to \"balance\" the design choice of using EC to derive the initial hyperedge attributes x_{k,E} in equation 4? \n\nOn the embedding space analysis, authors write \"traditional graph embedding results in a larger modality gap, which in turn leads to decreased performance\", but it seems instead that there is no modality gap - EEG and fMRI are located in overlapping regions of embedding space.\nAlso, what is the \"traditional graph embedding\" in L290 and fig. 2? Graph2Vec then UMAP ?\n\n\n\n## Clarity\n\nOverall, the structure of the paper is good. The illustration in Figure 1 is helful in understanding the approach. However, one major weakness is the lack of details in linking EEG and fMRI data to the model inputs.\n\nThe general, initial definition of the hypergraphs is clear, but the transition in Section 2.2.1 to the precise case of the EEG and fMRI graphs needs better framing, especially because the dimensionalities of the EEG and fMRI datasets are quite different. The number of nodes in both EEG and fMRI is very confusing. The supplementary section 6B mentions the 400-parcels Schaefer atlas, but goes on to list 44 regions from the \"AAL atlas\" - a different atlas altogether with 116 ROIs. So, is the paper using 400 or 44 regions for fMRI data to derive the \"FC matrix\"? Likewise, the EEG montage shown highlights 30 electrodes in red, from 62 according to the Babayan et al paper - traditionally meaning these are used for analysis. Is the paper using only 30 electrodes in EEG to derive the \"EC matrix\"?  Section 2.2.2 suggests that the fMRI BOLD time-series is Nxd, where N is the number of nodes (44 or 400?), and serves to derive the nodal attribute matrix X_{k,V}, while the EC matrix is given as dimension NxN. Because N is used in both EEG and fMRI, this implies that the same number of nodes are used in both, and, presumably spatially aligned somehow so that nodes are in 1:1 correspondance between modalities? No EEG source reconstruction is mentioned in the supplementary so this interpretation is also not certain. In the review I am chosing the interpret that the modalites are node-aligned for simplicity. This point should be spelled out explicitly.\n\nThus I assume that edges and nodes of both modalities are jointly combined into the 'initial hypergraphs', but this should be much more explicit as it is the basis of the method. The paper states \"If these initial hypergraphs {G′k} are directly utilized in the generative process to achieve multimodal connectivity, the robustness of the resultant generation may be compromised.\", but why is that? To me this is lacking a justification. The results of the FE-NETnoFEH ablation seem to bear our your assertion, but it would be useful to have an intuition.\n\nSection 2.2.2 mentions the bold time series B is \"employed to characterize the nodal attributes\" (X_{k_V}) - how? what is used from the B time-series? GNNs are very sensitive to node feature choice and their scaling, so it would be useful to specify. Likewise, where does the EC matrix come from? I would assume a dependency measure such as coherence but this is not described. Then in L110 S_k is used without being defined, making it more difficult to understand what X is supposed to contain. Please define all symbols when they are used first.\n\nA few more points on clarity\n- the correlation coefficients assigned to the nodes seem not to be used later in the paper\n- in 2.2.3 walks and paths are used interchangeably, while they are not (on paths each node can only by visited once) - please specify\n- I am not sure the second part of equation 13 is useful, as Z_v and Z_e are not used later\n- eq. 15, L168 user lower case t\n\nMinor points\nFigure 3 - left panel title is partly covered\nL226 \"datase tand\"\n\n## Significance\n\nThe approach could be used to any number of multimodal datasets where spatio-temporal correlations are of interest.\n\n## Originality\n\nL32: \"joint modeling of fMRI and EEG is a rarely explored area\" that \"has not yielded satisfactory results\" is a mischaracterisation of the field. While the combination of both modalities is certainly less explored than each modality alone, EEG-fMRI analysis is quite common, with over 1000 journal papers on pubmed alone and 27K papers on Google scholar, and dates back a while. See for instance Menon and Crottaz-Herbette's 2005 review \"Combined EEG and fMRI studies of human brain function\", Herrmann and Debener's 2008 \"Simultaneous recording of EEG and BOLD responses: a historical perspective\", or Helmut Lauf's 2012 \"A personalized history of EEG–fMRI integration\", for accounts from pioneers dating back to at least 1993 with Ives and colleagues \"Monitoring the patient's EEG during echo planar MRI\". EEG-fMRI clinical usefulness is also well-established in epilepsy. I recommend positioning more precisely the knowledge gap that the paper is meant to address.\n\nL58 also claims that \"this is the first attempt to model asynchronous EEG and fMRI data\", meaning in this context EEG and fMRI acquired not at the same time.\nThis is repeated again in L322 \"For the first time, we attempt to model asynchronous fMRI and EEG data...\".\nThis is factually incorrect. There have been very numerous studies where EEG and fMRI acquired at different times have been modelled, going back at least to 1996 (Gerloff et al., Coregistration of EEG and fMRI in a Simple Motor Task).\nFor example: joint ICA has been used to jointly model EEG and fMRI activations in Calhoun et al., A feature-based approach to combine functional MRI, structural MRI and EEG brain imaging data, 2006; CCA (a), then Multi-set CCA (b) has been used to jointly model fMRI, sMRI and EEG in (b) Correa et al., Fusion of fMRI, sMRI, and EEG data using canonical correlation analysis, 2009, and (b) Sui et al., Combination of FMRI-SMRI-EEG Data Improves Discrimination of Schizophrenia Patients by Ensemble Feature Selection, 2014; Then coupled matrix and tensor factorization for EEG and fMRI in Acar et al., Tensor-based fusion of EEG and FMRI to understand neurological changes in schizophrenia, 2017, etc... and several works with deep learning are also easy to find, including many on EEG-to-fMRI prediction.\nThis work is enabled by multiple databases that have both fMRI and EEG data recorded at different times, such as the PEARL-neuro database (https://www.nature.com/articles/s41597-024-03106-5) or the Leipzig mind-brain-body database (https://www.nature.com/articles/sdata2018308) (ref 21 in the paper).\nThe authors should tone down this claim of novelty, or make it more specific.\n\nThe lack of direct architectural comparison (not only benchmarking) with multimodal neural networks taking EEG and fMRI connectivity as input (or even neural networks taking multimodal MRI connectivity as input) makes it difficult to establish the core differences with existing offers.\n\nQUESTIONS:\n- Provide improved description of how EEG and fMRI data are taken as input - will improve rating \n- Revise and improve claims of originality with better situation in the state of the art - will improve rating\n- It is not clear why fMRI is used for initial node attributes and EEG for hyperedge attributes. Could we swap EEG and fMRI in the inputs?\n\nLIMITATIONS:\nNo, there is no limitations section or discussion of limitations, either technical or social. Predicting verbal intelligence or logical intelligence, while convenient as an engineering exercise because prediction targets are provided, does of course raise ethical questions related to potential bias and fairness, as well as benefits and harms. It is not essential to go in depth but this should be at least acknowledged.\nSee e.g. slides 40++ in \"Understanding Ethics in NLP Authoring and Reviewing\" <https://github.com/acl-org/ethics-tutorial/blob/main/slides/Understanding%20Ethics%20in%20NLP%20Authoring%20and%20Reviewing%20%E2%80%93%20Presentation%20Deck.pdf>.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents FE-NET, a novel framework for jointly modeling fMRI and EEG brain imaging data, specifically addressing the challenge of asynchronous acquisition. Evaluated on the LEMON dataset, FE-NET achieves superior performance compared to existing unimodal and multimodal approaches across various cognitive classification tasks. Extensive ablations further validate the effectiveness of both modules.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:  \n1.\tThe paper addresses the underexplored but practically important problem of modeling fMRI and EEG data which can provide complementary information.  \n2.\tThe proposed FE-FEH outperforms a broad range of SOTA unimodal and multimodal baselines on multiple cognitive tasks, showing strong results across accuracy, precision, and specificity.  \n3.\tThe model outputs meaningful functional connectivity patterns associated with cognitive traits (e.g., DMN in verbal intelligence), supporting interpretability.  \n4.\tThe FEH module effectively bridges modality differences using a GAN framework and the FED module employs Neural ODEs to model temporal dynamics across modalities with mismatched sampling rates, enabling smooth representation evolution.\n\nWeaknesses:  \n1.\tThe use of GANs, Neural ODEs, and hypergraphs likely introduces high computational costs. It would be better to include an analysis of computational.  \n2.\tThere should be some analysis of training time, inference speed, or GPU usage compared to existing baselines.  \n3.\tPotential issues such as overfitting, dependency on high-quality preprocessing, and uncertain generalization to clinical or pathological populations are neither acknowledged nor discussed. It would be better to include a dedicated section discussing these points.  \n4.\tIt is unclear whether the model can handle common real-world challenges such as missing EEG channels, or varying time windows, limiting confidence in its practical reliability. It would be helpful to include relevant tests or discussions addressing these concerns.  \n5.\tThe paper lacks insights into the learned temporal representations. There is no analysis explaining what the Neural ODE dynamics capture across modalities or how time-dependent information contributes to classification performance.  \n6.\tAlthough the LEMON dataset is sizable, most experiments rely solely on it. The secondary dataset (CN-EPFL) is very small (only 20 subjects), which limits the claims regarding generalization. Including additional datasets would strengthen the robustness claims.\n\nQUESTIONS:\n1.\tGiven the complexity introduced by GANs, Neural ODEs, and hypergraphs, could you provide a detailed computational complexity analysis? And could you include an analysis comparing training time, inference speed, and GPU usage with existing baseline methods?  \n2.\tPotential issues such as overfitting, dependency on high-quality preprocessing, and uncertain generalization to clinical or pathological populations are currently not discussed. Could you include a dedicated section addressing these points?  \n3.\tIt is unclear if your model handles common practical issues like missing EEG channels or varying time windows. Could you provide relevant tests or a discussion addressing these real-world challenges?  \n4.\tThe paper lacks insight into what temporal representations are learned by the Neural ODEs. Could you provide an analysis or visualization explaining what dynamics the Neural ODEs capture across modalities and their contribution to classification?  \n5.\tAlthough the LEMON dataset is sizable, reliance on it and the very small CN-EPFL dataset limits generalization claims. Could you include experiments using additional datasets to strengthen claims of robustness?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}